|
1 |
+# Phase 0 — Feasibility report (WP37) |
|
2 |
+ |
|
3 |
+Date: 2026-11-01 (UTC). Author of record: S.-P. Boucher (UQO), pipeline executed by Claude Code. |
|
4 |
+ |
|
5 |
+## 1. Working directory and repository |
|
6 |
+ |
|
7 |
+- Project created at `Recherche-UQO/economie/hidden-architecture-grocery-prices/` from the UQO working-paper template (`_modele-working-paper`). Nothing pre-existing was overwritten; WP numbers 1–35 are taken, WP36 is an unnumbered stub (`economie/used-car-market-liquidity`), so **WP37** is free. |
|
8 |
+- Git initialised locally (`main`). Remote `gitsrv:srv/git/wp37_uqo.git` not yet created (creation on spbgit requires the PI's go-ahead; nothing is published externally). |
|
9 |
+- Mandate stored verbatim in `CLAUDE.md`. |
|
10 |
+ |
|
11 |
+## 2. Compute, toolchain |
|
12 |
+ |
|
13 |
+| Item | Observed | |
|
14 |
+|---|---| |
|
15 |
+| Machine | Laptop, 18 cores, 48 GiB RAM, 1.0 TiB free (sufficient: raw DB is 165 MB). MacLustr cluster (29 nodes) available but not required. | |
|
16 |
+| Python | 3.12 venv (`uv`), deps pinned in `requirements-lock.txt` (pandas, statsmodels, linearmodels, pyfixest, scikit-learn, lightgbm, duckdb, pulp, matplotlib, pytest…) | |
|
17 |
+| R | 4.x available (`/opt/homebrew/bin/R`) — fallback only | |
|
18 |
+| LaTeX | TeX Live, latexmk 4.88 ; pandoc present | |
|
19 |
+| Seeds | `config/settings.yaml: seed: 20261101` used by every stochastic step (bootstrap, CV folds, sampling) | |
|
20 |
+ |
|
21 |
+## 3. Data access — what actually exists |
|
22 |
+ |
|
23 |
+**Food-Ka is owned by the PI**; the production SQLite database (`M2U64:~/apps/food-ka/data/foodka.db`, WAL mode) was copied with a consistent `.backup` from a read-only connection (production untouched). Checksum verified. Provenance in `data/raw/README.md`. |
|
24 |
+ |
|
25 |
+Public cross-check (2026-11-01 ~08:00 UTC): homepage shows 50 760 products, 54 banners, 8 631 on promotion — identical to the snapshot's `active=1` rows. The mandate's figures (50 886 / 54 / 8 590) were the homepage values a few hours earlier; they are **source-feed listings, not unique products**. `robots.txt` disallows `/api/` to crawlers; the API was queried once for this cross-check only and is not a data source. |
|
26 |
+ |
|
27 |
+### 3.1 Schema (observed) |
|
28 |
+ |
|
29 |
+`products` (124 579 rows: 50 760 active, 73 819 deactivated) — uid = `source:external_id`, name, brand, category (18 canonical), category_raw, size_label, price, regular_price, price_label, on_sale, unit_price, unit_price_label, in_stock, details (JSON), first_seen, last_seen, updated_at, miss_count, active. |
|
30 |
+`price_log` (300 613 rows: uid, ts, price) — **event log**: one row at first observation and one row at every *change* of `price` (NULL = price disappeared from display). No row when the price is unchanged. `regular_price`, `on_sale`, `in_stock` are **not** historised. |
|
31 |
+`sync_log` (15 630 rows) — one row per connector pass (57 sources × ~255–308 passes, median gap **7.2 h**, 2026-08-12 → 2026-11-01), with found/added/updated/removed counts. |
|
32 |
+`product_links` (2 537 uids in 1 103 cross-banner groups) — Food-Ka's own brand+size+name matching (not GTIN based). |
|
33 |
+`off_cache` (25 630) — Open Food Facts barcodes attached by a conservative brand/name/size lookup (derived, not source-provided). |
|
34 |
+ |
|
35 |
+### 3.2 Data-generating process (from the connector code, read-only copy in `data/raw/foodka-src/`) |
|
36 |
+ |
|
37 |
+- 57 active source feeds = **54 with active products**, belonging to far fewer **retail companies**: Loblaw (maxi, provigo, club_entrepot + 3 flyers + inter_marche, axep? no: Axep/Richelieu/Marché Ami are Metro affiliates), Metro Inc. (metro, superc + 2 flyers + richelieu, marche_ami, axep), Sobeys (iga + flyer, tradition, bonichoix, rachelle_bery), Walmart, Costco, Adonis (Metro Inc.), Avril, independents, 4 pharmacies, SAQ. A `retailers` table (company → banner → feed → channel) will be built in Phase 1. |
|
38 |
+- **Two channels**: (a) *online storefront* feeds (Shopify/WooCommerce JSON, Loblaw/Metro/Walmart/Costco/IGA sites scraped through residential proxy pools without store selection); (b) *weekly circulars* via Flipp for postal code **H2X 1Y4 (Montréal)** with `valid_from`/`valid_to`, `original_price`, `dollars_off`, fine print. |
|
39 |
+- **Geography**: no store identifier anywhere. Circulars are regional (Montréal zone); online prices are whatever store the retailer's site assigned to the proxy session. → **Store-level geography is unavailable** (H4 and travel-cost components of Model E cannot be estimated from Food-Ka). |
|
40 |
+- **Identifiers**: Metro and Super C `external_id` are valid GS1 UPC-A codes (100 % check-digit valid, n = 1 954 active). Shopify/Walmart/Loblaw ids are internal SKUs (check-digit validity ≈ 10 %, i.e. chance). Exact GTIN matching is therefore only possible within Metro Inc.; everything else requires validated probabilistic matching. |
|
41 |
+ |
|
42 |
+### 3.3 Critical measurement finding — store-mixture noise in scraped banners |
|
43 |
+ |
|
44 |
+For anti-bot banners (club_entrepot, metro, maxi, superc, provigo, walmart) the event log shows **high-frequency oscillation among a small set of price values** (e.g. Metro mayonnaise 890 mL alternating 5.77 / 6.99 / 7.49 / 7.79 / 7.99 within the same day; 54–68 % of change events revert to the previous-but-one value; median 4 distinct values per product; week-to-week Jaccard of value sets ≈ 0.5). This is consistent with **each scrape being served by a different (unknown) store**, not with genuine intra-day repricing. Consequences, to be handled in Phase 1 and preregistered in Phase 3: |
|
45 |
+- a banner-level "posted price" at time *t* must be defined as a statistic of the daily/weekly value set (mode, max = likely shelf regular, min = likely best store/promo), with sensitivity analysis; |
|
46 |
+- intra-week volatility for these banners is **not** interpretable as price flexibility; H2 must be tested on weekly aggregates and separately on circular feeds (clean weekly cycles); |
|
47 |
+- the amplitude of the value set is itself a (store-unidentified) lower bound on **within-banner cross-store dispersion** — a usable, honestly-labelled quantity. |
|
48 |
+Shopify/WooCommerce feeds (single online store) show far fewer events and no such cycling. |
|
49 |
+ |
|
50 |
+### 3.4 Variable availability (summary; full inventory in Phase 1) |
|
51 |
+ |
|
52 |
+| Variable | Status | |
|
53 |
+|---|---| |
|
54 |
+| product name, brand, category, size_label | confirmed (brand missing for several feeds: walmart 87 %, adonis/tt/akhavan/bocoboco 100 %) | |
|
55 |
+| UPC/GTIN | confirmed for Metro/Super C only; derivable-unreliable via OFF barcode elsewhere | |
|
56 |
+| unit price ($/100 g, $/100 ml, $/un) | derivable (Food-Ka's parser); missing where size_label absent (giant_tiger 92 %, nuvo 100 %, most Flipp feeds) | |
|
57 |
+| current price history | confirmed (event log, 81 days) | |
|
58 |
+| regular/reference price history | **unavailable** (current state only); circulars carry `original_price` per item-week | |
|
59 |
+| promotion flag history | unavailable → must be **inferred** from price paths; validated against current `on_sale` and circular items | |
|
60 |
+| observation timestamps | confirmed (`sync_log` per source; product-level presence between events inferred) | |
|
61 |
+| circular validity | confirmed (Flipp feeds) | |
|
62 |
+| availability/in_stock | current only; unreliable (NULL for 23 %) | |
|
63 |
+| store / geography | **unavailable** | |
|
64 |
+| quantities / expenditure weights | unavailable (no transactions) | |
|
65 |
+ |
|
66 |
+## 4. External benchmarks |
|
67 |
+ |
|
68 |
+Statistics Canada WDS reachable: table 18-10-0004-01 (CPI monthly NSA, Quebec = member 11, Montréal = 13; "Food purchased from stores" = member 4; latest 2026-09) and 18-10-0245-01 (monthly average retail prices for 110 selected products, Quebec; latest 2026-08). Overlap with the Food-Ka window is only **Aug–Sep 2026** (2 months) — validation of QGPAI against official indices will necessarily be weak and labelled as such. |
|
69 |
+ |
|
70 |
+## 5. API credentials (minimal tests, 2026-11-01T08:02Z) |
|
71 |
+ |
|
72 |
+| Provider | Model | Result | |
|
73 |
+|---|---|---| |
|
74 |
+| Anthropic | claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5 | OK (end_turn, 18 in / 4 out tokens each) | |
|
75 |
+| OpenAI | gpt-5.4, gpt-5.4-mini, gpt-5.4-nano | OK | |
|
76 |
+ |
|
77 |
+Planned LLM use is limited and auxiliary: (i) second annotator for the stratified match-validation sample (human = PI adjudication remains the reference), (ii) nothing in estimation. Every call is logged with model id, tokens and cost. |
|
78 |
+ |
|
79 |
+## 6. GO / NO-GO gates |
|
80 |
+ |
|
81 |
+| Gate | Decision | Basis | |
|
82 |
+|---|---|---| |
|
83 |
+| G1 Authorised data access | **GO** | owner export, read-only copy, checksum | |
|
84 |
+| G2 Longitudinal panel | **GO (partial)** | 81-day event log, ~3.3 passes/day/source; reconstructable daily panel; regular-price history absent | |
|
85 |
+| G3 Cross-banner matching | **GO (partial)** | exact UPC only Metro↔Super C; probabilistic matching + validation elsewhere; expect a few thousand matched products, not 50 k | |
|
86 |
+| G4 Promotion dynamics | **GO (partial)** | circulars: clean weekly cycles with reference prices; online feeds: promotion state inferred from price paths and store-mixture noise | |
|
87 |
+| G5 Competitive spillovers | **GO descriptive only** | weekly banner-level responses; no exogenous variation identified a priori → no causal claims | |
|
88 |
+| G6 Store geography / travel costs | **NO-GO** | no store identifiers; H4 untestable; Model E limited to banner-count constraints; prospective plan required | |
|
89 |
+| G7 Price indices vs StatCan | **GO (weak)** | 2-month overlap only | |
|
90 |
+| G8 Compute | **GO** | laptop sufficient | |
|
91 |
+ |
|
92 |
+## 7. Expected compute |
|
93 |
+ |
|
94 |
+Whole pipeline (panel reconstruction, matching, fixed-effects models, bootstraps, MIP baskets, rolling-origin forecasts) is expected to run in well under one hour on the laptop; no cluster job required. |