| |
1 | +# Phase 0 — Feasibility report (WP37) |
| |
2 | + |
| |
3 | +Date: 2026-11-01 (UTC). Author of record: S.-P. Boucher (UQO), pipeline executed by Claude Code. |
| |
4 | + |
| |
5 | +## 1. Working directory and repository |
| |
6 | + |
| |
7 | +- Project created at `Recherche-UQO/economie/hidden-architecture-grocery-prices/` from the UQO working-paper template (`_modele-working-paper`). Nothing pre-existing was overwritten; WP numbers 1–35 are taken, WP36 is an unnumbered stub (`economie/used-car-market-liquidity`), so **WP37** is free. |
| |
8 | +- Git initialised locally (`main`). Remote `gitsrv:srv/git/wp37_uqo.git` not yet created (creation on spbgit requires the PI's go-ahead; nothing is published externally). |
| |
9 | +- Mandate stored verbatim in `CLAUDE.md`. |
| |
10 | + |
| |
11 | +## 2. Compute, toolchain |
| |
12 | + |
| |
13 | +| Item | Observed | |
| |
14 | +|---|---| |
| |
15 | +| Machine | Laptop, 18 cores, 48 GiB RAM, 1.0 TiB free (sufficient: raw DB is 165 MB). MacLustr cluster (29 nodes) available but not required. | |
| |
16 | +| Python | 3.12 venv (`uv`), deps pinned in `requirements-lock.txt` (pandas, statsmodels, linearmodels, pyfixest, scikit-learn, lightgbm, duckdb, pulp, matplotlib, pytest…) | |
| |
17 | +| R | 4.x available (`/opt/homebrew/bin/R`) — fallback only | |
| |
18 | +| LaTeX | TeX Live, latexmk 4.88 ; pandoc present | |
| |
19 | +| Seeds | `config/settings.yaml: seed: 20261101` used by every stochastic step (bootstrap, CV folds, sampling) | |
| |
20 | + |
| |
21 | +## 3. Data access — what actually exists |
| |
22 | + |
| |
23 | +**Food-Ka is owned by the PI**; the production SQLite database (`M2U64:~/apps/food-ka/data/foodka.db`, WAL mode) was copied with a consistent `.backup` from a read-only connection (production untouched). Checksum verified. Provenance in `data/raw/README.md`. |
| |
24 | + |
| |
25 | +Public cross-check (2026-11-01 ~08:00 UTC): homepage shows 50 760 products, 54 banners, 8 631 on promotion — identical to the snapshot's `active=1` rows. The mandate's figures (50 886 / 54 / 8 590) were the homepage values a few hours earlier; they are **source-feed listings, not unique products**. `robots.txt` disallows `/api/` to crawlers; the API was queried once for this cross-check only and is not a data source. |
| |
26 | + |
| |
27 | +### 3.1 Schema (observed) |
| |
28 | + |
| |
29 | +`products` (124 579 rows: 50 760 active, 73 819 deactivated) — uid = `source:external_id`, name, brand, category (18 canonical), category_raw, size_label, price, regular_price, price_label, on_sale, unit_price, unit_price_label, in_stock, details (JSON), first_seen, last_seen, updated_at, miss_count, active. |
| |
30 | +`price_log` (300 613 rows: uid, ts, price) — **event log**: one row at first observation and one row at every *change* of `price` (NULL = price disappeared from display). No row when the price is unchanged. `regular_price`, `on_sale`, `in_stock` are **not** historised. |
| |
31 | +`sync_log` (15 630 rows) — one row per connector pass (57 sources × ~255–308 passes, median gap **7.2 h**, 2026-08-12 → 2026-11-01), with found/added/updated/removed counts. |
| |
32 | +`product_links` (2 537 uids in 1 103 cross-banner groups) — Food-Ka's own brand+size+name matching (not GTIN based). |
| |
33 | +`off_cache` (25 630) — Open Food Facts barcodes attached by a conservative brand/name/size lookup (derived, not source-provided). |
| |
34 | + |
| |
35 | +### 3.2 Data-generating process (from the connector code, read-only copy in `data/raw/foodka-src/`) |
| |
36 | + |
| |
37 | +- 57 active source feeds = **54 with active products**, belonging to far fewer **retail companies**: Loblaw (maxi, provigo, club_entrepot + 3 flyers + inter_marche, axep? no: Axep/Richelieu/Marché Ami are Metro affiliates), Metro Inc. (metro, superc + 2 flyers + richelieu, marche_ami, axep), Sobeys (iga + flyer, tradition, bonichoix, rachelle_bery), Walmart, Costco, Adonis (Metro Inc.), Avril, independents, 4 pharmacies, SAQ. A `retailers` table (company → banner → feed → channel) will be built in Phase 1. |
| |
38 | +- **Two channels**: (a) *online storefront* feeds (Shopify/WooCommerce JSON, Loblaw/Metro/Walmart/Costco/IGA sites scraped through residential proxy pools without store selection); (b) *weekly circulars* via Flipp for postal code **H2X 1Y4 (Montréal)** with `valid_from`/`valid_to`, `original_price`, `dollars_off`, fine print. |
| |
39 | +- **Geography**: no store identifier anywhere. Circulars are regional (Montréal zone); online prices are whatever store the retailer's site assigned to the proxy session. → **Store-level geography is unavailable** (H4 and travel-cost components of Model E cannot be estimated from Food-Ka). |
| |
40 | +- **Identifiers**: Metro and Super C `external_id` are valid GS1 UPC-A codes (100 % check-digit valid, n = 1 954 active). Shopify/Walmart/Loblaw ids are internal SKUs (check-digit validity ≈ 10 %, i.e. chance). Exact GTIN matching is therefore only possible within Metro Inc.; everything else requires validated probabilistic matching. |
| |
41 | + |
| |
42 | +### 3.3 Critical measurement finding — store-mixture noise in scraped banners |
| |
43 | + |
| |
44 | +For anti-bot banners (club_entrepot, metro, maxi, superc, provigo, walmart) the event log shows **high-frequency oscillation among a small set of price values** (e.g. Metro mayonnaise 890 mL alternating 5.77 / 6.99 / 7.49 / 7.79 / 7.99 within the same day; 54–68 % of change events revert to the previous-but-one value; median 4 distinct values per product; week-to-week Jaccard of value sets ≈ 0.5). This is consistent with **each scrape being served by a different (unknown) store**, not with genuine intra-day repricing. Consequences, to be handled in Phase 1 and preregistered in Phase 3: |
| |
45 | +- a banner-level "posted price" at time *t* must be defined as a statistic of the daily/weekly value set (mode, max = likely shelf regular, min = likely best store/promo), with sensitivity analysis; |
| |
46 | +- intra-week volatility for these banners is **not** interpretable as price flexibility; H2 must be tested on weekly aggregates and separately on circular feeds (clean weekly cycles); |
| |
47 | +- the amplitude of the value set is itself a (store-unidentified) lower bound on **within-banner cross-store dispersion** — a usable, honestly-labelled quantity. |
| |
48 | +Shopify/WooCommerce feeds (single online store) show far fewer events and no such cycling. |
| |
49 | + |
| |
50 | +### 3.4 Variable availability (summary; full inventory in Phase 1) |
| |
51 | + |
| |
52 | +| Variable | Status | |
| |
53 | +|---|---| |
| |
54 | +| product name, brand, category, size_label | confirmed (brand missing for several feeds: walmart 87 %, adonis/tt/akhavan/bocoboco 100 %) | |
| |
55 | +| UPC/GTIN | confirmed for Metro/Super C only; derivable-unreliable via OFF barcode elsewhere | |
| |
56 | +| unit price ($/100 g, $/100 ml, $/un) | derivable (Food-Ka's parser); missing where size_label absent (giant_tiger 92 %, nuvo 100 %, most Flipp feeds) | |
| |
57 | +| current price history | confirmed (event log, 81 days) | |
| |
58 | +| regular/reference price history | **unavailable** (current state only); circulars carry `original_price` per item-week | |
| |
59 | +| promotion flag history | unavailable → must be **inferred** from price paths; validated against current `on_sale` and circular items | |
| |
60 | +| observation timestamps | confirmed (`sync_log` per source; product-level presence between events inferred) | |
| |
61 | +| circular validity | confirmed (Flipp feeds) | |
| |
62 | +| availability/in_stock | current only; unreliable (NULL for 23 %) | |
| |
63 | +| store / geography | **unavailable** | |
| |
64 | +| quantities / expenditure weights | unavailable (no transactions) | |
| |
65 | + |
| |
66 | +## 4. External benchmarks |
| |
67 | + |
| |
68 | +Statistics Canada WDS reachable: table 18-10-0004-01 (CPI monthly NSA, Quebec = member 11, Montréal = 13; "Food purchased from stores" = member 4; latest 2026-09) and 18-10-0245-01 (monthly average retail prices for 110 selected products, Quebec; latest 2026-08). Overlap with the Food-Ka window is only **Aug–Sep 2026** (2 months) — validation of QGPAI against official indices will necessarily be weak and labelled as such. |
| |
69 | + |
| |
70 | +## 5. API credentials (minimal tests, 2026-11-01T08:02Z) |
| |
71 | + |
| |
72 | +| Provider | Model | Result | |
| |
73 | +|---|---|---| |
| |
74 | +| Anthropic | claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5 | OK (end_turn, 18 in / 4 out tokens each) | |
| |
75 | +| OpenAI | gpt-5.4, gpt-5.4-mini, gpt-5.4-nano | OK | |
| |
76 | + |
| |
77 | +Planned LLM use is limited and auxiliary: (i) second annotator for the stratified match-validation sample (human = PI adjudication remains the reference), (ii) nothing in estimation. Every call is logged with model id, tokens and cost. |
| |
78 | + |
| |
79 | +## 6. GO / NO-GO gates |
| |
80 | + |
| |
81 | +| Gate | Decision | Basis | |
| |
82 | +|---|---|---| |
| |
83 | +| G1 Authorised data access | **GO** | owner export, read-only copy, checksum | |
| |
84 | +| G2 Longitudinal panel | **GO (partial)** | 81-day event log, ~3.3 passes/day/source; reconstructable daily panel; regular-price history absent | |
| |
85 | +| G3 Cross-banner matching | **GO (partial)** | exact UPC only Metro↔Super C; probabilistic matching + validation elsewhere; expect a few thousand matched products, not 50 k | |
| |
86 | +| G4 Promotion dynamics | **GO (partial)** | circulars: clean weekly cycles with reference prices; online feeds: promotion state inferred from price paths and store-mixture noise | |
| |
87 | +| G5 Competitive spillovers | **GO descriptive only** | weekly banner-level responses; no exogenous variation identified a priori → no causal claims | |
| |
88 | +| G6 Store geography / travel costs | **NO-GO** | no store identifiers; H4 untestable; Model E limited to banner-count constraints; prospective plan required | |
| |
89 | +| G7 Price indices vs StatCan | **GO (weak)** | 2-month overlap only | |
| |
90 | +| G8 Compute | **GO** | laptop sufficient | |
| |
91 | + |
| |
92 | +## 7. Expected compute |
| |
93 | + |
| |
94 | +Whole pipeline (panel reconstruction, matching, fixed-effects models, bootstraps, MIP baskets, rolling-origin forecasts) is expected to run in well under one hour on the laptop; no cluster job required. |