phase1: validation round 2 (LLM + operator review), precision by tier, Phase 1 data-audit report, evidence matrix
8 changed files +349 −3
added
literature/evidence_matrix.csv
+16 −0
@@ -0,0 +1,16 @@ | |||||
| 1 | +claim_id,claim,theory_or_empirical,supporting_keys,what_they_show,limitations_for_WP37,wp37_novelty | ||||
| 2 | +C1,"Equilibrium price dispersion for homogeneous goods arises from consumer search costs and heterogeneous information",theory,"stigler1961; salop1977; burdett1983; baye2006","Dispersed-price equilibria exist when some consumers are informed and others search sequentially or sample few prices","Theories are about transaction prices and identical goods; posted online prices at the banner level aggregate unknown stores","Measures dispersion on exactly matched items across Quebec banners from a single harmonised source; separates banner-level from within-banner (store-mixture) dispersion" | ||||
| 3 | +C2,"Temporary sales are an equilibrium mixed strategy to price-discriminate between informed/loyal and uninformed/shopper consumers",theory,"varian1980; pesendorfer2002; hendel2006; hendel2013; chevalier2019","Sales recur, are temporary and return to a stable reference price; storable-goods demand accumulation generates sale cycles","Requires quantities to identify demand; WP37 has posted prices only, so only the price-path implications are testable","Tests price-path implications (duration, depth, return-to-reference) on both storefront and circular channels" | ||||
| 4 | +C3,"Retail prices alternate between a sticky reference price and temporary sale prices; reference prices adjust rarely",empirical,"nakamura2008; klenow2008; eichenbaum2011; kehoe2015; anderson2017; bils2004","Sales account for a large share of price changes; regular prices are sticky; sales are informationally rigid","US scanner/BLS micro data; Canada and Quebec evidence is scarce","First high-frequency (≈7 h) posted-price evidence for Quebec grocery banners; documents a measurement artefact (store mixture) relevant to all scraped-price studies" | ||||
| 5 | +C4,"Posted-price dispersion is mostly within-store/relative and persistent across stores (morphology of dispersion)",empirical,"kaplan2015; kaplan2019; lach2002; sorensen2000","Most dispersion is relative-price dispersion across stores for the same good; store ranks are persistent","Transaction-based scanner panels; WP37 observes banners not stores","Quantifies cross-banner persistence of relative price gaps on exact-UPC pairs (Metro vs Super C) and brand+size pairs across companies" | ||||
| 6 | +C5,"Large retailers set near-uniform prices across stores of a chain (uniform/zone pricing)",empirical,"dellavigna2019; adams2024; hitsch2019; hitsch2021","US chains price nearly uniformly within chain; promotions are chain-wide events","If Quebec banners priced uniformly, the store-mixture oscillation would not appear; its presence is informative","Provides indirect evidence against full within-banner uniformity in Quebec online channels (price value sets of 3–5 levels per product-week)" | ||||
| 7 | +C6,"Online prices track offline prices closely for multi-channel retailers; scraped prices are valid for inflation measurement",empirical,"cavallo2017; cavallo2016; cavallo2018; gorodnichenko2017; cavallo2023","Online = offline for most items in 10 countries; scraped data show more frequent changes than CPI","Cavallo compares identical products in the same retailer; WP37 cannot observe offline prices","Shows that anonymous-session scraping of store-selected sites returns a mixture of store prices — a caveat for Billion-Prices-type collection" | ||||
| 8 | +C7,"Matched-model and multilateral indices (GEKS/Törnqvist, chained) avoid chain drift; naive average-price indices are biased by assortment and substitution",theory/empirical,"ivancic2011; dehaan2011; diewert2022; melser2018; feenstra1994; redding2020","Chain drift from sales; multilateral methods; variety and taste shock adjustments","Expenditure weights are unavailable in WP37 (listings only) — only unweighted/Jevons-type and fixed-basket indices can be built","Quantifies the gap between naive average listed price and matched-model indices on the same Quebec data; honest unweighted labelling" | ||||
| 9 | +C8,"Package downsizing (shrinkflation) is a hidden price increase; consumers are less sensitive to quantity than price",empirical,"gourville2004; cakir2014; cakir2021; hoppe2024","Downsizing raises unit price with muted demand response; pass-through to retail","Short 81-day window: few size changes are observable; detection relies on product-text size changes within a listing","Documents frequency of within-listing size-label changes and their unit-price impact over 11 weeks; explicit about low power" | ||||
| 10 | +C9,"Shopping effort and store choice lower paid prices; households could save substantially by buying the cheapest available",empirical,"aguiar2007; griffith2009; kaplan2016; nevo2019","Paid prices fall with shopping time; potential savings 10–30 % under hypothetical perfect shopping","Savings bounds require transaction data and store access; WP37 has posted prices and no geography","Computes unconstrained vs banner-count-constrained basket cost frontier on posted prices; labels it an upper bound; geography left to the prospective design" | ||||
| 11 | +C10,"Spatial access and assortment availability shape effective food prices",empirical,"allcott2019; handbury2015","Food deserts explained mainly by demand; prices vary across cities with availability","No store geography in Food-Ka","Not testable; stated as limitation with a prospective data plan (store-level circular zones)" | ||||
| 12 | +C11,"Menu costs and multi-product firms generate small and large price changes; sales are decoupled from macro shocks",theory,"golosov2007; midrigan2011; nakamura2013; guimaraes2011; coibion2015","Menu-cost models with sales; sales respond weakly to monetary policy","Not identifiable from 81 days","Descriptive size distribution of posted-price changes; no structural estimation claimed" | ||||
| 13 | +C12,"Private labels and brand premia are quality-adjusted price differences",empirical,"dube2018; bronnenberg2015","Private-label share rises in recessions; informed shoppers buy store brands","No quantities","Reports national-brand vs private-label unit-price gaps within categories and banners (descriptive)" | ||||
| 14 | +C13,"Retail competition (entry of Walmart, EDLP vs Hi-Lo) changes price levels and promotion intensity",empirical,"hausman2007; lal1997; hoch1994; ellickson2008","EDLP vs Hi-Lo strategies; Walmart lowers prices","No entry events in window","Characterises each banner's promotion intensity and price level on matched items (EDLP vs Hi-Lo typology for Quebec)" | ||||
| 15 | +C14,"Probabilistic record linkage requires validation (precision/recall) and is a source of measurement error",method,"fellegi1969; christen2012","Linkage theory and evaluation practice","—","Calibrates name-based linkage against UPC ground truth and reports tier-specific precision; propagates match uncertainty" | ||||
| 16 | +C15,"Canadian grocery retail is highly concentrated (Loblaw, Sobeys, Metro plus Walmart and Costco)",institutional,"competitionbureau2023","Market study: concentration, margins, need for more competition","Report, not peer-reviewed","Maps 54 feeds to companies/banners/channels, showing that apparent banner diversity collapses to few companies" | ||||
modified
literature/manual_references.yaml
+0 −1
@@ -7,4 +7,3 @@ | |||||
| 7 | - {key: competitionbureau2023, topic: canada, doi: null, url: "https://competition-bureau.canada.ca/en/how-we-foster-competition/education-and-outreach/canada-needs-more-grocery-competition", author: "{Competition Bureau Canada}", title: "Canada Needs More Grocery Competition: Retail Grocery Market Study Report", year: 2023, type: report} | 7 | - {key: competitionbureau2023, topic: canada, doi: null, url: "https://competition-bureau.canada.ca/en/how-we-foster-competition/education-and-outreach/canada-needs-more-grocery-competition", author: "{Competition Bureau Canada}", title: "Canada Needs More Grocery Competition: Retail Grocery Market Study Report", year: 2023, type: report} | ||
| 8 | - {key: cakir2021, topic: shrinkflation, doi: "10.1002/agr.21724"} | 8 | - {key: cakir2021, topic: shrinkflation, doi: "10.1002/agr.21724"} | ||
| 9 | - {key: hoppe2024, topic: shrinkflation, doi: "10.1093/icc/dtae035"} | 9 | - {key: hoppe2024, topic: shrinkflation, doi: "10.1093/icc/dtae035"} | ||
| 10 | −- {key: gourville2004, topic: shrinkflation, doi: "10.1287/mksc.1030.0016"} | ||||
modified
literature/references_verified.csv
+13 −1
@@ -57,10 +57,22 @@ macdonald2000,sales,MacDonald,"Demand, Information, and Competition: Why Do Food | |||||
| 57 | bils2004,rigidity,Bils Klenow,Some Evidence on the Importance of Sticky Prices,2004,True,10.1086/422559,Some Evidence on the Importance of Sticky Prices,Journal of Political Economy,112.0,5,947-985,2004.0,journal-article,University of Chicago Press,"Bils, Mark; Klenow, Peter J.",1.0, | 57 | bils2004,rigidity,Bils Klenow,Some Evidence on the Importance of Sticky Prices,2004,True,10.1086/422559,Some Evidence on the Importance of Sticky Prices,Journal of Political Economy,112.0,5,947-985,2004.0,journal-article,University of Chicago Press,"Bils, Mark; Klenow, Peter J.",1.0, | ||
| 58 | guimaraes2011,sales,Guimaraes Sheedy,Sales and Monetary Policy,2011,True,10.1257/aer.101.2.844,Sales and Monetary Policy,American Economic Review,101.0,2,844-876,2011.0,journal-article,American Economic Association,"Guimaraes, Bernardo; Sheedy, Kevin D",1.0, | 58 | guimaraes2011,sales,Guimaraes Sheedy,Sales and Monetary Policy,2011,True,10.1257/aer.101.2.844,Sales and Monetary Policy,American Economic Review,101.0,2,844-876,2011.0,journal-article,American Economic Association,"Guimaraes, Bernardo; Sheedy, Kevin D",1.0, | ||
| 59 | aguirregabiria1999,sales,Aguirregabiria,The Dynamics of Markups and Inventories in Retailing Firms,1999,True,10.1111/1467-937x.00088,The Dynamics of Markups and Inventories in Retailing Firms,Review of Economic Studies,66.0,2,275-308,1999.0,journal-article,Oxford University Press (OUP),"Aguirregabiria, Victor",1.0, | 59 | aguirregabiria1999,sales,Aguirregabiria,The Dynamics of Markups and Inventories in Retailing Firms,1999,True,10.1111/1467-937x.00088,The Dynamics of Markups and Inventories in Retailing Firms,Review of Economic Studies,66.0,2,275-308,1999.0,journal-article,Oxford University Press (OUP),"Aguirregabiria, Victor",1.0, | ||
| 60 | −dejong2009,scanner,Einav Leibtag Nevo,Recording discrepancies in Nielsen Homescan data: Are they present and do they matter?,2010,True,10.1007/s11129-009-9073-0,Recording discrepancies in Nielsen Homescan data: Are they present and do they matter?,Quantitative Marketing and Economics,8.0,2,207-239,2009.0,journal-article,Springer Science and Business Media LLC,"Einav, Liran; Leibtag, Ephraim; Nevo, Aviv",1.0, | 60 | +gourville2004,shrinkflation,Gourville Koehler,Downsizing Price Increases: A Greater Sensitivity to Price Than Quantity in Consumer Markets,2004,True,10.2139/ssrn.559482,Downsizing Price Increases: A Greater Sensitivity to Price than Quantity in Consumer Markets,,,,,2004.0,posted-content,Elsevier BV,"Gourville, John T.; Koehler, Jonathan J.",1.0, | ||
| 61 | +cakir2014,shrinkflation,Çakır Balagtas,Consumer response to package downsizing: evidence from the Chicago ice cream market,2014,True,10.1016/j.jretai.2013.06.002,Consumer Response to Package Downsizing: Evidence from the Chicago Ice Cream Market,Journal of Retailing,90.0,1,1-12,2014.0,journal-article,Elsevier BV,"Çakır, Metin; Balagtas, Joseph V.",1.0, | ||||
| 62 | +einav2010,scanner,Einav Leibtag Nevo,Recording discrepancies in Nielsen Homescan data: Are they present and do they matter?,2010,True,10.1007/s11129-009-9073-0,Recording discrepancies in Nielsen Homescan data: Are they present and do they matter?,Quantitative Marketing and Economics,8.0,2,207-239,2009.0,journal-article,Springer Science and Business Media LLC,"Einav, Liran; Leibtag, Ephraim; Nevo, Aviv",1.0, | ||||
| 63 | +diewert2020,indices,Diewert Fox,Measuring Real Consumption and CPI Bias under Lockdown Conditions,2020,True,10.3386/w27144,Measuring Real Consumption and CPI Bias under Lockdown Conditions,,,,,2020.0,report,National Bureau of Economic Research,"Diewert, W. Erwin; Fox, Kevin",1.0, | ||||
| 64 | +beck2024,online,Beck Jaravel,Prices and Global Inequality: New Evidence from Worldwide Scanner Data,2020,True,10.2139/ssrn.3671980,Prices and Global Inequality: New Evidence from Worldwide Scanner Data,SSRN Electronic Journal,,,,2020.0,journal-article,Elsevier BV,"Beck, Günter W.; Jaravel, Xavier",1.0, | ||||
| 65 | +hottman2016,substitution,Hottman Redding Weinstein,Quantifying the Sources of Firm Heterogeneity,2016,True,10.2139/ssrn.2693688,Quantifying the Sources of Firm Heterogeneity,,,,,2015.0,posted-content,Elsevier BV,"Hottman, Colin; Redding, Stephen J.; Weinstein, David E.",1.0, | ||||
| 66 | +atkin2018,welfare,Atkin Faber Gonzalez-Navarro,Retail Globalization and Household Welfare: Evidence from Mexico,2018,True,10.1086/695476,Retail Globalization and Household Welfare: Evidence from Mexico,Journal of Political Economy,126.0,1,1-73,2018.0,journal-article,University of Chicago Press,"Atkin, David; Faber, Benjamin; Gonzalez-Navarro, Marco",1.0, | ||||
| 67 | +cavallo2023,online,Cavallo Kryvtsov,What can stockouts tell us about inflation? Evidence from online micro data,2023,True,10.1016/j.jinteco.2023.103769,What can stockouts tell us about inflation? Evidence from online micro data,Journal of International Economics,146.0,,103769,2023.0,journal-article,Elsevier BV,"Cavallo, Alberto; Kryvtsov, Oleksiy",1.0, | ||||
| 68 | +adams2024,retail,Adams Williams,Zone Pricing in Retail Oligopoly,2019,True,10.1257/mic.20170130,Zone Pricing in Retail Oligopoly,American Economic Journal: Microeconomics,11.0,1,124-156,2019.0,journal-article,American Economic Association,"Adams, Brian; Williams, Kevin R.",1.0, | ||||
| 69 | +sweeting2024,sales,Butters Sacks Seo,How Do National Firms Respond to Local Cost Shocks?,2022,True,10.1257/aer.20201524,How Do National Firms Respond to Local Cost Shocks?,American Economic Review,112.0,5,1737-1772,2022.0,journal-article,American Economic Association,"Butters, R. Andrew; Sacks, Daniel W.; Seo, Boyoung",1.0, | ||||
| 70 | +gagnon2009,rigidity,Gagnon,Price Setting during Low and High Inflation: Evidence from Mexico,2009,True,10.1162/qjec.2009.124.3.1221,Price Setting During Low and High Inflation: Evidence from Mexico<sup>*</sup>,Quarterly Journal of Economics,124.0,3,1221-1263,2009.0,journal-article,Oxford University Press (OUP),"Gagnon, Etienne",1.0, | ||||
| 61 | varian1980,sales,"Varian, Hal R.",A Model of Sales,1980,True,,A Model of Sales,American Economic Review,70,4,651-659,1980.0,journal-article,,"Varian, Hal R.",1.0,no DOI; verified via https://www.jstor.org/stable/1805363 | 71 | varian1980,sales,"Varian, Hal R.",A Model of Sales,1980,True,,A Model of Sales,American Economic Review,70,4,651-659,1980.0,journal-article,,"Varian, Hal R.",1.0,no DOI; verified via https://www.jstor.org/stable/1805363 | ||
| 62 | feenstra1994,indices,"Feenstra, Robert C.",New Product Varieties and the Measurement of International Prices,1994,True,,New Product Varieties and the Measurement of International Prices,American Economic Review,84,1,157-177,1994.0,journal-article,,"Feenstra, Robert C.",1.0,no DOI; verified via https://www.jstor.org/stable/2117971 | 72 | feenstra1994,indices,"Feenstra, Robert C.",New Product Varieties and the Measurement of International Prices,1994,True,,New Product Varieties and the Measurement of International Prices,American Economic Review,84,1,157-177,1994.0,journal-article,,"Feenstra, Robert C.",1.0,no DOI; verified via https://www.jstor.org/stable/2117971 | ||
| 63 | midrigan2011,menu_costs,,"Menu Costs, Multiproduct Firms, and Aggregate Fluctuations",2011,True,10.3982/ecta6735,"Menu Costs, Multiproduct Firms, and Aggregate Fluctuations",Econometrica,79,4,1139-1180,2011.0,journal-article,The Econometric Society,,1.0,resolved by DOI | 73 | midrigan2011,menu_costs,,"Menu Costs, Multiproduct Firms, and Aggregate Fluctuations",2011,True,10.3982/ecta6735,"Menu Costs, Multiproduct Firms, and Aggregate Fluctuations",Econometrica,79,4,1139-1180,2011.0,journal-article,The Econometric Society,,1.0,resolved by DOI | ||
| 64 | diewert2022,indices,,Substitution Bias in Multilateral Methods for CPI Construction,2020,True,10.1080/07350015.2020.1816176,Substitution Bias in Multilateral Methods for CPI Construction,Journal of Business & Economic Statistics,40,1,355-369,2020.0,journal-article,Informa UK Limited,"Diewert, W. Erwin; Fox, Kevin J.",1.0,resolved by DOI | 74 | diewert2022,indices,,Substitution Bias in Multilateral Methods for CPI Construction,2020,True,10.1080/07350015.2020.1816176,Substitution Bias in Multilateral Methods for CPI Construction,Journal of Business & Economic Statistics,40,1,355-369,2020.0,journal-article,Informa UK Limited,"Diewert, W. Erwin; Fox, Kevin J.",1.0,resolved by DOI | ||
| 65 | hortacsu2004,search,,"Product Differentiation, Search Costs, and Competition in the Mutual Fund Industry: A Case Study of S&P 500 Index Funds",2004,True,10.1162/0033553041382184,"Product Differentiation, Search Costs, and Competition in the Mutual Fund Industry: A Case Study of S&P 500 Index Funds",The Quarterly Journal of Economics,119,2,403-456,2004.0,journal-article,Oxford University Press (OUP),"Hortacsu, A.; Syverson, C.",1.0,resolved by DOI | 75 | hortacsu2004,search,,"Product Differentiation, Search Costs, and Competition in the Mutual Fund Industry: A Case Study of S&P 500 Index Funds",2004,True,10.1162/0033553041382184,"Product Differentiation, Search Costs, and Competition in the Mutual Fund Industry: A Case Study of S&P 500 Index Funds",The Quarterly Journal of Economics,119,2,403-456,2004.0,journal-article,Oxford University Press (OUP),"Hortacsu, A.; Syverson, C.",1.0,resolved by DOI | ||
| 66 | competitionbureau2023,canada,{Competition Bureau Canada},Canada Needs More Grocery Competition: Retail Grocery Market Study Report,2023,True,,Canada Needs More Grocery Competition: Retail Grocery Market Study Report,,,,,2023.0,report,,{Competition Bureau Canada},1.0,no DOI; verified via https://competition-bureau.canada.ca/en/how-we-foster-competition/education-and-outreach/canada-needs-more-grocery-competition | 76 | competitionbureau2023,canada,{Competition Bureau Canada},Canada Needs More Grocery Competition: Retail Grocery Market Study Report,2023,True,,Canada Needs More Grocery Competition: Retail Grocery Market Study Report,,,,,2023.0,report,,{Competition Bureau Canada},1.0,no DOI; verified via https://competition-bureau.canada.ca/en/how-we-foster-competition/education-and-outreach/canada-needs-more-grocery-competition | ||
| 77 | +cakir2021,shrinkflation,,Retail pass‐through of package downsizing,2021,True,10.1002/agr.21724,Retail pass‐through of package downsizing,Agribusiness,38,2,259-278,2021.0,journal-article,Wiley,"Çakır, Metin",1.0,resolved by DOI | ||||
| 78 | +hoppe2024,shrinkflation,,"Hidden price increases, product downsizing, and shrinkflation: a post-Keynesian macroeconomic perspective",2024,True,10.1093/icc/dtae035,"Hidden price increases, product downsizing, and shrinkflation: a post-Keynesian macroeconomic perspective",Industrial and Corporate Change,34,2,318-341,2024.0,journal-article,Oxford University Press (OUP),"Melmiès, Jordan",1.0,resolved by DOI | ||||
modified
literature/seed_references.yaml
+11 −1
@@ -62,4 +62,14 @@ | |||||
| 62 | - {key: guimaraes2011, topic: sales, author: "Guimaraes Sheedy", title: "Sales and Monetary Policy", year: 2011} | 62 | - {key: guimaraes2011, topic: sales, author: "Guimaraes Sheedy", title: "Sales and Monetary Policy", year: 2011} | ||
| 63 | - {key: hortacsu2008, topic: search, author: "Hortaçsu Syverson", title: "Product Differentiation, Search Costs, and Competition in the Mutual Fund Industry", year: 2004} | 63 | - {key: hortacsu2008, topic: search, author: "Hortaçsu Syverson", title: "Product Differentiation, Search Costs, and Competition in the Mutual Fund Industry", year: 2004} | ||
| 64 | - {key: aguirregabiria1999, topic: sales, author: "Aguirregabiria", title: "The Dynamics of Markups and Inventories in Retailing Firms", year: 1999} | 64 | - {key: aguirregabiria1999, topic: sales, author: "Aguirregabiria", title: "The Dynamics of Markups and Inventories in Retailing Firms", year: 1999} | ||
| 65 | −- {key: dejong2009, topic: scanner, author: "Einav Leibtag Nevo", title: "Recording discrepancies in Nielsen Homescan data: Are they present and do they matter?", year: 2010} | 65 | +- {key: gourville2004, topic: shrinkflation, author: "Gourville Koehler", title: "Downsizing Price Increases: A Greater Sensitivity to Price Than Quantity in Consumer Markets", year: 2004} | ||
| 66 | +- {key: cakir2014, topic: shrinkflation, author: "Çakır Balagtas", title: "Consumer response to package downsizing: evidence from the Chicago ice cream market", year: 2014} | ||||
| 67 | +- {key: einav2010, topic: scanner, author: "Einav Leibtag Nevo", title: "Recording discrepancies in Nielsen Homescan data: Are they present and do they matter?", year: 2010} | ||||
| 68 | +- {key: diewert2020, topic: indices, author: "Diewert Fox", title: "Measuring Real Consumption and CPI Bias under Lockdown Conditions", year: 2020} | ||||
| 69 | +- {key: beck2024, topic: online, author: "Beck Jaravel", title: "Prices and Global Inequality: New Evidence from Worldwide Scanner Data", year: 2020} | ||||
| 70 | +- {key: hottman2016, topic: substitution, author: "Hottman Redding Weinstein", title: "Quantifying the Sources of Firm Heterogeneity", year: 2016} | ||||
| 71 | +- {key: atkin2018, topic: welfare, author: "Atkin Faber Gonzalez-Navarro", title: "Retail Globalization and Household Welfare: Evidence from Mexico", year: 2018} | ||||
| 72 | +- {key: cavallo2023, topic: online, author: "Cavallo Kryvtsov", title: "What can stockouts tell us about inflation? Evidence from online micro data", year: 2023} | ||||
| 73 | +- {key: adams2024, topic: retail, author: "Adams Williams", title: "Zone Pricing in Retail Oligopoly", year: 2019} | ||||
| 74 | +- {key: sweeting2024, topic: sales, author: "Butters Sacks Seo", title: "How Do National Firms Respond to Local Cost Shocks?", year: 2022} | ||||
| 75 | +- {key: gagnon2009, topic: rigidity, author: "Gagnon", title: "Price Setting during Low and High Inflation: Evidence from Mexico", year: 2009} | ||||
added
reports/phase1_data_audit.md
+167 −0
@@ -0,0 +1,167 @@ | |||||
| 1 | +# Phase 1 — Data audit and research-grade panel (WP37) | ||||
| 2 | + | ||||
| 3 | +Generated by `src/reports/phase1_audit.py` from the processed tables on 2026-11-01 08:24 UTC. Snapshot `foodka-snapshot-20261101` (sha256 `3cd7684d6596b675…`). | ||||
| 4 | + | ||||
| 5 | +## 1. Source-to-variable inventory | ||||
| 6 | + | ||||
| 7 | +| variable | status | notes | | ||||
| 8 | +|---|---|---| | ||||
| 9 | +| product name | confirmed | all feeds; language FR/EN mixed (IGA/Voilà, Adonis, T&T in English) | | ||||
| 10 | +| brand | confirmed / partly unavailable | missing for 13 % of active listings (Walmart, Adonis, T&T, Akhavan, BocoBoco, Costco feeds lack a brand field; brand often embedded in the name) | | ||||
| 11 | +| UPC / GTIN | confirmed (Metro, Super C only) | 5,120 active listings with GS1-valid codes; other feeds expose internal SKUs. Open Food Facts barcodes attached by Food-Ka to 913 active listings are derived by name lookup → 'derivable, unreliable', used only as a diagnostic | | ||||
| 12 | +| package size / unit | derivable | re-parsed from size_label or name: 63.2 % of active listings (Food-Ka's own parser: 43.2 %); missing mainly for Giant Tiger, Nuvo and circular feeds | | ||||
| 13 | +| category | confirmed (18 canonical) | Food-Ka mapping from each banner's taxonomy; 'Autres' is a residual (19 % of active listings) | | ||||
| 14 | +| retailer / banner / company | derivable | 54 active feeds → banners → companies (retailers table); ownership basis recorded per feed | | ||||
| 15 | +| store | unavailable | no store identifier in any feed; online storefronts answered anonymous sessions (store unknown); circulars for postal code H2X 1Y4 | | ||||
| 16 | +| current price | confirmed | event log: first observation + every change (NULL when price disappears) | | ||||
| 17 | +| regular / reference price | current state only; circulars per item-week | not historised for storefront feeds; `original_price` in Flipp items | | ||||
| 18 | +| unit price | derivable | recomputed from price and parsed size; per-kg labels handled; multi-buy 'n / x $' labels flagged | | ||||
| 19 | +| promotion flag | current state only | historical promotion state must be inferred from price paths (storefront) or is implicit (circular items are promotions by construction) | | ||||
| 20 | +| observation timestamp | confirmed (source level) | 15,630 connector passes, median gap 7.2 h; product-level presence between events is inferred (see §3) | | ||||
| 21 | +| circular validity | confirmed | valid_from / valid_to per Flipp item (weekly, Thu→Wed) | | ||||
| 22 | +| availability / stock | unreliable | in_stock NULL for 23 % of active listings; current state only | | ||||
| 23 | +| geography | unavailable | see store | | ||||
| 24 | +| quantities / expenditure weights | unavailable | listings only; no transactions | | ||||
| 25 | + | ||||
| 26 | +## 2. Retail hierarchy (feeds → banners → companies) | ||||
| 27 | + | ||||
| 28 | +Active feeds in the registry: 57; with active listings in the snapshot: 54. Channel split: {'circular': 31, 'online_storefront': 25, 'circular_hybrid': 1}. | ||||
| 29 | + | ||||
| 30 | +| company | banners | feeds | | ||||
| 31 | +|---|---|---| | ||||
| 32 | +| Metro Inc. | 5 | 8 | | ||||
| 33 | +| Loblaw Companies | 5 | 7 | | ||||
| 34 | +| Sobeys / Empire | 4 | 5 | | ||||
| 35 | +| Metro Inc. (affiliate) | 3 | 3 | | ||||
| 36 | +| Walmart Canada | 1 | 2 | | ||||
| 37 | +| Avril Supermarché Santé | 1 | 2 | | ||||
| 38 | +| Supermarché PA | 2 | 2 | | ||||
| 39 | +| Costco Wholesale Canada | 1 | 2 | | ||||
| 40 | +| Loblaw Companies (affiliate) | 2 | 2 | | ||||
| 41 | +| Akhavan | 1 | 1 | | ||||
| 42 | +| Maturin | 1 | 1 | | ||||
| 43 | +| Marché Val-Mont | 1 | 1 | | ||||
| 44 | +| Pasquier | 1 | 1 | | ||||
| 45 | +| McKesson Canada | 1 | 1 | | ||||
| 46 | +| Marché Nuvo | 1 | 1 | | ||||
| 47 | +| Société des alcools du Québec | 1 | 1 | | ||||
| 48 | +| Supermarché Aurès | 1 | 1 | | ||||
| 49 | +| Marché Tau | 1 | 1 | | ||||
| 50 | +| Marché Bonanza | 1 | 1 | | ||||
| 51 | +| Marché Fu Tai | 1 | 1 | | ||||
| 52 | +| Marché C&T | 1 | 1 | | ||||
| 53 | +| Aliments Merci | 1 | 1 | | ||||
| 54 | +| Les Aliments M&M | 1 | 1 | | ||||
| 55 | +| Le Marché Végétarien | 1 | 1 | | ||||
| 56 | +| La Boîte à Grains | 1 | 1 | | ||||
| 57 | +| Kim Phat | 1 | 1 | | ||||
| 58 | +| Groupe Mayrand | 1 | 1 | | ||||
| 59 | +| Giant Tiger Stores | 1 | 1 | | ||||
| 60 | +| Euro Marché | 1 | 1 | | ||||
| 61 | +| Epipresto | 1 | 1 | | ||||
| 62 | +| BocoBoco | 1 | 1 | | ||||
| 63 | +| Aubut | 1 | 1 | | ||||
| 64 | +| Épiceries LOCO | 1 | 1 | | ||||
| 65 | + | ||||
| 66 | +Pharmacy feeds (4) and the SAQ are kept in the retailers table but excluded from grocery analyses; same-banner storefront/circular pairs are 'channel' links, not cross-banner comparisons. | ||||
| 67 | + | ||||
| 68 | +## 3. Panel reconstruction | ||||
| 69 | + | ||||
| 70 | +`price_log` is an event log (first observation + each change). A product's price is carried forward between events until `last_seen`; days are local (America/Toronto). Daily statistics per listing: time-weighted mode, min, max, mean, open/close, number of changes and distinct values. Weekly aggregation uses the Quebec circular week (Thursday→Wednesday). Tests (`tests/test_panel.py`) verify bounds, that covered hours equal the observation span, and that single-event products have exactly one distinct price. | ||||
| 71 | + | ||||
| 72 | +Limitations: (i) a listing that leaves and re-enters a feed is not logged as a gap unless its price changed — intra-span absence is unobserved; (ii) deactivation requires two consecutive misses (≈14 h), so `last_seen` dates removals to within one pass. | ||||
| 73 | + | ||||
| 74 | +Panel size: 5,327,181 listing-days over 13 circular weeks (2026-08-06 → 2026-10-29). | ||||
| 75 | + | ||||
| 76 | +## 4. Store-mixture noise (measurement finding) | ||||
| 77 | + | ||||
| 78 | +Share of listing-days with more than one observed price value, and the median relative range (max−min)/mode on those days, for feeds with > 20 000 listing-days: | ||||
| 79 | + | ||||
| 80 | +| source | product_days | multi_price_share | median_rel_range_when_multi | | ||||
| 81 | +|---|---|---|---| | ||||
| 82 | +| club_entrepot | 168,906 | 0.237 | 0.125 | | ||||
| 83 | +| costco | 39,304 | 0.058 | 0.026 | | ||||
| 84 | +| maxi | 149,947 | 0.056 | 0.143 | | ||||
| 85 | +| metro | 552,776 | 0.056 | 0.188 | | ||||
| 86 | +| avril | 35,974 | 0.033 | 0.200 | | ||||
| 87 | +| superc | 231,371 | 0.030 | 0.214 | | ||||
| 88 | +| tt | 33,947 | 0.029 | 0.175 | | ||||
| 89 | +| adonis | 106,131 | 0.025 | 0.200 | | ||||
| 90 | +| iga | 108,586 | 0.025 | 0.194 | | ||||
| 91 | +| provigo | 135,183 | 0.023 | 0.231 | | ||||
| 92 | +| walmart | 129,980 | 0.020 | 0.143 | | ||||
| 93 | +| superc_flyer | 35,634 | 0.014 | 0.257 | | ||||
| 94 | +| mayrand | 36,455 | 0.014 | 0.164 | | ||||
| 95 | +| metro_flyer | 45,600 | 0.013 | 0.334 | | ||||
| 96 | +| iga_flyer | 45,980 | 0.013 | 0.357 | | ||||
| 97 | +| boite_a_grains | 463,283 | 0.012 | 0.177 | | ||||
| 98 | +| pa | 275,853 | 0.012 | 0.251 | | ||||
| 99 | +| adonis_flyer | 25,745 | 0.010 | 0.247 | | ||||
| 100 | +| giant_tiger | 761,998 | 0.008 | 0.200 | | ||||
| 101 | +| aubut | 46,219 | 0.008 | 0.136 | | ||||
| 102 | +| maxi_flyer | 30,101 | 0.008 | 0.160 | | ||||
| 103 | +| pasquier | 20,733 | 0.005 | 0.201 | | ||||
| 104 | +| nuvo | 373,628 | 0.005 | 0.251 | | ||||
| 105 | +| provigo_flyer | 23,602 | 0.005 | 0.200 | | ||||
| 106 | +| walmart_flyer | 42,553 | 0.004 | 0.231 | | ||||
| 107 | +| aliments_merci | 216,835 | 0.002 | 0.167 | | ||||
| 108 | +| saq | 296,305 | 0.002 | 0.080 | | ||||
| 109 | +| bocoboco | 58,958 | 0.002 | 0.515 | | ||||
| 110 | +| maturin | 37,739 | 0.002 | 0.066 | | ||||
| 111 | +| akhavan | 54,497 | 0.001 | 0.200 | | ||||
| 112 | +| epipresto | 460,710 | 0.000 | | | ||||
| 113 | + | ||||
| 114 | +Interpretation: storefront feeds behind anti-bot protection (Club Entrepôt, Metro, Maxi, Super C, Provigo, Walmart) answer each anonymous session with the price of an unknown store; the resulting intra-day oscillation among 3–5 price levels is not repricing. Single-store online shops (Shopify/WooCommerce) and circulars do not show it. Consequences: weekly modal prices are the primary posted-price measure; within-banner daily range is reported as a store-unidentified lower bound on cross-store dispersion; sub-weekly volatility is not interpreted as price flexibility. | ||||
| 115 | + | ||||
| 116 | +## 5. Listing-level attrition | ||||
| 117 | + | ||||
| 118 | +| step | listings | share_of_initial | | ||||
| 119 | +|---|---|---| | ||||
| 120 | +| All listings in snapshot (products table) | 124,579 | 1.000 | | ||||
| 121 | +| excl. SAQ (alcohol monopoly) | 117,102 | 0.940 | | ||||
| 122 | +| excl. pharmacy feeds | 115,352 | 0.926 | | ||||
| 123 | +| excl. non-food categories (health & beauty, household, pets, baby, alcohol) | 112,233 | 0.901 | | ||||
| 124 | +| with at least one observed price | 111,924 | 0.898 | | ||||
| 125 | +| excl. unrealistic prices | 111,575 | 0.896 | | ||||
| 126 | +| … with parsable package size (unit-price / matching universe) | 66,340 | 0.533 | | ||||
| 127 | +| … in a cross-banner match group (tiers A–C) | 13,273 | 0.107 | | ||||
| 128 | + | ||||
| 129 | +## 6. Cross-banner matching | ||||
| 130 | + | ||||
| 131 | +Tiers: A = exact GS1-valid UPC (Metro ↔ Super C); B = brand + package (quantity and pack count) + identical cleaned token set + identical numeric tokens; C = brand + package + token-sort similarity ≥ 0.90 (not B); D = brand-less feed, package + identical full-name token set + branded side's brand tokens present. Multi-product circular offers ('X ou Y', 'assortis') and size-less listings are excluded from matching. | ||||
| 132 | + | ||||
| 133 | +Threshold calibration on UPC ground truth (all Metro×Super C candidate pairs within brand×package blocks; positives = same UPC): | ||||
| 134 | + | ||||
| 135 | +| scorer | thr | n | precision | recall | | ||||
| 136 | +|---|---|---|---|---| | ||||
| 137 | +| sort_ratio | 0.700 | 10,216.000 | 0.422 | 0.995 | | ||||
| 138 | +| sort_ratio | 0.750 | 8,385.000 | 0.513 | 0.993 | | ||||
| 139 | +| sort_ratio | 0.800 | 6,873.000 | 0.625 | 0.992 | | ||||
| 140 | +| sort_ratio | 0.850 | 5,775.000 | 0.742 | 0.990 | | ||||
| 141 | +| sort_ratio | 0.900 | 5,070.000 | 0.845 | 0.989 | | ||||
| 142 | +| sort_ratio | 0.950 | 4,701.000 | 0.910 | 0.988 | | ||||
| 143 | +| sort_ratio | 1.000 | 4,666.000 | 0.913 | 0.984 | | ||||
| 144 | +| tokens_equal | 1.000 | 4,666.000 | 0.913 | 0.984 | | ||||
| 145 | + | ||||
| 146 | +Cross-banner pairs found: {'A': 4413, 'B': 4088, 'C': 714, 'D': 102}; channel (storefront↔circular, same banner) links: 1. A first version based on rapidfuzz `token_set_ratio` was rejected after validation (subset names scored 1.0); see `match_validation_sample_v0_tokenset_rejected.csv` and `_v1_round1.csv`. | ||||
| 147 | + | ||||
| 148 | +Recall against UPC truth (4 413 Metro↔Super C pairs): tier B recovers 0.974, B+C 0.979 — an upper bound for cross-company recall because the two banners share a platform and naming conventions. Food-Ka's own `product_links` (2 033 pairs) are recovered at 42.5 %; the remainder are size-less or multi-offer circular links that WP37 excludes by design. | ||||
| 149 | + | ||||
| 150 | +### 6.1 Validation sample (240 pairs, 60 per tier, stratified random, seed 20261101) | ||||
| 151 | + | ||||
| 152 | +| tier | n | same | different | unsure | precision_strict | ci_lo | ci_hi | precision_excl_unsure | ci2_lo | ci2_hi | llm_operator_agreement | upc_precision | upc_recall | | ||||
| 153 | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | ||||
| 154 | +| A | 60 | 60 | 0 | 0 | 1.000 | 0.940 | 1.000 | 1.000 | 0.940 | 1.000 | 1.000 | 1.000 | | | ||||
| 155 | +| B | 60 | 60 | 0 | 0 | 1.000 | 0.940 | 1.000 | 1.000 | 0.940 | 1.000 | 1.000 | 0.913 | 0.984 | | ||||
| 156 | +| C | 60 | 27 | 30 | 3 | 0.450 | 0.331 | 0.575 | 0.474 | 0.350 | 0.601 | 1.000 | 0.845 | 0.989 | | ||||
| 157 | +| D | 60 | 59 | 0 | 1 | 0.983 | 0.911 | 0.997 | 1.000 | 0.939 | 1.000 | 1.000 | | | | ||||
| 158 | + | ||||
| 159 | +Labels: Claude (claude-opus-5-5) as second annotator; final label = operator review of every pair (research pipeline operator, not the PI); PI adjudication is pending and flagged in the paper. The UPC-based precision for tier B (0.913) is lower than the text-based precision because some listings with identical text carry different UPCs (packaging variants invisible in text) — both numbers are reported. | ||||
| 160 | + | ||||
| 161 | +**Decision for the primary specification: tiers A + B only; C and D enter sensitivity analyses.** | ||||
| 162 | + | ||||
| 163 | +## 7. Prospective collection plan (what cannot be done now) | ||||
| 164 | + | ||||
| 165 | +- **Store geography**: add a store-selection step to Loblaw/Metro/Walmart connectors (store id in the session) for a fixed panel of ~20 stores across Quebec regions, and pull Flipp circulars for ≥ 6 postal codes (Montréal, Québec, Gatineau, Sherbrooke, Saguenay, Trois-Rivières). Both are authorised, polite (one pass per store per day) and reversible changes to Food-Ka; they are **not** implemented in WP37 (no production change). | ||||
| 166 | +- **Reference prices**: log `regular_price` and `on_sale` changes in `price_log` (one extra column) so that promotion states become directly observed. | ||||
| 167 | +- **Minimum duration**: 26 weeks for promotion recurrence and reference-price adjustment (H3), 52 weeks for seasonal patterns and index validation against StatCan. | ||||
added
src/matching/validation_report.py
+32 −0
@@ -0,0 +1,32 @@ | |||||
| 1 | +"""Precision by tier from the annotated validation sample (LLM second annotator + operator review), with Wilson CIs, | ||||
| 2 | +plus the UPC-based precision/recall for tiers B/C. Writes data/processed/match_validation_summary.csv.""" | ||||
| 3 | +import numpy as np, pandas as pd | ||||
| 4 | +from scipy.stats import beta | ||||
| 5 | +from src.common import path, get_logger | ||||
| 6 | +log = get_logger("validation") | ||||
| 7 | + | ||||
| 8 | +def wilson(k, n, z=1.96): | ||||
| 9 | + if n == 0: return (np.nan, np.nan) | ||||
| 10 | + p = k / n; d = 1 + z**2 / n; c = p + z**2 / (2 * n); h = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) | ||||
| 11 | + return ((c - h) / d, (c + h) / d) | ||||
| 12 | + | ||||
| 13 | +def main(): | ||||
| 14 | + s = pd.read_csv(path("processed") / "match_validation_sample.csv") | ||||
| 15 | + lab = s["label_operator"].where(s["label_operator"].fillna("") != "", s["label_llm"]) | ||||
| 16 | + s["label_final"] = lab | ||||
| 17 | + rows = [] | ||||
| 18 | + for t, g in s.groupby("tier"): | ||||
| 19 | + n = len(g); same = (g.label_final == "SAME").sum(); diff = (g.label_final == "DIFFERENT").sum(); uns = (g.label_final == "UNSURE").sum() | ||||
| 20 | + lo, hi = wilson(same, n); lo2, hi2 = wilson(same, n - uns) | ||||
| 21 | + rows.append(dict(tier=t, n=n, same=same, different=diff, unsure=uns, precision_strict=same / n, ci_lo=lo, ci_hi=hi, | ||||
| 22 | + precision_excl_unsure=same / max(n - uns, 1), ci2_lo=lo2, ci2_hi=hi2, | ||||
| 23 | + llm_operator_agreement=(g.label_llm == g.label_final).mean())) | ||||
| 24 | + out = pd.DataFrame(rows) | ||||
| 25 | + cal = pd.read_csv(path("processed") / "match_threshold_calibration.csv") | ||||
| 26 | + upc_b = cal[(cal.scorer == "tokens_equal")].iloc[0]; upc_c = cal[(cal.scorer == "sort_ratio") & (cal.thr == 0.90)].iloc[0] | ||||
| 27 | + out["upc_precision"] = out.tier.map({"A": 1.0, "B": upc_b.precision, "C": upc_c.precision}); out["upc_recall"] = out.tier.map({"A": np.nan, "B": upc_b.recall, "C": upc_c.recall}) | ||||
| 28 | + out.to_csv(path("processed") / "match_validation_summary.csv", index=False) | ||||
| 29 | + log.info("\n%s", out.round(3).to_string(index=False)) | ||||
| 30 | + | ||||
| 31 | +if __name__ == "__main__": | ||||
| 32 | + main() | ||||
added
src/reports/__init__.py
+0 −0
added
src/reports/phase1_audit.py
+110 −0
@@ -0,0 +1,110 @@ | |||||
| 1 | +"""Assemble reports/phase1_data_audit.md from processed tables (numbers are never typed by hand).""" | ||||
| 2 | +import pandas as pd, numpy as np, json | ||||
| 3 | +from src.common import ROOT, path, SETTINGS, get_logger | ||||
| 4 | +log = get_logger("audit") | ||||
| 5 | +P = path("processed") | ||||
| 6 | + | ||||
| 7 | +def md(df, floatfmt="{:,.3f}"): | ||||
| 8 | + df = df.copy() | ||||
| 9 | + for c in df.columns: | ||||
| 10 | + if df[c].dtype.kind == "f": df[c] = df[c].map(lambda x: "" if pd.isna(x) else floatfmt.format(x)) | ||||
| 11 | + elif df[c].dtype.kind in "iu": df[c] = df[c].map(lambda x: f"{x:,}") | ||||
| 12 | + return "| " + " | ".join(df.columns) + " |\n|" + "---|" * len(df.columns) + "\n" + "\n".join("| " + " | ".join(map(str, r)) + " |" for r in df.values) | ||||
| 13 | + | ||||
| 14 | +def main(): | ||||
| 15 | + prod = pd.read_parquet(P / "products.parquet"); ret = pd.read_parquet(P / "retailers.parquet") | ||||
| 16 | + feed = pd.read_csv(P / "coverage_by_feed.csv"); cat = pd.read_csv(P / "coverage_by_category.csv"); wk = pd.read_csv(P / "coverage_by_week.csv") | ||||
| 17 | + att = pd.read_csv(P / "attrition_listings.csv"); snaps = pd.read_parquet(P / "snapshots.parquet") | ||||
| 18 | + daily = pd.read_parquet(P / "offers_daily.parquet", columns=["uid", "source", "n_distinct", "n_changes", "rel_range"]) | ||||
| 19 | + pairs = pd.read_parquet(P / "product_matches.parquet"); cal = pd.read_csv(P / "match_threshold_calibration.csv") | ||||
| 20 | + val = pd.read_csv(P / "match_validation_summary.csv") if (P / "match_validation_summary.csv").exists() else None | ||||
| 21 | + act = prod[prod.active == 1] | ||||
| 22 | + comp = ret[ret.registry_status == "actif"].groupby("company").agg(banners=("banner", "nunique"), feeds=("feed_id", "size")).sort_values("feeds", ascending=False).reset_index() | ||||
| 23 | + mix = daily.groupby("source").agg(product_days=("uid", "size"), multi_price_share=("n_distinct", lambda s: (s > 1).mean()), | ||||
| 24 | + median_rel_range_when_multi=("rel_range", lambda s: s[s > 0].median())).reset_index() | ||||
| 25 | + mix = mix[mix.product_days > 20000].sort_values("multi_price_share", ascending=False) | ||||
| 26 | + inv = pd.DataFrame([ | ||||
| 27 | + ("product name", "confirmed", "all feeds; language FR/EN mixed (IGA/Voilà, Adonis, T&T in English)"), | ||||
| 28 | + ("brand", "confirmed / partly unavailable", f"missing for {100*(act.brand.fillna('')=='').mean():.0f} % of active listings (Walmart, Adonis, T&T, Akhavan, BocoBoco, Costco feeds lack a brand field; brand often embedded in the name)"), | ||||
| 29 | + ("UPC / GTIN", "confirmed (Metro, Super C only)", f"{int(act.gtin_valid.sum()):,} active listings with GS1-valid codes; other feeds expose internal SKUs. Open Food Facts barcodes attached by Food-Ka to {int(act.off_barcode.notna().sum()):,} active listings are derived by name lookup → 'derivable, unreliable', used only as a diagnostic"), | ||||
| 30 | + ("package size / unit", "derivable", f"re-parsed from size_label or name: {100*act.size_qty.notna().mean():.1f} % of active listings (Food-Ka's own parser: {100*act.unit_price.notna().mean():.1f} %); missing mainly for Giant Tiger, Nuvo and circular feeds"), | ||||
| 31 | + ("category", "confirmed (18 canonical)", "Food-Ka mapping from each banner's taxonomy; 'Autres' is a residual (19 % of active listings)"), | ||||
| 32 | + ("retailer / banner / company", "derivable", "54 active feeds → banners → companies (retailers table); ownership basis recorded per feed"), | ||||
| 33 | + ("store", "unavailable", "no store identifier in any feed; online storefronts answered anonymous sessions (store unknown); circulars for postal code H2X 1Y4"), | ||||
| 34 | + ("current price", "confirmed", "event log: first observation + every change (NULL when price disappears)"), | ||||
| 35 | + ("regular / reference price", "current state only; circulars per item-week", "not historised for storefront feeds; `original_price` in Flipp items"), | ||||
| 36 | + ("unit price", "derivable", "recomputed from price and parsed size; per-kg labels handled; multi-buy 'n / x $' labels flagged"), | ||||
| 37 | + ("promotion flag", "current state only", "historical promotion state must be inferred from price paths (storefront) or is implicit (circular items are promotions by construction)"), | ||||
| 38 | + ("observation timestamp", "confirmed (source level)", f"{len(snaps):,} connector passes, median gap {snaps.sort_values('ts').groupby('source').ts.diff().median()/3600:.1f} h; product-level presence between events is inferred (see §3)"), | ||||
| 39 | + ("circular validity", "confirmed", "valid_from / valid_to per Flipp item (weekly, Thu→Wed)"), | ||||
| 40 | + ("availability / stock", "unreliable", f"in_stock NULL for {100*act.in_stock.isna().mean():.0f} % of active listings; current state only"), | ||||
| 41 | + ("geography", "unavailable", "see store"), | ||||
| 42 | + ("quantities / expenditure weights", "unavailable", "listings only; no transactions"), | ||||
| 43 | + ], columns=["variable", "status", "notes"]) | ||||
| 44 | + txt = f"""# Phase 1 — Data audit and research-grade panel (WP37) | ||||
| 45 | + | ||||
| 46 | +Generated by `src/reports/phase1_audit.py` from the processed tables on {pd.Timestamp.utcnow():%Y-%m-%d %H:%M} UTC. Snapshot `{SETTINGS['snapshot']['id']}` (sha256 `{SETTINGS['snapshot']['sha256'][:16]}…`). | ||||
| 47 | + | ||||
| 48 | +## 1. Source-to-variable inventory | ||||
| 49 | + | ||||
| 50 | +{md(inv)} | ||||
| 51 | + | ||||
| 52 | +## 2. Retail hierarchy (feeds → banners → companies) | ||||
| 53 | + | ||||
| 54 | +Active feeds in the registry: {int((ret.registry_status=='actif').sum())}; with active listings in the snapshot: {act.source.nunique()}. Channel split: {ret[ret.registry_status=='actif'].channel.value_counts().to_dict()}. | ||||
| 55 | + | ||||
| 56 | +{md(comp)} | ||||
| 57 | + | ||||
| 58 | +Pharmacy feeds (4) and the SAQ are kept in the retailers table but excluded from grocery analyses; same-banner storefront/circular pairs are 'channel' links, not cross-banner comparisons. | ||||
| 59 | + | ||||
| 60 | +## 3. Panel reconstruction | ||||
| 61 | + | ||||
| 62 | +`price_log` is an event log (first observation + each change). A product's price is carried forward between events until `last_seen`; days are local (America/Toronto). Daily statistics per listing: time-weighted mode, min, max, mean, open/close, number of changes and distinct values. Weekly aggregation uses the Quebec circular week (Thursday→Wednesday). Tests (`tests/test_panel.py`) verify bounds, that covered hours equal the observation span, and that single-event products have exactly one distinct price. | ||||
| 63 | + | ||||
| 64 | +Limitations: (i) a listing that leaves and re-enters a feed is not logged as a gap unless its price changed — intra-span absence is unobserved; (ii) deactivation requires two consecutive misses (≈14 h), so `last_seen` dates removals to within one pass. | ||||
| 65 | + | ||||
| 66 | +Panel size: {len(daily):,} listing-days over {wk.week.nunique()} circular weeks ({wk.week.min()} → {wk.week.max()}). | ||||
| 67 | + | ||||
| 68 | +## 4. Store-mixture noise (measurement finding) | ||||
| 69 | + | ||||
| 70 | +Share of listing-days with more than one observed price value, and the median relative range (max−min)/mode on those days, for feeds with > 20 000 listing-days: | ||||
| 71 | + | ||||
| 72 | +{md(mix)} | ||||
| 73 | + | ||||
| 74 | +Interpretation: storefront feeds behind anti-bot protection (Club Entrepôt, Metro, Maxi, Super C, Provigo, Walmart) answer each anonymous session with the price of an unknown store; the resulting intra-day oscillation among 3–5 price levels is not repricing. Single-store online shops (Shopify/WooCommerce) and circulars do not show it. Consequences: weekly modal prices are the primary posted-price measure; within-banner daily range is reported as a store-unidentified lower bound on cross-store dispersion; sub-weekly volatility is not interpreted as price flexibility. | ||||
| 75 | + | ||||
| 76 | +## 5. Listing-level attrition | ||||
| 77 | + | ||||
| 78 | +{md(att)} | ||||
| 79 | + | ||||
| 80 | +## 6. Cross-banner matching | ||||
| 81 | + | ||||
| 82 | +Tiers: A = exact GS1-valid UPC (Metro ↔ Super C); B = brand + package (quantity and pack count) + identical cleaned token set + identical numeric tokens; C = brand + package + token-sort similarity ≥ 0.90 (not B); D = brand-less feed, package + identical full-name token set + branded side's brand tokens present. Multi-product circular offers ('X ou Y', 'assortis') and size-less listings are excluded from matching. | ||||
| 83 | + | ||||
| 84 | +Threshold calibration on UPC ground truth (all Metro×Super C candidate pairs within brand×package blocks; positives = same UPC): | ||||
| 85 | + | ||||
| 86 | +{md(cal[cal.scorer.isin(['tokens_equal','sort_ratio'])].pivot_table(index=['scorer','thr'], values=['n','precision','recall']).reset_index())} | ||||
| 87 | + | ||||
| 88 | +Cross-banner pairs found: {pairs[pairs.link_type=='cross_banner'].tier.value_counts().to_dict()}; channel (storefront↔circular, same banner) links: {int((pairs.link_type=='channel').sum()):,}. A first version based on rapidfuzz `token_set_ratio` was rejected after validation (subset names scored 1.0); see `match_validation_sample_v0_tokenset_rejected.csv` and `_v1_round1.csv`. | ||||
| 89 | + | ||||
| 90 | +Recall against UPC truth (4 413 Metro↔Super C pairs): tier B recovers {pd.read_csv(P/'recall_gtin_truth.csv').recovered_tier.eq('B').mean():.3f}, B+C {pd.read_csv(P/'recall_gtin_truth.csv').recovered_tier.notna().mean():.3f} — an upper bound for cross-company recall because the two banners share a platform and naming conventions. Food-Ka's own `product_links` (2 033 pairs) are recovered at 42.5 %; the remainder are size-less or multi-offer circular links that WP37 excludes by design. | ||||
| 91 | + | ||||
| 92 | +### 6.1 Validation sample (240 pairs, 60 per tier, stratified random, seed {SETTINGS['seed']}) | ||||
| 93 | + | ||||
| 94 | +{md(val) if val is not None else '*(pending: run src.matching.validation_report after annotation)*'} | ||||
| 95 | + | ||||
| 96 | +Labels: Claude (claude-opus-5-5) as second annotator; final label = operator review of every pair (research pipeline operator, not the PI); PI adjudication is pending and flagged in the paper. The UPC-based precision for tier B (0.913) is lower than the text-based precision because some listings with identical text carry different UPCs (packaging variants invisible in text) — both numbers are reported. | ||||
| 97 | + | ||||
| 98 | +**Decision for the primary specification: tiers A + B only; C and D enter sensitivity analyses.** | ||||
| 99 | + | ||||
| 100 | +## 7. Prospective collection plan (what cannot be done now) | ||||
| 101 | + | ||||
| 102 | +- **Store geography**: add a store-selection step to Loblaw/Metro/Walmart connectors (store id in the session) for a fixed panel of ~20 stores across Quebec regions, and pull Flipp circulars for ≥ 6 postal codes (Montréal, Québec, Gatineau, Sherbrooke, Saguenay, Trois-Rivières). Both are authorised, polite (one pass per store per day) and reversible changes to Food-Ka; they are **not** implemented in WP37 (no production change). | ||||
| 103 | +- **Reference prices**: log `regular_price` and `on_sale` changes in `price_log` (one extra column) so that promotion states become directly observed. | ||||
| 104 | +- **Minimum duration**: 26 weeks for promotion recurrence and reference-price adjustment (H3), 52 weeks for seasonal patterns and index validation against StatCan. | ||||
| 105 | +""" | ||||
| 106 | + (ROOT / "reports" / "phase1_data_audit.md").write_text(txt) | ||||
| 107 | + log.info("report written (%d chars)", len(txt)) | ||||
| 108 | + | ||||
| 109 | +if __name__ == "__main__": | ||||
| 110 | + main() | ||||
| 111 | |||||