# Referee 3 — Data engineering and reproducibility (WP38)

Adversarial audit of the pipeline, 2026-11-01. Scope: code under src/, Makefile, config, ledger/manifest, cached artefacts, and the consistency between text and outputs. Nothing outside reports/ was modified; no pipeline step or LLM call was rerun. All numbers below were measured with .venv/bin/python on the artefacts present at commit 47d768d.

# 1. Five most serious threats

# T1 — Match-validation precision is mis-computed (and the paper repeats the wrong number)

src/validation/match_validation.py:90-92 builds cons = llm_a.where(llm_a == llm_b) (NaN when annotators disagree) and then reports yes = (cons[tier] == "yes").mean(). A NaN compares as False, so the denominator is all 60 pairs, while the n printed beside it is the consensus count (56/54/52/48). The same bug affects no and precision_by_tier_comparable_only.

Check run: recomputed from data/processed/match_validation_sample.csv.

paper/sections/data.tex:24 states "among consensus pairs the share judged the same product is 0.92 … 0.90 … 0.77 … 0.17", reports/phase2_data_audit.md §2.5, reports/limitations.md L-10 ("precision 0.90–0.92") and the abstract ("matching precision 0.90–0.92") all inherit the mismatch. The direction is conservative, but the text describes a statistic that is not what the code computes.

# T2 — The audit sample is not reproducible from the pipeline and leaks across the clean/sibling boundary

category_audit.sample() draws 25 listings per category from f_analysis_assort with seed 20261101 and then refuses to overwrite. The sample on disk contains 1 listing (uid 09cadb97…, store cafewilliamspartivento.com) that is outside the current assortment sample because that store is a non-representative sibling (f_sibling_dup = True). The sample was therefore drawn before src/clean/siblings.py existed in the clean chain (ledger: clean.listings 17:48:50Z, category_audit.annotate 17:56:56Z, clean.siblings first at 17:52:34Z, second clean.listings 17:52:57Z — the sample itself is not ledgered). Re-running make clean + category_audit sample from a fresh clone will produce a different 500-listing sample, so the 0.46 precision, the 474-listing classifier hold-out and A-002's primacy decision cannot be regenerated bit-for-bit. Neither category_audit_sample.csv, match_validation_sample.csv nor category_train_sample.csv is registered in logs/manifest.json (59 entries, none under data/processed/*.csv), so there is no checksum tying the published figures to the cached labels.

Related leakage: uids are disjoint between audit and training (0 overlap, verified), but 92.2 % of audit stores also appear in the training sample. With merchants publishing near-duplicate titles, the "held-out" 0.88 accuracy is a within-merchant estimate, not the merchant-holdout the mandate (Phase 5) asks for.

# T3 — H4 is built on survivors only; its docstring and "exits" are false by construction

src/models/h4_dynamics.py:47 reads li[li.f_published | (li.active == 0)] with the comment "all live-store listings incl. inactive". But listings_clean.parquet is filtered to f_published & f_live_store in src/clean/listings.py:44; verified: 390 108 rows, active ∈ {1} only, and 0 rows with last_seen before the outcome window (2026-09-28). Consequently (i) baseline portfolios (W36–W39) are reconstructed from listings that survived to the snapshot, so a category whose listings were all withdrawn before the snapshot looks "absent at baseline" and any later listing becomes a spurious "entry"; (ii) region_level_exits is [] in both namespaces because every baseline listing is, by construction, still present in W40–W44. The 73 618 inactive listings (source audit §0.2.3) exist in listings_raw.parquet and were simply never loaded.

# T4 — Reproducibility claims in the paper/README do not match what is versioned

• paper/sections/reproducibility.tex says "the labelled audit samples … the run ledger with every API call and its cost, and the SHA-256 of every output are in the repository". .gitignore lines 4 and 31 ignore data/processed/* and logs/; git ls-files logs data/processed returns only data/processed/.gitkeep. The ledger records aggregate token counts per annotation run, not per call, and no cost.
• README.md TL;DR reports "Cramér's V 0.107 vs 0.075 … trimmed-listing p = 0.46 … ρ = −0.20, CI [−0.62; 0.34]". The rule namespace gives V = 0.1049 / 0.0747 / p_T = 0.492; the model namespace (declared primary) gives 0.1114 / 0.0858 / 0.445 and ρ = 0.00 [−0.63; 0.60]. 0.107 matches neither; the README mixes a stale H1 run with rule-label H2 while the paper abstract correctly uses the model namespace.
• .gitignore ignores data/processed/* but not data/processed_model/ (asymmetric glob), so 19 label-dependent artefacts are tracked, including store_panel.parquet (1 557 merchant domains with FSA centroid lat/lng and median_price) and h5_groups.parquet (2 203 listing-level prices). The paper states the catalogue data are not redistributed.
• Neither the category_model steps (sample|annotate|embed|train) nor src.models.summary (hypothesis registry) appear in the Makefile, so WP38_LABEL=model make features … fails on a fresh clone (no category_model.parquet), and make all fails at validate without the cached CSVs that are not in git.
• Stale documentation: reports/data_dictionary.md still defines region_final as "region_db, else region_pip" and region_source as "db | pip | unknown" (A-001 made city the first rule, 393 stores); data/raw/README.md points external data to data/raw/external/; reports/00_source_audit.md §0.1.1 says "No empirical result depends on an LLM" although the primary labelling (A-002) rests on claude-sonnet-5-5 labels and OpenAI embeddings.

# T5 — Label namespaces and sensitivity samples: one silent misnomer, one silent fallback, no test coverage

• src/clean/stores.py:98 defines region_dbfirst = region_pip.fillna(region_city).fillna(region_db) with the comment "this is the PIP-first sensitivity variant"; region_category.py:34, h1_raa.py:41 and robustness.py:44 ("S_R0_dbfirst_regions") use it for S_R0. The preregistration (§ samples) defines S_R0 as "stores with a Fabri-Ka region only (region_source = db)". Verified: region_dbfirst == region_pip for 100 % of stores with a PIP value. So the S_R0 rows of Table h1 (label "S_R0_dbfirst_regions") are PIP-first regions on the 1 223 DB-labelled stores — not the preregistered sample, and the table label says the opposite.
• stores.py:23-45 globs every .gpkg/.shp/.geojson under data/external/ and accepts the first layer whose name column covers ≥ 15 regions; if the MERN file is missing it silently falls back to qc_er_2021_statcan_cbf.gpkg (pre-2021 Estrie/Montérégie boundaries, the very artefact A-001 removed), logging only a filename.
• Namespace separation itself is correct (verified: data/processed_model/ holds only label-dependent files; label-independent store_panel.parquet is byte-identical in both; audit/matching artefacts live only in base dirs; category_model.parquet is written to BASE_PROC). But ledger() (common.py:45) never records LABEL: 139 ledger lines, 0 with a label field, and steps such as models.h1_raa appear 9 times and reports.make_tables 18 times with no way to tell which namespace they wrote. tests/test_panel.py (7 tests, all pass in 0.85 s) covers none of: audit/train disjointness, namespace separation, S_R0 definition, consensus-precision arithmetic, H4 inputs.

# 2. Checks actually run

# 3. Unresolved limitations

• Determinism was verified only indirectly (manifest hashes match disk; seeds set everywhere; thread-pool results consumed in order); no two-run comparison exists because a rerun was out of scope.
• LLM annotations are intrinsically non-reproducible (reasoning models, thinking: adaptive); only caching protects them, and the caches are unversioned and unhashed (T2/T4).
• region_pip rests on FSA centroids geocoded upstream (zippopotam/GeoNames); the 40 straddling FSAs flagged in fsa_to_region_2021.csv are never joined to stores_master.
• The sample-flow step "of an enabled store with ≥ 1 published listing" removes 0 rows; the table does not say that f_live_store is redundant in this snapshot.

# 4. Remediation plan

Must-fix (before submission)

1. match_validation.report: compute c = cons[df.tier == t].dropna() and report yes = (c == "yes").mean(), no, unsure, n = len(c); same for comparable_only. Regenerate experiments/match_validation.json, paper/tables/match_validation.tex, phase2_data_audit.md, limitations.md L-10, data.tex:24, abstract.
2. Register the three LLM sample CSVs in logs/manifest.json (register_output in sample() and after each annotate()), add their sha256 to reports/reproducibility.md, and either commit them (they contain no personal data) or publish them as a release asset; drop the uid from the sibling store or document it as a frozen pre-sibling draw.
3. Reconcile reproducibility.tex and README.md with git: track logs/run_ledger.jsonl and logs/manifest.json (or say they are published separately); fix the README numbers to the primary namespace (0.111 / 0.086; p = 0.45; ρ = 0.00 [−0.63; 0.60]).
4. .gitignore: replace data/processed/* by data/processed*/ (or deliberately track both with a privacy review of store_panel.parquet / h5_groups.parquet); state the policy in data/raw/README.md.
5. H4: load listings_raw.parquet filtered to live, non-sibling stores and listing_status == 'published' regardless of active, so that baseline presence includes withdrawn listings; recompute entries/exits; or rewrite the docstring and the paper to say "entries among surviving listings; exits unobservable".
6. Rename region_dbfirst → region_pipfirst and the robustness key to S_R0_pipfirst_regions; either add the preregistered "Fabri-Ka label only" variant or amend the preregistration explicitly (A-003).

Should-fix 7. Add label=LABEL to every ledger() record and a cost_usd field computed from the model price list; add a make model-labels target (sample annotate embed train) and a make registry target for summary.py; make make validate skip gracefully when CSVs are absent. 8. load_regions_layer: take the layer path from settings.yaml (data/external/geo/qc_regions_admin_mern_bdat100k.gpkg) and raise if missing. 9. Tests: audit∩train = ∅ and audit ⊂ assortment; data/processed_model contains no audit/matching files; store_panel identical across namespaces; H4 inputs include active == 0; consensus-precision denominators; region_final precedence city > db > pip. 10. Store-level holdout for the classifier (GroupKFold by store_id, and an audit-store-disjoint evaluation) reported next to the 0.88.

Nice-to-have 11. Regenerate reports/data_dictionary.md with the A-001 definitions (pull DESC for region_final, region_source, region_conflict from one place shared with stores.py); fix data/raw/README.md, 00_source_audit.md §0.1.1 and data/external/README.md. 12. Break the listings ↔ siblings cycle (compute siblings from listings_raw + norm_title) so make clean runs each step once; split the tables figures recipe. 13. Join the 40 straddling FSAs as a geo_uncertain flag and add it to the robustness list already promised in data/external/README.md.