# Referee report 1 — Econometrics (adversarial), WP38 "The Economic Geography of Digital Production"

Scope: paper/main.pdf (sections, tables_model/, tables/), research/preregistration.md, research/decisions.md, src/models/*.py, src/validation/robustness.py, experiments_model/*.json (primary) and experiments/*.json. Checks were run read-only with .venv/bin/python on the saved artefacts (data/processed_model/h5_groups.parquet, store_category.parquet, store_panel.parquet, h4_panel.parquet, h2_region_table.csv); scripts in /tmp/wp38ref/. Nothing in the pipeline was rerun.

Bottom line: two of the five confirmatory conclusions do not survive replication with corrected code (H5-T1 and the power-law-vs-log-normal statement inside H3), the cell-level H1 claims are mostly bootstrap artefacts, and two preregistered procedures (H1 decision rule, H2 interval) are not what the paper reports. H3's thresholds, H5-T2, H4 and the H7 negative result replicate.

# 1. Five most serious threats

# T1. H5-T1 (regional price structure): the wild cluster bootstrap is mis-implemented for crossed group FE / store clusters, and the result is one region (one store)

src/models/h5_hedonic.py:82-83 builds yb = yhat_r + e_r * w on the group-demeaned design and computes eb = yb - Xw @ bb without re-projecting the bootstrap disturbance off the 1,064 match-group fixed effects. Because stores (clusters) cut across groups, e_r * w is not group-demeaned; the coefficient bb is unaffected (Xw' M_D = Xw') but the cluster-robust meat at line 83 uses residuals contaminated by the group means of e_r*w, which inflates the bootstrap variance, shrinks the bootstrap Wald draws, and makes the observed Wald look extreme. The paper and preregistration describe this as "restricted residuals, Rademacher by store" (methods.tex, §Matched products), which it is only in name.

Second, the joint Wald of 164.4 (14 df) is 65.6 from Outaouais alone (r::Outaouais b = 0.224, SE 0.028, 5 stores, of which laboiteagrains.com contributes 84 of 110 observations). The next largest squared t is 4.6 (Saguenay).

Falsification (/tmp/wp38ref/h5_check.py, h5_res.py; same seed 20261101, B = 1,999 unless stated):

H5-T2 (reseller) survives the corrected bootstrap: b = −0.116, t = −4.37, p = 0.0045 FE-reprojected (0.0005 code, 0.0012 paper); without Outaouais stores −0.128, p = 0.005. Holm on the corrected family {H1 0.445, H2 1.0, H3 0, T1 0.100, T2 0.0045} gives T2 adj. 0.018 (supported) and T1 adj. 0.30 (not supported). The abstract's "region effects are jointly significant under a wild cluster bootstrap (p = 0.005)" is not supportable; what the data show is one Outaouais bulk-food reseller pricing 20 log points above Montréal.

# T2. H3 "power-law tail favoured over log-normal (Vuong z = 4.99)": the log-normal was never fitted

src/models/h3_longtail.py:44-45 sets mu, sig to the sample mean and SD of log(t) on the tail and then applies a truncation normaliser to the density. That is not the MLE of a truncated log-normal (Clauset–Shalizi–Newman §5 is explicit that the alternative must be fitted on the same tail by maximum likelihood); the mis-fitted log-normal has log-likelihood −2149.4 vs −2088.7 for the power law, which manufactures the z of 4.99.

Falsification (/tmp/wp38ref/h3_check.py, same tail x_min = 165, n = 288, α = 1.929):

The two families are indistinguishable on this tail (the usual CSN outcome; the MLE drifts to the degenerate μ ≪ 0, σ ≫ 1 log-normal that mimics a power law). results.tex ("favoured over a log-normal… Vuong z = 4.99"), Table 3, the abstract ("with a power-law tail") and the H3 registry note must be changed to "not distinguishable". The Gini/top-share thresholds and their store-bootstrap intervals replicate and H3 remains supported.

# T3. H1 cell-level claims (12 advantages, 64 "robust absences") are mostly store-bootstrap artefacts with no multiplicity control

src/models/h1_raa.py:55-65 draws 2,000 store resamples with replacement, replaces empty cells by 1e-12 (line 61) and calls a cell "significant" if its percentile interval excludes 1 — 340 cells, no family-wise control. The paper then narrates the top cells and concludes "the absences are more robust than the advantages" (results.tex, H1 subsection). The same resampling is acknowledged to bias V upward (bootstrap mean 0.137 vs observed 0.111, my replication), and the RAA ratio with sparse cells inherits that bias asymmetrically toward intervals below 1.

Falsification (/tmp/wp38ref/h1_check.py, scheme M, 500 boots per run): observed 10 cells with CI > 1 and 68 with CI < 1. Under three merchant reshuffles of region labels (the paper's own null) the counts are 4/50, 2/59 and 4/53. So roughly 50–60 of the 64 "significant absences" are expected under the null, and the excess of advantages is 6–10 cells over a baseline of 2–4, with no identification of which ones. Several named cells rest on ≤ 16 stores (Gaspésie fashion: 16 stores in the cell's region, 3 in the cell). The heat-map dots, Table raa_top and the sentence quoted above should be re-expressed with a permutation-calibrated or Holm/BH-adjusted criterion, or presented as descriptive only.

# T4. H1 decision rule, power under L/T, and the "in every variant" claim

(a) Preregistration §4 required p < 0.05 under M and T, Holm-adjusted. src/models/summary.py:23 correctly sets p_raw(H1) = max(pM, pT) = 0.445, Holm 0.89 — "not supported" by the frozen rule. Line 30 then overrides this with a non-preregistered label "partially supported" whenever pM < 0.05 ≤ pT. The paper should say the preregistered H1 failed and report the merchant-equal result as a post-hoc decomposition.

(b) "Indistinguishable from chance under listing weights" is a power statement, not evidence of absence. From experiments_model/h1_raa.json: null SD ≈ 0.0043 under M (excess 5.9 SD), 0.027 under L (0.8 SD), 0.0155 under T (0.1 SD): the merchant-reshuffle null is 3.6–6× noisier under L/T because the four largest stores (24,104 / 20,210 / 18,147 / 12,864 listings) hold 25.2% of S_R listings. The L result also flips with the region-recovery rule: S_R0 L p = 0.0025 (Table 1, row 5) vs S_R L p = 0.21 — i.e. it depends on where the three recovered mega-stores (qublivre.ca Montérégie via city, remorquetrailer.com Capitale-Nationale via pip) land. The text's "as good as random" should become "the test has no power once a few stores dominate".

(c) The abstract says the merchant-equal V "exceeds 2,000 merchant reshuffles … in every sample variant". The variants use 1,000 permutations (robustness.py:22) and excl_Montreal gives p = 0.009 (9 permutations exceed), experiments_model/robustness.json. Also robustness.py:54-69 applies the rule-vs-consensus confusion matrix to the classifier labels when WP38_LABEL=model, so the "82% of draws reject" figure for the primary labelling perturbs the wrong labels.

# T5. H2: interval not as preregistered, power overstated, and the unrarefied association is withheld from the text

src/models/h2_scale.py:17-24 accepts boot_y for the preregistered "region-level bootstrap combined with store-level resampling of the outcome" but line 46 never passes it; the reported CI is a region-only bootstrap. Falsification (/tmp/wp38ref/h2_check.py, n = 16): code CI [−0.63; 0.59]; Fisher-z [−0.51; 0.51]; adding the Monte-Carlo error of the rarefied mean [−0.61; 0.59]; adding the full 15-store draw dispersion [−0.68; 0.65]. Conclusion unchanged, but the paper's statement should match what was run. Power: with n = 16 the 80%-power detectable |ρ| is 0.66 (Fisher-z), not "≳ 0.5" (prereg) — the design can only detect very large associations. Rarefying Montréal to 15 of 237 stores discards most of the data: the unrarefied merchant-equal entropy has ρ = 0.49 with log population (exact p = 0.046, Table 2 rows 5–7, 0.49–0.52 with all three scale covariates), and its rank correlation with the rarefied series is only 0.79. The abstract's "no detectable association with population, business counts, income" is true of the preregistered rarefied statistic only; the text should say that the unrarefied statistic points to a moderate positive association that the family does not confirm. (The Fisher-z p = 1.0 at summary.py:25 and the H3 placeholder p = 0 at line 26 enter Holm only for ordering; I verified they do not change the other adjusted p-values. Table 2 note "Nord-du-Québec has no located store" is wrong — it has 3.)

# 2. Checks that passed

• H4 LPM: alternating demeaning (h4_dynamics.py:35-39) converges by 5 iterations (max |store mean| 4e-8, 2e-18 at 50); sparse exact two-way FE gives b = 0.06504 = reported 0.065; SE 0.0186 replicates. Entry rate rises from 0.17% (density decile 1) to 1.05% (decile 10). Exploratory status is appropriate; 200 placebo draws make p = 0.01 the floor.
• H1 permutation design (store-level relabelling, store counts preserved) is correct for the stated null; p-value uses (k+1)/(N+1).
• H3 Gini 0.849 [0.811; 0.878], top-10% 0.789 replicate.
• H7 negative result and QDACI withholding follow the preregistered rules; nothing to contest.
• H5 identification: 920 of 1,064 groups span ≥ 2 regions (1,915 obs); Abitibi, Gaspésie and Mauricie have 1–3 cross-region groups.

# 3. Unresolved limitations (could not be tested here)

• Match false-positive rate (≈ 10%) interacts with the Outaouais effect: whether laboiteagrains.com groups are true matches is unverifiable without the annotation file.
• Dependence of the H5 bootstrap on the single-cluster regions (Abitibi, Côte-Nord): with Rademacher weights a dummy identified by one cluster has a degenerate bootstrap law (MacKinnon–Webb 2018); merging small regions is the only fix I could run.
• The rarefied H2 outcome could not be recomputed with full store-level resampling because the 2,000 draw arrays are not saved.
• The confusion-reassignment robustness for the classifier labelling cannot be corrected without a classifier-vs-consensus confusion matrix.

# 4. Remediation plan

Must fix before circulation

1. h5_hedonic.py:82-86: after yb = yhat_r + e_r * w (and yb2), demean the bootstrap disturbance by group — u = e_r * w; u -= pd.Series(u).groupby(d.group.values).transform("mean").values — before computing eb; rerun; report T1 p ≈ 0.10, Wald decomposition by region, and the no-Outaouais / merged-regions variants in Table 5. Rewrite abstract, §Results H5, Table 5, registry: T1 not supported; T2 supported.
2. h3_longtail.py:41-50: fit the truncated (discrete) log-normal by MLE on the same support (or use the powerlaw package, distribution_compare('power_law','lognormal')); report z ≈ −0.2 to −0.6, "not distinguishable". Remove "power-law tail" from abstract and conclusion.
3. H1 cells: add a permutation-calibrated significance count (number of cells with CI excluding 1 under ≥ 100 reshuffles) to Table 1 and delete "the absences are more robust than the advantages"; present named cells as descriptive.
4. H1 status: report "not supported under the preregistered rule (Holm p = 0.89); merchant-equal component significant" and drop "partially supported" from the registry (summary.py:30). Correct "exceeds 2,000 reshuffles in every variant" (1,000; excl_Montréal p = 0.009).

Should fix 5. h2_scale.py:46: pass store-level outcome draws (save them in features/diversity.py) or state that the CI is region-only; replace "≳ 0.5" by 0.66; report the unrarefied ρ ≈ 0.5 (p ≈ 0.05) in the text, not only in the table. 6. robustness.py:54-69: under WP38_LABEL=model, skip or rebuild the confusion run with a classifier-vs-consensus matrix. 7. Fix the Table 2 note (Nord-du-Québec has 3 stores, excluded by the 15-store rule).

Nice to have 8. H5: merge regions with < 5 matched stores into one dummy in the primary specification; add a leave-one-store-out plot for region effects; consider webb weights for G = 143. 9. H1 L/T: report the null SD or an excess-V/SD ratio so readers see the power difference across schemes. 10. H4: 1,000 placebo draws; G/(G−1) factor in lpm().