# F4-conf-04 — Verification of a five-item reference list

# Question

For each of five cited arXiv studies, I checked whether (i) the title in the entry matches the document, (ii) the year matches the first-submission year or the year of the version in the extract, and (iii) the finding the manuscript attributes to it is stated in the extract or follows arithmetically from it. A reference that fails any of these checks is unverifiable.

# Data and exclusions

Inputs were data/reference_list.json (5 items) and five extracts under data/sources/; data/README.md was also read. Line counts are in n_input_rows. All SHA-256 hashes match the README. No reference was excluded (out/exclusions.csv): n_analysed = 5. The only evidence used was each extract's header ("Title", "Bibliographic record") and its body. Problems found in the sources are listed under data_issues.

# Method

out/analysis.py does the following:

• Entry parsing. It splits each entry into authors, year, title and arXiv id.
• Title check. It compares the entry title with the header title and the body title after normalisation (case, diacritics, punctuation, "&"→"and").
• Year check. It reads the first-submission year and the year of the reproduced version from the "Bibliographic record" line.
• Finding check. It finds and prints the table cells and sentences that decide each finding, then compares them with every element of cited_for: magnitude, SE/CI, significance, timing and population/design.

The decisions on the finding checks were made by me after reading each extract in full. The script encodes those decisions as checks against the exact printed cells. A header-only baseline (title and year checks only) is stored as baseline_unverifiable_title_year_only.

# Results

• R1, Armendariz 2026: verified. Title and year (2026) match. Table 6, 5 km column, prints ATT -0.431** with SE (0.210) and -35.0%. The text states significance at the 5% level, and the abstract describes the staggered DiD on hospital records (reference_checks.R1).
• R2, Li et al. 2026: verified. In Table 2's ATE column: $1,153 with 95% CI [166, 2,141], 1.47 pp, and -1.99 weeks. The abstract gives the preregistered RCT and N = 23,549 (reference_checks.R2).
• R3, Liu & Zhao: unverifiable, fails all three checks (reference_checks.R3):
  • The entry title "Restaurants, Cafés and Neighbourhood Change: Commercial Entry and House Prices in Greater London" is a different title from "Does Entry of Food-and-Drink Establishments Raise Local House Prices? Event-Study Evidence from London".
  • The year 2024 does not match either submission year (v1 and v2 are both 2026).
  • The finding is contradicted. The extract reports 0.005 log points at τ=0, rising gradually to 0.037/0.034 at τ=4/5 ("prices do not jump immediately"; Section 3.2 and the Table 3 clean-onset row). This rules out a 3.4–3.7% jump in the entry year followed by no further rise.
• R4, Kanayama et al. 2025: verified. Year 2025 equals the v1 year. Table 2 prints -0.388 (0.132) and -0.732 (0.091). The abstract places the study in Japan's spot labour market (reference_checks.R4).
• R5, Tossou 2025: verified. Table 3 column (4), described as the preferred specification, prints -0.0962 (0.0129) and % change -0.3685, i.e. 36.85% of the mean of 0.261 (reference_checks.R5).

Summary: unverifiable_references = ["R3"] and n_verified = 4. The header-only baseline also flags only R3 (baseline_unverifiable_title_year_only).

# Uncertainty

No statistical estimation is involved, and random_seed is null. The checks are deterministic, but the following judgement calls affect them:

• R1 SE conflict. The same CS-DR 5 km estimate has SE 0.210 in Table 6 but 0.218 in Table A19. I treated Table 6, the main table, as decisive. The 5% significance (**) is printed in both tables.
• R4 SE may be a copy. The SE of the own-wage elasticity (0.091) is identical to the SE printed for "Affected employment". That looks like a possible duplication in the original paper. The cited SE matches what is printed, but the extract cannot confirm it is the true SE of the ratio.
• R5 rounding. Recomputing -0.0962/0.261 gives -0.3686 against the printed -0.3685. The difference comes from the mean being rounded and does not affect the verdict.

# Limitations and open issues

• Only the excerpts were available, so material outside them was not checked.
• R1 has further internal inconsistencies (608 vs 640 control regions; -30%/+47% in Section 6.1 vs 35% in the abstract). They do not affect the cited finding and are listed in data_issues.
• R5's abstract headline (0.0855, 18.79%) comes from a different table that is not reproduced in the extract. A reader citing the abstract would quote a different number.
• feasibility = "complete".