1# F4-conf-04 — Checking a reference list against the source extracts23## Question4For each of five references (R1–R5) in a draft manuscript, we checked three things against the provided arXiv extract: (i) whether the title matches, (ii) whether the year matches the first-submission year or the year of the reproduced version, and (iii) whether every element of the finding attributed to the reference ("cited_for") is stated in the extract or follows arithmetically from it. A reference that fails any of the three checks is unverifiable.56## Data and exclusions7- Inputs: `data/reference_list.json` (5 items) and five extracts under `data/sources/`. `data/README.md` was read for provenance. Line counts are in `n_input_rows`.8- The SHA-256 of every extract matches the value in the README (`out/state/data_audit.md`).9- No references were excluded (`out/exclusions.csv`: 5 − 0 = 5). All five were assessed (`n_analysed` = 5).10- Problems found in the inputs are listed in `data_issues`. None of them changes a verdict:11 - Armendariz: Table 6 gives SE 0.210, Table A19 gives 0.218 for the same estimate. Table A19 also gives two different control counts (608 and 640). The abstract's 35% fatal / "comparable" narcotics figures disagree with Table 7 and Section 6.1, which give about 30% and 47%.12 - Kanayama et al.: the printed SE of 0.091 for the own-wage elasticity looks implausible (see Uncertainty).13 - Liu & Zhao: the Table 3 header is garbled in the extract.14 - Tossou, Li et al.: footnote and table-layout rendering defects.1516## Method17`out/analysis.py` reads only `data/` and does the following:18- **Header fields.** It parses each extract's "Title:" line, its "Bibliographic record:" line and its body title line.19- **Title check.** Titles are compared after normalisation: case, diacritics, punctuation, and "&" vs "and".20- **Year check.** The year is compared with the first and the last submission date in the "Bibliographic record" line.21- **Finding check.** For each reference the script reads the specific table cells and sentences that decide the check, compares them with the cited values exactly as printed, and prints those lines.22 - For R3, the timing claim is tested against the stacked event-time coefficients reported in Section 3.2.23- **Baseline.** As a first step we ran a bibliographic-only check (title and year only). It is stored under `baseline_*` keys.24- No randomness is involved (`random_seed` = null).2526## Results27- **Unverifiable:** only R3 (`unverifiable_references` = ["R3"]). Four references pass all three checks (`n_verified` = 4). The bibliographic-only baseline also flags only R3 (`baseline_unverifiable_references_bibliographic_only`).28- **R1, Armendariz 2026:** title, year and finding all pass (`reference_checks.R1`). Table 6, 5 km column: ATT −0.431**, SE (0.210), −35.0%. The text says the estimate is significant at the 5% level and describes a staggered DiD on hospital records.29- **R2, Li, Pearson-Merkowitz & Yokum 2026:** all checks pass (`reference_checks.R2`). The Table 2 average-treatment-effect column gives $1,153 (95% CI [166, 2,141]), 1.47 pp and −1.99 weeks. The abstract gives N = 23,549 in a preregistered RCT.30- **R3, Liu & Zhao:** fails all three checks (`reference_checks.R3`).31 - Title: the entry's title ("Restaurants, Cafés and Neighbourhood Change: …") is different from the document's title ("Does Entry of Food-and-Drink Establishments Raise Local House Prices? Event-Study Evidence from London").32 - Year: the entry says 2024, but v1 and v2 were both submitted in 2026.33 - Finding: the source reports 0.005 log points (about 0.5%) in the event year, rising gradually to 0.037 and 0.034 at τ = 4 and 5 (Section 3.2; Table 3, clean-onset row). This contradicts "a 3.4–3.7% jump in the year of entry with no further appreciation".34- **R4, Kanayama, Miyaji & Otani 2025:** all checks pass (`reference_checks.R4`).35 - Year: 2025 is the v1 year. The reproduced version (v3) is from 2026, and the rule accepts either.36 - Finding: Table 2 prints −0.388 (0.132) and −0.732 (0.091), and the abstract places the study in Japan's spot labour market.37- **R5, Tossou 2025:** all checks pass (`reference_checks.R5`). Table 3, column (4), which the text calls the preferred specification, prints −0.0962 (0.0129) and a % change of −0.3685. This equals −0.0962 / 0.261 and matches the stated 36.85% reduction.3839## Uncertainty40The checks are deterministic, so there is no sampling uncertainty. Three verdicts depend on judgement:41- **R4.** The finding counts as supported because the cited SE is printed in the extract. However, the printed SE of 0.091 for the own-wage elasticity is the same as the SE of "Affected employment". A delta-method approximation from the printed components gives about 0.42, which suggests a possible error in the source itself (`data_issues`). A stricter reviewer might want this SE confirmed before citing it.42- **R1.** The cited SE of 0.210 comes from Table 6, while Table A19 prints 0.218 for the same estimate. Both values imply significance at the 5% level (t = 2.05 vs 1.98), so the cited claim holds either way.43- **R3.** The verdict does not depend on judgement: each of its three checks fails on its own.4445## Limitations and open issues46- Only the extracts were admissible, so no other versions or bibliographic databases were consulted. Elements outside the extract could not be checked, but none of the cited findings needed them.47- The source inconsistencies listed in `data_issues` (Armendariz SE and control-count mismatches, Kanayama SE plausibility) cannot be resolved from the extracts. We report them; we did not correct them.48- The task was completed in full (`feasibility` = "complete").49