F4-conf-02 — Are the six review assertions supported by their cited sources?
Question
For each of six assertions (A1–A5 cite Armendariz 2026, arXiv:2609.27732v1, on drug consumption rooms (DCRs); A6 cites Li, Pearson-Merkowitz & Yokum 2026, arXiv:2606.14621v1, on the RESEA programme), I decide from the cited extract alone whether it is supported, not_supported or not_assessable. Each verdict comes with a locator and a one-sentence reason (verdicts, locators, reasons).
Data and exclusions
Inputs: data/assertions.json (6 items), data/README.md (14 lines), the DCR extract (134 lines) and the RESEA extract (58 lines) (n_input_rows). The SHA-256 hashes of both extracts match the README (out/state/data_audit.md). No assertion was excluded (out/exclusions.csv), so all 6 were assessed (n_analysed). I found and recorded these issues in the inputs (data_issues):
- The DCR paper gives two different SEs for the same CS-DR 5 km ATT: 0.210 in Table 6 and 0.218 in Table A19.
- The Table A19 note says there are 640 control regions; everywhere else the count is 608.
- The DCR text's percentages for sub-outcomes don't agree with each other. The abstract says fatal hospitalisations fall by 35%, while Section 6.1 says about 30%. The text also calls the HIV estimates "close to zero" although the 5 km HIV ATT is −0.340.
- In the RESEA text, the confidence intervals are written as positive magnitudes, and "p ¡ 0.001" is a rendering defect.
None of these issues decides a verdict.
Process note: through a working-directory mistake, two of my own state files (contract.md, log.md) were briefly written to data/out/state/. I moved them to out/state/ and removed the directory. The original data files were never touched, and their hashes were re-checked (out/state/checks.md, check 10).
Method
out/analysis.py parses the relevant table rows from the extracts and prints them. It then applies explicit rules: the sign, magnitude, SE and stars must match; percentages are converted as 100·(exp(b)−1); 95% CIs are b ± 1.96·SE; for the subgroup claim it runs a regex search for sex terms. The script writes results.json and uses no randomness (random_seed = null). As a baseline, I first ran a naive check that only asks whether every number in an assertion appears verbatim in the cited extract (baseline_verdicts_verbatim_number_match). It agrees with the final verdicts on 4 of 6 assertions (baseline_agreement_with_verdicts). It gets A2 wrong because it ignores sign, and A5 wrong because it ignores population.
Results
| id | verdict (verdicts) |
locator (locators) |
decisive element (reasons, supporting_calculations) |
|---|---|---|---|
| A1 | supported | DCR, Table 6, row "ATT", column "5 km" | −0.431** (0.210), % change −35.0%; the text states 5% significance |
| A2 | not_supported | DCR, Table 7, row "HIV", column "1 km" | the printed ATT is +0.063 (SE 0.801), i.e. positive, not negative |
| A3 | not_supported | DCR, Table 6, row "% change", column "3 km" | −0.283 log points = −24.7% (implied −24.65%), not −28.3% |
| A4 | not_supported | DCR, Table 7, row "Hepatitis", column "5 km" | −0.406 ± 1.96·0.250 = [−0.896, 0.084], which includes 0; the asserted [−0.66, −0.16] is ±1 SE |
| A5 | not_assessable | not reported in arxiv-2609.27732v1 | the extract never breaks results down by sex (the regex finds no hits) |
| A6 | supported | RESEA, Table 2, row "Weeks on UI", column "Average treatment effect estimate" | −1.99 weeks, CI [−2.24, −1.75], control mean 17.3; 1.99/17.3 = 11.50% |
Counts: 2 supported (n_supported), 3 not supported (n_not_supported), 1 not assessable (n_not_assessable). Feasibility: complete (feasibility).
Uncertainty
- Critical value. A4 depends on how the CI is built. With 1.96 the interval is [−0.896, 0.084]; with 2.0 it is [−0.906, 0.094] (
supporting_calculations.A4). Both include zero, and the estimate has no star in Table 7, so the verdict is the same either way. The asserted interval matches only a ±1 SE (≈68%) band. A1's 5% significance claim rests on the printed ** and the text. With Table 6's SE the |z| is 2.052 (supporting_calculations.A1.z), above both 1.96 and 2.0. With Table A19's SE of 0.218 the |z| is 1.977. That is still above 1.96 but just below 2.0, which is noted indata_issues. A1 cites Table 6, and the ** is printed in both tables, so the verdict stays supported. - Log-to-percent conversion. For A1 and A3 the conversion is printed in Table 6 and matches 100(exp(b)−1). A3's 28.3% only comes out if log points are read as percent changes, which is not a reasonable alternative convention when the source prints −24.7%. For A6 the ratio 1.99/17.3 is computed directly and is also stated in the source.
- A2 and A5 do not depend on any convention.
Limitations and open issues
- The verdicts are based only on the extracts, as the task requires. A5 might be assessable from omitted parts of the paper, but those are not admissible here.
- The internal inconsistencies in the DCR source (two SEs for one estimate; 608 vs 640 control regions; narrative percentages) are reported, not resolved.
- The verdicts are rule-based judgements encoded in
out/analysis.py. They are not statistical estimates, so no sampling uncertainty applies to them.