F4-conf-01 — Extracting treatment effects and uncertainty from two published tables
Question
Extract, exactly as printed, specified point estimates, standard errors (SEs), 95% confidence-interval (CI) bounds and sample counts from (a) Armendariz (2026), arXiv:2609.27732v1 (Swiss drug consumption rooms, staggered DiD, Tables 6–7) and (b) Li, Pearson-Merkowitz & Yokum (2026), arXiv:2606.14621v1 (Rhode Island RESEA RCT, Table 2). Attach a locator to each value. Derive one 95% CI from an SE and one SE from a CI. Identify any requested quantity that the source does not report. Nothing is re-estimated.
Data and exclusions
Inputs read: data/README.md (10 lines), data/sources/arxiv-2609.27732v1.txt (134 lines) and data/sources/arxiv-2606.14621v1.txt (58 lines). These two extracts are the only evidence used. Values come from the tables named in each key: DCR Table 6 and Table 7, and RESEA Table 2. DCR Appendix Table A19 and RESEA Table 3 were read only for cross-checks and supplied no reported values.
Method
out/analysis.py parses the pipe-delimited tables. It locates each cell by row label and column header, and it prints every matched source line. The standard error for a coefficient is taken from the parenthesised line directly below that coefficient. Significance stars are removed from the number and kept for the significance flag. Thousands separators are removed (e.g. "17,250" → 17250, "2,141" → 2141). Decimals are kept as printed. One limitation of the format: JSON number literals drop trailing zeros, so the printed 0.210 is stored as 0.21 and −0.340 as −0.34. The printed strings are shown in the table below and in the script output.
The script also runs consistency checks. It confirms that each star is consistent with a normal z-test using coef/SE, and that the derived wage SE reproduces the printed p-value. It compares duplicate reports of the same estimate across tables and compares the running text with the tables.
Results
| key | value (as printed) | locator |
|---|---|---|
| dcr_att_5km | -0.431 (**) | 2609.27732v1, Table 6, row "ATT", col "5 km" |
| dcr_att_5km_se | 0.210 | same cell, SE line |
| dcr_att_3km | -0.283 | Table 6, "ATT", "3 km" |
| dcr_att_3km_se | 0.279 | SE line |
| dcr_obs_5km | 17,250 → 17250 | Table 6, "Observations", "5 km" |
| dcr_narcotics_att_3km | 0.549 (***) | Table 7, "Narcotics", "3 km" |
| dcr_narcotics_att_3km_se | 0.187 | SE line |
| dcr_fatal_att_1km | -1.207 (***) | Table 7, "Drug-related (fatal)", "1 km" |
| dcr_fatal_att_1km_se | 0.137 | SE line |
| dcr_hiv_att_5km | -0.340 (no star) | Table 7, "HIV", "5 km" |
| dcr_hiv_att_5km_se | 0.294 | SE line |
| dcr_hiv_att_5km_p_value | null — not reported (only stars are printed) | — |
| dcr_hiv_att_5km_significant_10pct | false (no star) | Table 7 |
| dcr_att_5km_ci_low / high | derived: −0.843 / −0.019 | Table 6 (derived) |
| ri_wage_ate | $1,153 → 1153 | 2606.14621v1, Table 2, "Wage", "Average treatment effect estimate" |
| ri_wage_ate_ci_low / high | 166 / 2,141 → 2141 | same cell |
| ri_wage_ate_se_reported | false | Table 2 prints p and 95% CI only |
| ri_wage_ate_se_derived | derived: 503.8 | Table 2 (derived) |
| ri_control_mean_wage | $33,043 → 33043 | Table 2, "Wage", "Control" |
| ri_reemployment_ate_pp | 1.47 | Table 2, "Reemployment", ATE |
| ri_weeks_ui_ate | -1.99 | Table 2, "Weeks on UI", ATE |
| ri_weeks_ui_ate_ci_low / high | -2.24 / -1.75 | same cell |
Full locator strings are in out/results.json. In total 26 reported or derived items were assessed (n_analysed). Feasibility: complete. The single requested quantity that the source does not print, the HIV exact p-value, is correctly null.
Uncertainty
- Standard errors: all DCR values (Tables 6, 7 and A19) are SEs clustered at the MedStat-region level, printed in parentheses. Significance is shown only by stars (*** p<0.01, ** p<0.05, * p<0.10), and no exact p-values are printed.
- Confidence intervals: all RESEA ATE values are given as two-sided p-values plus 95% CIs, with no SEs. According to the Table 2 caption, the ATE CIs follow "the methodology described in the main text", which is not part of the extract.
- Derived DCR CI: −0.431 ± 1.96 × 0.210 = [−0.843, −0.019]. This assumes the clustered-SE ATT is asymptotically normal and uses no small-sample or t correction.
- With 50 treated regions and 608 controls, a t critical value would widen the interval only slightly.
- Using 2.0 instead of 1.96 would give [−0.851, −0.011].
- The interval excludes 0, which matches the printed ** (z-based p ≈ 0.040).
- Derived RESEA SE: (2141 − 166)/(2 × 1.96) = 503.8 USD. This assumes a symmetric Wald interval with a normal critical value; with N = 23,549 a t value is effectively the same.
- The printed CI is almost symmetric around the estimate: its midpoint is 1153.5 against the printed 1153.
- The implied p-value (0.022) equals the printed p=0.022, which supports the assumption.
- A normal-approximation p-value for the HIV 5 km ATT can be computed (≈0.25). It is not reported as a result, because the source does not print it and computing it would need a distributional assumption.
Limitations and open issues
Data issues detected in the sources (also listed in results.json → data_issues):
- SE conflict for the same estimate: the 5 km CS-DR ATT −0.431 has SE 0.210 in Table 6 but 0.218 in Appendix Table A19. Table 6 was used, as the keys specify. The derived CI would be [−0.858, −0.004] with 0.218.
- Control-group count conflict: Tables 6–7 give 608 never-treated control regions, while the Table A19 note gives 640.
- Abstract vs. Table 7: the DCR abstract says fatal hospitalisations "fall by 35%". Table 7 (5 km) gives −0.354 log points, which is about −29.8% (Section 6.1 itself says "approximately 30%"). The abstract appears to treat log points as percent.
- Text vs. Table 7 on HIV: the DCR text calls the HIV estimates "close to zero", but the 5 km estimate is −0.340, which is insignificant but not small.
- Sign convention in RESEA text: the running text gives the weeks-on-UI CI as positive magnitudes [1.75, 2.24]. Table 2's signed interval [−2.24, −1.75] was used.
- Rendering defect: "p ¡ 0.001" appears in the RESEA text. RESEA Tables 2–3 were re-set by the extract editor, and their values are stated to be verbatim.
All star markers checked were consistent with coef/SE under a normal approximation. No values were taken from memory of the papers.