F4-conf-03 — Harmonising effect sizes reported under incompatible definitions
Question
Re-express five printed treatment effects from four arXiv extracts on the target definition stated for each item in data/effects.json, using only quantities printed in the cited extract and an explicit transformation; where the target is unreachable, state the missing quantity.
Data and exclusions
Inputs: data/effects.json (5 items) and four source extracts (line counts in n_input_rows; SHA-256 hashes match data/README.md, see out/state/data_audit.md). All 5 items were assessed (n_analysed = 5); none excluded (out/exclusions.csv). Source anomalies are listed in data_issues (abstract vs Section 6.1 vs Table 7 inconsistency for fatal hospitalisations in the DCR paper; differing SE 0.210/0.218 and 608/640 control-region counts; "about 4%" note vs printed 0.047 MW change; Benin text citing an absent "table 2" and calling relative changes "percentage changes"; RESEA abstract rounding 1.47 pp to 1.5 pp). None changes a harmonised value.
Method
Task conventions applied (harmonised.*.transformation): log-point b → 100·(exp(b)−1) %; pp effect Δ with control level p → 100·Δ/p %; elasticity η with proportional treatment change g → 100·η·g %; LPM coefficient on 0–1 scale → ×100 pp. out/analysis.py locates each printed input by a unique regex match in the extract, prints that source line, and computes the value. No randomness (random_seed = null).
Results
- E1 (DCR overall ATT, 5 km, Table 6, b = −0.431): 100·(exp(−0.431)−1) = −35.01 % of the counterfactual rate (
harmonised.E1.value); matches the printed "% change" −35.0%. - E2 (RESEA reemployment ATE 1.47 pp; control mean 75.0%, Table 2): 100·1.47/75.0 = +1.96 % of the control-group mean (
harmonised.E2.value). - E3 (MW employment elasticity −0.388; "% MW changes" 0.047, Table 2): 100·(−0.388)·0.047 = −1.82 % employment (
harmonised.E3.value, −1.8236). - E4 (DCR fatal hospitalisations ATT, 5 km, Table 7, b = −0.354): not harmonisable (
harmonised.E4.harmonisable= false,harmonised.E4.value= null). An absolute change per 100,000 per year equals R0·(exp(−0.354)−1) = −0.298·R0, but the baseline/counterfactual fatal hospitalisation rate R0 is not printed in the extract (harmonised.E4.missing_quantity). - E5 (Benin LPM, Table 3, column (4), β = −0.0962): −9.62 percentage points (
harmonised.E5.value).
Summary: n_harmonised = 4, n_not_harmonisable = 1, feasibility = "complete" (every item has a value or a justified impossibility statement).
Uncertainty
No standard errors or confidence intervals were propagated; only point estimates are harmonised. Exact conversions: E5 (pure rescaling) and E1 (exact mapping of a log-point coefficient to a percent change under the convention, though the ATT of a log outcome maps only approximately to the percent change in the average rate). Approximate: E2 (ratio uses the raw aggregate control mean while the ATE is block-adjusted; the ratio estimator is itself only approximately unbiased); E3 (linear elasticity extrapolation 100·η·g is a first-order approximation; the note's "about 4%" vs the printed 0.047 adds ambiguity — with 0.04 the value would be −1.55%, see data_issues). E4 has no value.
Limitations and open issues
- E4 cannot be computed from the extract: it requires the baseline level of the fatal drug-related hospitalisation rate per 100,000 inhabitants per year for the treated 5 km regions (
harmonised.E4.missing_quantity). No external or remembered value was substituted. - Source internal inconsistencies are recorded, not resolved (
data_issues); in particular the DCR abstract's "35%" fall in fatal hospitalisations is not supported by Table 7 (−29.8% implied), which agrees with Section 6.1 ("approximately 30%"). - E3 depends on reading 0.047 as the proportional MW change, as printed in the same table per the target definition.