# F4-conf-03 — Harmonising effect sizes reported under incompatible definitions ## Question Five treatment effects from four open-access arXiv extracts are reported under different definitions (log-point ATT, percentage-point ATE, elasticity, linear-probability coefficient). The task is to re-express each printed point estimate on the target definition given in `data/effects.json`, using only numbers printed in the cited extract and a stated transformation. Where that is impossible, the report says so and names the quantity that is missing. ## Data and exclusions Inputs: `data/effects.json` (5 items), `data/README.md` and the four extracts under `data/sources/`. Line counts are in `n_input_rows`. All SHA-256 hashes match the README (see `out/state/data_audit.md`). All 5 items were assessed (`n_analysed` = 5) and none were excluded (`out/exclusions.csv`). Problems found in the inputs are listed in `data_issues`. The main ones: - the DCR abstract's "35%" fall in fatal hospitalisations disagrees with Table 7 (−29.8%) and with Section 6.1 ("approximately 30%"); - the MW Table 2 note says "about 4%" but the table prints 0.047; - small rounding mismatches in the DCR Table A19 and Benin Table 3 "% change" rows; - the Benin text refers to a "table 2" estimate that is not in the extract; - the RESEA control mean is a raw mean, while the ATE is adjusted for randomisation week. ## Method `out/analysis.py` finds each required table cell by regular expression in the extract, prints the source line it uses, and applies the conventions in TASK.md: - E1: log points to relative %, 100·(exp(b)−1), with b = −0.431 (DCR Table 6, ATT, 5 km). - E2: pp to relative % of the control mean, 100·Δ/p, with Δ = 1.47 pp and p = 75.0% (RESEA Table 2, Reemployment). - E3: elasticity times the printed proportional MW change, 100·η·g, with η = −0.388 and g = 0.047 (MW Table 2). - E4: an absolute change per 100,000 would be R0·(exp(b)−1) with b = −0.354 (DCR Table 7, fatal, 5 km). This needs the baseline rate R0. - E5: 0–1 probability to percentage points, 100·β, with β = −0.0962 (Benin Table 3, col. 4). No randomness is involved (`random_seed` = null). The printed estimates are recorded under `baseline_reported_estimates`, which serves as the "no transformation" baseline. ## Results - **E1** (`harmonised.E1.value`): −35.0141% change in the drug-related hospitalisation rate. This agrees with the paper's printed "% change" of −35.0%. - **E2** (`harmonised.E2.value`): +1.96% relative to the control-group reemployment rate. - **E3** (`harmonised.E3.value`): −1.8236% change in employment for the 4.7% minimum-wage rise printed in the table. - **E4** (`harmonised.E4`): **not harmonisable** (value null). The extract never prints a baseline level (counterfactual or pre-opening mean) of the fatal drug-related hospitalisation rate per 100,000 per year, so the absolute change cannot be computed (`harmonised.E4.missing_quantity`). From the printed numbers, only the relative change of −29.8% can be reached; this appears in `harmonised.E4.transformation`. - **E5** (`harmonised.E5.value`): −9.62 percentage points. - Counts: 4 items harmonised (`n_harmonised`) and 1 not harmonisable (`n_not_harmonisable`). Every item has either a value or a justified statement of impossibility, so `feasibility` = "complete". ## Uncertainty No standard errors or confidence intervals were propagated; all values are point estimates only. How exact each conversion is: - **E5** is an exact rescaling. - **E1** is exact under the log-point convention, given that the coefficient is printed to 3 decimals. Rounding of the printed b to ±0.0005 moves the result by about ±0.03 pp. - **E2** is exact arithmetic. However, it is an approximation to a counterfactual relative effect, because the raw control mean (75.0%) stands in for the counterfactual level of the week-adjusted ATE. - **E3** is a first-order (linear-elasticity) approximation. It also depends on using g = 0.047 rather than the note's "about 4%". - **E4** has no value. ## Limitations and open issues - E4 cannot be expressed in absolute units without the baseline fatal hospitalisation rate per 100,000 per year for the 5 km treated regions, which this extract does not print. No external or remembered value was substituted. - The source inconsistencies listed in `data_issues` were recorded but do not change any harmonised value, because each item uses the specific table cell named in `effects.json`. - Values are kept to 4 decimals. The printed inputs keep their original precision.