# F4-conf-03 — Harmonising effect sizes reported under incompatible definitions

# Question

For each of five effects (data/effects.json), re-express the printed point estimate on the stated target definition, using only quantities printed in the cited extract and an explicit transformation. Where the target cannot be reached, say so and name the missing quantity.

# Data and exclusions

Inputs: data/effects.json (5 items), data/README.md, and four source extracts under data/sources/ (line counts are in n_input_rows). Only the extracts were used. Nothing from the full papers or from memory was used. Numbers printed in the text that are based on omitted tables (for example, the Benin "table 2" estimate 0.0855/0.455) were not used.

# Method

out/analysis.py finds each required table row by its printed text, prints the source line, parses the printed values and applies the conventions in TASK.md:

• log points → percent: 100·(exp(b) − 1);
• percentage points → relative percent: 100·Δ/p, where p is the control mean;
• elasticity → percent: 100·η·g, where g is the printed proportional change in the minimum wage;
• 0–1 probability scale → percentage points: 100·β.

# Results

In total, 4 items are harmonised and 1 cannot be harmonised. Under TASK.md's definition the task is complete, because every item has either a value or a justified statement of impossibility.

E4 cannot be computed. The target is an absolute change in hospitalisations per 100,000 per year. Getting there needs the baseline (counterfactual) level Y0 of the fatal drug-related hospitalisation rate for the treated 5 km regions, and the extract does not print it. The relative change can be computed, 100(exp(−0.354)−1) = −29.8%, but it is not the requested estimand. No value was substituted.

# Uncertainty

No standard errors or confidence intervals were propagated; all values are point estimates only.

• Exact conversions:
  • E5 is a pure rescaling.
  • E1 is exact under the log-point convention: it is the exact transformation of the point estimate. It is not an unbiased estimate of the expected percent change, because of Jensen's inequality.
• Approximate conversions:
  • E3 uses a first-order (linear) approximation. The elasticity is applied to a 4.7% change, and the elasticity is itself an averaged six-month estimate.
  • E2 divides a block-adjusted (randomisation-week) ATE by the raw aggregate control-group mean (75.0%). That mean is only an approximation of the counterfactual level; the raw treatment-control difference is just 0.26 pp. The inputs are also rounded (75.0%, 1.47 pp), so E2 is reliable only to about ±0.01 percentage points of relative change.

# Limitations and open issues

• E3: the Table 2 note says the 2023 changes were "about 4%", but the table prints % MW changes = 0.047. I used 0.047, as the target definition requires. Using 4% would give −1.55%. The abstract's "2% employment decline in affected bins" is a different quantity.
• E4 (DCR paper), inconsistencies in the source:
  • The abstract says fatal hospitalisations "fall by 35%", while Section 6.1 says "approximately 30%". The Table 7 coefficient implies −29.8%.
  • Table A19 prints SE 0.218 for the same CS-DR estimate that Table 6 prints with SE 0.210.
  • The A19 note says 640 control regions, while the tables say 608.
  • None of these affect the point estimates used.
• E5 (Benin paper): the text cites an omitted "table 2", and footnote 11 is garbled in the rendering. Table 3 itself is internally consistent: −0.0962/0.261 = −0.3686, against a printed −0.3685.
• The full list is in data_issues in out/results.json.