# F4-conf-03 — Harmonising effect sizes reported under incompatible definitions

# Question

The task is to re-express five printed treatment effects (data/effects.json) on a stated target definition. Only quantities printed in the cited extract may be used, and each transformation must be stated explicitly. Where the target definition cannot be reached from the printed quantities, I say so and name the quantity that is missing.

# Data and exclusions

• Files read: the four source extracts under data/sources/ (66, 91, 58 and 134 lines), data/README.md (16 lines) and data/effects.json (5 items).
• Evidence used: only numbers printed in the extracts. No outside values were used, including remembered values from the full papers.
• Exclusions: none of the five items was excluded.
• Data issues: inconsistencies and rendering problems are listed under data_issues in results.json and summarised under Limitations.

# Method

I used the conventions stated in TASK.md.

• E1 (log points → % of the counterfactual rate): 100·(exp(b) − 1), with b = −0.431 (Armendariz, Table 6, row ATT, column 5 km).
• E2 (pp → % of the control mean): 100·Δ/p, with Δ = 1.47 pp and p = 75.0%, the Control column (RESEA, Table 2, row Reemployment).
• E3 (elasticity → % change for the observed minimum-wage hike): 100·η·g, with η = −0.388 and g = 0.047 (Kanayama et al., Table 2, rows "Employment elasticity w.r.t MW" and "% MW changes").
• E4 (log points → absolute change per 100,000 per year): this would need R0·(exp(b) − 1), with b = −0.354 (Table 7, row "Drug-related (fatal)", column 5 km). The level R0 is not printed anywhere in the extract.
• E5 (LPM coefficient on a 0–1 scale → pp): 100·β, with β = −0.0962 (Tossou, Table 3, row Treatment, column (4)).

out/analysis.py finds each table row by pattern match, prints the source lines it uses, and recomputes all values.

# Results

Totals: 4 items harmonised, 1 not harmonisable. The task counts as complete, because every item has either a value or a justified statement of impossibility.

Supporting checks for E4 and E5. These checks are not target values:

• The relative change implied by E4's coefficient is 100·(exp(−0.354) − 1) = −29.81%. This agrees with the paper's Section 6.1 ("approximately 30%"), but not with its abstract ("fall by 35%").
• For E5, the relative change is −0.0962/0.261 = −36.86%, against a printed "% Change" of −0.3685. This relative change is a different quantity from the pp target.

# Uncertainty

No standard errors or confidence intervals were propagated; the task asks for point estimates only.

• Exact conversions: E1 is an exact transformation of the log-point coefficient. E5 is an exact rescaling.
• E2 (approximate): the arithmetic is exact, but the denominator is the raw aggregate control mean (75.0%). That is not a block-adjusted counterfactual level for the ATE.
• E3 (approximate): the formula η·g is a linear, first-order approximation. The elasticity is a six-month average from an event study, and g is a summary (average) of 47 prefecture-level changes.
• Rounding of printed inputs: all inputs are printed to 2–4 significant digits, which limits the precision of the outputs.

# Limitations and open issues

1. E4 cannot be harmonised. The extract prints only log-point ATTs and no outcome level, so an absolute change per 100,000 cannot be derived from it. I did not use any guessed or remembered baseline.

2. Kanayama (E3): the Table 2 note says "about 4%" for the minimum-wage increase, but the table prints 0.047. I used the printed 0.047, as the target definition requires.

   • Using 0.04 instead would give about −1.55%.
   • The 0.047 figure is consistent with missing plus excess jobs (−0.03 + 0.012 = −0.018) and with the abstract's "2% employment decline".

3. Armendariz (E1, E4):

   • The same 5 km ATT has a standard error of 0.210 in Table 6 but 0.218 in Table A19.
   • The number of control regions is 608 in the main tables but 640 in the Table A19 note.
   • The abstract's 35% for fatal hospitalisations does not match the Table 7 coefficient (which implies −29.8%).
   • Notes refer to "four" sub-categories, although five outcomes are reported, and the fatal outcome is not defined in the extract.

   None of these changes the point values above.

4. RESEA (E2): the abstract rounds the effect to 1.5 pp. I used the Table 2 value of 1.47 pp.

5. Tossou (E5):

   • The headline figures (0.0855, "18.79 percentage change", mean 0.455) cite a "table 2" that is not in the extract.
   • The author calls relative changes "percentage change", which is easy to confuse with percentage points. The pp value reported here (−9.62) is a different quantity from the paper's −36.85% relative change.
   • Footnote markers in the extract are garbled by the rendering.

6. Comparability across studies: the harmonised values are on comparable scales, but the studies remain incomparable in population, design and treatment intensity. Harmonising the units does not make the effects comparable as estimates.