F2-conf-01 — Effect of a school-level programme on student scores, with valid uncertainty
Scientific question
A programme was assigned at the school level: every student in a treated school is exposed, no student in a control school is. Estimate the average effect of the programme on the student test score and report a valid measure of its sampling uncertainty. The estimand τ is the difference between the mean score of students in treated schools and the mean score of students in control schools (student-weighted means, i.e. the OLS coefficient of score on treated with an intercept). Uncertainty must reflect the design: the programme varies only between schools, and students within the same school share unobserved school-level influences; schools are independent of each other. Decide, at the 5% level (two-sided), whether the data support a non-zero effect.
Inputs
data/schools.csv— simulated data (a synthetic dataset generated for this exercise; it is not observed data). Columns:school_id(integer school identifier),student_id(integer student identifier, unique),treated(1 if the student's school received the programme, 0 otherwise — constant within a school),score(test score, points). One row per student; rows are in random order. The design is balanced: every school has the same number of students.
Information-availability rules
- Use only
data/schools.csv. No other input exists.
Required outputs
out/results.jsonwith the keys:
| key | type | unit | meaning |
|---|---|---|---|
task_id |
string | – | "F2-conf-01" |
tau_hat |
number | points | estimated programme effect (OLS coefficient of score on treated, with intercept) |
se_iid |
number | points | conventional OLS standard error of tau_hat assuming independent, homoskedastic errors — the naïve baseline |
se_cluster |
number | points | standard error of tau_hat that is valid under the two-level design described above (your main uncertainty measure) |
p_value |
number | – | two-sided p-value of H0: τ = 0 using se_cluster |
tau_ci_low, tau_ci_high |
number | points | 95% confidence interval for τ built from se_cluster |
effect_significant_5pct |
boolean | – | true if and only if H0: τ = 0 is rejected at the 5% level with the valid uncertainty measure |
icc_hat |
number | – | estimated intraclass correlation of the score residual (share of residual variance that is between schools, after removing the treatment effect) |
n_schools |
integer | – | number of distinct schools |
n_treated_schools |
integer | – | number of treated schools |
n_analysed |
integer | – | students used in the estimate |
feasibility |
string | – | "complete", "partial" or "infeasible" |
feasibility_reason |
string or null | – | explanation if not complete |
data_issues |
array of {"issue","handling"} | – | data problems detected and how handled (empty list if none) |
n_input_rows |
object | – | {"schools.csv": rows read} |
random_seed |
integer or null | – | seed used, if any (e.g. for a bootstrap) |
out/report.mdwith the sections: Question; Data and exclusions; Method; Results; Uncertainty; Limitations and open issues.out/analysis.py— running~/wp29/venv/bin/python out/analysis.pyfrom the workspace root must regenerateout/results.jsonfromdata/only, deterministically.out/claims.json— a list of the central claims of the report: [{"id": "C1", "text": "...", "results_keys": ["..."], "status": "supported"|"unsupported"|"not_evaluated", "caveat": "..."}]. Include one claim about whether the programme has a non-zero effect at the 5% level.
Constraints
- No network access. Python:
~/wp29/venv/bin/python(numpy, pandas, scipy, statsmodels available). Do not modifydata/. - This task is designed to be completed in well under 15 minutes of work.