# F2-conf-01 — Effect of a school-level programme on student scores

## Question
A programme was assigned to whole schools. The goal is to estimate τ, the difference in mean student score between treated and control schools (the OLS coefficient of `score` on `treated` with an intercept), with uncertainty that respects the clustered design, and to decide at the 5% level (two-sided) whether τ ≠ 0.

## Data and exclusions
`data/schools.csv` (simulated): 1000 rows read (`n_input_rows`), 40 schools (`n_schools`) of 25 students each (`students_per_school`), 20 treated schools (`n_treated_schools`). No missing values, no duplicate students, and treatment is constant within every school. Four validity rules were applied and removed 0 rows (`out/exclusions.csv`), so 1000 students were analysed (`n_analysed`). `data_issues` is empty. Full audit: `out/state/data_audit.md`.

## Method
- Point estimate: OLS of score on treated with an intercept (`tau_hat`), which equals the difference in student-weighted means (`mean_score_treated` − `mean_score_control`).
- Baseline: the conventional homoskedastic iid OLS standard error (`se_iid`, with `p_value_iid`).
- Main uncertainty measure: a cluster-robust (CR1) sandwich standard error clustered by school (`se_cluster`), with the usual small-sample factor G/(G−1)·(N−1)/(N−K). The test statistic (`t_stat_cluster`) is compared with a t distribution on G−1 = 39 df (`df_cluster`). The p-value is `p_value` and the 95% CI is tau_hat ± t₀.₉₇₅,₃₉ (`t_crit_975`) × se_cluster (`tau_ci_low`, `tau_ci_high`).
- Cross-check: a Welch t-test on the 40 school means (`se_school_means_welch`, `p_school_means_welch`).
- ICC: one-way ANOVA estimator on the OLS residuals with schools as groups. The between-school mean square uses G−2 df because two arm means were removed (`icc_hat`, `sigma2_between_resid`, `sigma2_within_resid`). Implied design effect: 1+(m−1)·ICC (`design_effect_from_icc`).
- Nothing random is used (`random_seed` = null). `out/analysis.py` regenerates all results.

## Results
- `tau_hat` = 0.961 points (treated mean 51.28 vs control mean 50.32).
- `se_cluster` = 0.729; `p_value` = 0.195; 95% CI [−0.513, 2.436] (`tau_ci_low`, `tau_ci_high`).
- `effect_significant_5pct` = false: with valid design-based uncertainty, the data do **not** support a non-zero effect at the 5% level.
- Naive baseline: `se_iid` = 0.282, with `p_value_iid` = 0.0007. This would wrongly suggest a highly significant effect.
- `icc_hat` = 0.240, giving `design_effect_from_icc` = 6.77.

## Uncertainty
The clustered SE is 2.58 times the iid SE (`se_ratio_cluster_over_iid`). This is close to the 2.60 implied by the estimated ICC (`sqrt_design_effect`), so the inflation comes from the shared school-level variation. The school-means Welch test gives a similar SE of 0.738 and p = 0.201 (`se_school_means_welch`, `p_school_means_welch`), so the conclusion does not depend on the choice between these two cluster-valid methods. The effective sample size is about 1000/6.8 ≈ 150 students. With 40 clusters split evenly, CR1 with t(G−1) inference is standard and adequate.

## Limitations and open issues
- The data are simulated. Reading τ as a causal effect assumes schools were assigned to treatment as if at random, which cannot be checked without covariates.
- Failing to reject H0 does not show that the effect is zero. The CI is compatible with effects from about −0.5 to +2.4 points, so the study is under-powered for effects of about 1 point.
- The ICC is a point estimate; its sampling uncertainty is not quantified.
- Using G−2 instead of G−1 df, or other small-sample corrections (CR2/wild bootstrap), would change the p-value only slightly (compare the Welch cross-check). The 5% decision is not close to the boundary.