# F2-conf-01 — Programme effect on student scores with cluster-valid uncertainty

## Question
A programme was assigned to whole schools. We estimate τ, the difference between student-weighted mean scores in treated and control schools (OLS coefficient of `score` on `treated` with intercept), give a sampling uncertainty that respects the school-level design, and test H0: τ = 0 at the 5% level (two-sided).

## Data and exclusions
`data/schools.csv` (synthetic; SHA-256 0f1d8378…c88a) has 1000 rows (`n_input_rows`). No missing values, duplicate rows or duplicate student IDs were found. `out/exclusions.csv` therefore records two filtering steps that each removed 0 rows. All `n_analysed` = 1000 students were used. They come from `n_schools` = 40 schools of 25 students each, and `n_treated_schools` = 20 of those schools are treated. Treatment is constant within every school. `data_issues` is empty. Full audit: `out/state/data_audit.md`.

## Method
- Baseline: OLS of score on treated with the conventional iid, homoskedastic standard error (`se_iid`).
- Main analysis: the same OLS point estimate (`tau_hat`), with a cluster-robust variance clustered on `school_id`. This uses the CR1 small-sample correction from statsmodels: G/(G−1)·(N−1)/(N−K). The SE allows any correlation among students in the same school and assumes schools are independent. Because there are only 40 clusters, the p-value and 95% CI use a t distribution with G−1 = 39 degrees of freedom (`df_cluster`).
- Cross-check: a Welch two-sample t-test on the 40 school means (`school_means_welch_se`, `school_means_welch_p`).
- ICC: I subtracted each arm's mean score to get residuals, then applied a one-way ANOVA variance-components estimator by school (2 between-school df used up by the arm means). `icc_hat` = σ²_between / (σ²_between + σ²_within). It implies a design effect of 1 + (25−1)·ICC (`design_effect_from_icc`).
- The method is deterministic, so no random seed is used (`random_seed` = null).

## Results
- Mean score was 51.28 in treated schools (`mean_score_treated`) and 50.32 in control schools (`mean_score_control`).
- `tau_hat` = 0.961 points.
- Valid (cluster) inference: `se_cluster` = 0.729, `t_stat_cluster` = 1.32, `p_value` = 0.195. The 95% CI is [`tau_ci_low`, `tau_ci_high`] = [−0.513, 2.436].
- **H0: τ = 0 is not rejected at the 5% level (`effect_significant_5pct` = false).** The data do not support a non-zero programme effect.
- The naive baseline would have concluded otherwise: `se_iid` = 0.282, `baseline_p_value_iid` = 0.00068, CI [`baseline_ci_iid_low`, `baseline_ci_iid_high`] = [0.408, 1.515]. That conclusion is invalid because it treats 1000 correlated students as independent observations.

## Uncertainty
- `icc_hat` = 0.240: about a quarter of residual score variance lies between schools.
- With 25 students per school, this gives a design effect of about 6.77 (`design_effect_from_icc`). That matches the observed ratio `se_ratio_cluster_to_iid` = 2.58, whose square is about 6.68.
- The school-means Welch test gives almost the same SE and p-value (`school_means_welch_se` = 0.738, `school_means_welch_p` = 0.201), which supports the cluster-robust result.
- The effective sample size is about 1000/6.8 ≈ 150 students, and the information really comes from 40 schools.

## Limitations and open issues
- The data are synthetic. Reading τ as causal assumes that schools were assigned to treatment as if at random; the data cannot verify this.
- Cluster-robust inference is asymptotic in the number of clusters. With 40 balanced clusters (20/20), CR1 with t(39) is generally adequate, and the school-means analysis agrees closely. A wild-cluster bootstrap was not run because the task does not require it.
- Not rejecting H0 does not show the effect is zero. The 95% CI includes effects from about −0.5 to +2.4 points.
- No covariates were available, so precision could not be improved through adjustment.