1# F2-conf-01 — School-level programme effect23## Question4Estimate τ, the difference in mean test score between students in treated and control schools (OLS coefficient of `score` on `treated` with an intercept). Report sampling uncertainty that reflects the fact that treatment was assigned to whole schools, and decide at the 5% level (two-sided) whether τ ≠ 0.56## Data and exclusions7I used only `data/schools.csv`. It has 1000 rows (`n_input_rows`) from 40 schools (`n_schools`), 20 of them treated (`n_treated_schools`), with 25 students in every school (`students_per_school`). There are no missing values, no duplicate students, and `treated` is constant within each school. I applied three filter rules (missing values, duplicate students, schools where treatment varies). None removed any rows (`out/exclusions.csv`), so `n_analysed` = 1000 and `data_issues` is empty. The full audit is in `out/state/data_audit.md`.89## Method10- Point estimate: OLS of score on treated, with an intercept (`tau_hat`).11- Naïve baseline: the conventional OLS standard error, which assumes independent, homoskedastic errors (`se_iid`). It is not valid for this design.12- Main uncertainty measure: a cluster-robust standard error clustered by school (CR1, the statsmodels small-sample correction) (`se_cluster`). Tests and the 95% CI use a t distribution with G−1 = 39 degrees of freedom (`t_df`), which allows for there being only 40 clusters.13- Cross-check: OLS on the 40 school means, with the conventional SE and 38 df (`school_means_se`, `school_means_p_value`). Because the design is balanced, this gives the same point estimate.14- ICC: a one-way ANOVA estimator applied to the OLS residuals, so the treatment effect is removed first. The between-school mean square uses G−2 df (`icc_hat`).15- No random numbers are used, so `random_seed` is null.1617## Results18- `tau_hat` = 0.961 points. This equals `mean_treated` 51.282 minus `mean_control` 50.321.19- `se_cluster` = 0.729, giving t = 1.32 (`t_stat_cluster`) and p = 0.195 (`p_value`).20- The 95% CI is [−0.513, 2.436] points (`tau_ci_low`, `tau_ci_high`).21- **The data do not support a non-zero effect at the 5% level** (`effect_significant_5pct` = false).22- The school-means cross-check agrees: SE 0.738, p 0.201 (`school_means_se`, `school_means_p_value`).2324## Uncertainty25The naïve `se_iid` = 0.282 understates the uncertainty by a factor of 2.58 (`se_ratio_cluster_to_iid`). The residual intraclass correlation is `icc_hat` = 0.240. With 25 students per school, this implies a design effect of 1 + 24 × 0.240 = 6.77 (`design_effect_from_icc`), and √6.77 ≈ 2.60 matches the observed SE ratio. Inference based on `se_iid` would wrongly reject H0: its implied t is about 3.4. That reading is invalid because the programme varies only between schools.2627## Limitations and open issues28- The data are synthetic. Reading τ as a causal effect assumes that schools were assigned to treatment as if at random, which I cannot check from these data.29- 40 clusters is a moderate number. To guard against small-sample bias in the cluster-robust SE, I used CR1 with a t(39) reference and cross-checked with the exact school-means analysis. A wild-cluster bootstrap was not run, but the two analytic approaches agree.30- An effect that is not significant is not evidence that the effect is zero. The CI includes effects of up to about 2.4 points.31