1# F2-conf-01 — School-level programme effect23## Question4The programme was assigned to whole schools. What is its average effect on student test scores (τ = student-weighted mean score in treated schools minus the mean in control schools), how uncertain is that estimate given that assignment varies only between schools, and is τ different from zero at the 5% level (two-sided)?56## Data and exclusions7`data/schools.csv` (simulated data), 1000 rows read (`n_input_rows`). No missing values, duplicate students, or schools with mixed treatment status. Every school has 25 students. No rows were excluded (`out/exclusions.csv`), so `n_analysed` = 1000 students in `n_schools` = 40 schools, `n_treated_schools` = 20. `data_issues` is empty.89## Method10- Point estimate: OLS of `score` on `treated` with an intercept (`tau_hat`).11- Naïve baseline: the usual OLS standard error, which assumes independent, homoskedastic errors (`se_iid`).12- Main uncertainty measure: a cluster-robust standard error with schools as clusters (CR1, using the usual small-sample correction G/(G−1)·(N−1)/(N−K)) (`se_cluster`). The p-value and 95% CI use a t distribution with G−1 = 39 degrees of freedom (`diagnostics.t_df`), which is conservative with 40 clusters.13- ICC: one-way ANOVA (method-of-moments) estimator applied to OLS residuals, i.e. after removing each arm's mean, with school as the grouping factor (`icc_hat`).14- Cross-check: two-sample t-test on the 40 school means (`diagnostics.school_means_*`). No randomness is used (`random_seed` = null).1516## Results17- Estimated effect `tau_hat` = 0.961 points.18- Naïve SE `se_iid` = 0.282 (naïve p = 0.0007, `diagnostics.p_value_iid`). This would wrongly suggest a highly significant effect.19- Cluster SE `se_cluster` = 0.729, which is 2.58 times the naïve SE (`diagnostics.se_ratio_cluster_over_iid`). t = 1.32 (`diagnostics.t_stat_cluster`), `p_value` = 0.195.20- 95% CI [`tau_ci_low`, `tau_ci_high`] = [−0.51, 2.44] points. It contains 0.21- `effect_significant_5pct` = false. Using the valid uncertainty measure, the data do not support a non-zero effect at the 5% level.22- `icc_hat` = 0.240.2324## Uncertainty25The residual ICC is 0.24 with 25 students per school. That gives a Moulton design effect of 1 + 24·0.24 ≈ 6.77 (`diagnostics.design_effect_moulton`), whose square root, 2.60 (`diagnostics.sqrt_design_effect`), closely matches the observed SE inflation of 2.58. So the effective sample size is closer to 40 schools than to 1000 students. The school-means t-test gives the same answer: SE 0.738 (`diagnostics.school_means_se`) and p = 0.201 (`diagnostics.school_means_ttest_p_df38`).2627## Limitations and open issues28- With 40 clusters, CR1 with t(39) reference is generally adequate; wild-cluster bootstrap was not run (not required, design balanced and the aggregated t-test agrees).29- Not rejecting H0 does not show the effect is zero. The CI is consistent with effects from about −0.5 to +2.4 points.30- A causal reading of τ relies on schools being assigned at random. The dataset does not document how schools were assigned, so this is assumed, not verified.31- The data are synthetic.32