F2-conf-01 — School-level programme effect
Question
Estimate the average effect τ of a programme assigned at school level on student test scores (difference in student-weighted mean score between treated and control schools = OLS slope of score on treated), with uncertainty that respects the clustered design, and test H0: τ = 0 at the 5% level (two-sided).
Data and exclusions
data/schools.csv (synthetic): 1000 rows read (n_input_rows), 40 schools (n_schools) of 25 students each (students_per_school), 20 treated (n_treated_schools). No missing values, no duplicate students, treatment constant within school; no rows excluded (out/exclusions.csv), so n_analysed = 1000. data_issues is empty.
Method
OLS of score on an intercept and treated. Naïve baseline: conventional homoskedastic SE (se_iid). Main uncertainty measure: cluster-robust (CR1, school clusters, small-sample correction G/(G−1)·(N−1)/(N−K)) SE (se_cluster), with inference from a t distribution with G−1 = 39 degrees of freedom (cluster_df). The residual intraclass correlation (icc_hat) uses the one-way ANOVA estimator on OLS residuals grouped by school. Cross-check: a two-sample t-test on the 40 school means. No randomness (random_seed = null).
Results
- Estimated effect
tau_hat= 0.961 points. - Cluster-robust SE
se_cluster= 0.729; t = 1.32 (t_stat_cluster);p_value= 0.195. - 95% CI [
tau_ci_low,tau_ci_high] = [−0.513, 2.436]. effect_significant_5pct= false: at the 5% level the data do not support a non-zero effect.icc_hat= 0.235.
Uncertainty
The naïve SE se_iid = 0.282 would give p_value_iid = 0.0007, which looks highly significant. It treats 1000 students as independent, but treatment varies only across 40 schools and residuals are correlated within school (icc_hat = 0.235). The cluster SE is 2.58 times larger (se_ratio_cluster_to_iid), which agrees with the Moulton design effect 1+(25−1)·ICC = 6.64 (design_effect_from_icc; √6.64 ≈ 2.58). The school-means analysis gives the same answer (SE check_se_school_means = 0.738, p check_p_school_means_pooled_t = 0.201).
Limitations and open issues
- With 40 clusters (20 per arm, balanced) CR1 with t(39) should be reliable. Other reasonable choices (school-means t-test) give similar p-values (≈0.20), so the conclusion does not depend on which one is used.
- Reading τ̂ as causal assumes schools were assigned to treatment as if at random. The data are simulated and that assignment was not verified.
- Not rejecting H0 does not show the effect is zero: the CI includes effects up to about 2.4 points.