# F2-conf-01 — Effect of a school-level programme on student scores

# Question

A programme was assigned to whole schools. The goal is to estimate τ, the difference in mean student score between treated and control schools (the OLS coefficient of score on treated with an intercept), with uncertainty that respects the clustered design, and to decide at the 5% level (two-sided) whether τ ≠ 0.

# Data and exclusions

data/schools.csv (simulated): 1000 rows read (n_input_rows), 40 schools (n_schools) of 25 students each (students_per_school), 20 treated schools (n_treated_schools). No missing values, no duplicate students, and treatment is constant within every school. Four validity rules were applied and removed 0 rows (out/exclusions.csv), so 1000 students were analysed (n_analysed). data_issues is empty. Full audit: out/state/data_audit.md.

# Method

• Point estimate: OLS of score on treated with an intercept (tau_hat), which equals the difference in student-weighted means (mean_score_treated − mean_score_control).
• Baseline: the conventional homoskedastic iid OLS standard error (se_iid, with p_value_iid).
• Main uncertainty measure: a cluster-robust (CR1) sandwich standard error clustered by school (se_cluster), with the usual small-sample factor G/(G−1)·(N−1)/(N−K). The test statistic (t_stat_cluster) is compared with a t distribution on G−1 = 39 df (df_cluster). The p-value is p_value and the 95% CI is tau_hat ± t₀.₉₇₅,₃₉ (t_crit_975) × se_cluster (tau_ci_low, tau_ci_high).
• Cross-check: a Welch t-test on the 40 school means (se_school_means_welch, p_school_means_welch).
• ICC: one-way ANOVA estimator on the OLS residuals with schools as groups. The between-school mean square uses G−2 df because two arm means were removed (icc_hat, sigma2_between_resid, sigma2_within_resid). Implied design effect: 1+(m−1)·ICC (design_effect_from_icc).
• Nothing random is used (random_seed = null). out/analysis.py regenerates all results.

# Results

• tau_hat = 0.961 points (treated mean 51.28 vs control mean 50.32).
• se_cluster = 0.729; p_value = 0.195; 95% CI [−0.513, 2.436] (tau_ci_low, tau_ci_high).
• effect_significant_5pct = false: with valid design-based uncertainty, the data do not support a non-zero effect at the 5% level.
• Naive baseline: se_iid = 0.282, with p_value_iid = 0.0007. This would wrongly suggest a highly significant effect.
• icc_hat = 0.240, giving design_effect_from_icc = 6.77.

# Uncertainty

The clustered SE is 2.58 times the iid SE (se_ratio_cluster_over_iid). This is close to the 2.60 implied by the estimated ICC (sqrt_design_effect), so the inflation comes from the shared school-level variation. The school-means Welch test gives a similar SE of 0.738 and p = 0.201 (se_school_means_welch, p_school_means_welch), so the conclusion does not depend on the choice between these two cluster-valid methods. The effective sample size is about 1000/6.8 ≈ 150 students. With 40 clusters split evenly, CR1 with t(G−1) inference is standard and adequate.

# Limitations and open issues

• The data are simulated. Reading τ as a causal effect assumes schools were assigned to treatment as if at random, which cannot be checked without covariates.
• Failing to reject H0 does not show that the effect is zero. The CI is compatible with effects from about −0.5 to +2.4 points, so the study is under-powered for effects of about 1 point.
• The ICC is a point estimate; its sampling uncertainty is not quantified.
• Using G−2 instead of G−1 df, or other small-sample corrections (CR2/wild bootstrap), would change the p-value only slightly (compare the Welch cross-check). The 5% decision is not close to the boundary.