# F2-conf-01 — School-level programme effect

# Question

Estimate τ, the difference in mean test score between students in treated and control schools (OLS coefficient of score on treated with an intercept). Report sampling uncertainty that reflects the fact that treatment was assigned to whole schools, and decide at the 5% level (two-sided) whether τ ≠ 0.

# Data and exclusions

I used only data/schools.csv. It has 1000 rows (n_input_rows) from 40 schools (n_schools), 20 of them treated (n_treated_schools), with 25 students in every school (students_per_school). There are no missing values, no duplicate students, and treated is constant within each school. I applied three filter rules (missing values, duplicate students, schools where treatment varies). None removed any rows (out/exclusions.csv), so n_analysed = 1000 and data_issues is empty. The full audit is in out/state/data_audit.md.

# Method

• Point estimate: OLS of score on treated, with an intercept (tau_hat).
• Naïve baseline: the conventional OLS standard error, which assumes independent, homoskedastic errors (se_iid). It is not valid for this design.
• Main uncertainty measure: a cluster-robust standard error clustered by school (CR1, the statsmodels small-sample correction) (se_cluster). Tests and the 95% CI use a t distribution with G−1 = 39 degrees of freedom (t_df), which allows for there being only 40 clusters.
• Cross-check: OLS on the 40 school means, with the conventional SE and 38 df (school_means_se, school_means_p_value). Because the design is balanced, this gives the same point estimate.
• ICC: a one-way ANOVA estimator applied to the OLS residuals, so the treatment effect is removed first. The between-school mean square uses G−2 df (icc_hat).
• No random numbers are used, so random_seed is null.

# Results

• tau_hat = 0.961 points. This equals mean_treated 51.282 minus mean_control 50.321.
• se_cluster = 0.729, giving t = 1.32 (t_stat_cluster) and p = 0.195 (p_value).
• The 95% CI is [−0.513, 2.436] points (tau_ci_low, tau_ci_high).
• The data do not support a non-zero effect at the 5% level (effect_significant_5pct = false).
• The school-means cross-check agrees: SE 0.738, p 0.201 (school_means_se, school_means_p_value).

# Uncertainty

The naïve se_iid = 0.282 understates the uncertainty by a factor of 2.58 (se_ratio_cluster_to_iid). The residual intraclass correlation is icc_hat = 0.240. With 25 students per school, this implies a design effect of 1 + 24 × 0.240 = 6.77 (design_effect_from_icc), and √6.77 ≈ 2.60 matches the observed SE ratio. Inference based on se_iid would wrongly reject H0: its implied t is about 3.4. That reading is invalid because the programme varies only between schools.

# Limitations and open issues

• The data are synthetic. Reading τ as a causal effect assumes that schools were assigned to treatment as if at random, which I cannot check from these data.
• 40 clusters is a moderate number. To guard against small-sample bias in the cluster-robust SE, I used CR1 with a t(39) reference and cross-checked with the exact school-means analysis. A wild-cluster bootstrap was not run, but the two analytic approaches agree.
• An effect that is not significant is not evidence that the effect is zero. The CI includes effects of up to about 2.4 points.