# F2-conf-01 — School-level programme effect on student scores

# Question

Estimate τ, the difference in mean student test score between treated and control schools (the OLS coefficient of score on treated with an intercept). Report sampling uncertainty that matches the design: treatment was assigned to whole schools, and students in the same school share school-level shocks. Then test H0: τ = 0 at the 5% level, two-sided.

# Data and exclusions

data/schools.csv has 1000 rows: 40 schools with 25 students each, 20 of them treated. The data are simulated. Checks run: no missing values, no duplicate student_id, treated constant within every school, balanced school sizes. No problems were found and no rows were excluded (n_analysed = 1000).

# Method

• Point estimate: OLS of score on treated with an intercept, using all students.
• Naïve baseline (se_iid): the conventional OLS standard error, which assumes independent, homoskedastic errors.
• Main SE (se_cluster): a cluster-robust (Liang–Zeger) sandwich SE clustered by school_id, with the CR1 small-sample correction G/(G−1)·(N−1)/(N−K). Tests and the 95% CI use a t distribution with G−1 = 39 degrees of freedom, a standard adjustment when there are few clusters.
• Cross-check: a two-sample t-test on the 40 school means. With balanced schools this gives the same point estimate and is an exact analogue of the cluster analysis.
• ICC: a one-way ANOVA (method-of-moments) estimator applied to the OLS residuals, so the treatment effect is removed first. Between-school mean square uses G−2 df; within-school mean square uses N−G df.

# Results

Treated schools score about 0.96 points higher on average, but this difference is not statistically significant at the 5% level under design-valid inference (p = 0.195), and the 95% CI includes zero.

# Uncertainty

The cluster-robust SE is 2.58 times the naïve iid SE. About 24% of residual variance lies between schools, so 25 students in one school carry much less independent information than 25 students drawn at random. A rough design-effect argument (√6.77 ≈ 2.6) agrees with the observed SE ratio.

The naïve analysis would give p ≈ 0.0007 and wrongly declare a highly significant effect. That result is an artefact of treating 1000 correlated students as independent observations. In effect the sample size for inference is 40 schools.

The school-means t-test gives SE 0.738 and p = 0.201, which agrees with the cluster result.

# Limitations and open issues

• With G = 40 clusters (20 per arm), cluster-robust SEs can be somewhat biased downward. The t(39) reference and the CR1 correction reduce this, and the school-means t-test agrees closely. No wild-cluster bootstrap was run, so no random seed was used.
• The ICC is a method-of-moments estimate from a modest number of schools, so it is itself imprecise.
• "Not significant" does not show that the effect is zero. The CI runs from about −0.5 to +2.4 points, so moderate positive effects are still possible.
• The estimand is the student-weighted difference in means. With equal school sizes this is the same as the school-weighted difference.
• The data are simulated.