# F2-conf-01 — School-level programme effect on student scores

# Question

The programme was assigned to whole schools. We estimate τ = the student-weighted mean score in treated schools minus the mean in control schools (the OLS coefficient of score on treated with an intercept). We report a standard error that accounts for students in the same school being correlated, and test H0: τ = 0 two-sided at the 5% level.

# Data and exclusions

• data/schools.csv: 1000 rows read, 1000 students analysed, no exclusions.
• 40 schools, 25 students each (balanced). 20 schools treated, 20 control.
• Checks run: no missing values, no duplicate student_id, no fully duplicated rows, treated is 0/1 and constant within every school, and every school has the same size. No data issues found (data_issues = []).

# Method

• Point estimate: OLS score ~ 1 + treated.
• Naïve baseline: the conventional OLS SE, which assumes errors are i.i.d. and homoskedastic.
• Main uncertainty measure: a cluster-robust (Liang–Zeger) SE clustered by school, with the CR1 small-sample correction G/(G−1)·(N−1)/(N−K). Inference uses a t distribution with G−1 = 39 degrees of freedom, because there are only 40 clusters.
• Cross-check: a Welch two-sample t-test on the 40 school means. With equal cluster sizes, this gives the same point estimate.
• ICC: one-way ANOVA (method-of-moments) estimator on the OLS residuals, i.e. after removing the treatment effect. The estimate is truncated at 0, though truncation was not needed here.
• No randomness is involved (random_seed = null). out/analysis.py regenerates out/results.json.

# Results

On average, students in treated schools scored about 0.96 points higher. This difference is not statistically significant at the 5% level once school-level clustering is accounted for (p ≈ 0.20, and the 95% CI includes 0). The data do not show that the programme has a non-zero effect.

# Uncertainty

The clustered SE is 2.58 times the naïve SE. This matches the estimated ICC of 0.235 with 25 students per school: the implied design effect is about 6.6, and √6.6 ≈ 2.58. The naïve analysis would give t = 3.41 and p = 0.0007, a false claim of significance, because it treats 1000 correlated students as independent when the treatment actually varies across only 40 schools. The school-means Welch test gives a similar answer (SE 0.738, t = 1.30, p = 0.20). So the conclusion does not depend on which design-consistent method is used.

# Limitations and open issues

• With 40 clusters, CR1 with t(G−1) inference is standard and should be roughly correct in size. Wild-cluster bootstrap or CR2/Bell–McCaffrey corrections were not run. Because the school-means test agrees closely, they are unlikely to change the conclusion.
• The estimate is design-based only in the sense of the question asked. It is causal only if schools were assigned to treatment at random (the data are synthetic and no assignment mechanism is documented). No covariates were available to adjust for imbalance or improve precision.
• A non-significant result is not evidence of no effect. The CI cannot rule out effects from about −0.5 to +2.4 points.