# F2-conf-01 — School-level programme effect on student scores

# Question

Estimate τ, the difference in mean test score between students in treated and control schools (the OLS coefficient of score on treated with an intercept). Give a standard error that matches the design, where treatment is assigned to whole schools, and test H0: τ = 0 at the 5% level (two-sided).

# Data and exclusions

• data/schools.csv: 1000 rows read, 1000 analysed. There are 40 schools with 25 students each; 20 schools are treated and 20 are controls.
• Checks run: missing values (none), duplicate student_id (none), duplicate rows (none), treated constant within each school (yes in all 40 schools), and equal school sizes (yes).
• No data issues were found and no rows were excluded (data_issues = []).

# Method

• Point estimate: OLS of score on treated with an intercept, using all students.
• Naïve SE (se_iid): the conventional homoskedastic OLS standard error. This is reported only as a baseline. It is not valid here, because it treats the 1000 students as independent.
• Main SE (se_cluster): a cluster-robust (CR1) sandwich standard error, clustered by school_id, with the usual small-sample correction G/(G−1)·(N−1)/(N−K). Because there are only G = 40 clusters, the p-value and 95% CI use a t distribution with G−1 = 39 degrees of freedom rather than the normal.
• Cross-check: a two-sample t-test on the 40 school means, which is an equivalent analysis at the level where treatment was randomised.
• ICC: a one-way ANOVA estimator applied to the OLS residuals, grouped by school: ICC = σ²_b/(σ²_b+σ²_w), where σ²_b = (MSB−MSW)/m and m = 25.
• No randomness is used (random_seed = null).

# Results

On average, students in treated schools scored about 0.96 points higher than students in control schools. With design-valid uncertainty, this difference is not statistically significant at the 5% level (p = 0.195), and the 95% CI includes zero.

# Uncertainty

• The naïve iid SE (0.282) would give p ≈ 0.0007 and a spurious "significant" conclusion.
• The clustered SE is 2.58 times larger. This implies a design effect of (0.729/0.282)² ≈ 6.68.
• That design effect matches the Kish formula 1 + (m−1)·ICC = 1 + 24·0.235 ≈ 6.64. Most of the uncertainty therefore comes from school-level variation, and the effective sample size is closer to about 150 students than to 1000.
• The school-means t-test gives a very similar result (SE 0.738, p = 0.201), which supports the clustered result.

# Limitations and open issues

• With only 40 clusters, CR1 standard errors can be slightly too small. Using t(39) critical values and the school-means cross-check reduces this concern, and both lead to the same conclusion.
• A non-significant result does not show that the effect is zero. The CI is consistent with effects from about −0.5 to +2.4 points, so the study cannot rule out a modest positive effect.
• The data are simulated. The causal interpretation of τ rests on the assumption that schools were assigned to treatment as if at random; this cannot be checked from the data, since no covariates are provided.