# F2-conf-01 — School-level programme effect on student scores

# Question

What is the average effect τ of a programme that was assigned to whole schools on student test scores? τ is the student-weighted difference in mean score, treated minus control. That is the OLS coefficient of score on treated with an intercept. The uncertainty must respect the clustered design. Is τ ≠ 0 at the 5% level (two-sided)?

# Data and exclusions

• Input: data/schools.csv, 1000 rows read. The data are simulated.
• Checks run: no missing values, no duplicate student_id, treated is constant within every school and takes only the values 0/1.
• Design: 40 schools with 25 students each (balanced), 20 treated and 20 control.
• No data issues were found, no rows were excluded, and all 1000 students were analysed.

# Method

• Point estimate: OLS of score on treated with an intercept.
• Naïve SE (se_iid): the conventional OLS SE, which assumes errors are independent and have equal variance. This is reported only as a baseline.
• Main SE (se_cluster): a cluster-robust (Liang–Zeger) sandwich SE, clustered by school_id, with the CR1 small-sample correction G/(G−1)·(N−1)/(N−K).
  • Schools are independent. Students within a school share school-level shocks, and treatment varies only between schools, so the school is the right unit for inference.
  • With only G = 40 clusters, the p-value and 95% CI use a t distribution with G−1 = 39 degrees of freedom rather than the normal.
• Cross-check: a two-sample t-test on the 40 school means. Because the design is balanced, it is essentially equivalent to the main analysis.
• ICC: a one-way ANOVA (method-of-moments) estimator applied to the OLS residuals. The between-school mean square uses G−2 df because two regression parameters were removed. A negative variance component would have been truncated at 0, but that did not happen here.

# Results

• Point estimate: students in treated schools scored about 0.96 points higher on average.
• Significance: with design-valid uncertainty, the difference is not statistically significant at the 5% level, and the 95% CI includes 0.
• Cross-check: the school-means t-test agrees (t = 1.30, p = 0.201).

# Uncertainty

• Naïve inference is wrong here. The naïve SE is 2.58 times smaller than the cluster SE. If used, it would give p ≈ 0.0007 and a spurious "significant" result.
• The ICC explains the gap. The estimated ICC of 0.24 with 25 students per school implies a design effect of 1 + 24·0.24 ≈ 6.8. Its square root, ≈ 2.60, matches the observed SE ratio.
• Effective sample size: there are effectively about 40 independent units, not 1000.

# Limitations and open issues

• Few clusters: with 40 clusters (20 per arm), CR1 with t(G−1) is a standard and reasonable choice. Alternatives such as CR2/Bell–McCaffrey or the wild cluster bootstrap were not run. Because the design is balanced and the school-means t-test agrees closely, they would be unlikely to change the conclusion.
• No covariates: none were available, so precision could not be improved by adjustment.
• Causal reading: interpreting τ̂ as causal assumes schools were randomised. The task describes assignment but does not state that it was random.
• Simulated data: the dataset is synthetic, so the results say nothing about any real programme.
• What the null result means: a non-significant result is not evidence of no effect. The CI is compatible with effects from about −0.5 to +2.4 points.