# F2-conf-01 — School-level programme effect

# Question

The programme was assigned to whole schools. What is its average effect on student test scores (τ = student-weighted mean score in treated schools minus the mean in control schools), how uncertain is that estimate given that assignment varies only between schools, and is τ different from zero at the 5% level (two-sided)?

# Data and exclusions

data/schools.csv (simulated data), 1000 rows read (n_input_rows). No missing values, duplicate students, or schools with mixed treatment status. Every school has 25 students. No rows were excluded (out/exclusions.csv), so n_analysed = 1000 students in n_schools = 40 schools, n_treated_schools = 20. data_issues is empty.

# Method

• Point estimate: OLS of score on treated with an intercept (tau_hat).
• Naïve baseline: the usual OLS standard error, which assumes independent, homoskedastic errors (se_iid).
• Main uncertainty measure: a cluster-robust standard error with schools as clusters (CR1, using the usual small-sample correction G/(G−1)·(N−1)/(N−K)) (se_cluster). The p-value and 95% CI use a t distribution with G−1 = 39 degrees of freedom (diagnostics.t_df), which is conservative with 40 clusters.
• ICC: one-way ANOVA (method-of-moments) estimator applied to OLS residuals, i.e. after removing each arm's mean, with school as the grouping factor (icc_hat).
• Cross-check: two-sample t-test on the 40 school means (diagnostics.school_means_*). No randomness is used (random_seed = null).

# Results

• Estimated effect tau_hat = 0.961 points.
• Naïve SE se_iid = 0.282 (naïve p = 0.0007, diagnostics.p_value_iid). This would wrongly suggest a highly significant effect.
• Cluster SE se_cluster = 0.729, which is 2.58 times the naïve SE (diagnostics.se_ratio_cluster_over_iid). t = 1.32 (diagnostics.t_stat_cluster), p_value = 0.195.
• 95% CI [tau_ci_low, tau_ci_high] = [−0.51, 2.44] points. It contains 0.
• effect_significant_5pct = false. Using the valid uncertainty measure, the data do not support a non-zero effect at the 5% level.
• icc_hat = 0.240.

# Uncertainty

The residual ICC is 0.24 with 25 students per school. That gives a Moulton design effect of 1 + 24·0.24 ≈ 6.77 (diagnostics.design_effect_moulton), whose square root, 2.60 (diagnostics.sqrt_design_effect), closely matches the observed SE inflation of 2.58. So the effective sample size is closer to 40 schools than to 1000 students. The school-means t-test gives the same answer: SE 0.738 (diagnostics.school_means_se) and p = 0.201 (diagnostics.school_means_ttest_p_df38).

# Limitations and open issues

• With 40 clusters, CR1 with t(39) reference is generally adequate; wild-cluster bootstrap was not run (not required, design balanced and the aggregated t-test agrees).
• Not rejecting H0 does not show the effect is zero. The CI is consistent with effects from about −0.5 to +2.4 points.
• A causal reading of τ relies on schools being assigned at random. The dataset does not document how schools were assigned, so this is assumed, not verified.
• The data are synthetic.