# F2-conf-01 — School-level programme effect on student scores

# Question

A programme was assigned to whole schools. We estimate τ, the difference between treated- and control-school mean student scores (OLS slope of score on treated with intercept). We also give a standard error that reflects the clustered design, where treatment varies only between schools and students in the same school share school-level shocks. Finally, we test H0: τ = 0 at the 5% level (two-sided).

# Data and exclusions

• data/schools.csv (synthetic). 1000 rows read (n_input_rows). The file has no missing values, no duplicate rows and no duplicate student ids. Its SHA-256 and the full audit are in out/state/data_audit.md.
• n_schools = 40 schools of 25 students each (students_per_school). n_treated_schools = 20. Treatment is constant within every school.
• No exclusions (out/exclusions.csv): n_analysed = 1000. data_issues is empty.

# Method

• Point estimate: OLS score ~ treated (statsmodels).
• Naïve baseline: conventional iid/homoskedastic OLS SE (se_iid).
• Main uncertainty measure: cluster-robust SE with school as the cluster (CR1: statsmodels' default small-sample factor G/(G−1)·(N−1)/(N−K)) (se_cluster). The p-value and 95% CI use a t distribution with G−1 = 39 degrees of freedom (t_df). This is a common conservative reference when there are few clusters.
• Sensitivity check: a two-sample t-test on the 40 school means. Because every school has the same size, this targets the same τ (school_mean_ttest_t, school_mean_ttest_p).
• ICC: a one-way ANOVA (method-of-moments) estimator applied to the OLS residuals grouped by school, so the treatment effect is removed first (icc_hat). Implied design effect = 1 + (m−1)·ICC (design_effect_from_icc).
• The method is deterministic, so no seed is used (random_seed = null).

# Results

• tau_hat = 0.961 points: treated-school students score about 0.96 points higher on average.
• se_iid = 0.282 versus se_cluster = 0.729. The clustered SE is 2.58 times larger (se_ratio_cluster_to_iid). That ratio fits the estimated icc_hat = 0.235, which implies a design effect of 6.64 (design_effect_from_icc) and √6.64 ≈ 2.58.
• Clustered test: t = 1.32 (t_stat_cluster), p_value = 0.195. 95% CI: [tau_ci_low, tau_ci_high] = [−0.51, 2.44].
• effect_significant_5pct = false: with design-valid uncertainty, the data do not support a non-zero effect at the 5% level.
• The naïve iid analysis would give p = 0.0007 (p_value_iid) and falsely declare significance. This shows why the clustered SE matters.

# Uncertainty

The CI runs from about −0.5 to +2.4 points. So the data are consistent with no effect and also with a positive effect of up to about 2.4 points. The school-means t-test gives the same conclusion (p = 0.201, school_mean_ttest_p). Inference rests on 40 independent schools, the effective number of independent units, not on 1000 students.

# Limitations and open issues

• The data are synthetic. Reading τ as a causal effect assumes schools were assigned to treatment as-good-as-randomly, which cannot be checked from these data.
• With G = 40 clusters, cluster-robust inference is generally regarded as reasonable but not exact. CR1 with t(39) and the school-means t-test agree here. Wild cluster bootstrap and CR2 were not computed because the task does not require them.
• icc_hat is a moment estimator and is noisy with 40 clusters. It is a descriptive quantity and does not drive the decision.
• Not significant does not mean no effect. The study cannot rule out effects of up to about 2.4 points.