# F2-conf-01 — School-level programme effect on student scores

# Question

A programme was assigned to whole schools. We estimate τ, the difference between the student-weighted mean score in treated and control schools (OLS slope of score on treated with an intercept), give an uncertainty measure that respects the school-level design, and test H0: τ = 0 at the 5% level (two-sided).

# Data and exclusions

Input: data/schools.csv, 1000 rows (n_input_rows), simulated data. Audit (out/state/data_audit.md): no missing values, no duplicates, unique student IDs, 40 schools (n_schools) of exactly 25 students (school_size), treatment constant within school, 20 treated schools (n_treated_schools). No rows excluded (out/exclusions.csv); n_analysed = 1000. data_issues is empty.

# Method

• Point estimate: OLS of score on treated with intercept (tau_hat).
• Naïve baseline: conventional homoskedastic iid OLS SE (se_iid).
• Main uncertainty: cluster-robust (Liang–Zeger, CR1 small-sample correction) SE clustered by school_id (se_cluster). Test statistic and 95% CI use a t distribution with G−1 = 39 df (df_cluster), which is appropriate with a moderate number of clusters.
• ICC: one-way random-effects ANOVA estimator on the OLS residuals, with G−2 between-school df (one lost to the treatment effect) (icc_hat).
• Sensitivity: two-sample t-test on the 40 school means (sens_school_means_se, sens_school_means_p).
• Deterministic; no randomness (random_seed = null).

# Results

• tau_hat = 0.961 points (treated mean 51.28 mean_treated vs control 50.32 mean_control).
• se_cluster = 0.729; t = 1.32 (t_stat_cluster); p_value = 0.195.
• 95% CI: [tau_ci_low = −0.513, tau_ci_high = 2.436].
• effect_significant_5pct = false: the data do not support a non-zero effect at the 5% level.
• icc_hat = 0.240; implied design effect 6.77 (design_effect_from_icc).

# Uncertainty

The naïve se_iid = 0.282 would give p = 0.0007 (p_value_iid) and a spurious "significant" result. Because the programme varies only between schools and students in a school share school-level shocks, the effective sample is closer to 40 schools than 1000 students: the clustered SE is 2.58× larger (se_ratio_cluster_over_iid), consistent with √(design effect) ≈ 2.60. The school-means t-test gives the same conclusion (SE 0.738, p = 0.201). Verification checks are listed in out/state/checks.md.

# Limitations and open issues

• Not rejecting H0 does not mean the effect is zero; the CI is compatible with effects from about −0.5 to +2.4 points.
• With 40 clusters, CR1 with t(39) is a standard and reasonably accurate choice; wild-cluster bootstrap was not used (not required; balanced design makes CR1 ≈ school-means t-test, which was checked).
• Causal interpretation relies on the school assignment being as-good-as-random, as in the simulated design; no covariates were available.
• The data are synthetic.