F2-conf-01 — School-level programme effect on student scores
Question
A programme was assigned to whole schools. What is its effect on student test scores, τ = treated-school minus control-school student-weighted mean score (OLS slope of score on treated), with uncertainty that reflects school-level assignment? Is τ ≠ 0 at the 5% level (two-sided)?
Data and exclusions
data/schools.csv (synthetic): 1000 rows (n_input_rows), no missing values, no duplicate students, treatment constant within each school. There are 40 schools (n_schools) with 25 students each; 20 are treated (n_treated_schools). No rows were excluded (out/exclusions.csv), so n_analysed = 1000. No data issues (data_issues = []).
Method
- Point estimate: OLS of score on treated, with an intercept (
tau_hat). - Naive baseline: conventional homoskedastic iid OLS standard error (
se_iid). - Main uncertainty: cluster-robust (CR1) standard error clustered by school, using statsmodels' small-sample correction (
se_cluster). Tests and the 95% CI use a t distribution with G−1 = 39 degrees of freedom (aux_df_cluster), which is conservative with 40 clusters. - ICC: one-way ANOVA variance-component estimator applied to the OLS residuals (
icc_hat). - Cross-check: Welch two-sample t-test on the 40 school means (
aux_school_means_welch_p). - The method is deterministic and uses no random seed (
random_seed= null).
Results
tau_hat= 0.961 points.se_cluster= 0.729, t = 1.32 (aux_t_stat_cluster),p_value= 0.195.- 95% CI: [
tau_ci_low,tau_ci_high] = [−0.513, 2.436]. effect_significant_5pct= false. At the 5% level, the data do not support a non-zero effect.icc_hat= 0.235.
Uncertainty
The naive se_iid = 0.282 would give p = 0.0007 (aux_p_value_iid), which looks highly significant. It ignores the fact that students in the same school are correlated. The cluster SE is 2.58 times larger (aux_se_ratio_cluster_over_iid). This matches the Moulton factor √(1+(25−1)·0.235) = √6.64 (aux_design_effect_moulton). The school-means Welch test reaches the same conclusion (p = 0.201, aux_school_means_welch_p). Because treatment varies only between schools, the effective sample size is close to 40 schools, not 1000 students.
Limitations and open issues
- A non-significant result does not show that the effect is zero. The CI is consistent with effects from about −0.5 to +2.4 points.
- Reading τ causally requires that schools were assigned to treatment as if at random. The task does not document the assignment mechanism.
- Cluster-robust inference relies on asymptotics in the number of clusters. With 40 balanced clusters this is generally adequate, and the t(39) reference plus the school-means cross-check supports it.
- The ICC is estimated with noise from 40 schools.