# F2-conf-01 — Programme effect on student scores with cluster-valid uncertainty

# Question

A programme was assigned to whole schools. We estimate τ, the difference between student-weighted mean scores in treated and control schools (OLS coefficient of score on treated with intercept), give a sampling uncertainty that respects the school-level design, and test H0: τ = 0 at the 5% level (two-sided).

# Data and exclusions

data/schools.csv (synthetic; SHA-256 0f1d8378…c88a) has 1000 rows (n_input_rows). No missing values, duplicate rows or duplicate student IDs were found. out/exclusions.csv therefore records two filtering steps that each removed 0 rows. All n_analysed = 1000 students were used. They come from n_schools = 40 schools of 25 students each, and n_treated_schools = 20 of those schools are treated. Treatment is constant within every school. data_issues is empty. Full audit: out/state/data_audit.md.

# Method

• Baseline: OLS of score on treated with the conventional iid, homoskedastic standard error (se_iid).
• Main analysis: the same OLS point estimate (tau_hat), with a cluster-robust variance clustered on school_id. This uses the CR1 small-sample correction from statsmodels: G/(G−1)·(N−1)/(N−K). The SE allows any correlation among students in the same school and assumes schools are independent. Because there are only 40 clusters, the p-value and 95% CI use a t distribution with G−1 = 39 degrees of freedom (df_cluster).
• Cross-check: a Welch two-sample t-test on the 40 school means (school_means_welch_se, school_means_welch_p).
• ICC: I subtracted each arm's mean score to get residuals, then applied a one-way ANOVA variance-components estimator by school (2 between-school df used up by the arm means). icc_hat = σ²_between / (σ²_between + σ²_within). It implies a design effect of 1 + (25−1)·ICC (design_effect_from_icc).
• The method is deterministic, so no random seed is used (random_seed = null).

# Results

• Mean score was 51.28 in treated schools (mean_score_treated) and 50.32 in control schools (mean_score_control).
• tau_hat = 0.961 points.
• Valid (cluster) inference: se_cluster = 0.729, t_stat_cluster = 1.32, p_value = 0.195. The 95% CI is [tau_ci_low, tau_ci_high] = [−0.513, 2.436].
• H0: τ = 0 is not rejected at the 5% level (effect_significant_5pct = false). The data do not support a non-zero programme effect.
• The naive baseline would have concluded otherwise: se_iid = 0.282, baseline_p_value_iid = 0.00068, CI [baseline_ci_iid_low, baseline_ci_iid_high] = [0.408, 1.515]. That conclusion is invalid because it treats 1000 correlated students as independent observations.

# Uncertainty

• icc_hat = 0.240: about a quarter of residual score variance lies between schools.
• With 25 students per school, this gives a design effect of about 6.77 (design_effect_from_icc). That matches the observed ratio se_ratio_cluster_to_iid = 2.58, whose square is about 6.68.
• The school-means Welch test gives almost the same SE and p-value (school_means_welch_se = 0.738, school_means_welch_p = 0.201), which supports the cluster-robust result.
• The effective sample size is about 1000/6.8 ≈ 150 students, and the information really comes from 40 schools.

# Limitations and open issues

• The data are synthetic. Reading τ as causal assumes that schools were assigned to treatment as if at random; the data cannot verify this.
• Cluster-robust inference is asymptotic in the number of clusters. With 40 balanced clusters (20/20), CR1 with t(39) is generally adequate, and the school-means analysis agrees closely. A wild-cluster bootstrap was not run because the task does not require it.
• Not rejecting H0 does not show the effect is zero. The 95% CI includes effects from about −0.5 to +2.4 points.
• No covariates were available, so precision could not be improved through adjustment.