# F2-conf-01 — School-level programme effect on student scores

# Question

A programme was assigned to whole schools. We estimate τ, the difference between the student-weighted mean test score in treated and control schools (OLS slope of score on treated with intercept), with an uncertainty measure that respects the clustered design (treatment varies only between schools; students share school-level shocks; schools independent), and test H0: τ = 0 at the 5% level (two-sided).

# Data and exclusions

data/schools.csv (simulated; SHA-256 0f1d8378…c88a) has 1000 rows (n_input_rows). Audit (out/state/data_audit.md): no missing values, no duplicate students, 40 schools (n_schools) of exactly 25 students (students_per_school), treatment constant within each school, 20 treated schools (n_treated_schools). No rows were excluded (out/exclusions.csv), so 1000 students were analysed (n_analysed); no data issues were found (data_issues = []).

# Method

• Point estimate: OLS of score on treated with an intercept (tau_hat), which equals the difference of arm means (mean_score_treated − mean_score_control).
• Naive baseline: conventional homoskedastic iid OLS standard error (se_iid, p-value baseline_p_value_iid). This ignores within-school correlation and is not valid for this design.
• Main uncertainty: cluster-robust (CR1) standard error clustered by school_id (se_cluster), with small-sample factor G/(G−1)·(N−1)/(N−K), and a t reference distribution with G−1 = 39 degrees of freedom (cluster_t_df) for the p-value (p_value) and 95% CI (tau_ci_low, tau_ci_high).
• ICC (icc_hat): one-way ANOVA estimator on the OLS residuals (after removing the treatment effect), between-school mean square on G−2 df, within on N−G df.
• Robustness check: two-sample pooled-variance t-test on the 40 school means (robust_school_mean_ttest_se, robust_school_mean_ttest_p).
• Nothing is random, so no seed is used (random_seed = null).

# Results

• Estimated effect tau_hat = 0.961 points (treated mean mean_score_treated = 51.28 vs control mean_score_control = 50.32).
• Cluster-robust SE se_cluster = 0.729; 95% CI [tau_ci_low, tau_ci_high] = [−0.513, 2.436]; p_value = 0.195.
• Decision: H0: τ = 0 is not rejected at the 5% level (effect_significant_5pct = false). The data do not provide statistically significant evidence of a non-zero programme effect.
• Residual ICC icc_hat = 0.240.

# Uncertainty

The naive iid SE se_iid = 0.282 would give baseline_p_value_iid = 0.0007 and a spurious "significant" finding. The cluster SE is design_effect_se_ratio = 2.58 times larger, in line with the Moulton factor √(1+(25−1)·0.240) ≈ 2.60 implied by icc_hat (check in out/state/checks.md). The school-means t-test gives nearly the same answer (robust_school_mean_ttest_se = 0.738, robust_school_mean_ttest_p = 0.201), so the conclusion does not depend on the CR1 implementation. With only 40 clusters, the effective sample size is closer to 40 than to 1000.

# Limitations and open issues

• Data are simulated; the causal reading of τ assumes schools were randomly assigned (not verifiable from the data).
• CR1 with t(G−1) is a standard but approximate small-sample correction; 40 balanced clusters (20/20) is generally adequate, and the exact-style school-means t-test agrees.
• A non-significant result is not evidence of zero effect: the CI includes effects up to ~2.4 points.
• No issues prevented completion (feasibility = "complete").
• Process note: during setup, state files were briefly created in data/out/ by a working-directory mistake and immediately moved to out/; data/schools.csv was verified unchanged (hash).