# F2-conf-01 — School-level programme effect on student scores

## Question
A programme was assigned to whole schools. We estimate τ, the difference between the student-weighted mean test score in treated and control schools (OLS slope of `score` on `treated` with intercept), with an uncertainty measure that respects the clustered design (treatment varies only between schools; students share school-level shocks; schools independent), and test H0: τ = 0 at the 5% level (two-sided).

## Data and exclusions
`data/schools.csv` (simulated; SHA-256 0f1d8378…c88a) has 1000 rows (`n_input_rows`). Audit (`out/state/data_audit.md`): no missing values, no duplicate students, 40 schools (`n_schools`) of exactly 25 students (`students_per_school`), treatment constant within each school, 20 treated schools (`n_treated_schools`). No rows were excluded (`out/exclusions.csv`), so 1000 students were analysed (`n_analysed`); no data issues were found (`data_issues` = []).

## Method
- Point estimate: OLS of score on treated with an intercept (`tau_hat`), which equals the difference of arm means (`mean_score_treated` − `mean_score_control`).
- Naive baseline: conventional homoskedastic iid OLS standard error (`se_iid`, p-value `baseline_p_value_iid`). This ignores within-school correlation and is not valid for this design.
- Main uncertainty: cluster-robust (CR1) standard error clustered by `school_id` (`se_cluster`), with small-sample factor G/(G−1)·(N−1)/(N−K), and a t reference distribution with G−1 = 39 degrees of freedom (`cluster_t_df`) for the p-value (`p_value`) and 95% CI (`tau_ci_low`, `tau_ci_high`).
- ICC (`icc_hat`): one-way ANOVA estimator on the OLS residuals (after removing the treatment effect), between-school mean square on G−2 df, within on N−G df.
- Robustness check: two-sample pooled-variance t-test on the 40 school means (`robust_school_mean_ttest_se`, `robust_school_mean_ttest_p`).
- Nothing is random, so no seed is used (`random_seed` = null).

## Results
- Estimated effect `tau_hat` = 0.961 points (treated mean `mean_score_treated` = 51.28 vs control `mean_score_control` = 50.32).
- Cluster-robust SE `se_cluster` = 0.729; 95% CI [`tau_ci_low`, `tau_ci_high`] = [−0.513, 2.436]; `p_value` = 0.195.
- **Decision:** H0: τ = 0 is not rejected at the 5% level (`effect_significant_5pct` = false). The data do not provide statistically significant evidence of a non-zero programme effect.
- Residual ICC `icc_hat` = 0.240.

## Uncertainty
The naive iid SE `se_iid` = 0.282 would give `baseline_p_value_iid` = 0.0007 and a spurious "significant" finding. The cluster SE is `design_effect_se_ratio` = 2.58 times larger, in line with the Moulton factor √(1+(25−1)·0.240) ≈ 2.60 implied by `icc_hat` (check in `out/state/checks.md`). The school-means t-test gives nearly the same answer (`robust_school_mean_ttest_se` = 0.738, `robust_school_mean_ttest_p` = 0.201), so the conclusion does not depend on the CR1 implementation. With only 40 clusters, the effective sample size is closer to 40 than to 1000.

## Limitations and open issues
- Data are simulated; the causal reading of τ assumes schools were randomly assigned (not verifiable from the data).
- CR1 with t(G−1) is a standard but approximate small-sample correction; 40 balanced clusters (20/20) is generally adequate, and the exact-style school-means t-test agrees.
- A non-significant result is not evidence of zero effect: the CI includes effects up to ~2.4 points.
- No issues prevented completion (`feasibility` = "complete").
- Process note: during setup, state files were briefly created in `data/out/` by a working-directory mistake and immediately moved to `out/`; `data/schools.csv` was verified unchanged (hash).