1# F2-conf-01 — School-level programme effect on student scores23## Question4Estimate τ, the difference in mean student test score between treated and control schools (the OLS coefficient of `score` on `treated` with an intercept). Report sampling uncertainty that matches the design: treatment was assigned to whole schools, and students in the same school share school-level shocks. Then test H0: τ = 0 at the 5% level, two-sided.56## Data and exclusions7`data/schools.csv` has 1000 rows: 40 schools with 25 students each, 20 of them treated. The data are simulated. Checks run: no missing values, no duplicate `student_id`, `treated` constant within every school, balanced school sizes. No problems were found and no rows were excluded (n_analysed = 1000).89## Method10- **Point estimate:** OLS of `score` on `treated` with an intercept, using all students.11- **Naïve baseline (`se_iid`):** the conventional OLS standard error, which assumes independent, homoskedastic errors.12- **Main SE (`se_cluster`):** a cluster-robust (Liang–Zeger) sandwich SE clustered by `school_id`, with the CR1 small-sample correction G/(G−1)·(N−1)/(N−K). Tests and the 95% CI use a t distribution with G−1 = 39 degrees of freedom, a standard adjustment when there are few clusters.13- **Cross-check:** a two-sample t-test on the 40 school means. With balanced schools this gives the same point estimate and is an exact analogue of the cluster analysis.14- **ICC:** a one-way ANOVA (method-of-moments) estimator applied to the OLS residuals, so the treatment effect is removed first. Between-school mean square uses G−2 df; within-school mean square uses N−G df.1516## Results17| quantity | value |18|---|---|19| τ̂ | 0.961 points |20| SE, iid (naïve) | 0.282 |21| SE, cluster-robust (CR1) | 0.729 |22| t (cluster), df = 39 | 1.32 |23| p-value (cluster) | 0.195 |24| 95% CI (cluster) | [−0.513, 2.436] |25| residual ICC | 0.240 |26| design effect 1+(m−1)ICC | 6.77 |2728Treated schools score about 0.96 points higher on average, but **this difference is not statistically significant at the 5% level** under design-valid inference (p = 0.195), and the 95% CI includes zero.2930## Uncertainty31The cluster-robust SE is 2.58 times the naïve iid SE. About 24% of residual variance lies between schools, so 25 students in one school carry much less independent information than 25 students drawn at random. A rough design-effect argument (√6.77 ≈ 2.6) agrees with the observed SE ratio.3233The naïve analysis would give p ≈ 0.0007 and wrongly declare a highly significant effect. That result is an artefact of treating 1000 correlated students as independent observations. In effect the sample size for inference is 40 schools.3435The school-means t-test gives SE 0.738 and p = 0.201, which agrees with the cluster result.3637## Limitations and open issues38- With G = 40 clusters (20 per arm), cluster-robust SEs can be somewhat biased downward. The t(39) reference and the CR1 correction reduce this, and the school-means t-test agrees closely. No wild-cluster bootstrap was run, so no random seed was used.39- The ICC is a method-of-moments estimate from a modest number of schools, so it is itself imprecise.40- "Not significant" does not show that the effect is zero. The CI runs from about −0.5 to +2.4 points, so moderate positive effects are still possible.41- The estimand is the student-weighted difference in means. With equal school sizes this is the same as the school-weighted difference.42- The data are simulated.43