# F2-conf-01 — School-level programme effect on student scores

## Question
Estimate τ, the difference in mean student test score between treated and control schools (the OLS coefficient of `score` on `treated` with an intercept). Report sampling uncertainty that matches the design: treatment was assigned to whole schools, and students in the same school share school-level shocks. Then test H0: τ = 0 at the 5% level, two-sided.

## Data and exclusions
`data/schools.csv` has 1000 rows: 40 schools with 25 students each, 20 of them treated. The data are simulated. Checks run: no missing values, no duplicate `student_id`, `treated` constant within every school, balanced school sizes. No problems were found and no rows were excluded (n_analysed = 1000).

## Method
- **Point estimate:** OLS of `score` on `treated` with an intercept, using all students.
- **Naïve baseline (`se_iid`):** the conventional OLS standard error, which assumes independent, homoskedastic errors.
- **Main SE (`se_cluster`):** a cluster-robust (Liang–Zeger) sandwich SE clustered by `school_id`, with the CR1 small-sample correction G/(G−1)·(N−1)/(N−K). Tests and the 95% CI use a t distribution with G−1 = 39 degrees of freedom, a standard adjustment when there are few clusters.
- **Cross-check:** a two-sample t-test on the 40 school means. With balanced schools this gives the same point estimate and is an exact analogue of the cluster analysis.
- **ICC:** a one-way ANOVA (method-of-moments) estimator applied to the OLS residuals, so the treatment effect is removed first. Between-school mean square uses G−2 df; within-school mean square uses N−G df.

## Results
| quantity | value |
|---|---|
| τ̂ | 0.961 points |
| SE, iid (naïve) | 0.282 |
| SE, cluster-robust (CR1) | 0.729 |
| t (cluster), df = 39 | 1.32 |
| p-value (cluster) | 0.195 |
| 95% CI (cluster) | [−0.513, 2.436] |
| residual ICC | 0.240 |
| design effect 1+(m−1)ICC | 6.77 |

Treated schools score about 0.96 points higher on average, but **this difference is not statistically significant at the 5% level** under design-valid inference (p = 0.195), and the 95% CI includes zero.

## Uncertainty
The cluster-robust SE is 2.58 times the naïve iid SE. About 24% of residual variance lies between schools, so 25 students in one school carry much less independent information than 25 students drawn at random. A rough design-effect argument (√6.77 ≈ 2.6) agrees with the observed SE ratio.

The naïve analysis would give p ≈ 0.0007 and wrongly declare a highly significant effect. That result is an artefact of treating 1000 correlated students as independent observations. In effect the sample size for inference is 40 schools.

The school-means t-test gives SE 0.738 and p = 0.201, which agrees with the cluster result.

## Limitations and open issues
- With G = 40 clusters (20 per arm), cluster-robust SEs can be somewhat biased downward. The t(39) reference and the CR1 correction reduce this, and the school-means t-test agrees closely. No wild-cluster bootstrap was run, so no random seed was used.
- The ICC is a method-of-moments estimate from a modest number of schools, so it is itself imprecise.
- "Not significant" does not show that the effect is zero. The CI runs from about −0.5 to +2.4 points, so moderate positive effects are still possible.
- The estimand is the student-weighted difference in means. With equal school sizes this is the same as the school-weighted difference.
- The data are simulated.