# F2-conf-01 — School-level programme effect on student scores

## Question
Estimate τ, the difference in mean test score between students in treated and control schools (the OLS coefficient of `score` on `treated` with an intercept). Give a standard error that matches the design, where treatment is assigned to whole schools, and test H0: τ = 0 at the 5% level (two-sided).

## Data and exclusions
- `data/schools.csv`: 1000 rows read, 1000 analysed. There are 40 schools with 25 students each; 20 schools are treated and 20 are controls.
- Checks run: missing values (none), duplicate `student_id` (none), duplicate rows (none), `treated` constant within each school (yes in all 40 schools), and equal school sizes (yes).
- No data issues were found and no rows were excluded (`data_issues` = []).

## Method
- **Point estimate:** OLS of `score` on `treated` with an intercept, using all students.
- **Naïve SE (`se_iid`):** the conventional homoskedastic OLS standard error. This is reported only as a baseline. It is not valid here, because it treats the 1000 students as independent.
- **Main SE (`se_cluster`):** a cluster-robust (CR1) sandwich standard error, clustered by `school_id`, with the usual small-sample correction G/(G−1)·(N−1)/(N−K). Because there are only G = 40 clusters, the p-value and 95% CI use a t distribution with G−1 = 39 degrees of freedom rather than the normal.
- **Cross-check:** a two-sample t-test on the 40 school means, which is an equivalent analysis at the level where treatment was randomised.
- **ICC:** a one-way ANOVA estimator applied to the OLS residuals, grouped by school: ICC = σ²_b/(σ²_b+σ²_w), where σ²_b = (MSB−MSW)/m and m = 25.
- No randomness is used (`random_seed` = null).

## Results
| quantity | value |
|---|---|
| τ̂ (points) | 0.961 |
| SE, naïve iid | 0.282 |
| SE, clustered by school (CR1) | 0.729 |
| t (cluster), df = 39 | 1.32 |
| p-value (cluster) | 0.195 |
| 95% CI (cluster) | [−0.513, 2.436] |
| residual ICC | 0.235 |

On average, students in treated schools scored about 0.96 points higher than students in control schools. With design-valid uncertainty, this difference is **not statistically significant at the 5% level** (p = 0.195), and the 95% CI includes zero.

## Uncertainty
- The naïve iid SE (0.282) would give p ≈ 0.0007 and a spurious "significant" conclusion.
- The clustered SE is 2.58 times larger. This implies a design effect of (0.729/0.282)² ≈ 6.68.
- That design effect matches the Kish formula 1 + (m−1)·ICC = 1 + 24·0.235 ≈ 6.64. Most of the uncertainty therefore comes from school-level variation, and the effective sample size is closer to about 150 students than to 1000.
- The school-means t-test gives a very similar result (SE 0.738, p = 0.201), which supports the clustered result.

## Limitations and open issues
- With only 40 clusters, CR1 standard errors can be slightly too small. Using t(39) critical values and the school-means cross-check reduces this concern, and both lead to the same conclusion.
- A non-significant result does not show that the effect is zero. The CI is consistent with effects from about −0.5 to +2.4 points, so the study cannot rule out a modest positive effect.
- The data are simulated. The causal interpretation of τ rests on the assumption that schools were assigned to treatment as if at random; this cannot be checked from the data, since no covariates are provided.