# F2-conf-01 — School-level programme effect on student scores

## Question
What is the average effect τ of a programme that was assigned to whole schools on student test scores? τ is the student-weighted difference in mean score, treated minus control. That is the OLS coefficient of `score` on `treated` with an intercept. The uncertainty must respect the clustered design. Is τ ≠ 0 at the 5% level (two-sided)?

## Data and exclusions
- Input: `data/schools.csv`, 1000 rows read. The data are simulated.
- Checks run: no missing values, no duplicate `student_id`, `treated` is constant within every school and takes only the values 0/1.
- Design: 40 schools with 25 students each (balanced), 20 treated and 20 control.
- No data issues were found, no rows were excluded, and all 1000 students were analysed.

## Method
- **Point estimate:** OLS of `score` on `treated` with an intercept.
- **Naïve SE (`se_iid`):** the conventional OLS SE, which assumes errors are independent and have equal variance. This is reported only as a baseline.
- **Main SE (`se_cluster`):** a cluster-robust (Liang–Zeger) sandwich SE, clustered by `school_id`, with the CR1 small-sample correction G/(G−1)·(N−1)/(N−K).
  - Schools are independent. Students within a school share school-level shocks, and treatment varies only between schools, so the school is the right unit for inference.
  - With only G = 40 clusters, the p-value and 95% CI use a t distribution with G−1 = 39 degrees of freedom rather than the normal.
- **Cross-check:** a two-sample t-test on the 40 school means. Because the design is balanced, it is essentially equivalent to the main analysis.
- **ICC:** a one-way ANOVA (method-of-moments) estimator applied to the OLS residuals. The between-school mean square uses G−2 df because two regression parameters were removed. A negative variance component would have been truncated at 0, but that did not happen here.

## Results
| quantity | value |
|---|---|
| τ̂ | 0.961 points |
| SE, naïve iid | 0.282 |
| SE, cluster (CR1) | 0.729 |
| t (cluster) | 1.32 (df 39) |
| p-value (cluster, t₃₉) | 0.195 |
| 95% CI (cluster) | [−0.513, 2.436] |
| ICC of residual | 0.240 |
| Significant at 5%? | **No** |

- **Point estimate:** students in treated schools scored about 0.96 points higher on average.
- **Significance:** with design-valid uncertainty, the difference is not statistically significant at the 5% level, and the 95% CI includes 0.
- **Cross-check:** the school-means t-test agrees (t = 1.30, p = 0.201).

## Uncertainty
- **Naïve inference is wrong here.** The naïve SE is 2.58 times smaller than the cluster SE. If used, it would give p ≈ 0.0007 and a spurious "significant" result.
- **The ICC explains the gap.** The estimated ICC of 0.24 with 25 students per school implies a design effect of 1 + 24·0.24 ≈ 6.8. Its square root, ≈ 2.60, matches the observed SE ratio.
- **Effective sample size:** there are effectively about 40 independent units, not 1000.

## Limitations and open issues
- **Few clusters:** with 40 clusters (20 per arm), CR1 with t(G−1) is a standard and reasonable choice. Alternatives such as CR2/Bell–McCaffrey or the wild cluster bootstrap were not run. Because the design is balanced and the school-means t-test agrees closely, they would be unlikely to change the conclusion.
- **No covariates:** none were available, so precision could not be improved by adjustment.
- **Causal reading:** interpreting τ̂ as causal assumes schools were randomised. The task describes assignment but does not state that it was random.
- **Simulated data:** the dataset is synthetic, so the results say nothing about any real programme.
- **What the null result means:** a non-significant result is not evidence of no effect. The CI is compatible with effects from about −0.5 to +2.4 points.