1# F2-conf-01 — School-level programme effect on student scores23## Question4What is the average effect τ of a programme that was assigned to whole schools on student test scores? τ is the student-weighted difference in mean score, treated minus control. That is the OLS coefficient of `score` on `treated` with an intercept. The uncertainty must respect the clustered design. Is τ ≠ 0 at the 5% level (two-sided)?56## Data and exclusions7- Input: `data/schools.csv`, 1000 rows read. The data are simulated.8- Checks run: no missing values, no duplicate `student_id`, `treated` is constant within every school and takes only the values 0/1.9- Design: 40 schools with 25 students each (balanced), 20 treated and 20 control.10- No data issues were found, no rows were excluded, and all 1000 students were analysed.1112## Method13- **Point estimate:** OLS of `score` on `treated` with an intercept.14- **Naïve SE (`se_iid`):** the conventional OLS SE, which assumes errors are independent and have equal variance. This is reported only as a baseline.15- **Main SE (`se_cluster`):** a cluster-robust (Liang–Zeger) sandwich SE, clustered by `school_id`, with the CR1 small-sample correction G/(G−1)·(N−1)/(N−K).16 - Schools are independent. Students within a school share school-level shocks, and treatment varies only between schools, so the school is the right unit for inference.17 - With only G = 40 clusters, the p-value and 95% CI use a t distribution with G−1 = 39 degrees of freedom rather than the normal.18- **Cross-check:** a two-sample t-test on the 40 school means. Because the design is balanced, it is essentially equivalent to the main analysis.19- **ICC:** a one-way ANOVA (method-of-moments) estimator applied to the OLS residuals. The between-school mean square uses G−2 df because two regression parameters were removed. A negative variance component would have been truncated at 0, but that did not happen here.2021## Results22| quantity | value |23|---|---|24| τ̂ | 0.961 points |25| SE, naïve iid | 0.282 |26| SE, cluster (CR1) | 0.729 |27| t (cluster) | 1.32 (df 39) |28| p-value (cluster, t₃₉) | 0.195 |29| 95% CI (cluster) | [−0.513, 2.436] |30| ICC of residual | 0.240 |31| Significant at 5%? | **No** |3233- **Point estimate:** students in treated schools scored about 0.96 points higher on average.34- **Significance:** with design-valid uncertainty, the difference is not statistically significant at the 5% level, and the 95% CI includes 0.35- **Cross-check:** the school-means t-test agrees (t = 1.30, p = 0.201).3637## Uncertainty38- **Naïve inference is wrong here.** The naïve SE is 2.58 times smaller than the cluster SE. If used, it would give p ≈ 0.0007 and a spurious "significant" result.39- **The ICC explains the gap.** The estimated ICC of 0.24 with 25 students per school implies a design effect of 1 + 24·0.24 ≈ 6.8. Its square root, ≈ 2.60, matches the observed SE ratio.40- **Effective sample size:** there are effectively about 40 independent units, not 1000.4142## Limitations and open issues43- **Few clusters:** with 40 clusters (20 per arm), CR1 with t(G−1) is a standard and reasonable choice. Alternatives such as CR2/Bell–McCaffrey or the wild cluster bootstrap were not run. Because the design is balanced and the school-means t-test agrees closely, they would be unlikely to change the conclusion.44- **No covariates:** none were available, so precision could not be improved by adjustment.45- **Causal reading:** interpreting τ̂ as causal assumes schools were randomised. The task describes assignment but does not state that it was random.46- **Simulated data:** the dataset is synthetic, so the results say nothing about any real programme.47- **What the null result means:** a non-significant result is not evidence of no effect. The CI is compatible with effects from about −0.5 to +2.4 points.48