Verification checks (P4)
Run by an independent script (statsmodels OLS for AR(1), hand-recomputed DM/RMSE/revision).
| check | outcome |
|---|---|
| clean re-run of analysis.py regenerates byte-identical results.json, forecasts.csv, table1.csv | PASS |
| 80 rows, unique keys | PASS |
| vintage_used<=origin_date<target_date | FAIL |
| AR1 coefs match statsmodels (2026-07-31) | PASS |
| AR1 h3 forecast 2026-07-31 matches | PASS |
| model h3 forecast 2026-07-31 recomputed | PASS |
| rmse/mae naive h1 | PASS |
| rmse/mae snaive h1 | PASS |
| rmse/mae ar1 h1 | PASS |
| rmse/mae model h1 | PASS |
| DM h1 recomputed | PASS |
| rel_improvement h1 | PASS |
| bootstrap CI ordered h1 | PASS |
| LRV positive h1 | PASS |
| rmse/mae naive h3 | PASS |
| rmse/mae snaive h3 | PASS |
| rmse/mae ar1 h3 | PASS |
| rmse/mae model h3 | PASS |
| DM h3 recomputed | PASS |
| rel_improvement h3 | PASS |
| bootstrap CI ordered h3 | PASS |
| LRV positive h3 | PASS |
| actuals match first_release.csv | PASS |
| target = last(V)+h | PASS |
| revision (d) recomputed | PASS |
| all RMSE>0, MAE<=RMSE | PASS |
| conclusion codes consistent | PASS |
| n_analysed = 20 = 80 - 0 exclusions / 4 models | PASS |
Magnitudes: all RMSEs 0.55–1.67 pp of YoY growth, comparable to the spread of first-release YoY (2.4–6.7%); model RMSE below naive at both horizons, consistent with baseline ordering in table1.csv.
Note on the FAIL above (and gate check forecasts_chronology)
vintage_used <= origin_date holds for all 80 rows (vintage_used = origin_date). origin_date < target_date fails for all 40 h = 1 rows (and only those): TASK.md defines the target as last(V)+h with target_date written as YYYY-MM-01, and because last(V) is at least one month before the month of V, the h = 1 target month always begins on or before the month-end origin (e.g. origin 2026-04-30, last = 2026-03, target 2026-04-01; origin 2025-10-31, last = 2025-08, target 2025-09-01). This is not information leakage: the target value is not published in vintage V and the actual comes from a later vintage (first_release_vintage > origin for all 20 targets; verified below). The gate rule origin_date < target_date is therefore inconsistent with the task's own target definition for h = 1. I did not relabel target dates to pass the gate, since that would misstate the target month. Reported as an open issue.
| first_release_vintage > origin_date for all 80 rows (no leakage of the actual) | PASS |