Verification checks (P4)
Script: out/state/verify.py; output: out/state/verify_output.txt.
| id | check | outcome |
|---|---|---|
| C1 | Delete results/forecasts/table/figure, re-run analysis.py, SHA-256 of results.json identical | PASS |
| C2 | forecasts.csv has 80 rows, unique (origin,horizon,model) | PASS |
| C3 | vintage_used == origin_date for all rows (no future vintage) | PASS |
| C4 | first_release.csv consistent with shipped vintages and YoY formula | PASS |
| C5 | RMSE recomputed independently from forecasts.csv equals results.json | PASS (0 mismatches) |
| C6 | DM statistics recomputed independently with explicit Bartlett weights (h=3: 2/3, 1/3); LRV > 0 | PASS (2.3021, 1.8645) |
| C7 | naive forecast equals last YoY of vintage (spot check 2026-04-30) | PASS |
| C8 | rel_improvement = 1 − RMSE_model/RMSE_naive | PASS |
| C9 | MAE ≤ RMSE for all model/horizon pairs | PASS |
| C10 | every target month > last(V) and not published in vintage V | PASS (also asserted in analysis.py) |
| C11 | revision for 2026-01 recomputed by hand from RSAFS_current | PASS |
| C12 | Magnitudes: model RMSE (0.55/1.05) below naive (1.00/1.42) and in the same order of magnitude as the baselines; snaive worst at h=1 as expected for a 12-month-old value | PASS (plausible) |
| C13 | Gate forecasts_chronology (rule origin_date < target_date) |
FAIL — structural: all 40 h=1 rows have target_date (YYYY-MM-01 of last(V)+1) ≤ month-end origin_date. This follows from TASK.md's definition target = last(V)+h with target_date formatted YYYY-MM-01; for the six two-month-delay origins the h=1 target month is entirely before the origin date. No date convention compatible with TASK.md satisfies the rule. Information validity is not affected: the target month is not present in vintage V (C10) and vintage_used ≤ origin_date (C3). Reported, not "fixed", because altering targets would violate TASK.md. |