| 1 |
Clean re-run (deleted results.json, forecasts.csv, table1.csv, figure1.png) regenerates byte-identical results.json/forecasts.csv/table1.csv (SHA-256 compared), done twice (before and after adding HLN sensitivity keys) |
PASS |
| 2 |
forecasts.csv: 80 rows, unique (origin,horizon,model); vintage_used == origin_date |
PASS |
| 3 |
Information rule: last(V) = target − h precedes origin date for every row; each forecast reads only RSAFS vintage V (code inspection: only vintages/RSAFS_vintage_{V} enters FUNCS); RSAFS_current used only in estimand (d) block |
PASS |
| 4 |
actual_first_release equals first_release.csv yoy for every target |
PASS |
| 5 |
naive, snaive recomputed independently; AR(1) recomputed with statsmodels OLS (sample 1993-02…last(V)) and iterated — matches to 1e-5 |
PASS |
| 6 |
RMSE/MAE recomputed from forecasts.csv for all 8 (model,h); RMSE ≥ MAE in all cases |
PASS |
| 7 |
DM statistic and p-value recomputed independently (Bartlett weights 2/3,1/3 at h=3; lag 0 at h=1); LRV > 0 at both horizons (0.809, 2.119) |
PASS |
| 8 |
rel_improvement = 1 − RMSE_model/RMSE_naive recomputed |
PASS |
| 9 |
Bootstrap 95% CIs of RMSE difference contain the point estimate (h1, h3) |
PASS |
| 10 |
Estimand (d) recomputed independently from RSAFS_current.csv |
PASS |
| 11 |
Counts: n_raw pairs 80 − 0 removed = 80 analysed; n_analysed (model) = 20 |
PASS |
| 12 |
n_input_rows equal wc -l − 1 for all 23 files |
PASS |
| 13 |
Magnitudes vs baseline: model RMSE (0.55 h1, 1.14 h3) of same order as baselines (0.94–1.67); errors in percent; h3 errors > h1 errors for naive/AR1/model as expected |
PASS |
| 14 |
conclusion codes consistent with RMSE ordering and p<0.05 rule (h1: p=0.0151 → significant; h3: p=0.122 → not significant) |
PASS |