1# F3-dev-01 — Rolling-origin forecasts of the US unemployment rate: AR(2) on differences vs naive23## Question4Over the 36 monthly forecast origins from 2023-07-01 to 2026-06-01 (`n_origins`, `first_origin`, `last_origin`), does a simple autoregressive model forecast the US civilian unemployment rate (FRED UNRATE, %, SA) better than the naive no-change forecast at horizons of 1 and 3 months? Forecasts use expanding windows; accuracy is measured by out-of-sample RMSE and MAE and by the relative improvement 1 − RMSE_model/RMSE_naive.56## Data and exclusions7- Input: `data/UNRATE.csv`, 945 data rows, 1948-01 to 2026-09 (`n_input_rows`); SHA-256 matches the README. Audit details in `out/state/data_audit.md`.8- One missing value: 2025-10-01 (empty field) (`data_issues`). Handling: the naive forecast at origin 2025-10-01 uses the 2025-09-01 value; AR estimation drops regressions involving the gap (complete cases); for forecasting, the missing level is replaced by the model's own forecast from earlier data. Two (origin, horizon) pairs target 2025-10-01 (origin 2025-09-01 at h=1; origin 2025-07-01 at h=3). Both stay in `out/forecasts.csv` with an empty `actual` and are left out of the error statistics. As a result, 35 pairs are evaluated at each horizon (`n_evaluated_h1`, `n_evaluated_h3`).9- Observation-level exclusion ledger (`out/exclusions.csv`): 945 raw rows → −1 missing → −3 observations after the last origin, which are used only as targets → 941 observations in the last training window (`n_analysed`).10- The file is the final (revised) vintage, so this exercise is pseudo-out-of-sample and has no real-time validity.1112## Method13- **Naive:** the forecast at every horizon is the latest non-missing observation dated ≤ origin.14- **Model `AR2_diff`** (`model_name`, `model_description`): Δy_t = c + φ1 Δy_{t−1} + φ2 Δy_{t−2} + e_t, estimated by OLS on all complete cases from 1948 up to the origin and re-estimated at each origin. Forecasts for h = 1..3 come from iterating the difference equation and cumulating onto the last level. I fixed the lag order (2) and the transformation (first differences) before looking at any out-of-sample result. No model selection was done on the evaluation window. Every estimation step, transformation and gap fill uses only data dated ≤ origin; a perturbation test confirmed there is no look-ahead (`out/state/checks.md`, check 11).15- The procedure is deterministic, with no random numbers (`random_seed` = null). `out/analysis.py` regenerates every output.16- Uncertainty: a Diebold–Mariano test on squared-error loss differences, using a HAC variance with h−1 lags, the Harvey–Leybourne–Newbold small-sample correction, and a t(n−1) reference (`diag_dm_stat_h*`, `diag_dm_pvalue_two_sided_h*`).1718## Results19| horizon | n | RMSE naive | RMSE AR2_diff | MAE naive | MAE AR2_diff | 1 − RMSE ratio |20|---|---|---|---|---|---|---|21| 1 | 35 | 0.1171 | 0.1199 | 0.0914 | 0.0957 | −0.0239 |22| 3 | 35 | 0.1586 | 0.1598 | 0.1200 | 0.1213 | −0.0076 |2324Keys: `rmse_naive_h1`, `rmse_model_h1`, `mae_naive_h1`, `mae_model_h1`, `rel_improvement_rmse_h1`, and likewise for `_h3`. Values are in percentage points, rounded from results.json.2526- At h=1 the AR model is about 2.4% worse than the naive forecast in RMSE (`rel_improvement_rmse_h1` = −0.024). At h=3 it is about 0.8% worse (`rel_improvement_rmse_h3` = −0.008). MAE points the same way at both horizons.27- **The answer is no: over this window, the simple AR model did not beat the naive forecast at either horizon.**28- The two forecasts are almost the same. The estimated AR coefficients at the last origin are small (`diag_ar_coef_last_origin` ≈ [0.0003, 0.037, −0.042]), so the model behaves almost like a random walk and its forecasts stay within 0.1 pp of the naive forecasts.2930## Uncertainty31- Diebold–Mariano, h=1: statistic 2.53, two-sided p = 0.016 (`diag_dm_stat_h1`, `diag_dm_pvalue_two_sided_h1`). Taken at face value, the AR model's h=1 loss is significantly larger than the naive loss. Treat this with caution. The forecasts differ by only hundredths of a point, the actuals are rounded to 0.1 pp, and there are only 35 errors, so the test statistic rests on very small loss differences.32- Diebold–Mariano, h=3: statistic 1.36, p = 0.18 (`diag_dm_stat_h3`, `diag_dm_pvalue_two_sided_h3`). There is no significant difference at h=3. The overlapping 3-month errors are handled through the HAC variance.33- The main finding is robust in direction (the model does not beat naive). The size of the differences (≤ 2.4% of RMSE) is small compared with sampling variability over a 3-year window.3435## Limitations and open issues36- Only one model specification was evaluated, and its specification was fixed in advance. Other AR variants might do better or worse; I did not explore them, to avoid choosing a model on the evaluation window.37- The estimation sample includes the 2020 COVID spike (+10.4 pp in one month). This probably pulls the OLS AR coefficients towards zero, which makes the model nearly a random walk. A later estimation start or outlier dummies might change the results. I did not try these, because choosing them after seeing the results would be data snooping.38- The evaluation uses the final-vintage data, so it says nothing about real-time performance.39- The 2025-10 observation is missing. Each horizon therefore has one unevaluated pair, and the forecasts at nearby origins depend on the gap-filling rule.40- The evaluation window is short (36 origins) and fairly calm (unemployment between 3.5% and 4.5% over 2023-07 to 2026-09, a figure checked directly on the data and not stored in results.json), so the results may not hold in other periods.41- The `diag_*` keys are extra diagnostics beyond the required schema.42