# F3-dev-01 — Rolling-origin forecasts of the US unemployment rate: AR(2) on differences vs naive

# Question

Over the 36 monthly forecast origins from 2023-07-01 to 2026-06-01 (n_origins, first_origin, last_origin), does a simple autoregressive model forecast the US civilian unemployment rate (FRED UNRATE, %, SA) better than the naive no-change forecast at horizons of 1 and 3 months? Forecasts use expanding windows; accuracy is measured by out-of-sample RMSE and MAE and by the relative improvement 1 − RMSE_model/RMSE_naive.

# Data and exclusions

• Input: data/UNRATE.csv, 945 data rows, 1948-01 to 2026-09 (n_input_rows); SHA-256 matches the README. Audit details in out/state/data_audit.md.
• One missing value: 2025-10-01 (empty field) (data_issues). Handling: the naive forecast at origin 2025-10-01 uses the 2025-09-01 value; AR estimation drops regressions involving the gap (complete cases); for forecasting, the missing level is replaced by the model's own forecast from earlier data. Two (origin, horizon) pairs target 2025-10-01 (origin 2025-09-01 at h=1; origin 2025-07-01 at h=3). Both stay in out/forecasts.csv with an empty actual and are left out of the error statistics. As a result, 35 pairs are evaluated at each horizon (n_evaluated_h1, n_evaluated_h3).
• Observation-level exclusion ledger (out/exclusions.csv): 945 raw rows → −1 missing → −3 observations after the last origin, which are used only as targets → 941 observations in the last training window (n_analysed).
• The file is the final (revised) vintage, so this exercise is pseudo-out-of-sample and has no real-time validity.

# Method

• Naive: the forecast at every horizon is the latest non-missing observation dated ≤ origin.
• Model AR2_diff (model_name, model_description): Δy_t = c + φ1 Δy_{t−1} + φ2 Δy_{t−2} + e_t, estimated by OLS on all complete cases from 1948 up to the origin and re-estimated at each origin. Forecasts for h = 1..3 come from iterating the difference equation and cumulating onto the last level. I fixed the lag order (2) and the transformation (first differences) before looking at any out-of-sample result. No model selection was done on the evaluation window. Every estimation step, transformation and gap fill uses only data dated ≤ origin; a perturbation test confirmed there is no look-ahead (out/state/checks.md, check 11).
• The procedure is deterministic, with no random numbers (random_seed = null). out/analysis.py regenerates every output.
• Uncertainty: a Diebold–Mariano test on squared-error loss differences, using a HAC variance with h−1 lags, the Harvey–Leybourne–Newbold small-sample correction, and a t(n−1) reference (diag_dm_stat_h*, diag_dm_pvalue_two_sided_h*).

# Results

Keys: rmse_naive_h1, rmse_model_h1, mae_naive_h1, mae_model_h1, rel_improvement_rmse_h1, and likewise for _h3. Values are in percentage points, rounded from results.json.

• At h=1 the AR model is about 2.4% worse than the naive forecast in RMSE (rel_improvement_rmse_h1 = −0.024). At h=3 it is about 0.8% worse (rel_improvement_rmse_h3 = −0.008). MAE points the same way at both horizons.
• The answer is no: over this window, the simple AR model did not beat the naive forecast at either horizon.
• The two forecasts are almost the same. The estimated AR coefficients at the last origin are small (diag_ar_coef_last_origin ≈ [0.0003, 0.037, −0.042]), so the model behaves almost like a random walk and its forecasts stay within 0.1 pp of the naive forecasts.

# Uncertainty

• Diebold–Mariano, h=1: statistic 2.53, two-sided p = 0.016 (diag_dm_stat_h1, diag_dm_pvalue_two_sided_h1). Taken at face value, the AR model's h=1 loss is significantly larger than the naive loss. Treat this with caution. The forecasts differ by only hundredths of a point, the actuals are rounded to 0.1 pp, and there are only 35 errors, so the test statistic rests on very small loss differences.
• Diebold–Mariano, h=3: statistic 1.36, p = 0.18 (diag_dm_stat_h3, diag_dm_pvalue_two_sided_h3). There is no significant difference at h=3. The overlapping 3-month errors are handled through the HAC variance.
• The main finding is robust in direction (the model does not beat naive). The size of the differences (≤ 2.4% of RMSE) is small compared with sampling variability over a 3-year window.

# Limitations and open issues

• Only one model specification was evaluated, and its specification was fixed in advance. Other AR variants might do better or worse; I did not explore them, to avoid choosing a model on the evaluation window.
• The estimation sample includes the 2020 COVID spike (+10.4 pp in one month). This probably pulls the OLS AR coefficients towards zero, which makes the model nearly a random walk. A later estimation start or outlier dummies might change the results. I did not try these, because choosing them after seeing the results would be data snooping.
• The evaluation uses the final-vintage data, so it says nothing about real-time performance.
• The 2025-10 observation is missing. Each horizon therefore has one unevaluated pair, and the forecasts at nearby origins depend on the gap-filling rule.
• The evaluation window is short (36 origins) and fairly calm (unemployment between 3.5% and 4.5% over 2023-07 to 2026-09, a figure checked directly on the data and not stored in results.json), so the results may not hold in other periods.
• The diag_* keys are extra diagnostics beyond the required schema.