# F3-dev-01 — Rolling-origin forecasts of the US unemployment rate

# Question

Over the 36 expanding-window forecast origins 2023-07-01 … 2026-06-01 (n_origins, first_origin, last_origin), does a simple autoregressive-type model forecast the US civilian unemployment rate (FRED UNRATE, %, SA, final vintage) more accurately than the naive no-change forecast at horizons of 1 and 3 months? Accuracy is measured by out-of-sample RMSE and MAE, and the relative improvement is 1 − RMSE_model/RMSE_naive.

# Data and exclusions

• data/UNRATE.csv has 945 data rows (n_input_rows), covering 1948-01 to 2026-09. Its SHA-256 matches the README. There are no duplicate dates and no calendar gaps. Full audit: out/state/data_audit.md.
• One value is missing: 2025-10-01 (data_issues). It was not imputed. Observation ledger (out/exclusions.csv): 945 rows → drop 1 missing → 944 → keep only observations dated ≤ the last origin → 941 observations in the last training window (n_analysed).
• Forecast pairs: the ledger out/forecasts.csv holds all 144 rows. The missing month is the target of two pairs: (origin 2025-09-01, h=1) and (origin 2025-07-01, h=3). Both stay in the ledger with an empty actual and are left out of the error statistics. That gives 35 evaluated pairs per horizon (n_evaluated_h1, n_evaluated_h3).
• At origin 2025-10-01 (whose own value is missing), the naive forecast uses the 2025-09-01 value. The ARIMA model forecasts from its Kalman-filter state given the data through that origin.

# Method

• Naive baseline: the latest non-missing observation dated ≤ t, used for every horizon.
• Model (model_name = ARIMA210): see model_description. It is an ARIMA(2,1,0) without a constant, i.e. an AR(2) on monthly changes. It is fitted by Gaussian maximum likelihood (statsmodels SARIMAX, Kalman filter, with missing months treated as missing). The sample starts 1948-01-01 and ends at each origin t, and the model is re-fitted at every origin. 3-step forecasts are built recursively.
• I fixed this specification before computing any evaluation error (out/state/log.md). No other model was tried on the evaluation window. All estimation uses only observations dated ≤ t. No randomness is involved (random_seed = null).
• Uncertainty: a Diebold–Mariano test of equal mean squared error (keys dm_stat_h*, dm_pvalue_h*). It uses a Newey–West variance with h−1 lags, the Harvey–Leybourne–Newbold small-sample correction and a t(n−1) reference. A positive statistic means the model's squared errors are larger than the naive ones.
• Replication: ~/wp29/venv/bin/python out/analysis.py.

# Results

(Units: percentage points.) The ARIMA(2,1,0) model did not beat the naive forecast at either horizon:

• At h=1 its RMSE is about 2.3% higher than naive (rel_improvement_rmse_h1).
• At h=3 its RMSE is about 0.5% higher (rel_improvement_rmse_h3).
• Its MAE is also higher at both horizons (mae_model_h1, mae_model_h3).

The estimated AR coefficients on monthly changes are close to zero. In the last window they are 0.037 and −0.042 (ar1_coef_last_window, ar2_coef_last_window). So the model's forecasts barely differ from the naive ones: by less than 0.2 pp in every case (out/state/checks.md).

# Uncertainty

• h=1: the Diebold–Mariano statistic is 2.46 with p = 0.019 (dm_stat_h1, dm_pvalue_h1). Taken at face value, the model's loss is significantly higher than naive at the 5% level.
• h=3: the statistic is 1.09 with p = 0.28 (dm_stat_h3, dm_pvalue_h3), so there is no significant difference.
• Caveats on these tests:
  • The model nests the random walk, and its forecasts are almost the same as naive. The standard DM test is not designed for nested models (Clark–West would be the usual alternative, and it was not computed). So these p-values are approximate.
  • With 35 evaluation points, and overlapping errors at h=3, all accuracy estimates are imprecise.
  • The absolute differences in RMSE are below 0.003 pp.

# Limitations and open issues

• Final vintage: the evaluation uses revised data, so it is not a real-time assessment.
• One specification only: a single model was fixed in advance. The result does not show that no AR-type model can beat naive. The 2020 COVID spike (14.8% in 2020-04) is kept in every estimation window. It probably pulls the AR coefficients toward zero, which makes the model nearly a random walk. Alternatives such as a 2020 dummy, a later sample start or robust estimation were deliberately not explored, because choosing among them on this evaluation window would be selection on test data.
• Missing month: the 2025-10 observation is missing. It removes one evaluated pair per horizon, and the training windows after 2025-10 contain a gap, which the Kalman filter handles.
• Short window: 36 origins in a relatively calm period (the rate stayed between about 3.5% and 4.6%). The results may not carry over to recessions.