# F3-dev-01 — Can an AR model beat the naive forecast of US unemployment? (Rolling origins 2023-07 to 2026-06)

# Question

Over the 36 monthly forecast origins 2023-07-01 … 2026-06-01, does a simple autoregressive model forecast the US unemployment rate (FRED UNRATE, SA, %) better than the naive no-change forecast at horizons h = 1 and h = 3 months? We compare out-of-sample RMSE and MAE and report 1 − RMSE_model/RMSE_naive.

# Data and exclusions

• data/UNRATE.csv: 945 data rows, 1948-01 to 2026-09. Dates are unique, in order and have no calendar gaps.
• Missing value: 2025-10-01 has an empty field (no October 2025 observation). How it was handled:
  • The 2025-10 target is not scored. Two (origin, horizon) pairs are affected: origin 2025-09 at h=1 and origin 2025-07 at h=3. They stay in forecasts.csv with an empty actual. This leaves 35 evaluated pairs per horizon.
  • In estimation, any regression row whose dependent or lagged difference involves 2025-10 is dropped.
  • At origin 2025-10, the naive forecast uses 2025-09 (4.4). The AR model iterates forward from 2025-09, adding one extra step.
  • At later origins, 2025-10 is linearly interpolated from 2025-09 and 2025-11, both dated ≤ origin. This is used only to build lag inputs and is never used as an estimation observation.
• COVID-19 outlier: unemployment rose from 4.4 (2020-03) to 14.8 (2020-04). In the primary specification this outlier was kept, a choice made in advance. A sensitivity check is reported below.
• Every error is computed on the final vintage, so none of these results are valid in real time.

# Method

• Naive: the forecast for all h is the latest observation dated ≤ origin.
• Model AR2_diff: an AR(2) with intercept on monthly first differences Δy_t.
  • Estimated by OLS on an expanding window that starts at the beginning of the series and ends at the origin. It is re-estimated at every origin using only data dated ≤ origin.
  • h-step forecasts come from iterating the difference equation and adding the steps to the last level.
  • The specification was fixed before looking at any results, with no model selection. The last training window (origin 2026-06) has 935 regression observations.
• Uncertainty: Diebold–Mariano test on squared-error loss differentials, with HAC variance (h−1 lags) and the Harvey–Leybourne–Newbold small-sample correction, compared against a t(n−1) distribution.
• Reproducibility: out/analysis.py writes results.json, forecasts.csv and supplementary.json. It is deterministic and uses no random numbers.

# Results

The AR model does not beat the naive forecast. It is slightly worse on both RMSE and MAE at both horizons: about 2.3% higher RMSE at h=1 and 0.8% at h=3. Its absolute error was smaller than the naive error at only 10 of 35 origins (28.6%) at each horizon.

Why: the estimated AR coefficients are close to zero. At the last origin they are intercept 0.0003, lag 1 0.037 and lag 2 −0.042, and they are almost the same at the first origin. The 2020 spike dominates the full-sample OLS fit and pushes them towards zero. As a result the model is close to a random walk, and its forecasts differ from the naive one by only a few thousandths of a point. The data are recorded to 0.1 pp.

# Uncertainty

• h=1: DM-HLN statistic = 2.50, two-sided p = 0.017. Positive means the model's loss is higher. The model's (small) inferiority is nominally significant at the 5% level.
• h=3: statistic = 1.36, p = 0.18, so there is no significant difference.
• Treat these tests with caution:
  • n = 35, and the two forecasts are nearly identical, so the loss differentials are tiny (mean 0.00065 pp² at h=1).
  • The DM test's assumptions are only approximately met.
  • The economic difference is negligible.
• Sensitivity (not pre-specified; report only): I dropped 2020-03…2020-12 differences from estimation. The AR coefficients then become substantial (lag 1 0.08, lag 2 0.28 at the last origin), but out-of-sample performance gets worse:
  • h=1: RMSE 0.1186, relative improvement −0.013.
  • h=3: RMSE 0.1756, relative improvement −0.108.
• The conclusion "no improvement over naive" therefore holds up under this alternative.

# Limitations and open issues

• The evaluation covers only 36 origins in one fairly calm, slowly rising unemployment period. The results may not carry over to recessions.
• Final-vintage data were used, not real-time vintages. Seasonal-factor revisions may change the errors.
• Only one model specification was evaluated. The primary model's near-zero coefficients largely reflect the influence of the COVID outlier, so the comparison is effectively "random walk with tiny drift vs random walk".
• The missing 2025-10 value reduces each horizon to 35 evaluated pairs. Interpolating it as a lag input is a modelling choice.
• The HAC/DM inference is approximate in small samples.