1# F3-dev-01 — Can an AR model beat the naive forecast of US unemployment? (Rolling origins 2023-07 to 2026-06)23## Question4Over the 36 monthly forecast origins 2023-07-01 … 2026-06-01, does a simple autoregressive model forecast the US unemployment rate (FRED UNRATE, SA, %) better than the naive no-change forecast at horizons h = 1 and h = 3 months? We compare out-of-sample RMSE and MAE and report 1 − RMSE_model/RMSE_naive.56## Data and exclusions7- `data/UNRATE.csv`: 945 data rows, 1948-01 to 2026-09. Dates are unique, in order and have no calendar gaps.8- **Missing value:** 2025-10-01 has an empty field (no October 2025 observation). How it was handled:9 - The 2025-10 target is not scored. Two (origin, horizon) pairs are affected: origin 2025-09 at h=1 and origin 2025-07 at h=3. They stay in `forecasts.csv` with an empty `actual`. This leaves **35 evaluated pairs per horizon**.10 - In estimation, any regression row whose dependent or lagged difference involves 2025-10 is dropped.11 - At origin 2025-10, the naive forecast uses 2025-09 (4.4). The AR model iterates forward from 2025-09, adding one extra step.12 - At later origins, 2025-10 is linearly interpolated from 2025-09 and 2025-11, both dated ≤ origin. This is used only to build lag inputs and is never used as an estimation observation.13- **COVID-19 outlier:** unemployment rose from 4.4 (2020-03) to 14.8 (2020-04). In the primary specification this outlier was kept, a choice made in advance. A sensitivity check is reported below.14- Every error is computed on the final vintage, so none of these results are valid in real time.1516## Method17- **Naive:** the forecast for all h is the latest observation dated ≤ origin.18- **Model `AR2_diff`:** an AR(2) with intercept on monthly first differences Δy_t.19 - Estimated by OLS on an expanding window that starts at the beginning of the series and ends at the origin. It is re-estimated at every origin using only data dated ≤ origin.20 - h-step forecasts come from iterating the difference equation and adding the steps to the last level.21 - The specification was fixed before looking at any results, with no model selection. The last training window (origin 2026-06) has 935 regression observations.22- **Uncertainty:** Diebold–Mariano test on squared-error loss differentials, with HAC variance (h−1 lags) and the Harvey–Leybourne–Newbold small-sample correction, compared against a t(n−1) distribution.23- **Reproducibility:** `out/analysis.py` writes `results.json`, `forecasts.csv` and `supplementary.json`. It is deterministic and uses no random numbers.2425## Results26| horizon | n | RMSE naive | RMSE AR2_diff | MAE naive | MAE AR2_diff | 1 − RMSE ratio |27|---|---|---|---|---|---|---|28| h=1 | 35 | 0.1171 | 0.1199 | 0.0914 | 0.0957 | −0.0235 |29| h=3 | 35 | 0.1586 | 0.1598 | 0.1200 | 0.1213 | −0.0076 |3031The AR model **does not beat** the naive forecast. It is slightly worse on both RMSE and MAE at both horizons: about 2.3% higher RMSE at h=1 and 0.8% at h=3. Its absolute error was smaller than the naive error at only 10 of 35 origins (28.6%) at each horizon.3233Why: the estimated AR coefficients are close to zero. At the last origin they are intercept 0.0003, lag 1 0.037 and lag 2 −0.042, and they are almost the same at the first origin. The 2020 spike dominates the full-sample OLS fit and pushes them towards zero. As a result the model is close to a random walk, and its forecasts differ from the naive one by only a few thousandths of a point. The data are recorded to 0.1 pp.3435## Uncertainty36- **h=1:** DM-HLN statistic = 2.50, two-sided p = 0.017. Positive means the model's loss is higher. The model's (small) inferiority is nominally significant at the 5% level.37- **h=3:** statistic = 1.36, p = 0.18, so there is no significant difference.38- Treat these tests with caution:39 - n = 35, and the two forecasts are nearly identical, so the loss differentials are tiny (mean 0.00065 pp² at h=1).40 - The DM test's assumptions are only approximately met.41 - The economic difference is negligible.42- **Sensitivity (not pre-specified; report only):** I dropped 2020-03…2020-12 differences from estimation. The AR coefficients then become substantial (lag 1 0.08, lag 2 0.28 at the last origin), but out-of-sample performance gets *worse*:43 - h=1: RMSE 0.1186, relative improvement −0.013.44 - h=3: RMSE 0.1756, relative improvement −0.108.45- The conclusion "no improvement over naive" therefore holds up under this alternative.4647## Limitations and open issues48- The evaluation covers only 36 origins in one fairly calm, slowly rising unemployment period. The results may not carry over to recessions.49- Final-vintage data were used, not real-time vintages. Seasonal-factor revisions may change the errors.50- Only one model specification was evaluated. The primary model's near-zero coefficients largely reflect the influence of the COVID outlier, so the comparison is effectively "random walk with tiny drift vs random walk".51- The missing 2025-10 value reduces each horizon to 35 evaluated pairs. Interpolating it as a lag input is a modelling choice.52- The HAC/DM inference is approximate in small samples.53