SPB Git

spb/wp29_uqo

Public
2.4 KB

# Task card — F3-dev-01

  • Family / split: F3 Forecasting and temporal information / development
  • Question: rolling-origin (expanding window) forecasts of the US unemployment rate (FRED UNRATE) at h = 1 and h = 3 over 36 origins (2023-07 … 2026-06); naive no-change baseline vs one model of the executor's choice; estimands = out-of-sample RMSE/MAE per horizon and relative improvement.
  • Data type: observed (real public data, final vintage), 945 monthly rows 1948-01 … 2026-09; one genuine missing month (2025-10).
  • Scope / prerequisites: expanding-window evaluation, exact naive baseline, forecast ledger with chronology, handling of a missing observation under a stated rule.
  • Allowed information: at origin t, observations dated ≤ t only (estimation, selection and transformations included).
  • Expected artifacts: out/forecasts.csv (144 rows: origin_date, target_date, horizon, forecast, actual, model), results.json (24 keys), report.md (6 sections), analysis.py, claims.json.
  • Rubric summary: required criteria — naive RMSE/MAE at both horizons exact (rel 1e-6); counts (36 origins, 35 evaluated pairs per horizon); forecasts.csv chronology (targets = origin + h, naive rows = last observation ≤ origin, actuals = data); model RMSE/MAE consistent with the ledger; relative improvement consistent; the missing month reported in data_issues. Critical checks — leakage (model RMSE below the information-feasible bound; naive rows using post-origin values; perfect-foresight rows), fabrication (row count, actuals, n_input_rows), false_completion (feasibility). Baseline = rmse_naive_h1.
  • Oracle provenance: independent_code (pure-Python naive baseline and chronology table in ~/wp29-oracles/F3-dev-01/build.py); designer = WP29 F3 authoring agent; no human check.
  • Difficulty rationale: easy-moderate; difficulty is care (chronology, missing month, consistency between ledger and summary), not volume.
  • Budget class: benchmark (4.00 USD cap, 25 min wall, 80 turns per session).
  • Anticipated failure modes: dropping the origin whose month is missing instead of using the last available value; silently imputing 2025-10; computing errors on all 36 pairs with a NaN; wrong horizon alignment; reporting the model's in-sample fit; omitting the ledger.
  • Contamination risk: medium (famous series, textbook comparison) — but reference values depend on the exact 2026 window and the stated rules.
Files · 32c4ccff60dd76c81da4f5b63d24bd0b55824e43
Menu

Theme