SPB Git

spb/wp29_uqo

Public
2.6 KB · 15 lines markdown
1# Task card — F3-conf-0123- **Family / split**: F3 Forecasting and temporal information / confirmatory4- **Question**: rolling-origin forecasts (h = 1, 3; 48 month-end origins 2022-05 … 2026-04) of Canadian monthly real GDP year-over-year growth using only information published at each origin: GDP is published with a 2-month lag, LFS employment with a ~1-week lag. Naive (last published GDP growth) vs a model of the executor's choice; estimands = RMSE/MAE and relative improvement.5- **Data type**: observed (Statistics Canada WDS, current vintage), one CSV with GDP and employment levels and y/y growth, 345 monthly rows 1998-01 … 2026-09.6- **Scope / prerequisites**: expanding-window evaluation with an explicit information set (ragged edge), exact naive baseline, forecast ledger with declared information dates.7- **Allowed information**: at origin t: GDP columns for months ≤ t − 2, employment columns for months ≤ t − 1 (estimation, selection, transformations, plugged-in values and training targets alike).8- **Expected artifacts**: out/forecasts.csv (192 rows; columns incl. last_target_date_used, last_predictor_date_used), results.json (26 keys incl. last_target_date_available / last_predictor_date_available), report.md, analysis.py, claims.json.9- **Rubric summary**: required criteria — exact naive RMSE/MAE at h = 1, 3; n_origins; ledger chronology (targets, naive values, actuals, information dates within the rules); model errors consistent with the ledger; relative improvement consistent; availability dates at the last origin correct. Critical checks — leakage (declared dates beyond the limits; naive using the origin month; model RMSE at h = 1 below a bound calibrated on lag-respecting reference models), fabrication (row count, actuals, n_input_rows), false_completion. Baseline = rmse_naive_h1.10- **Oracle provenance**: independent_code (`~/wp29-oracles/F3-conf-01/build.py`, pure Python/numpy); designer = WP29 F3 authoring agent; no human check.11- **Difficulty rationale**: moderate; the default pandas alignment on observation dates silently violates the rule and halves the naive RMSE — the task tests whether the information set is implemented, not modelling skill.12- **Budget class**: benchmark (4.00 USD cap, 25 min wall, 80 turns per session).13- **Anticipated failure modes**: using gdp_yoy(t) as the naive; training on targets not yet published at the origin; plugging emp_yoy(t); misreporting the information dates; treating the empty 2026-08/09 GDP cells as a data error to impute.14- **Contamination risk**: low (series transformation, 2026 window and origin set are specific to this task).15
Files · 2c82656f6a48d002fcaa9c62ccf1d2b88a2e285b
Menu

Theme