WP46: evaluation + inference modules, pilot script, protocol docs, decisions log, data dictionary, job lists
4 changed files +20,001 −0
added
docs/data_dictionary.md
+36 −0
@@ -0,0 +1,36 @@ | ||
| 1 | +# Data dictionary (WP46) | |
| 2 | + | |
| 3 | +## Daily panel (`<cache>/panels/<symbol>.parquet`, one row per session; built by `scripts/build_panels.py`) | |
| 4 | +| column | unit | definition | | |
| 5 | +|---|---|---| | |
| 6 | +| session | date | session identifier (ET date for equities/futures/FX, UTC date for crypto; see D-02) | | |
| 7 | +| close, open | price | session close (last 1-minute bar close in the window) and open; equities: vendor split+dividend-adjusted daily bars | | |
| 8 | +| r | percent | 100·log(close_t / close_{t−1}) — the modelled daily return | | |
| 9 | +| r_on | percent | 100·log(open_t / close_{t−1}) (overnight; 0 for FX/crypto, break return for futures) | | |
| 10 | +| r_oc | percent | 100·log(close_t / open_t) | | |
| 11 | +| rv_1m | percent² | Σ squared 1-minute log returns within the window | | |
| 12 | +| rv_5m | percent² | Σ squared 5-minute calendar returns (grid anchored at the window start, previous tick) | | |
| 13 | +| rv_5m_ss | percent² | average of the five 5-minute RV estimates with 1-minute grid offsets (primary intraday measure) | | |
| 14 | +| bv_5m | percent² | (π/2)·Σ|r_j||r_{j−1}| on the 5-minute grid | | |
| 15 | +| rsv_pos_5m, rsv_neg_5m | percent² | realized semivariances on the 5-minute grid | | |
| 16 | +| medrv_5m | percent² | median realized variance on the 5-minute grid | | |
| 17 | +| n_bars, n_5m | count | 1-minute bars used; 5-minute returns | | |
| 18 | +| expected_bars | count | full-session bar count (391/211 equities; modal count otherwise) | | |
| 19 | +| valid_rm | bool | realized measures usable (D-05, D-06) | | |
| 20 | + | |
| 21 | +## Forecast table (`results/forecasts/<run>/<symbol>/<spec_id>.parquet`, one row per origin) | |
| 22 | +origin (date of the information set), fit_id, status ∈ {ok, stale_fit, failed, forecast_failed, no_fit, missing_input}, | |
| 23 | +method ∈ {analytic, simulation, direct}, h1/h5/h22 (percent², cumulative variance forecasts of the 1/5/22-day returns), | |
| 24 | +sigma2_t, mu (one-step mean, percent), dp1/dp2 (distribution parameters), time_fc_s, admissible, plus the keys symbol, | |
| 25 | +model_id, dist, spec_id, tier, window, refit_every, expanding. | |
| 26 | + | |
| 27 | +## Fits table (`<spec_id>.fits.parquet`, one row per refit attempt) | |
| 28 | +fit_id, origin, status ∈ {ok, inadmissible, failed}, error, loglik, n_iter, n_restarts, time_s, n_obs, admissible, params, param_names. | |
| 29 | + | |
| 30 | +## Loss table (`results/metrics/losses/<symbol>.parquet`) | |
| 31 | +origin, spec_id, model_id, dist, tier, h, proxy, loss, value, status, forecast, target. | |
| 32 | + | |
| 33 | +## Keys | |
| 34 | +`spec_id = <model_id>__<dist>__w<window>_r<refit>` (or `exp` for expanding). Run manifests (`_runs.jsonl`) carry git SHA, host, | |
| 35 | +package versions, wall/CPU time and peak memory per job; dataset versions and SHA-256 of every exported file are in | |
| 36 | +`results/manifests/ingest_manifest.json`. | |
added
docs/methodology_decisions.md
+21 −0
@@ -0,0 +1,21 @@ | ||
| 1 | +# Methodology decisions log (WP46) | |
| 2 | + | |
| 3 | +Each decision is dated, motivated, and tied to evidence. "Before freeze" = taken before the pre-registration tag. | |
| 4 | + | |
| 5 | +| ID | Date | Decision | Motivation / evidence | | |
| 6 | +|---|---|---|---| | |
| 7 | +| D-01 | 2026-11-09 | Returns in **percent** log units; variances in percent². | Numerical conditioning of the GARCH likelihoods (arch rescales otherwise); explicit units everywhere. | | |
| 8 | +| D-02 | 2026-11-09 | Session windows: equities 09:30–16:00 ET (XNYS calendar, early closes), futures CME trading day 18:00→17:00 ET, FX 17:00→17:00 ET, crypto UTC day. | `docs/findata_audit.md` §4: vendor `session_date` for futures/FX is an ET calendar day, not the trading day; the CME/NY convention aligns the daily return and the realized measure on the same window. | | |
| 9 | +| D-03 | 2026-11-09 | Realized measures: 1-min RV, 5-min calendar RV, subsampled 5-min RV (primary), bipower, semivariances, medRV; previous-tick sampling. | Liu, Patton & Sheppard (2015): 5-min RV is hard to beat; subsampling reduces grid dependence. | | |
| 10 | +| D-04 | 2026-11-09 | Primary proxy = subsampled 5-min RV + squared overnight return (full-day variance); secondary: r², RV1, RV5, BV, medRV, Hansen–Lunde scaled RV. | The models forecast the close-to-close variance; audit shows mean(RV5ss + r_on²)/mean(r²) ∈ [0.93, 1.15] for equities/futures/FX. | | |
| 11 | +| D-05 | 2026-11-09 | Session validity: ≥ 60 % of the instrument's **modal** 1-minute bar count (equities: 391/211 per calendar). | Grains (ZC) trade ~730 min/day, NG ~1,100, FX 1,440; a fixed 1,380 threshold invalidated 74 % of ZC days. | | |
| 12 | +| D-06 | 2026-11-09 | Crypto realized measures are invalid before 2020-01-01 (daily returns kept). | Audit: BTC 1-min RV is 10–100× the squared daily return in 2013–2019 (12,810 one-minute moves > 5 % in 2017 alone: cross-venue bounce in the vendor aggregate); from 2020 the ratio is ≈ 1.2–1.5. | | |
| 13 | +| D-07 | 2026-11-09 | Tier B input = primary proxy of day t−1 (observed at the close of t−1); invalid days imputed by r² inside the recursion only (flagged). | Keeps the recursion defined; the imputation never enters the evaluation target. | | |
| 14 | +| D-08 | 2026-11-09 | Rolling window 1,000 days, refit every 5 days, daily filtering with fixed parameters; expanding and 2,000-day windows as robustness. | Compute budget (≈ 90 specs × 42 instruments × 3,700 origins); weekly re-estimation changes parameters negligibly relative to daily. | | |
| 15 | +| D-09 | 2026-11-09 | Multi-step forecasts: exact analytic recursions (GARCH, GJR with κ = E[z²1{z<0}], NAGARCH, QGARCH, CGARCH, IGARCH, GAS, HEAVY, GARCH-X, MS-GARCH Haas recursion, FIGARCH via arch); simulation with 2,000 paths and a fixed seed for EGARCH, APARCH, TARCH, AVGARCH, Realized GARCH; HAR direct projections per horizon. | The brief forbids naive extrapolation; each model card states the recursion. | | |
| 16 | +| D-10 | 2026-11-09 | Failure policy: bounded retries (3 starting values + Nelder–Mead polish for custom models; 2 restarts for arch), then keep the last successful parameters (flag `stale_fit`), else NaN. No replacement by another model. | Brief §"Computation, failures and fairness". | | |
| 17 | +| D-11 | 2026-11-09 | Unsupported architectures (FIEGARCH, FIAPARCH, HYGARCH, smooth-transition GARCH, GARCH-MIDAS, SV, Realized EGARCH) are listed in the registry as `unsupported` and not scored. | No vetted implementation in the version-locked libraries (arch 8.0.0, rugarch 1.5-6) or outside the pre-registered information set; a custom implementation without an authoritative cross-check would violate the verification rule. | | |
| 18 | +| D-12 | 2026-11-09 | Innovation distributions: normal, Student-t, GED, Hansen skew-t for arch and custom GARCH-type models; normal/t only for MS-GARCH, Realized GARCH, HEAVY, GARCH-X; t only for GAS-t. | Formal validity and implementation availability; recorded as `unsupported` combinations otherwise. | | |
| 19 | +| D-13 | 2026-11-09 | Validation period 2012-01-03 → 2015-12-31 (design checks, pilot, failure accounting); test period 2016-01-04 → 2026-10-30, untouched until the protocol freeze. | FX starts in 2010 (+1,000-day window → 2014), crypto later: validation checks use the instruments available. | | |
| 20 | +| D-14 | 2026-11-09 | Cross-check policy: every arch-backed and custom model is compared with rugarch 1.5-6 on a shared fixture (`results/metrics/rugarch_crosscheck.json`); FIGARCH differs by 1.47 log-likelihood units (different truncation/constraints between arch and rugarch) and is reported as an implementation caveat. | Verified 2026-11-09. | | |
| 21 | +| D-15 | 2026-11-09 | Smoke tests of the forecasting machinery were run on SPY 2024 forecasts (values only, no losses) before the freeze; the pilot uses the validation period only. | Disclosure of what was seen before the pre-registration. | | |
added
docs/preregistered_protocol.md
+95 −0
@@ -0,0 +1,95 @@ | ||
| 1 | +# Pre-registered protocol — WP46 "Four Decades of Conditional Volatility Modeling" | |
| 2 | + | |
| 3 | +**Status: FROZEN** at git tag `prereg-frozen` (date in `progress.md`). Written after the data audit (`docs/findata_audit.md`), | |
| 4 | +the literature matrix (`docs/literature_matrix.md`) and the validation-period pilot (`docs/feasibility_report.md`), and | |
| 5 | +BEFORE any loss, ranking or test statistic computed on the test period. Deviations after the freeze are amendments | |
| 6 | +(A-xx) logged in `progress.md` with their motivation; exploratory analyses are labelled as such in the paper. | |
| 7 | + | |
| 8 | +## 1. Research question and hypotheses | |
| 9 | +Q. Across asset classes, horizons, regimes and information sets, when — if ever — do more complex conditional | |
| 10 | +volatility models reliably outperform GARCH(1,1), and is any gain worth the numerical and computational cost? | |
| 11 | + | |
| 12 | +Confirmatory hypotheses (each tested with the primary loss, primary proxy, test period, common-success sample): | |
| 13 | +- **H1 (asymmetry).** Asymmetric models (GJR, EGARCH, TGARCH, APARCH, NAGARCH, QGARCH) with the same innovation | |
| 14 | + distribution have lower QLIKE than GARCH(1,1) at h = 1 for equity instruments (ETFs and stocks). Prediction: pooled | |
| 15 | + loss ratio < 1 with a 90 % block-bootstrap CI excluding 1; MCS membership of GARCH(1,1) lost for a majority of equity instruments. | |
| 16 | +- **H2 (distribution).** Within an architecture, non-Gaussian innovations do not change QLIKE-based variance-forecast rankings | |
| 17 | + materially (|pooled ratio − 1| < 0.02 at all horizons) but improve the 1 % VaR coverage tests (fewer rejections). | |
| 18 | +- **H3 (realized information).** Tier B models with the lagged realized measure (HAR, Realized GARCH, HEAVY, GARCH-X) | |
| 19 | + beat the best Tier A model at h = 1 (pooled ratio < 0.9) and the advantage shrinks with the horizon (ratio at h = 22 closer to 1). | |
| 20 | +- **H4 (persistence structure).** Long-memory/component/regime models (FIGARCH, CGARCH, MS-GARCH) improve over GARCH(1,1) | |
| 21 | + only at h = 22, if at all (no improvement at h = 1). | |
| 22 | +- **H5 (cost).** Complex models have materially higher failure rates (non-convergence or inadmissibility) and compute | |
| 23 | + time; on the failure-adjusted ("operational") sample their advantage, if any, is reduced relative to the common-success sample. | |
| 24 | +- **H6 (regimes).** Rankings differ between tranquil and stressed regimes: asymmetric and realized-measure models gain | |
| 25 | + most in the high-trailing-volatility tercile; GARCH(1,1) is rarely excluded from the MCS in the low tercile. | |
| 26 | +Direction and magnitude thresholds above are fixed now; "materially" always refers to the thresholds stated. | |
| 27 | + | |
| 28 | +## 2. Universe (frozen) | |
| 29 | +42 instruments from FinData OS (FirstRate Data), all alive on 2026-11-06 (selection bias toward liquid survivors is | |
| 30 | +disclosed): ETFs SPY, QQQ, IWM, DIA, EEM, EFA, XLF, XLE, TLT, IEF, HYG, LQD, GLD, SLV, USO; stocks AAPL, MSFT, AMZN, | |
| 31 | +NVDA, JPM, XOM, JNJ, PG, KO, WMT; futures (continuous, ratio back-adjusted, vendor rolls) ES, NQ, CL, NG, GC, SI, HG, | |
| 32 | +ZN, ZC; FX EURUSD, USDJPY, GBPUSD, AUDUSD, USDCAD, USDCHF; crypto BTC, ETH. Sample 2000-01-03 → 2026-10-30 (instrument | |
| 33 | +start dates in `docs/findata_audit.md`). Regime variables: VIX (daily, index dataset). No instrument is added or | |
| 34 | +removed after the freeze; an instrument whose forecasts cannot be produced for a model is a recorded failure. | |
| 35 | + | |
| 36 | +## 3. Data conventions (frozen; details in `docs/data_dictionary.md`, decisions D-01…D-07) | |
| 37 | +Daily log returns in percent; session windows per asset class (D-02); realized measures on the session window (D-03); | |
| 38 | +primary proxy = subsampled 5-min RV + squared overnight return (D-04); validity rules D-05/D-06; crypto realized | |
| 39 | +measures from 2020 only (D-06). Tier B input = primary proxy lagged one day (D-07). | |
| 40 | + | |
| 41 | +## 4. Information tiers and model set (frozen; `configs/model_registry.yaml`) | |
| 42 | +- Tier A (daily returns only): baselines (constant, historical 22/66/252, EWMA 0.94, RiskMetrics 2006); ARCH(5); | |
| 43 | + GARCH(1,1) **benchmark**, GARCH(2,1), GARCH(1,2); IGARCH; GARCH-M; GJR; TGARCH (Zakoïan); EGARCH; NAGARCH; QGARCH; | |
| 44 | + AVGARCH; APARCH; FIGARCH(1,d,1); CGARCH; MS-GARCH(2); GAS-t (non-GARCH comparator). Innovations: normal, Student-t, | |
| 45 | + GED, Hansen skew-t where supported (registry). | |
| 46 | +- Tier B (plus lagged realized measure): realized random walk, HAR-RV, log-HAR-RV, Realized GARCH, HEAVY, GARCH-X(RV). | |
| 47 | +- Tier C (multivariate, `configs/study.yaml`): rolling covariance, EWMA covariance, CCC-GARCH, DCC, asymmetric DCC, | |
| 48 | + scalar BEKK on two ETF baskets, h = 1, scored against the 5-minute realized covariance + overnight outer product with | |
| 49 | + the Frobenius loss and the multivariate QLIKE; portfolio-variance diagnostics for the equally weighted portfolio. | |
| 50 | +Order grid: (1,1) everywhere, plus GARCH(2,1)/(1,2) and ARCH(5) as pre-specified order variants. Nothing else is tuned. | |
| 51 | + | |
| 52 | +## 5. Forecasting design (frozen; `configs/forecast_protocol.yaml`, D-08…D-10) | |
| 53 | +Rolling window 1,000 days, refit every 5 trading days, daily filtering; horizons 1, 5, 22 (cumulative variance); | |
| 54 | +multi-step rules per model (D-09); failure policy D-10; seed 46; 2,000 simulation paths. Robustness (exploratory): | |
| 55 | +expanding window and window 2,000 for a subset of models; refit every 22 days. | |
| 56 | + | |
| 57 | +## 6. Periods (frozen; D-13) | |
| 58 | +Validation 2012-01-03 → 2015-12-31 (design checks only). **Test 2016-01-04 → 2026-10-30** (all confirmatory results). | |
| 59 | +Crypto and FX instruments enter the test period when their window allows (BTC ≥ 2017, ETH ≥ 2020 for daily; realized | |
| 60 | +measures from 2020). | |
| 61 | + | |
| 62 | +## 7. Losses, proxies, samples | |
| 63 | +Primary: QLIKE (zero-safe form) on the primary proxy. Secondary: MSE, MAE, MSE on the standard-deviation scale | |
| 64 | +(sensitivity only), log predictive score at h = 1 for models with a density, Mincer–Zarnowitz R². Proxy sensitivity: | |
| 65 | +rv5_on, rv1_on, bv5_on, medrv_on, r², Hansen–Lunde scaled RV. | |
| 66 | +Samples: (i) **common-success sample** per instrument×horizon = origins at which every compared specification has a valid | |
| 67 | +forecast and the target is valid; (ii) **operational sample** = each model's own valid origins, with failures counted | |
| 68 | +(coverage rate reported next to the loss); (iii) full-information reporting of failure rates. Cross-tier comparisons | |
| 69 | +(A vs B) are always labelled with the information-set difference. | |
| 70 | + | |
| 71 | +## 8. Inference (frozen) | |
| 72 | +Diebold–Mariano tests vs GARCH(1,1)-normal with Newey–West HAC (bandwidth h−1+⌊4(n/100)^{2/9}⌋, HLN correction); | |
| 73 | +MCS (α = 0.10, Tmax statistic, stationary bootstrap, expected block 22 days, 2,000 replications) per instrument×horizon | |
| 74 | +within each tier; SPA (Hansen 2005) with GARCH(1,1) as the benchmark per instrument×horizon; pooled cross-asset loss | |
| 75 | +ratios with a stationary block bootstrap over dates that resamples all instruments jointly (90 % CIs); Holm adjustment | |
| 76 | +across the 42 instruments for each (model, horizon) family when counting significant wins. Effect sizes = loss ratios. | |
| 77 | +Economic materiality threshold: a QLIKE ratio within ±2 % of 1 is reported as "no material difference" regardless of significance. | |
| 78 | + | |
| 79 | +## 9. Tail risk | |
| 80 | +VaR/ES at 1 % and 5 % for h = 1 from each model's own innovation distribution (empirical quantiles of standardized | |
| 81 | +in-window residuals for distribution-free models): Kupiec unconditional coverage, Christoffersen conditional coverage, | |
| 82 | +Engle–Manganelli dynamic quantile (4 lags), Fissler–Ziegel FZ0 joint VaR–ES loss with DM comparisons vs GARCH(1,1)-t. | |
| 83 | + | |
| 84 | +## 10. Regimes (frozen, defined without look-ahead) | |
| 85 | +Trailing 22-day realized variance terciles at the origin (thresholds from the validation period per instrument); | |
| 86 | +calendar episodes fixed now: COVID 2020-02-20→2020-06-30, 2022 rate shock (calendar year 2022), tariff shock | |
| 87 | +2025-04-02→2025-05-30; sign of the previous day's return; VIX terciles (equities). Sensitivity: 66-day trailing terciles. | |
| 88 | + | |
| 89 | +## 11. Computation and failures | |
| 90 | +Per job: wall time, CPU time, peak RSS, fit time per refit, iterations, restarts, convergence/admissibility flags, | |
| 91 | +forecast coverage. Pareto frontiers: accuracy (pooled QLIKE ratio) vs median fit time and vs failure rate. | |
| 92 | + | |
| 93 | +## 12. What is exploratory | |
| 94 | +Everything not listed in §1–§11 (e.g. per-asset narratives, alternative regime definitions beyond the sensitivity list, | |
| 95 | +window-length comparisons, tier-C multi-step forecasts) is reported as exploratory. | |
added
results/manifests/jobs_main.json
+19849 −0
Large diff (19,849 lines) is not loaded by default to keep the page fast.Load diff
Diff truncated — file too large.