PHASE 2 — Baselines and immutable splits (2026-11-02)
Completed
- Chronological session splits with warm-up (21 sessions) and a one-session embargo at the start of every evaluation
block (
pipeline.session_split,build_indices); train origins =origins_per_sessionrandom eligible slots, evaluation origins = fixed stride. Main protocol frozen inconfigs/main.yaml/configs/final.yaml: train ≤ 2022-12-31, validation 2023, development test 2024, untouched final 2025; 60 held-out instruments. - Single causal pass producing targets and all baseline inputs per origin (
baselines/features.py, parallel). - Baselines (
configs/baselines.yaml): random walk (trailing RMS), seasonal volatility, EWMA-t, HAR-RV (minute scale), GARCH(1,1)-t on 5-minute returns (analytic term structure, own filter), ridge, elastic net, LightGBM (two-part location / log-variance models with a Student-t assembled from residual kurtosis); MLP and TCN with the Student-t head. - Scoring (
eval/scores.py,pipeline.score_horizon/summarise/compare): CRPS (sample + closed form), WIS, quantile loss, PIT/KS, coverage, sharpness, QLIKE, tail Brier, directional accuracy, day-cluster bootstrap, HAC DM, Holm/BH. - Known constraint: LightGBM and torch cannot share a process on macOS (libomp) → baselines and neural stages are
separate subprocesses (
scripts/run_experiment.py --stage).
Pilot evidence (20 instruments, 2022-09-01 → 2022-12-30; test = 2022-12-16 → 12-30, 4 659 origins, 10 sessions)
Scaled CRPS of the cumulative return (95 % coverage in parentheses) and relative CRPS change vs the random walk with HAC Diebold–Mariano t (H = 5 / 15 / 30 / 60):
| Model | H=5 | H=15 | H=30 | H=60 | Δ vs random walk |
|---|---|---|---|---|---|
| naive_rw | 0.4803 (0.96) | 0.4874 (0.96) | 0.4952 (0.96) | 0.4819 (0.97) | |
| seasonal_vol | 0.4746 (0.94) | 0.4793 (0.94) | 0.4870 (0.93) | 0.4767 (0.93) | -1.98% (t=-5.7) / -2.51% (t=-6.6) / -2.69% (t=-5.8) / -2.12% (t=-3.4) |
| ewma | 0.4699 (0.97) | 0.4775 (0.96) | 0.4868 (0.96) | 0.4773 (0.96) | -2.83% (t=-8.1) / -2.79% (t=-6.7) / -2.62% (t=-5.2) / -1.77% (t=-2.3) |
| har_rv | 0.4833 (0.99) | 0.4761 (0.96) | 0.4845 (0.95) | 0.4728 (0.95) | -0.24% (t=-0.7) / -3.20% (t=-8.1) / -3.25% (t=-7.0) / -3.04% (t=-4.6) |
| garch_t | 0.4714 (0.97) | 0.4783 (0.98) | 0.4876 (0.98) | 0.4789 (0.98) | -2.58% (t=-8.9) / -2.63% (t=-8.0) / -2.45% (t=-7.2) / -1.60% (t=-3.7) |
| ridge | 0.4734 (0.91) | 0.4765 (0.94) | 0.4842 (0.94) | 0.4723 (0.94) | -2.06% (t=-3.7) / -3.07% (t=-5.7) / -3.29% (t=-5.5) / -2.96% (t=-4.1) |
| elastic_net | 0.4732 (0.91) | 0.4764 (0.94) | 0.4841 (0.94) | 0.4725 (0.94) | -2.14% (t=-3.8) / -3.10% (t=-5.8) / -3.31% (t=-5.6) / -2.91% (t=-3.9) |
| lightgbm | 0.5090 (0.89) | 0.5034 (0.92) | 0.5116 (0.92) | 0.5019 (0.92) | +3.20% (t=3.7) / +1.62% (t=2.0) / +1.59% (t=1.5) / +2.49% (t=1.8) |
| mlp | 0.4689 (0.95) | 0.4769 (0.95) | 0.4847 (0.94) | 0.4743 (0.94) | -3.08% (t=-9.1) / -2.96% (t=-8.3) / -3.01% (t=-7.2) / -2.47% (t=-4.2) |
| tcn | 0.4687 (0.96) | 0.4767 (0.96) | 0.4838 (0.95) | 0.4743 (0.95) | -3.11% (t=-9.8) / -2.94% (t=-7.8) / -3.16% (t=-7.3) / -2.50% (t=-4.0) |
| mrpt_small_1m_only | 0.4693 (0.96) | 0.4765 (0.96) | 0.4842 (0.95) | 0.4745 (0.95) | -3.00% (t=-8.9) / -2.99% (t=-7.8) / -3.16% (t=-6.8) / -2.42% (t=-3.9) |
| mrpt_small | 0.4697 (0.96) | 0.4763 (0.96) | 0.4831 (0.95) | 0.4736 (0.95) | -2.96% (t=-8.8) / -3.07% (t=-7.9) / -3.36% (t=-7.2) / -2.62% (t=-4.2) |
| Reading (pilot only, not a finding): every volatility-aware model improves on the random walk by 2–3 % in CRPS; | |||||
| the random walk is over-dispersed (95 % coverage 0.96–0.97, 50 % coverage 0.61); LightGBM over-fits the location | |||||
| on 4 942 training origins and is worse than the random walk; HAR-RV at H = 5 is miscalibrated (99 % coverage). | |||||
Baselines wall-clock: 22 s for all eight on the pilot (origin frames 4 s). Tables: results/metrics/pilot/leaderboard_test.csv, |
|||||
compare_{crps,wis,qlike}_vs_naive_rw_test.csv. |
Go / no-go
GO. Splits are immutable in the configs; baselines produce calibrated-looking distributions and score through the same code path as the neural models (identical origins, identical realised targets, asserted at run time).