PHASE 5 — Main experiments (development test 2024) — 2026-11-02/03
Run (configs/main.yaml, M3U96b, launched 19:02 ET, finished 23:06 ET; results/logs/main_chain.out)
- 240 pooled instruments, 2018-01-02 → 2025-12-31 grids; origins: train 586 202 (2 per instrument-session, 2018–2022), validation 1 040 345 (2023, stride 15), test 986 468 (2024, stride 15), final 1 008 425 (2025, untouched).
- Baselines stage: origin frames 153 s (test) + train; fits + scoring 1 581 s total for 8 baselines.
- Neural stage (batch 64, 12 000 steps = 768 000 examples, seed 1, early stopping on validation NLL every 1 000 steps, patience 4):
| Model | Params | Train+predict wall | Best val NLL | Examples/s (train, incl. validation) |
|---|---|---|---|---|
| MRPT-S (A+B+C) | 3 455 619 | 2 372 s | −6.4535 | ≈ 450 |
| MRPT-S (A only) | 3 200 259 | 1 509 s | −6.4520 | ≈ 600 |
| MRPT-B (A+B+C) | 20 718 339 | 6 684 s | −6.4526 | ≈ 180 |
| TCN (A) | 231 459 | 1 329 s | −6.4518 | |
| MLP (A, last 60 bars) | 301 827 | 1 219 s | −6.4497 | |
| All five reached the full 12 000 steps (no early stop); validation NLL plateaued from ≈ 7 000 steps for MRPT-S. |
Headline scores (scaled CRPS of the cumulative return; day-cluster 95 % CI; leaderboard_test.csv)
| Model | H=5 | H=15 | H=30 | H=60 | Δ vs random walk (H=5 / 60) |
|---|---|---|---|---|---|
| Random walk | 0.4699 [0.4648, 0.4754] | 0.4575 | 0.4469 | 0.4347 [0.4271, 0.4441] | — |
| EWMA-t | 0.4547 | 0.4425 | 0.4326 | 0.4214 | −3.65 % / −3.47 % |
| GARCH(1,1)-t | 0.4558 | 0.4430 | 0.4333 | 0.4232 | −3.60 % / −3.40 % |
| HAR-RV | 0.4615 | 0.4403 | 0.4295 | 0.4166 | −2.19 % / −4.89 % |
| Ridge / elastic net | 0.4603 | 0.4405 | 0.4297 | 0.4167 | −2.88 % / −4.89 % |
| LightGBM | 0.4591 | 0.4415 | 0.4293 | 0.4166 | −2.88 % / −4.85 % |
| MLP (1m) | 0.4538 | 0.4421 | 0.4319 | 0.4197 | −3.94 % / −4.07 % |
| TCN (1m) | 0.4538 | 0.4427 | 0.4327 | 0.4207 | −3.93 % / −3.73 % |
| MRPT-S 1m only | 0.4533 | 0.4416 | 0.4314 | 0.4192 | −4.02 % / −4.17 % |
| MRPT-S multi-res. | 0.4531 | 0.4414 | 0.4311 | 0.4187 | −4.05 % / −4.27 % |
| MRPT-B multi-res. | 0.4532 | 0.4414 | 0.4312 | 0.4189 | −4.04 % / −4.23 % |
| Every model beats the random walk (DM t between −25 and −72, all p < 1e-10 after Holm). |
Pre-specified contrasts (pairwise_test.csv; relative CRPS change, DM-HAC t, Holm-adjusted p)
| Contrast | H=5 | H=15 | H=30 | H=60 |
|---|---|---|---|---|
| H1 MRPT-S multi-res vs 1m-only | −0.04 % (t −5.3) | −0.04 % (t −4.5) | −0.05 % (t −4.5) | −0.10 % (t −5.3) |
| MRPT-B vs MRPT-S | +0.01 % (t 1.6, p 0.35) | −0.01 % (t −1.3) | −0.01 % (t −0.8) | +0.04 % (t 2.3, p 0.10) |
| MRPT-S vs HAR-RV | −1.91 % (t −58) | +0.34 % (t 12) | +0.53 % (t 12) | +0.65 % (t 10) |
| MRPT-S vs GARCH-t | −0.47 % (t −16) | −0.27 % (t −9) | −0.37 % (t −11) | −0.90 % (t −19) |
| MRPT-S vs EWMA-t | −0.42 % | −0.35 % | −0.47 % | −0.83 % |
| MRPT-S vs LightGBM | −1.21 % (t −22) | +0.09 % (t 2.0, p 0.19) | +0.53 % (t 12) | +0.61 % (t 10) |
| MRPT-S vs TCN | −0.13 % | −0.29 % | −0.39 % | −0.56 % |
| MRPT-S vs MLP | −0.12 % | −0.11 % | −0.12 % | −0.21 % |
| Reading: H1 is statistically detectable but economically negligible (a tenth of a percent of CRPS at 60 minutes). | ||||
| Model size does not matter (Base ≈ Small). The neural family wins at 5 minutes; the feature-based volatility | ||||
| regressions (HAR-RV, ridge, LightGBM) win from 15 minutes on, by 0.3–0.7 %. |
Calibration (leaderboard_test.csv, stress_calibration_test.csv)
Neural models and GARCH/EWMA are over-dispersed in the centre (50 % coverage 0.55–0.59, PIT KS 0.03–0.05) and near nominal at 95 %; HAR-RV / ridge / LightGBM are close to nominal from 15 minutes (0.49 / 0.80 / 0.96 at 60 min, KS 0.008) but badly under-dispersed at 5 minutes (50 % coverage 0.37, 95 % coverage 0.90). On stress days (top decile of cross-sectional σ_t, 100 846 origins) MRPT-S keeps 0.57 / 0.85 / 0.97 coverage (normal days 0.58 / 0.85 / 0.96): calibration does not deteriorate under stress, but the regime-conditioning hypothesis (H3) is tested in the ablations.
Subgroups (MRPT-S vs HAR-RV at H=60, subgroups_mrpt_small_vs_har_rv_test.csv)
HAR-RV's advantage concentrates in low-ADV stocks (+1.3 %), low-volatility instruments (+1.0 %), transport/utilities (+1.9 %), fixed-income ETFs (+1.8 %) and the "unknown"-sector group (+2.8 %); MRPT-S is better on broad-equity, sector and volatility ETFs (−0.3 to −1.1 %) and on high-ADV instruments (−0.03 %, n.s.). Afternoon origins favour HAR (+1.0 %).
Economic content (H5, exploratory; economic_test.csv)
Correlation between the predictive median and the realised 60-minute return: 0.003 (MRPT-S), 0.025 (MRPT-B), 0.014 (TCN), 0.007 (MLP); regression slopes between −1 and 1.2; the median's typical magnitude is 0.06–0.7 bp. With a 1 bp round-trip cost MRPT-S trades 17 % of origins at 60 minutes for a net −0.00 bp [−1.7, +1.8]; with 2.5–10 bp costs the traded share collapses below 4 % and every net return interval contains zero. No evidence of economic content.
Artifacts
results/metrics/main/: summary_*_test.json, per_origin_*_test.npz (node + laptop, not committed), leaderboard_test.csv,
compare_{crps,wis,qlike}_vs_naive_rw_test.csv, pairwise_test.csv, subgroups_*.csv, stress_calibration_test.csv,
economic_test.csv, <model>/train_log.jsonl, train_summary.json, best.pt (node only). Tables and figures:
paper/tables/main_*.tex, paper/figures/fig_main_*.