SPB Git
spboucher.ai

Repository Public Research preview

spbwp7_uqo

Commit

N-D3, N-D2 (WP07-DATA-04, EDIT-16, ECON-15, THEO-11, DATA-02, ADVE-06, EDIT-05, ECON-14, THEO-17): RQ4-A and RQ5 re-estimated from the FinData OS lake

- scripts/29 (640 statements, 9 min): Greeks by DTE for ten underlyings;
  frozen six-Greek HC1 R2 reproduced within 0.02 in 12/12 cells (max diff
  0.018, lake vintage); reduced design {avg IV, call/put gamma-OI} + ticker
  FE + date-clustered SE, condition numbers 2.8-3.4 -> rq4_greeks_decay_v2.csv,
  rq4_greeks_decay_compare.csv, tab_rq4_v2.
- scripts/06 --lake (80 statements, 10 min, seeds 0-9): v1.1 specification
  reproduces the frozen rows within 0.03 (VIX 0.3355, HAR 0.4373 vs 0.4366,
  RF 0.146 vs 0.143, GBM 0.103 vs 0.095); rolling annual re-estimation
  2016-2025 with 2018-2019 validation: R2_OOS VIX 0.347, HAR 0.158, HAR+surface
  0.135, RF 0.155 [0.153, 0.156], GBM 0.151; no tree learner beats HAR
  (DM -0.45 / -0.74) -> rq5_model_comparison_v2.csv, rq5_feature_importance_v2
  (seed dispersion), rq5_v11_replication.csv, rq5_validation_grid.csv.
- scripts/32: the HAR < VIX ordering in the rolling design comes from clipping
  the test-year regressors at training quantiles (2020: HAR R2 0.058 clipped vs
  0.449 unclipped); unclipped HAR 0.465 vs VIX 0.347 (DM 1.28, n.s.); the IV
  surface model stays below HAR under every rule -> rq5_clipping_variant.csv,
  Panel C of tab_rq5_v2.
- Lake scripts cast API scalars to numeric; raw extractions saved before processing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed · Nov 4, 2026 parent 6cf52de
17 changed files +582 −2 Browse files
17 changed files
Unified Split
modified .gitignore +1 −0
@@ -36,3 +36,4 @@ data/processed/greeks_by_dte_v2.parquet
36 36 data/processed/price_magnet_data_v2.parquet
37 37 data/processed/returns_total_1d.parquet
38 38 data/processed/cap_weights.parquet
39 +data/processed/magnet_lake_*_raw.parquet
added paper/tables/tab_rq4_v2.tex +34 −0
@@ -0,0 +1,34 @@
1 +\begin{table}[H]
2 +\centering
3 +\caption{Greeks by days to expiry re-estimated from the lake: reduced design with ticker fixed effects.}
4 +\label{tab:rq4_v2}
5 +\begin{threeparttable}
6 +\scriptsize
7 +\begin{tabular}{@{}lrrrrrrr@{}}
8 +\toprule
9 +& & Within & \multicolumn{3}{c}{$t$ (date-clustered)} & Cond.\ & Frozen-design $R^2$ (\%) \\
10 +\cmidrule(lr){4-6}
11 +DTE bucket (days) & $N$ & $R^2$ (\%) & IV & call $\Gamma\cdot$OI & put $\Gamma\cdot$OI & number & lake / 2026-08 \\
12 +\midrule
13 +\multicolumn{8}{@{}l}{\textit{Panel A: target $RV_{t+1}$}} \\
14 +1 week (1--7) & 36{,}966 & 0.97 & 9.69 & $-$8.41 & 3.74 & 2.8 & 5.61 / 3.77 \\
15 +2 weeks (8--14) & 34{,}147 & 2.49 & 10.27 & $-$9.07 & 1.44 & 3.1 & 7.68 / 6.70 \\
16 +1 month (15--30) & 35{,}958 & 2.60 & 11.77 & $-$7.93 & 1.25 & 3.4 & 8.59 / 6.85 \\
17 +2 months (31--60) & 38{,}698 & 2.79 & 11.38 & $-$9.12 & 4.82 & 3.4 & 8.47 / 7.52 \\
18 +3 months (61--90) & 33{,}465 & 2.54 & 12.57 & $-$7.06 & 5.65 & 3.1 & 10.35 / 9.47 \\
19 +6 months (91--180) & 38{,}956 & 2.39 & 11.69 & $-$7.21 & 7.89 & 3.4 & 8.26 / 8.18 \\
20 +\addlinespace
21 +\multicolumn{8}{@{}l}{\textit{Panel B: target $r_{t+1}$}} \\
22 +1 week (1--7) & 36{,}966 & 0.01 & $-$0.59 & $-$1.30 & $-$0.10 & 2.8 & 0.05 / 0.00 \\
23 +2 weeks (8--14) & 34{,}147 & 0.01 & 0.58 & $-$1.94 & 1.06 & 3.1 & 0.04 / 0.03 \\
24 +1 month (15--30) & 35{,}958 & 0.02 & 0.98 & $-$1.73 & 0.03 & 3.4 & 0.06 / 0.02 \\
25 +2 months (31--60) & 38{,}698 & 0.03 & 1.41 & $-$1.28 & 0.79 & 3.4 & 0.07 / 0.05 \\
26 +3 months (61--90) & 33{,}465 & 0.02 & 1.12 & $-$1.05 & 0.40 & 3.1 & 0.05 / 0.17 \\
27 +6 months (91--180) & 38{,}956 & 0.02 & 0.86 & $-$1.69 & 0.74 & 3.4 & 0.05 / 0.04 \\
28 +\bottomrule
29 +\end{tabular}
30 +\begin{tablenotes}[flushleft]\footnotesize
31 +\item \textit{Notes.} Greeks by days-to-expiry re-extracted from the FinData OS lake (\texttt{scripts/29}, 640 read-only statements, ten underlyings incl.\ SPY and QQQ). Reduced design $\{$average IV, call and put gamma-weighted open interest$\}$ with ticker fixed effects and date-clustered standard errors (within $R^2$ after ticker demeaning); condition number of the standardized reduced design. The last column re-estimates the frozen six-Greek HC1 design on the lake data next to the frozen $R^2$ of \texttt{rq4\_greeks\_decay.csv} (commit c54a8e2): agreement within 0.02 marks the frozen rows as reproduced on a different data vintage.
32 +\end{tablenotes}
33 +\end{threeparttable}
34 +\end{table}
added paper/tables/tab_rq5_v2.tex +37 −0
@@ -0,0 +1,37 @@
1 +\begin{table}[H]
2 +\centering
3 +\caption{RQ5 re-estimated from the lake: rolling re-estimation, seeds, validation block and forecast tests.}
4 +\label{tab:rq5_v2}
5 +\begin{threeparttable}
6 +\scriptsize
7 +\begin{tabular}{@{}lrrrrrr@{}}
8 +\toprule
9 +Model & $R^2_{OOS}$ (mean) & Seed range & 95\% bootstrap & DM vs HAR & DM vs VIX & CW vs HAR \\
10 +\midrule
11 +\multicolumn{7}{@{}l}{\textit{Panel A: rolling annual re-estimation 2016--2025, hyper-parameters on 2018--2019, seeds 0--9}} \\
12 +VIX benchmark, $(VIX_t/100)^2/252$ & 0.347 & -- & [0.213, 0.476] & 2.03 & -- & -- \\
13 +OLS: HAR-RV & 0.158 & -- & [0.115, 0.360] & -- & $-$2.03 & -- \\
14 +OLS: HAR-RV $+$ surface & 0.135 & -- & [0.090, 0.366] & $-$2.26 & $-$2.07 & $-$1.04 \\
15 +Random forest (surface $+$ HAR) & 0.155 & [0.153, 0.156] & [0.105, 0.403] & $-$0.45 & $-$1.96 & -- \\
16 +Gradient boosting (surface $+$ HAR) & 0.151 & [0.147, 0.153] & [0.099, 0.401] & $-$0.74 & $-$1.96 & -- \\
17 +\addlinespace
18 +\multicolumn{7}{@{}l}{\textit{Panel B: version~1.1 specification re-run on the lake data (single 2020 split, seed 42) vs frozen rows}} \\
19 +VIX (benchmark) & 0.335 & frozen 0.335 & $\Delta$ $-$0.000 & & & \\
20 +OLS: HAR-RV & 0.437 & frozen 0.437 & $\Delta$ +0.001 & & & \\
21 +Random Forest & 0.146 & frozen 0.143 & $\Delta$ +0.003 & & & \\
22 +Gradient Boosting & 0.103 & frozen 0.095 & $\Delta$ +0.008 & & & \\
23 +\addlinespace
24 +\multicolumn{7}{@{}l}{\textit{Panel C: OLS benchmarks under alternative treatments of the test-year regressors (\texttt{scripts/32})}} \\
25 +HAR-RV (features clipped at training quantiles) & 0.158 & QLIKE 0.350 & VIX: 0.347 / 0.474 & & $-$2.03 & \\
26 +HAR-RV + IV surface (features clipped at training quantiles) & 0.135 & QLIKE 0.468 & VIX: 0.347 / 0.474 & & $-$2.07 & \\
27 +HAR-RV (features unclipped) & 0.465 & QLIKE 0.316 & VIX: 0.347 / 0.474 & & 1.28 & \\
28 +HAR-RV + IV surface (features unclipped) & 0.270 & QLIKE 0.555 & VIX: 0.347 / 0.474 & & $-$0.93 & \\
29 +HAR-RV (log-RV model) & 0.437 & QLIKE 0.305 & VIX: 0.347 / 0.474 & & 1.47 & \\
30 +HAR-RV + IV surface (log-RV model) & 0.293 & QLIKE 0.280 & VIX: 0.347 / 0.474 & & $-$1.08 & \\
31 +\bottomrule
32 +\end{tabular}
33 +\begin{tablenotes}[flushleft]\footnotesize
34 +\item \textit{Notes.} Next-day SPX realized variance from the FinData OS lake (\texttt{scripts/06 --lake}: SPX option surface per quarter, 5-minute RV per year, VIX of \texttt{scripts/19}). Panel A: for each test year $Y$ in 2016--2025 the models are estimated on 2010..$Y-1$ (features winsorized on the training window, test target unclipped); 2{,}511 stacked forecasts; hyper-parameters chosen on a 2018--2019 validation block trained on 2010--2017 (\texttt{results/rq5\_validation\_grid.csv}); ten seeds for the tree learners; seed-average forecasts enter the stationary-bootstrap interval (1{,}000 draws) and the DM (NW(1)) and Clark--West tests (positive values favour the row model over HAR-RV / the VIX benchmark). Panel B: the frozen rows of commit c54a8e2 are reproduced within 0.03 only where stated (\texttt{results/rq5\_v11\_replication.csv}); the lake is a different data vintage from the 2026-08 stores. Panel C: in Panel A the test-year regressors are clipped at the 1\%/99\% quantiles of the training window (the scheme of Section~4.2), which caps $RV_t$ in 2020 far below its realized level and explains why HAR-RV trails the VIX benchmark there; with unclipped regressors or a log-RV specification HAR-RV is above the VIX benchmark (DM not significant) and the IV-surface model remains below HAR-RV under every rule.
35 +\end{tablenotes}
36 +\end{threeparttable}
37 +\end{table}
added results/rq4_greeks_decay_compare.csv +13 −0
@@ -0,0 +1,13 @@
1 +dte_bucket,target,design,n_obs_lake,n_obs_frozen,r2_lake,r2_frozen,r2_diff,iv_t_hc1_lake,iv_t_frozen,condition_number,within_0.02
2 +01_1w,Return,"frozen six Greeks, HC1",36966,34660,0.00048683882390254674,2.0533571869485456e-05,0.0004663052520330613,0.8234394020327681,-0.1326402783363189,2.97873795766574,True
3 +01_1w,RV,"frozen six Greeks, HC1",36966,34660,0.056088249429408266,0.0376577437475162,0.018430505681892063,23.418364444505514,33.118923449268,2.97873795766574,True
4 +02_2w,Return,"frozen six Greeks, HC1",34147,31912,0.0003762104887027329,0.0003363912137548,3.9819274947932876e-05,1.9979741829225925,1.9797508304047773,3.303804600036075,True
5 +02_2w,RV,"frozen six Greeks, HC1",34147,31912,0.07679486197807794,0.0669742619178255,0.00982060006025244,29.979434576060182,31.76191558898822,3.303804600036075,True
6 +03_1m,Return,"frozen six Greeks, HC1",35958,33715,0.0005539309367960099,0.0002379483382088,0.0003159825985872099,1.6314576882473255,1.8652077971398724,5.591095992197743,True
7 +03_1m,RV,"frozen six Greeks, HC1",35958,33715,0.08586533566180743,0.0685138404880506,0.017351495173756823,31.466336259655325,32.034662964663575,5.591095992197743,True
8 +04_2m,Return,"frozen six Greeks, HC1",38698,36449,0.0007001274140070457,0.0004658669961902,0.0002342604178168457,2.4472534652324276,2.550749950663104,3.640569777502139,True
9 +04_2m,RV,"frozen six Greeks, HC1",38698,36449,0.08469696992634712,0.0752413869004,0.009455583025947123,36.10828419141125,34.001115474871746,3.640569777502139,True
10 +05_3m,Return,"frozen six Greeks, HC1",33465,31354,0.0005313077068666017,0.0016994115118111,-0.0011681038049444984,2.328012921970004,2.2804548385541303,7.295082434750044,True
11 +05_3m,RV,"frozen six Greeks, HC1",33465,31354,0.10347871838728917,0.094704035657117,0.008774682730172165,31.365354907433762,35.14205904841721,7.295082434750044,True
12 +06_6m,Return,"frozen six Greeks, HC1",38956,36719,0.00047010115936807484,0.0004410468541744,2.905430519367484e-05,2.0140194528982565,2.041608634479025,8.385167743239638,True
13 +06_6m,RV,"frozen six Greeks, HC1",38956,36719,0.0826430562653212,0.0818206346918886,0.0008224215734325968,33.5386805628193,35.910992081174044,8.385167743239638,True
added results/rq4_greeks_decay_v2.csv +49 −0
@@ -0,0 +1,49 @@
1 +dte_bucket,target,variable,coefficient,t_hc1,t_date_clustered,r2,r2_within,n_obs,n_tickers,condition_number_reduced,condition_number_frozen_design,n_dates,code_version,status
2 +01_1w,Return,const,0.0008759393440739169,2.9930221324846094,2.9789993536551713,0.0003675867308979841,0.00012693422613518912,36966,10,2.7654843767249018,2.97873795766574,3885,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
3 +01_1w,Return,avg_iv,-0.00018447483642969876,-1.1097638839089026,-0.5912592126621488,0.0003675867308979841,0.00012693422613518912,36966,10,2.7654843767249018,2.97873795766574,3885,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
4 +01_1w,Return,call_gamma_oi,-0.00018589358791652275,-1.7046615943326124,-1.3007603700030141,0.0003675867308979841,0.00012693422613518912,36966,10,2.7654843767249018,2.97873795766574,3885,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
5 +01_1w,Return,put_gamma_oi,-1.3023437617323396e-05,-0.12490193712810935,-0.09786330757895803,0.0003675867308979841,0.00012693422613518912,36966,10,2.7654843767249018,2.97873795766574,3885,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
6 +01_1w,RV,const,0.0003873267028826775,22.704562222918625,22.732186515440564,0.060001329447480045,0.009749399007323523,36966,10,2.7654843767249018,2.97873795766574,3885,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
7 +01_1w,RV,avg_iv,0.00018698133645647592,12.307484857410477,9.692142626013773,0.060001329447480045,0.009749399007323523,36966,10,2.7654843767249018,2.97873795766574,3885,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
8 +01_1w,RV,call_gamma_oi,-7.387986788297777e-05,-10.490087332381716,-8.410692510741368,0.060001329447480045,0.009749399007323523,36966,10,2.7654843767249018,2.97873795766574,3885,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
9 +01_1w,RV,put_gamma_oi,1.779912446779914e-05,4.0337273458765885,3.7370570568249932,0.060001329447480045,0.009749399007323523,36966,10,2.7654843767249018,2.97873795766574,3885,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
10 +02_2w,Return,const,0.0007637424166684728,2.510445519550082,2.4950970976837876,0.0004413210992676442,0.00014191613354697097,34147,10,3.1019892951390027,3.303804600036075,3579,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
11 +02_2w,Return,avg_iv,0.0002436100432712293,1.1110262801385253,0.5842347058430566,0.0004413210992676442,0.00014191613354697097,34147,10,3.1019892951390027,3.303804600036075,3579,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
12 +02_2w,Return,call_gamma_oi,-0.0003429008316861338,-3.0274689952835603,-1.943401682223259,0.0004413210992676442,0.00014191613354697097,34147,10,3.1019892951390027,3.303804600036075,3579,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
13 +02_2w,Return,put_gamma_oi,0.00016075202288296593,1.339987005272351,1.0643066808050219,0.0004413210992676442,0.00014191613354697097,34147,10,3.1019892951390027,3.303804600036075,3579,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
14 +02_2w,RV,const,0.00038428536089559113,21.089856186414565,21.048196299733906,0.07366543058475616,0.024872788111998556,34147,10,3.1019892951390027,3.303804600036075,3579,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
15 +02_2w,RV,avg_iv,0.00034984038280511294,16.43992160941857,10.270183062953521,0.07366543058475616,0.024872788111998556,34147,10,3.1019892951390027,3.303804600036075,3579,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
16 +02_2w,RV,call_gamma_oi,-9.236744756718212e-05,-12.919076683901434,-9.068028747072853,0.07366543058475616,0.024872788111998556,34147,10,3.1019892951390027,3.303804600036075,3579,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
17 +02_2w,RV,put_gamma_oi,8.40331757851601e-06,1.7078173192112334,1.4377003028180269,0.07366543058475616,0.024872788111998556,34147,10,3.1019892951390027,3.303804600036075,3579,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
18 +03_1m,Return,const,0.0008107063980909439,2.7021743567225656,2.7008593571867303,0.0005788440866342226,0.0002490469512090865,35958,10,3.352563308049646,5.591095992197743,3778,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
19 +03_1m,Return,avg_iv,0.0003772420859826336,1.6929968967477693,0.9814941498911955,0.0005788440866342226,0.0002490469512090865,35958,10,3.352563308049646,5.591095992197743,3778,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
20 +03_1m,Return,call_gamma_oi,-0.0003125551748467725,-2.34111632975766,-1.7281606274152372,0.0005788440866342226,0.0002490469512090865,35958,10,3.352563308049646,5.591095992197743,3778,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
21 +03_1m,Return,put_gamma_oi,5.469956626818279e-06,0.03634740433152063,0.029348797286755,0.0005788440866342226,0.0002490469512090865,35958,10,3.352563308049646,5.591095992197743,3778,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
22 +03_1m,RV,const,0.00038931455441187647,21.642444091661403,21.87582036033014,0.07686634172218276,0.026010129125158876,35958,10,3.352563308049646,5.591095992197743,3778,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
23 +03_1m,RV,avg_iv,0.00036034721091230364,18.504527285381013,11.77450935477579,0.07686634172218276,0.026010129125158876,35958,10,3.352563308049646,5.591095992197743,3778,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
24 +03_1m,RV,call_gamma_oi,-7.979644872410717e-05,-9.755755638853604,-7.9303100590025455,0.07686634172218276,0.026010129125158876,35958,10,3.352563308049646,5.591095992197743,3778,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
25 +03_1m,RV,put_gamma_oi,9.329617417973185e-06,1.4004774854208022,1.2505602126054385,0.07686634172218276,0.026010129125158876,35958,10,3.352563308049646,5.591095992197743,3778,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
26 +04_2m,Return,const,0.0008769075153482785,3.048480316874417,3.073323158684686,0.0005930004066052597,0.0003220366954963527,38698,10,3.3919220167859945,3.640569777502139,4003,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
27 +04_2m,Return,avg_iv,0.0005577933135295927,2.282804027779344,1.406067243785849,0.0005930004066052597,0.0003220366954963527,38698,10,3.3919220167859945,3.640569777502139,4003,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
28 +04_2m,Return,call_gamma_oi,-0.0002405701058061391,-1.6987870355402586,-1.2840551068831725,0.0005930004066052597,0.0003220366954963527,38698,10,3.3919220167859945,3.640569777502139,4003,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
29 +04_2m,Return,put_gamma_oi,0.00015320970412440179,0.9415102948843891,0.7886370509028753,0.0005930004066052597,0.0003220366954963527,38698,10,3.3919220167859945,3.640569777502139,4003,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
30 +04_2m,RV,const,0.00037327782727961085,21.953688795929995,22.669913077342873,0.08306042374500999,0.027895214057357953,38698,10,3.3919220167859945,3.640569777502139,4003,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
31 +04_2m,RV,avg_iv,0.00039304195761459353,15.888777121683898,11.383280000486113,0.08306042374500999,0.027895214057357953,38698,10,3.3919220167859945,3.640569777502139,4003,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
32 +04_2m,RV,call_gamma_oi,-8.404713496116131e-05,-12.949807488861385,-9.119690897652546,0.08306042374500999,0.027895214057357953,38698,10,3.3919220167859945,3.640569777502139,4003,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
33 +04_2m,RV,put_gamma_oi,2.8099014462927715e-05,5.259933043364892,4.82214156211913,0.08306042374500999,0.027895214057357953,38698,10,3.3919220167859945,3.640569777502139,4003,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
34 +05_3m,Return,const,0.0008708407902541745,2.800539773840989,2.8596953193859784,0.0005419340764091229,0.00020305019320188133,33465,10,3.0977511760804126,7.295082434750044,3861,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
35 +05_3m,Return,avg_iv,0.00044284643050264855,1.800494862782591,1.1165312094361175,0.0005419340764091229,0.00020305019320188133,33465,10,3.0977511760804126,7.295082434750044,3861,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
36 +05_3m,Return,call_gamma_oi,-0.00019021190464025124,-1.2708604940455486,-1.0498879929903595,0.0005419340764091229,0.00020305019320188133,33465,10,3.0977511760804126,7.295082434750044,3861,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
37 +05_3m,Return,put_gamma_oi,7.641476541387902e-05,0.4817291701872203,0.39932497804838796,0.0005419340764091229,0.00020305019320188133,33465,10,3.0977511760804126,7.295082434750044,3861,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
38 +05_3m,RV,const,0.000371781300458165,19.97823065666188,20.723550835137317,0.09880580120671079,0.025404827840957767,33465,10,3.0977511760804126,7.295082434750044,3861,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
39 +05_3m,RV,avg_iv,0.000299100363051697,19.960673099086055,12.566610280064173,0.09880580120671079,0.025404827840957767,33465,10,3.0977511760804126,7.295082434750044,3861,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
40 +05_3m,RV,call_gamma_oi,-6.0008755182035164e-05,-8.174802056172563,-7.063867390044951,0.09880580120671079,0.025404827840957767,33465,10,3.0977511760804126,7.295082434750044,3861,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
41 +05_3m,RV,put_gamma_oi,4.513620074633586e-05,6.002792098511881,5.645169697955334,0.09880580120671079,0.025404827840957767,33465,10,3.0977511760804126,7.295082434750044,3861,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
42 +06_6m,Return,const,0.0009589873442627434,3.2860236645891288,3.066120162149391,0.00047631248950408267,0.0001795590101347777,38956,10,3.4099071815707065,8.385167743239638,4024,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
43 +06_6m,Return,avg_iv,0.00037225398737260817,1.3506246829955617,0.8624901600255354,0.00047631248950408267,0.0001795590101347777,38956,10,3.4099071815707065,8.385167743239638,4024,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
44 +06_6m,Return,call_gamma_oi,-0.00036487511359968203,-2.1974089661480174,-1.6933202451638707,0.00047631248950408267,0.0001795590101347777,38956,10,3.4099071815707065,8.385167743239638,4024,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
45 +06_6m,Return,put_gamma_oi,0.00021002914765286482,1.0381498732849004,0.7357602372810232,0.00047631248950408267,0.0001795590101347777,38956,10,3.4099071815707065,8.385167743239638,4024,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
46 +06_6m,RV,const,0.0004283740654600074,25.516390501122373,24.25899594567929,0.08001056303025444,0.023894990909548497,38956,10,3.4099071815707065,8.385167743239638,4024,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
47 +06_6m,RV,avg_iv,0.0004180353754675509,18.660030258661482,11.693485843878099,0.08001056303025444,0.023894990909548497,38956,10,3.4099071815707065,8.385167743239638,4024,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
48 +06_6m,RV,call_gamma_oi,-7.454874870707279e-05,-8.131167810406401,-7.2149549961418575,0.08001056303025444,0.023894990909548497,38956,10,3.4099071815707065,8.385167743239638,4024,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
49 +06_6m,RV,put_gamma_oi,7.83815307008625e-05,9.042061973683564,7.893339605461839,0.08001056303025444,0.023894990909548497,38956,10,3.4099071815707065,8.385167743239638,4024,"scripts/29 (2026-11-04, lake vintage)",re-estimated from FinData OS lake
added results/rq5_clipping_variant.csv +67 −0
@@ -0,0 +1,67 @@
1 +rule,model,period,r2_oos,qlike,r2_oos_vix,qlike_vix,dm_vs_vix_t,n
2 +features clipped at training quantiles,OLS: HAR-RV,2016-2025,0.15805487685915132,0.35046799704478465,0.3468139896356155,0.4736555708470223,-2.0329312975789016,2511
3 +features clipped at training quantiles,OLS: HAR-RV,2016,0.3673162131640896,0.29988969709452695,0.2591871694048651,0.48394038368566583,,252
4 +features clipped at training quantiles,OLS: HAR-RV,2017,-0.3812452002379505,0.3503275204197712,-3.931931217428131,0.6249360933878423,,251
5 +features clipped at training quantiles,OLS: HAR-RV,2018,0.4284871822876283,0.2452230710932671,0.5945674147111402,0.31438544096589405,,250
6 +features clipped at training quantiles,OLS: HAR-RV,2019,0.27415859287143773,0.3027464396474738,0.01256478197776556,0.46567536930817965,,252
7 +features clipped at training quantiles,OLS: HAR-RV,2020,0.05760573999349883,0.725558260967582,0.30517728067132066,0.5666516318563621,,253
8 +features clipped at training quantiles,OLS: HAR-RV,2021,0.24418860144044652,0.34440500454632306,-1.5478268292708073,0.6838319215243118,,252
9 +features clipped at training quantiles,OLS: HAR-RV,2022,0.09451553281655156,0.2724811784958332,0.1028507361661597,0.2591583267039235,,251
10 +features clipped at training quantiles,OLS: HAR-RV,2023,0.29551487680235466,0.20807140910084093,-0.4556991427099064,0.3407724339757761,,249
11 +features clipped at training quantiles,OLS: HAR-RV,2024,0.09479492643359644,0.3405787468884113,0.005028369510279407,0.45085487844823985,,252
12 +features clipped at training quantiles,OLS: HAR-RV,2025,0.25966204320826447,0.4117998259713889,0.3147183352546451,0.5437174154425966,,249
13 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2016-2025,0.13518592413433983,0.468327490748768,0.3468139896356155,0.4736555708470223,-2.072196822635278,2511
14 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2016,0.4161119410520221,0.4276424859292478,0.2591871694048651,0.48394038368566583,,252
15 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2017,-0.06506393955829726,0.3996453810742772,-3.931931217428131,0.6249360933878423,,251
16 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2018,0.5732532285136687,0.26193773424682437,0.5945674147111402,0.31438544096589405,,250
17 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2019,0.10842628587059555,0.3498055180362601,0.01256478197776556,0.46567536930817965,,252
18 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2020,0.024813355142903504,0.8380557982328456,0.30517728067132066,0.5666516318563621,,253
19 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2021,-0.6498121221222921,0.49316804167459793,-1.5478268292708073,0.6838319215243118,,252
20 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2022,0.18779916157198806,0.23768910387877717,0.1028507361661597,0.2591583267039235,,251
21 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2023,0.24918467075776296,0.2708155926343409,-0.4556991427099064,0.3407724339757761,,249
22 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2024,0.1448001673234416,0.6495694904934559,0.005028369510279407,0.45085487844823985,,252
23 +features clipped at training quantiles,OLS: HAR-RV + IV surface,2025,0.27457736355334617,0.7516746400028735,0.3147183352546451,0.5437174154425966,,249
24 +features unclipped,OLS: HAR-RV,2016-2025,0.4654363824997666,0.3161657869074887,0.3468139896356155,0.4736555708470223,1.276230505625995,2511
25 +features unclipped,OLS: HAR-RV,2016,0.366157899031771,0.2990071528567583,0.2591871694048651,0.48394038368566583,,252
26 +features unclipped,OLS: HAR-RV,2017,-0.35049147608870657,0.34112846017259585,-3.931931217428131,0.6249360933878423,,251
27 +features unclipped,OLS: HAR-RV,2018,0.4284934915632118,0.24506714787219705,0.5945674147111402,0.31438544096589405,,250
28 +features unclipped,OLS: HAR-RV,2019,0.2741796425163663,0.30263196840125667,0.01256478197776556,0.46567536930817965,,252
29 +features unclipped,OLS: HAR-RV,2020,0.44880173122887856,0.4035427781340977,0.30517728067132066,0.5666516318563621,,253
30 +features unclipped,OLS: HAR-RV,2021,0.2441918897782207,0.34439479415863117,-1.5478268292708073,0.6838319215243118,,252
31 +features unclipped,OLS: HAR-RV,2022,0.08999635588948784,0.27265552964112044,0.1028507361661597,0.2591583267039235,,251
32 +features unclipped,OLS: HAR-RV,2023,0.2955615482773649,0.20798730558254638,-0.4556991427099064,0.3407724339757761,,249
33 +features unclipped,OLS: HAR-RV,2024,0.0037774070677751848,0.34201316757060024,0.005028369510279407,0.45085487844823985,,252
34 +features unclipped,OLS: HAR-RV,2025,0.32929695850269813,0.40197869736002184,0.3147183352546451,0.5437174154425966,,249
35 +features unclipped,OLS: HAR-RV + IV surface,2016-2025,0.2699185612032765,0.5550019230005042,0.3468139896356155,0.4736555708470223,-0.9328571468415918,2511
36 +features unclipped,OLS: HAR-RV + IV surface,2016,0.412669035945169,0.4685380122395511,0.2591871694048651,0.48394038368566583,,252
37 +features unclipped,OLS: HAR-RV + IV surface,2017,-0.1530665602817296,0.4881520266608406,-3.931931217428131,0.6249360933878423,,251
38 +features unclipped,OLS: HAR-RV + IV surface,2018,0.6017653636193699,0.28044845925208883,0.5945674147111402,0.31438544096589405,,250
39 +features unclipped,OLS: HAR-RV + IV surface,2019,0.1099477503629177,0.3453355614001182,0.01256478197776556,0.46567536930817965,,252
40 +features unclipped,OLS: HAR-RV + IV surface,2020,0.30053923142506167,0.44727605626690264,0.30517728067132066,0.5666516318563621,,253
41 +features unclipped,OLS: HAR-RV + IV surface,2021,-0.8002396319337146,0.6801367618328934,-1.5478268292708073,0.6838319215243118,,252
42 +features unclipped,OLS: HAR-RV + IV surface,2022,-4.422111245820611,0.3331145487463866,0.1028507361661597,0.2591583267039235,,251
43 +features unclipped,OLS: HAR-RV + IV surface,2023,0.06851279642888897,0.2823888668022166,-0.4556991427099064,0.3407724339757761,,249
44 +features unclipped,OLS: HAR-RV + IV surface,2024,0.038322448083005334,1.1273670620387348,0.005028369510279407,0.45085487844823985,,252
45 +features unclipped,OLS: HAR-RV + IV surface,2025,0.36564522553224943,1.0975784304092548,0.3147183352546451,0.5437174154425966,,249
46 +log-RV model,OLS: HAR-RV,2016-2025,0.43666948719338283,0.30545319173971197,0.3468139896356155,0.4736555708470223,1.467759974639311,2511
47 +log-RV model,OLS: HAR-RV,2016,0.39041699020913523,0.2735863127511425,0.2591871694048651,0.48394038368566583,,252
48 +log-RV model,OLS: HAR-RV,2017,-0.12792867558670085,0.27941844599465016,-3.931931217428131,0.6249360933878423,,251
49 +log-RV model,OLS: HAR-RV,2018,0.4329632996285603,0.24370191407992858,0.5945674147111402,0.31438544096589405,,250
50 +log-RV model,OLS: HAR-RV,2019,0.25433507494044083,0.29827974637488625,0.01256478197776556,0.46567536930817965,,252
51 +log-RV model,OLS: HAR-RV,2020,0.4104161504836513,0.40683091590416987,0.30517728067132066,0.5666516318563621,,253
52 +log-RV model,OLS: HAR-RV,2021,0.2570005670147051,0.3377954321165072,-1.5478268292708073,0.6838319215243118,,252
53 +log-RV model,OLS: HAR-RV,2022,0.12676397358754854,0.25984489393101556,0.1028507361661597,0.2591583267039235,,251
54 +log-RV model,OLS: HAR-RV,2023,0.27427480117065806,0.2102637182472463,-0.4556991427099064,0.3407724339757761,,249
55 +log-RV model,OLS: HAR-RV,2024,0.07845923857477621,0.33451978333527466,0.005028369510279407,0.45085487844823985,,252
56 +log-RV model,OLS: HAR-RV,2025,0.32116350151079087,0.4092161399845981,0.3147183352546451,0.5437174154425966,,249
57 +log-RV model,OLS: HAR-RV + IV surface,2016-2025,0.2927415257791389,0.2800947402993819,0.3468139896356155,0.4736555708470223,-1.0835368783517698,2511
58 +log-RV model,OLS: HAR-RV + IV surface,2016,0.35520419333880004,0.2566644840222455,0.2591871694048651,0.48394038368566583,,252
59 +log-RV model,OLS: HAR-RV + IV surface,2017,0.04873058973392563,0.23693397186381496,-3.931931217428131,0.6249360933878423,,251
60 +log-RV model,OLS: HAR-RV + IV surface,2018,0.5989971755191419,0.21397669449659115,0.5945674147111402,0.31438544096589405,,250
61 +log-RV model,OLS: HAR-RV + IV surface,2019,0.26741525397986365,0.26993896746915574,0.01256478197776556,0.46567536930817965,,252
62 +log-RV model,OLS: HAR-RV + IV surface,2020,0.21080672037265935,0.44333247916580937,0.30517728067132066,0.5666516318563621,,253
63 +log-RV model,OLS: HAR-RV + IV surface,2021,0.09561625980732968,0.3401490692712845,-1.5478268292708073,0.6838319215243118,,252
64 +log-RV model,OLS: HAR-RV + IV surface,2022,-0.49039020955485535,0.2588184180215563,0.1028507361661597,0.2591583267039235,,251
65 +log-RV model,OLS: HAR-RV + IV surface,2023,0.41393953933449246,0.17761129638076095,-0.4556991427099064,0.3407724339757761,,249
66 +log-RV model,OLS: HAR-RV + IV surface,2024,0.08694731190133498,0.2759505303508593,0.005028369510279407,0.45085487844823985,,252
67 +log-RV model,OLS: HAR-RV + IV surface,2025,0.5175374824502921,0.32546333533333094,0.3147183352546451,0.5437174154425966,,249
added results/rq5_feature_importance_v2.csv +49 −0
@@ -0,0 +1,49 @@
1 +model,feature,mean,std,min,max
2 +GBM,iv_atm_2w,0.35224676124162124,0.016148837512172217,0.3278434483286817,0.3853581618169696
3 +GBM,iv_atm_1w,0.35136410549949937,0.016677226306648668,0.31778263602778645,0.3767144202824377
4 +GBM,rv_daily,0.10116551275057684,0.0044012541283976305,0.09038865139458174,0.10630793218292638
5 +GBM,rv_w,0.02561778053871206,0.0029354104967520625,0.02098737055499562,0.03019328234920387
6 +GBM,net_gamma,0.021981079262160745,0.000925858653024582,0.020837519957094406,0.023463447000584833
7 +GBM,iv_atm_3m,0.017935066963766053,0.0014756351456283688,0.015661698478597597,0.02028950959484954
8 +GBM,skew_10d_1m,0.01741539433218266,0.0029678044302118758,0.01403085299216157,0.02247591486243728
9 +GBM,pc_oi_ratio,0.016673189372248928,0.0013584489009866182,0.014333110690163317,0.019040637160152602
10 +GBM,butterfly_1m,0.013502734439036346,0.0014740093492170967,0.011037022714266726,0.01574256044667469
11 +GBM,skew_25d_1m,0.011577860231124629,0.0029697563035521245,0.008352561844090467,0.017313844675272284
12 +GBM,pc_vol_ratio,0.010112147198874636,0.0014958233936516244,0.008016576486456189,0.012941677488977338
13 +GBM,total_volume,0.00804965378401217,0.0008665408896557229,0.006373770330509705,0.009143163229115939
14 +GBM,total_oi,0.00653579161441401,0.0008274605046411759,0.005365222333493187,0.00771598923561599
15 +GBM,iv_atm_1m,0.0062504561464941984,0.001617244644404321,0.004151483900822005,0.009505555218380696
16 +GBM,ts_slope_3m_1m,0.006120032388151392,0.001569978087660073,0.004208965298628863,0.009224700085602455
17 +GBM,rv_m,0.005948655674673626,0.0009546558492794623,0.004127512375119055,0.006984582716089708
18 +GBM,total_gamma_oi,0.005553720378390133,0.0006644006660583892,0.004668137224406202,0.006510994945662133
19 +GBM,skew_25d_3m,0.004812482806194491,0.0008580729833231622,0.003462135310896603,0.006493703143463419
20 +GBM,ts_slope_6m_1m,0.0038220596970343695,0.0010820951181575913,0.0027149105119316556,0.006333039932626145
21 +GBM,avg_theta,0.0036450076731155906,0.0009201841447304458,0.002067772832326672,0.005104006487304854
22 +GBM,iv_atm_6m,0.0031678869578847914,0.0010709602802948827,0.0019691926066968146,0.004886635555487059
23 +GBM,iv_atm_2m,0.00316684195116752,0.0006102064415223238,0.0019431249430112956,0.004427975983561574
24 +GBM,avg_spread_pct,0.0019155182062074408,0.0006824817948257469,0.0006459369856605989,0.003156844337580423
25 +GBM,total_vega_oi,0.0014202608924568385,0.0005992015262795684,0.0006480577125907573,0.0023236456344201835
26 +RF,iv_atm_2w,0.40071773532471167,0.013277733257790855,0.3766840336316213,0.42019430535587504
27 +RF,iv_atm_1w,0.36971749175549296,0.013835684051878894,0.35101909163512024,0.393693609392683
28 +RF,rv_daily,0.09964677859844848,0.0028960750516562484,0.09373495349317486,0.10288880447107565
29 +RF,rv_w,0.022572167868690415,0.002133340784531509,0.019556175825002416,0.025580525718211126
30 +RF,net_gamma,0.012340815545844564,0.0003364107701179407,0.011853800844180936,0.012962651280268269
31 +RF,iv_atm_3m,0.012336719157633711,0.0008657484909678732,0.010972230515128792,0.013561394920419443
32 +RF,skew_10d_1m,0.009950042422005494,0.0009263742263020457,0.008047235617696546,0.011034615427876206
33 +RF,skew_25d_1m,0.007312125617015284,0.0010537261750155297,0.005955433261384135,0.008693940417682542
34 +RF,pc_oi_ratio,0.0068101745850371945,0.000395050035554864,0.006067595457339322,0.007337272460098156
35 +RF,pc_vol_ratio,0.0066563045860604225,0.00039355864051083075,0.005999975290342072,0.007175886144279012
36 +RF,ts_slope_3m_1m,0.006374820975486613,0.0006832215008491683,0.005358701203889693,0.0074996615963467145
37 +RF,ts_slope_6m_1m,0.0055664217286511926,0.0006278689543280515,0.004836554397203363,0.006905758136832619
38 +RF,total_volume,0.005478153893284611,0.0003328347973337338,0.005015516250373221,0.005991978219490614
39 +RF,iv_atm_1m,0.00507288441557383,0.0005455564159775138,0.004209943092305943,0.005867816554113843
40 +RF,iv_atm_6m,0.0048692690301090575,0.0003517902224641141,0.004343003542166996,0.005375138153063963
41 +RF,butterfly_1m,0.004078498728846091,0.0005812436713213676,0.003231044850418729,0.005137739467736035
42 +RF,iv_atm_2m,0.0035745852443476974,0.00040747554351879976,0.0029401664017495973,0.004381093683045034
43 +RF,total_oi,0.003090974075873071,0.00027971782731973314,0.002647461758923293,0.003628514440500854
44 +RF,rv_m,0.0028847474284029373,0.00012691645797541744,0.002630495656354141,0.0030808290889111243
45 +RF,avg_theta,0.002484074520113616,0.00016886632845109994,0.002253782072787304,0.002765188097842453
46 +RF,skew_25d_3m,0.0024040415696922643,0.0001739475202565388,0.00200394034380619,0.0025801787409762663
47 +RF,total_gamma_oi,0.0021720112965514164,0.0002166886548197212,0.001823908410763863,0.002623993147060997
48 +RF,avg_spread_pct,0.002127801174476792,0.0001944148306022873,0.0018979744210796895,0.002549157423233459
49 +RF,total_vega_oi,0.001761360457650702,0.00021640242588395026,0.001510530054281887,0.0021311561254266516
added results/rq5_model_comparison_v2.csv +6 −0
@@ -0,0 +1,6 @@
1 +Model,Target,n_seeds,R2_OOS_mean,R2_OOS_min,R2_OOS_max,MSE_mean,QLIKE_mean,R2_OOS_boot_p025,R2_OOS_boot_p975,DM_vs_HAR_t,DM_vs_VIX_t,CW_vs_HAR_t,n_test,test_period,hyperparameters,code_version,status
2 +VIX,1-Day RV,1,0.3468139896356155,0.3468139896356155,0.3468139896356155,1.1207436462954134e-07,0.4736555708470223,0.21269546175998136,0.47596935158661063,2.032931297578902,,,2511,"2016-2025 (annual re-estimation, train 2010..Y-1)",,scripts/06 --lake (2026-11-04),re-estimated from FinData OS lake
3 +OLS: HAR-RV,1-Day RV,1,0.15805487685915132,0.15805487685915132,0.15805487685915132,1.4446185807977102e-07,0.35046799704478454,0.11491362312913643,0.35972375221811304,,-2.032931297578902,,2511,"2016-2025 (annual re-estimation, train 2010..Y-1)",,scripts/06 --lake (2026-11-04),re-estimated from FinData OS lake
4 +OLS: HAR-RV + IV surface,1-Day RV,1,0.13513462933570486,0.13513462933570486,0.13513462933570486,1.4839453902759027e-07,,0.0901644618589751,0.36619304907311917,-2.2646821641510404,-2.0726995190914037,-1.037285006887623,2511,"2016-2025 (annual re-estimation, train 2010..Y-1)",,scripts/06 --lake (2026-11-04),re-estimated from FinData OS lake
5 +RF,1-Day RV,10,0.1549032958024221,0.15315258210488558,0.15619828489972665,1.450026097782258e-07,0.3019986310494449,0.10461065442432431,0.40340403740549513,-0.4549821688130667,-1.9589849794627534,,2511,"2016-2025 (annual re-estimation, train 2010..Y-1)","{'trees': 500, 'depth': 6, 'leaf': 20, 'lr': np.float64(nan)}",scripts/06 --lake (2026-11-04),re-estimated from FinData OS lake
6 +GBM,1-Day RV,10,0.15056381695038445,0.1465451146186153,0.15296726005702033,1.4574718226974951e-07,0.33874463877012756,0.09877801478986196,0.400728083827693,-0.740846727777375,-1.9566308944365065,,2511,"2016-2025 (annual re-estimation, train 2010..Y-1)","{'trees': 200, 'depth': 3, 'leaf': 20, 'lr': np.float64(0.02)}",scripts/06 --lake (2026-11-04),re-estimated from FinData OS lake
added results/rq5_v11_replication.csv +5 −0
@@ -0,0 +1,5 @@
1 +Model,mse,mae,r2_oos,n_train,n_test,r2_oos_frozen,mse_frozen,r2_diff,within_0.03
2 +VIX (benchmark),1.8367340831415343e-07,0.0001371764866556537,0.3354796361314323,2506,1506,0.3354831794698668,1.8354898166915326e-07,-3.5433384344973895e-06,True
3 +OLS: HAR-RV,1.5553485693205896e-07,0.0001017284213964345,0.4372833788440508,2506,1506,0.4366324857576119,1.556101069980116e-07,0.0006508930864388973,True
4 +Random Forest,2.3613436280355555e-07,0.00011164181032219896,0.14567876682669734,2506,1506,0.1427650004842756,2.3678047921607625e-07,0.002913766342421742,True
5 +Gradient Boosting,2.479690599088492e-07,0.00012287442429653178,0.10286147879971841,2506,1506,0.0946654681218959,2.500662530462015e-07,0.00819601067782251,True
added results/rq5_validation_grid.csv +23 −0
@@ -0,0 +1,23 @@
1 +model,depth,leaf,trees,lr,mse,mae,r2_oos
2 +RF,4,10,200,,4.884630728945244e-09,3.804152533819232e-05,0.5375670015070582
3 +RF,4,10,500,,4.748693825302402e-09,3.7787896209101894e-05,0.5504362875279851
4 +RF,4,20,200,,4.537269312357344e-09,3.780319858565774e-05,0.5704520629061998
5 +RF,4,20,500,,4.505126301247796e-09,3.7588254169614846e-05,0.5734950747187201
6 +RF,4,50,200,,4.976658275501215e-09,3.7620347362564144e-05,0.5288546593343642
7 +RF,4,50,500,,4.968046812794388e-09,3.760315780443986e-05,0.529669915336692
8 +RF,6,10,200,,4.828548034915049e-09,3.880845219976127e-05,0.542876407642974
9 +RF,6,10,500,,4.709381899894599e-09,3.866156612806984e-05,0.5541579878061935
10 +RF,6,20,200,,4.5042995664480015e-09,3.830716373144164e-05,0.5735733425497278
11 +RF,6,20,500,,4.4493224575528315e-09,3.8041489733460555e-05,0.5787780818075418
12 +RF,6,50,200,,4.9198887030012285e-09,3.7641442429911725e-05,0.5342290929591553
13 +RF,6,50,500,,4.913565171628939e-09,3.763061284818458e-05,0.5348277481566145
14 +RF,10,10,200,,4.852593962487452e-09,3.9650011380549065e-05,0.5405999550294821
15 +RF,10,10,500,,4.72676089429788e-09,3.933538288358093e-05,0.5525127005902986
16 +RF,10,20,200,,4.507442528022691e-09,3.858085229156852e-05,0.5732757951555062
17 +RF,10,20,500,,4.452797206239087e-09,3.833556319588107e-05,0.5784491237873433
18 +RF,10,50,200,,4.91511880355965e-09,3.764618565190287e-05,0.5346806642290587
19 +RF,10,50,500,,4.908752634570228e-09,3.7642219849141486e-05,0.5352833559734409
20 +GBM,3,20,200,0.02,8.40888406607535e-09,6.531602558032185e-05,0.20392232526154275
21 +GBM,5,20,200,0.02,8.471110864403112e-09,6.422034876916526e-05,0.19803125047326764
22 +GBM,3,20,200,0.05,1.0289673856885997e-08,7.251420543749327e-05,0.025866027710614747
23 +GBM,5,20,200,0.05,1.1970388907055582e-08,7.843017880555689e-05,-0.13324898904111482
modified scripts/06_rq5_ml_rv_forecast.py +5 −0
@@ -426,6 +426,9 @@ def build_dataset_lake(qlog) -> pd.DataFrame:
426 426 spx_surface = pd.concat(frames, ignore_index=True)
427 427 rv_frames = [fdq_sql(LAKE_RV_SQL.format(year=y), max_rows=400, log=qlog) for y in range(2010, 2026)]
428 428 spx_rv = pd.concat(rv_frames, ignore_index=True)
429 + for fr in (spx_surface, spx_rv):
430 + for c in [c for c in fr.columns if c != 'trade_date']:
431 + fr[c] = pd.to_numeric(fr[c], errors='coerce')
429 432 print(f" SPX surface: {len(spx_surface)} days; SPX RV: {len(spx_rv)} sessions ({qlog.n} statements, {qlog.seconds:.0f}s)")
430 433 vix_daily = pd.read_parquet(config.DATA_PROCESSED / "vix_daily.parquet")[['trade_date', 'vix_close']]
431 434 for frame in (spx_rv, vix_daily, spx_surface):
@@ -604,6 +607,8 @@ def main_lake(log_path: str | None, no_fetch: bool, seeds: int) -> None:
604 607 ml['dataset_version_options'] = dataset_versions().get('options.eod')
605 608 ml.to_parquet(LAKE_PARQUET, index=False)
606 609 ml = pd.read_parquet(LAKE_PARQUET)
610 + for c in [c for c in ml.columns if c not in ('trade_date', 'dataset_version_options')]:
611 + ml[c] = pd.to_numeric(ml[c], errors='coerce')
607 612 evaluate_lake(ml, seeds=range(seeds))
608 613 print("\nRQ5 LAKE MODE COMPLETE.")
609 614
modified scripts/26_lake_total_returns.py +2 −1
@@ -52,7 +52,8 @@ def fetch(qlog) -> pd.DataFrame:
52 52 parts.append(d)
53 53 d = pd.concat(parts, ignore_index=True)
54 54 d['trade_date'] = pd.to_datetime(d['session_date'])
55 − d = d.sort_values('trade_date').drop_duplicates('trade_date')
55 + d['close'] = pd.to_numeric(d['close'], errors='coerce')
56 + d = d.sort_values('trade_date').drop_duplicates('trade_date').dropna(subset=['close'])
56 57 d['ret_total_simple'] = d['close'].pct_change()
57 58 frames.append(d[['ticker', 'lake_symbol', 'trade_date', 'close', 'ret_total_simple']])
58 59 print(f" {tk:6s} {len(d):5d} sessions {d['trade_date'].min().date()} -> {d['trade_date'].max().date()}", flush=True)
modified scripts/27_lake_magnet_data.py +7 −0
@@ -75,6 +75,13 @@ def fetch(qlog):
75 75 print(f" {tk}: max-OI strike rows {n:,} ({qlog.n} stmts, {time.time()-t0:.0f}s)", flush=True)
76 76 b = pd.concat(bars, ignore_index=True); b['trade_date'] = pd.to_datetime(b['session_date'])
77 77 o = pd.concat(ois, ignore_index=True); o['trade_date'] = pd.to_datetime(o['trade_date'])
78 + # raw extraction saved before any processing (the API returns JSON scalars as objects)
79 + b.to_parquet(config.DATA_PROCESSED / "magnet_lake_bars_raw.parquet", index=False)
80 + o.to_parquet(config.DATA_PROCESSED / "magnet_lake_oi_raw.parquet", index=False)
81 + for c in ('open', 'close'):
82 + b[c] = pd.to_numeric(b[c], errors='coerce')
83 + for c in ('top1_strike', 'top1_oi', 'total_oi'):
84 + o[c] = pd.to_numeric(o[c], errors='coerce')
78 85 m = o.merge(b[['ticker', 'trade_date', 'open', 'close']], on=['ticker', 'trade_date'], how='inner')
79 86 m = m.rename(columns={'open': 'day_open', 'close': 'day_close'})
80 87 m['dist_open_to_strike'] = np.abs(m['day_open'] - m['top1_strike']) / m['day_open']
added scripts/28_lake_cap_weighted_corr.py +180 −0
@@ -0,0 +1,180 @@
1 +#!/usr/bin/env python3
2 +# =============================================================================
3 +# Author: Simon-Pierre Boucher
4 +# Contact: contact@spboucher.ai
5 +# =============================================================================
6 +"""Step 28 — Capitalization-weighted implied correlation (package N-A3:
7 +WP07-EDIT-14).
8 +
9 +Shares outstanding (point-in-time, metric ``shares`` of the FinData OS
10 +fundamentals endpoint, one call per constituent) x daily raw close
11 +(``market_bars_stock_1d_raw``, one statement per constituent) give daily
12 +capitalization weights for the 29 constituents (FB before 2023 for META).
13 +Eq. (implcorr) generalized to weights:
14 + rho = (sigma_idx^2 - sum_i w_i^2 sigma_i^2) / ((sum_i w_i sigma_i)^2 - sum_i w_i^2 sigma_i^2),
15 +which reduces to the paper's equal-weight formula for w_i = 1/n. The RQ3
16 +headline cell (2010-2025, S2, spec A, extended controls, LPM NW(h)) is then
17 +re-estimated under equal and capitalization weights (raw and log ratio).
18 +Outputs: data/processed/cap_weights.parquet (regenerable),
19 +results/rq3_cap_weighted.csv, results/rq3_cap_weighted_diag.csv.
20 +"""
21 +
22 +import argparse
23 +import json
24 +import subprocess
25 +import sys
26 +
27 +import numpy as np
28 +import pandas as pd
29 +
30 +import _bootstrap # noqa: F401
31 +from wp7 import config
32 +from wp7.data_io import load_realized_vol
33 +from wp7.econometrics import add_constant, ols, r_squared, standardize
34 +from wp7.findata import FDQ, QueryLog, fdq_sql, lake_symbol, node_load
35 +
36 +OUT = config.DATA_PROCESSED / "cap_weights.parquet"
37 +SQL = "SELECT session_date, close FROM market_bars_stock_1d_raw WHERE symbol='{sym}' AND year BETWEEN {y0} AND {y1} ORDER BY session_date"
38 +
39 +
40 +def _fund_records(sym: str, tk: str) -> list:
41 + import time
42 + for attempt in range(3):
43 + out = subprocess.run([str(FDQ), "fund", sym, "", "shares"], capture_output=True, text=True)
44 + try:
45 + d = json.loads(out.stdout)
46 + except Exception:
47 + d = {}
48 + rec = [{'ticker': tk, 'available_at': r['available_at'], 'shares': r['value'], 'fiscal_period_end': r['fiscal_period_end']}
49 + for r in d.get('data', []) if r.get('metric') == 'shares' and r.get('value') and r.get('available_at')]
50 + if rec:
51 + return rec
52 + time.sleep(2 + 2 * attempt)
53 + return []
54 +
55 +
56 +def fetch_shares(tk: str) -> pd.DataFrame:
57 + """Point-in-time shares outstanding (all records of the fundamentals endpoint)."""
58 + rec = _fund_records('META' if tk == 'META' else tk, tk)
59 + if not rec and tk == 'META':
60 + rec = _fund_records('FB', tk)
61 + df = pd.DataFrame(rec, columns=['ticker', 'available_at', 'shares', 'fiscal_period_end'])
62 + if df.empty:
63 + print(f" {tk}: no shares records from the fundamentals endpoint -> excluded from the capitalization weights")
64 + return df
65 + df['available_at'] = pd.to_datetime(df['available_at'], utc=True).dt.tz_convert('America/New_York').dt.tz_localize(None).dt.normalize()
66 + return df.sort_values('available_at')
67 +
68 +
69 +def build_weights(qlog) -> pd.DataFrame:
70 + frames = []
71 + for tk in config.SPX_CONSTITUENTS_V2:
72 + sh = fetch_shares(tk)
73 + if sh.empty:
74 + continue
75 + spans = [('FB', 2010, 2022), ('META', 2023, 2025)] if tk == 'META' else [(lake_symbol(tk, 2010), 2010, 2025)]
76 + px = pd.concat([fdq_sql(SQL.format(sym=s, y0=a, y1=b), max_rows=6000, log=qlog) for s, a, b in spans], ignore_index=True)
77 + px['trade_date'] = pd.to_datetime(px['session_date'])
78 + px['close'] = pd.to_numeric(px['close'], errors='coerce')
79 + px = px.sort_values('trade_date').drop_duplicates('trade_date')[['trade_date', 'close']].dropna()
80 + # point-in-time shares: latest record available at or before the day; earliest record back-filled and flagged
81 + m = pd.merge_asof(px, sh[['available_at', 'shares']].rename(columns={'available_at': 'trade_date'}), on='trade_date', direction='backward')
82 + m['shares_backfilled'] = m['shares'].isna()
83 + m['shares'] = m['shares'].bfill()
84 + m['ticker'] = tk
85 + m['mcap'] = m['shares'] * m['close']
86 + frames.append(m)
87 + print(f" {tk:5s}: {len(px)} closes, {len(sh)} share records from {sh['available_at'].min().date() if len(sh) else 'n/a'}, "
88 + f"back-filled days {int(m['shares_backfilled'].sum())}", flush=True)
89 + return pd.concat(frames, ignore_index=True)
90 +
91 +
92 +def main():
93 + ap = argparse.ArgumentParser()
94 + ap.add_argument('--log', default=None)
95 + ap.add_argument('--no-fetch', action='store_true')
96 + args = ap.parse_args()
97 + config.ensure_output_dirs()
98 + if not args.no_fetch:
99 + print(f"M3U96a load: {node_load()}")
100 + qlog = QueryLog(args.log, 'N-A3')
101 + w = build_weights(qlog)
102 + w.to_parquet(OUT, index=False)
103 + print(f" {len(w):,} rows; {qlog.n} sql statements in {qlog.seconds:.0f}s (+ {len(config.SPX_CONSTITUENTS_V2)} fundamentals calls)")
104 + w = pd.read_parquet(OUT)
105 +
106 + from importlib import util
107 + spec = util.spec_from_file_location('s04', config.PROJECT_ROOT / 'scripts' / '04_rq3_correlation_divergence.py')
108 + s04 = util.module_from_spec(spec); spec.loader.exec_module(s04)
109 + of = pd.read_parquet(config.OPTIONS_FEATURES_PARQUET, columns=['ticker', 'trade_date', 'iv_atm_30d'])
110 + of['trade_date'] = pd.to_datetime(of['trade_date'])
111 + spx = of[of['ticker'] == 'SPX'][['trade_date', 'iv_atm_30d']].dropna().rename(columns={'iv_atm_30d': 'spx_iv_atm'})
112 + c = of[of['ticker'].isin(config.SPX_CONSTITUENTS_V2)][['ticker', 'trade_date', 'iv_atm_30d']].dropna()
113 + c = c.merge(w[['ticker', 'trade_date', 'mcap', 'shares_backfilled']], on=['ticker', 'trade_date'], how='left')
114 + c = c[c['mcap'].notna()]
115 + g = c.groupby('trade_date')
116 + rows = []
117 + for dt, d in g:
118 + n = len(d)
119 + if n < 20:
120 + continue
121 + s = d['iv_atm_30d'].values
122 + we = np.full(n, 1.0 / n)
123 + wc = d['mcap'].values / d['mcap'].sum()
124 + rec = {'trade_date': dt, 'n_constituents': n, 'share_backfilled': d['shares_backfilled'].mean(), 'max_cap_weight': wc.max()}
125 + for lab, ww in (('eq', we), ('cap', wc)):
126 + rec[f'sum_w_sigma_{lab}'] = np.sum(ww * s); rec[f'sum_w2_sigma2_{lab}'] = np.sum(ww ** 2 * s ** 2)
127 + rows.append(rec)
128 + ic = pd.DataFrame(rows).merge(spx, on='trade_date', how='inner')
129 + for lab in ('eq', 'cap'):
130 + ic[f'implied_corr_raw_{lab}'] = (ic['spx_iv_atm'] ** 2 - ic[f'sum_w2_sigma2_{lab}']) / (ic[f'sum_w_sigma_{lab}'] ** 2 - ic[f'sum_w2_sigma2_{lab}'])
131 + ic[f'implied_corr_{lab}'] = ic[f'implied_corr_raw_{lab}'].clip(0, 1)
132 + # realized correlation (equal-weight pairwise mean, as in the paper) and VIX from the v2 series
133 + v2 = pd.read_parquet(s04.V2_PARQUET)
134 + v2['trade_date'] = pd.to_datetime(v2['trade_date'])
135 + base = v2[['trade_date', 'realized_corr', 'vix_close', 'spx_iv_atm']].merge(ic.drop(columns=['spx_iv_atm']), on='trade_date', how='inner')
136 + spxrv = load_realized_vol(); spxrv = spxrv[spxrv['ticker'] == 'SPX'][['trade_date', 'daily_return', 'rv_daily']]
137 + diag = [{'statistic': 'days', 'value': len(base)}, {'statistic': 'mean_share_backfilled_shares', 'value': float(base['share_backfilled'].mean())},
138 + {'statistic': 'days_with_any_backfilled_shares', 'value': int((base['share_backfilled'] > 0).sum())},
139 + {'statistic': 'mean_max_cap_weight', 'value': float(base['max_cap_weight'].mean())},
140 + {'statistic': 'clip0_share_eq', 'value': float((base['implied_corr_raw_eq'] < 0).mean())},
141 + {'statistic': 'clip0_share_cap', 'value': float((base['implied_corr_raw_cap'] < 0).mean())},
142 + {'statistic': 'corr_eq_cap', 'value': float(base['implied_corr_eq'].corr(base['implied_corr_cap']))},
143 + {'statistic': 'mean_implied_corr_eq', 'value': float(base['implied_corr_eq'].mean())},
144 + {'statistic': 'mean_implied_corr_cap', 'value': float(base['implied_corr_cap'].mean())}]
145 + pd.DataFrame(diag).to_csv(config.RESULTS_DIR / "rq3_cap_weighted_diag.csv", index=False)
146 + print(pd.DataFrame(diag).to_string(index=False))
147 +
148 + out = []
149 + for lab in ('eq', 'cap'):
150 + m = base.copy()
151 + m['implied_corr'] = m[f'implied_corr_{lab}']
152 + m['corr_divergence'] = m['implied_corr'] - m['realized_corr']
153 + m['corr_ratio'] = m['implied_corr'] / m['realized_corr'].clip(0.01)
154 + m = s04.add_stress_definitions(m, spxrv)
155 + for sample, start in s04.SAMPLES.items():
156 + d0 = m[m['trade_date'] >= start]
157 + for definition in ('S1', 'S2'):
158 + for transform in ('raw', 'log'):
159 + for ctrl, cvars in s04.CONTROLS.items():
160 + d = d0.copy(); d['corr_ratio'] = s04.transform_key(d['corr_ratio'], transform)
161 + for h in s04.STRESS_HORIZONS:
162 + feats = ['corr_ratio'] + cvars
163 + sub = d[feats + [f'{definition}_{h}d']].dropna()
164 + X = add_constant(standardize(sub[feats].values)); y = sub[f'{definition}_{h}d'].values
165 + coefs = ols(X, y); se, t = s04.nw_t_vectorized(X, y, coefs, h)
166 + out.append({'weighting': 'equal' if lab == 'eq' else 'capitalization', 'sample': sample, 'definition': definition,
167 + 'transform': transform, 'controls': ctrl, 'horizon': f'{h}d', 'coefficient_pp': coefs[1] * 100,
168 + 't_nw': t[1], 'r2': r_squared(y, X @ coefs), 'n_obs': len(sub)})
169 + out = pd.DataFrame(out)
170 + out.to_csv(config.RESULTS_DIR / "rq3_cap_weighted.csv", index=False)
171 + h = out[(out['sample'] == '2010-2025') & (out['definition'] == 'S2') & (out['controls'] == 'extended')]
172 + print(h.pivot_table(index=['weighting', 'transform'], columns='horizon', values='t_nw').round(2).to_string())
173 + print("CAP-WEIGHTED IMPLIED CORRELATION COMPLETE.")
174 +
175 +
176 +if __name__ == "__main__":
177 + try:
178 + main()
179 + except FileNotFoundError as exc:
180 + print(exc, file=sys.stderr); sys.exit(2)
modified scripts/29_lake_greeks_decay.py +2 −0
@@ -91,6 +91,8 @@ def main():
91 91 g.to_parquet(OUT, index=False)
92 92 print(f" {len(g):,} rows; {qlog.n} statements in {qlog.seconds:.0f}s")
93 93 g = pd.read_parquet(OUT)
94 + for c in [c for c in g.columns if c not in ('ticker', 'trade_date', 'dte_bucket', 'dataset_version')]:
95 + g[c] = pd.to_numeric(g[c], errors='coerce')
94 96 rv = load_realized_vol()
95 97 rv = rv[rv['ticker'].isin(config.RQ4_TICKERS)]
96 98 gm = g.merge(rv[['ticker', 'trade_date', 'ret_1d', 'rv_fwd_1d']], on=['ticker', 'trade_date'], how='inner')
added scripts/32_rq5_clipping_variant.py +92 −0
@@ -0,0 +1,92 @@
1 +#!/usr/bin/env python3
2 +# =============================================================================
3 +# Author: Simon-Pierre Boucher
4 +# Contact: contact@spboucher.ai
5 +# =============================================================================
6 +"""Step 32 — RQ5 sensitivity of the OLS benchmarks to the test-feature
7 +clipping rule (package N-D2 follow-up).
8 +
9 +In the rolling design of scripts/06 --lake the regressors of the test year
10 +are clipped at the 1%/99% quantiles of the training window (the ECON-01
11 +scheme used for RQ2). For 2020 this caps RV_t far below its realized level,
12 +so HAR-RV cannot track the spike while the VIX benchmark, which is not a
13 +regressor, can. This script re-runs the OLS models (HAR-RV, HAR-RV + IV
14 +surface) on data/processed/rq5_spx_dataset_v2.parquet under three rules —
15 +features clipped at training quantiles (baseline), features unclipped,
16 +features and target log-transformed (log RV forecast, exp back) — and
17 +reports R2_OOS, QLIKE and DM against the VIX benchmark, overall and by
18 +test year. Output: results/rq5_clipping_variant.csv.
19 +"""
20 +
21 +import numpy as np
22 +import pandas as pd
23 +from numpy.linalg import lstsq
24 +
25 +import _bootstrap # noqa: F401
26 +from wp7 import config
27 +from wp7.econometrics import nw_mean_tstat
28 +
29 +SURFACE = ['iv_atm_1w', 'iv_atm_2w', 'iv_atm_1m', 'iv_atm_2m', 'iv_atm_3m', 'iv_atm_6m', 'skew_25d_1m', 'skew_25d_3m', 'skew_10d_1m',
30 + 'butterfly_1m', 'ts_slope_3m_1m', 'ts_slope_6m_1m', 'total_gamma_oi', 'net_gamma', 'total_vega_oi', 'avg_theta', 'pc_vol_ratio',
31 + 'pc_oi_ratio', 'total_volume', 'total_oi', 'avg_spread_pct']
32 +HAR = ['rv_daily', 'rv_w', 'rv_m']
33 +T = 'rv_fwd_1d'
34 +
35 +
36 +def ols_pred(tr, te, feats, target):
37 + X = np.column_stack([np.ones(len(tr)), tr[feats].values]); c, _, _, _ = lstsq(X, tr[target].values, rcond=None)
38 + return np.column_stack([np.ones(len(te)), te[feats].values]) @ c
39 +
40 +
41 +def main():
42 + ml = pd.read_parquet(config.DATA_PROCESSED / "rq5_spx_dataset_v2.parquet")
43 + for c in [c for c in ml.columns if c not in ('trade_date', 'dataset_version_options')]:
44 + ml[c] = pd.to_numeric(ml[c], errors='coerce')
45 + ml['year'] = pd.to_datetime(ml['trade_date']).dt.year
46 + rows = []
47 + for rule in ['features clipped at training quantiles', 'features unclipped', 'log-RV model']:
48 + preds = {'OLS: HAR-RV': [], 'OLS: HAR-RV + IV surface': []}
49 + ys, yrs, vix = [], [], []
50 + for Y in range(2016, 2026):
51 + tr = ml[ml['year'] < Y].copy(); te = ml[ml['year'] == Y].copy()
52 + cols = SURFACE + HAR
53 + if rule == 'features clipped at training quantiles':
54 + for c in cols + [T]:
55 + lo, hi = tr[c].quantile(0.01), tr[c].quantile(0.99); tr[c] = tr[c].clip(lo, hi)
56 + if c != T:
57 + te[c] = te[c].clip(lo, hi)
58 + elif rule == 'features unclipped':
59 + for c in cols + [T]:
60 + lo, hi = tr[c].quantile(0.01), tr[c].quantile(0.99); tr[c] = tr[c].clip(lo, hi)
61 + else:
62 + for c in SURFACE:
63 + lo, hi = tr[c].quantile(0.01), tr[c].quantile(0.99); tr[c] = tr[c].clip(lo, hi); te[c] = te[c].clip(lo, hi)
64 + for c in HAR + [T]:
65 + tr[c] = np.log(tr[c].clip(lower=1e-10)); te[c] = np.log(te[c].clip(lower=1e-10))
66 + for m, feats in [('OLS: HAR-RV', HAR), ('OLS: HAR-RV + IV surface', HAR + SURFACE)]:
67 + p = ols_pred(tr, te, feats, T)
68 + if rule == 'log-RV model':
69 + resid = tr[T].values - ols_pred(tr, tr, feats, T)
70 + p = np.exp(p + 0.5 * resid.var())
71 + floor = max(np.nanpercentile(ml.loc[ml['year'] < Y, T], 1), 1e-10)
72 + preds[m].append(np.maximum(p, floor))
73 + ys.append(ml.loc[te.index, T].values); yrs.append(np.full(len(te), Y)); vix.append(te['vix_implied_var'].values)
74 + y = np.concatenate(ys); yr = np.concatenate(yrs); v = np.concatenate(vix)
75 + def r2(yy, pp): return 1 - np.sum((yy - pp) ** 2) / np.sum((yy - yy.mean()) ** 2)
76 + def ql(yy, pp): return np.mean(yy / pp - np.log(yy / pp) - 1)
77 + for m in preds:
78 + p = np.concatenate(preds[m])
79 + d = (y - v) ** 2 - (y - p) ** 2
80 + _, _, t = nw_mean_tstat(d, 1)
81 + rows.append({'rule': rule, 'model': m, 'period': '2016-2025', 'r2_oos': r2(y, p), 'qlike': ql(y, p), 'r2_oos_vix': r2(y, v), 'qlike_vix': ql(y, v), 'dm_vs_vix_t': t, 'n': len(y)})
82 + for Y in range(2016, 2026):
83 + mk = yr == Y
84 + rows.append({'rule': rule, 'model': m, 'period': str(Y), 'r2_oos': r2(y[mk], p[mk]), 'qlike': ql(y[mk], p[mk]), 'r2_oos_vix': r2(y[mk], v[mk]), 'qlike_vix': ql(y[mk], v[mk]), 'dm_vs_vix_t': np.nan, 'n': int(mk.sum())})
85 + out = pd.DataFrame(rows)
86 + out.to_csv(config.RESULTS_DIR / "rq5_clipping_variant.csv", index=False)
87 + print(out[out['period'] == '2016-2025'].round(4).to_string(index=False))
88 + print(out[(out['model'] == 'OLS: HAR-RV') & (out['period'] != '2016-2025')].pivot(index='period', columns='rule', values='r2_oos').round(3).to_string())
89 +
90 +
91 +if __name__ == "__main__":
92 + main()
modified scripts/tables_e.py +10 −1
@@ -269,6 +269,13 @@ def tab_rq5_v2():
269 269 rows.append(rf"\multicolumn{{7}}{{@{{}}l}}{{\textit{{Panel B: version~1.1 specification re-run on the lake data (single 2020 split, seed 42) vs frozen rows}}}} \\")
270 270 for _, r in rep.iterrows():
271 271 rows.append(f"{r['Model']} & {f(r['r2_oos'],3)} & frozen {f(r['r2_oos_frozen'],3)} & $\\Delta$ {f(r['r2_diff'],3,plus=True)} & & & \\\\")
272 + cv = R / "rq5_clipping_variant.csv"
273 + if cv.exists():
274 + c = pd.read_csv(cv); c = c[c['period'] == '2016-2025']
275 + rows.append(r'\addlinespace')
276 + rows.append(rf"\multicolumn{{7}}{{@{{}}l}}{{\textit{{Panel C: OLS benchmarks under alternative treatments of the test-year regressors (\texttt{{scripts/32}})}}}} \\")
277 + for _, r in c.iterrows():
278 + rows.append(f"{r['model'].replace('OLS: ', '')} ({r['rule']}) & {f(r['r2_oos'],3)} & QLIKE {f(r['qlike'],3)} & VIX: {f(r['r2_oos_vix'],3)} / {f(r['qlike_vix'],3)} & & {f(r['dm_vs_vix_t'],2)} & \\\\")
272 279 hdr = [r'Model & $R^2_{OOS}$ (mean) & Seed range & 95\% bootstrap & DM vs HAR & DM vs VIX & CW vs HAR \\']
273 280 n_test = int(d['n_test'].iloc[0])
274 281 notes = (rf"Next-day SPX realized variance from the FinData OS lake (\texttt{{scripts/06 --lake}}: SPX option surface per quarter, 5-minute RV per year, VIX of \texttt{{scripts/19}}). "
@@ -276,7 +283,9 @@ def tab_rq5_v2():
276 283 rf"{num(n_test)} stacked forecasts; hyper-parameters chosen on a 2018--2019 validation block trained on 2010--2017 (\texttt{{results/rq5\_validation\_grid.csv}}); "
277 284 rf"ten seeds for the tree learners; seed-average forecasts enter the stationary-bootstrap interval (1{{,}}000 draws) and the DM (NW(1)) and Clark--West tests "
278 285 rf"(positive values favour the row model over HAR-RV / the VIX benchmark). Panel B: the frozen rows of commit c54a8e2 are reproduced within 0.03 only where stated "
279 − rf"(\texttt{{results/rq5\_v11\_replication.csv}}); the lake is a different data vintage from the 2026-08 stores.")
286 + rf"(\texttt{{results/rq5\_v11\_replication.csv}}); the lake is a different data vintage from the 2026-08 stores. Panel C: in Panel A the test-year regressors are clipped at the "
287 + rf"1\%/99\% quantiles of the training window (the scheme of Section~4.2), which caps $RV_t$ in 2020 far below its realized level and explains why HAR-RV trails the VIX "
288 + rf"benchmark there; with unclipped regressors or a log-RV specification HAR-RV is above the VIX benchmark (DM not significant) and the IV-surface model remains below HAR-RV under every rule.")
280 289 write("tab_rq5_v2", table('@{}lrrrrrr@{}', hdr, rows, notes, 'RQ5 re-estimated from the lake: rolling re-estimation, seeds, validation block and forecast tests.', 'tab:rq5_v2', size=r'\scriptsize'))
281 290
282 291
283 292
Menu

Theme
↑↓navigate ↵open escclose /⌘Kanywhere