SPB Git

spb/wp41_uqo

Public
7.1 KB · 101 lines latex
1\section{Results}2\label{sec:results}34\subsection{Pilot}5The pilot (20 instruments, September--December 2022, test on the last ten sessions, \num{4659} origins) validated6the pipeline from export to LaTeX table and gave a first, deliberately unweighted, reading: every volatility-aware7model improved on the random walk by 2--3\,\% in CRPS and the multi-resolution MRPT, its one-minute-only variant, a8TCN and an MLP were within 0.2\,\% of one another (Table~\ref{tab:pilot_crps_scaled}, Appendix). Nothing was tuned on it.910\subsection{Development test (2024): all models beat the random walk, by about four percent}11Table~\ref{tab:main_crps_scaled} reports the scaled CRPS of the cumulative return on the \num{986468} evaluation12origins of 2024 (240 pooled instruments, 24 origins per session), with trading-day cluster-bootstrap intervals, and13Table~\ref{tab:main_cmp_crps} the relative change against the random walk with its Diebold--Mariano statistic.14Figure~\ref{fig:main_crps} summarises both. Three facts stand out.1516First, the random walk with a trailing one-minute scale is beaten by every model at every horizon, by173.5--4.9\,\%, with DM statistics between $-25$ and $-72$: minute-level history carries real information about the18\emph{dispersion} of the next hour, and the improvement is essentially a volatility improvement (directional19accuracy of every location forecast stays at 0.50).2021Second, the neural family (MRPT-S, MRPT-B, TCN, MLP) is the best at five minutes22(scaled CRPS \crpsmainmrptsmallHfive{} for MRPT-S against \crpsmainharrvHfive{} for HAR-RV, a difference of23\pwmainmrptsmallvsharrvHfive{}\,\% with $t=\tpwmainmrptsmallvsharrvHfive{}$), whereas the feature-based volatility24regressions---HAR-RV, ridge, elastic net and LightGBM---are the best from fifteen minutes on25(\crpsmainharrvHsixzero{} against \crpsmainmrptsmallHsixzero{} at sixty minutes; MRPT-S is worse by26\pwmainmrptsmallvsharrvHsixzero{}\,\%, $t=\tpwmainmrptsmallvsharrvHsixzero{}$). MRPT-S beats GARCH(1,1)-$t$, EWMA, the TCN27and the MLP at every horizon (by 0.1--0.9\,\%, all Holm-adjusted $p<10^{-6}$).2829Third, and central to the paper, \textbf{Hypothesis~\ref{h1} is rejected in economic terms}: the multi-resolution30MRPT-S improves on its one-minute-only twin by \pwmainmrptsmallvsmrptsmallonemonlyHfive{}\,\% at five minutes and31\pwmainmrptsmallvsmrptsmallonemonlyHsixzero{}\,\% at sixty minutes. The differences are statistically detectable32($t=\tpwmainmrptsmallvsmrptsmallonemonlyHsixzero{}$ at $H=60$, \num{986468} origins over 252 days) but they are an order33of magnitude smaller than the gap between any neural model and the random walk, and smaller than the gap between34the neural models and HAR-RV at sixty minutes. Five days of 5-minute aggregates and twenty days of 30-minute35aggregates add almost nothing that the last 390 one-minute bars do not already contain. Size does not matter36either: the 20.7-million-parameter Base model and the 3.5-million-parameter Small model are indistinguishable37(\pwmainmrptbasevsmrptsmallHsixzero{}\,\% at $H=60$, Holm $p=\ppwmainmrptbasevsmrptsmallHsixzero{}$).383940\begin{figure}[htbp]\centering41\includegraphics[width=\textwidth]{figures/fig_main_crps.pdf}42\caption{Development test 2024 (240 instruments, \num{986468} origins). Panel A: scaled CRPS of the cumulative-return43forecast by horizon. Panel B: relative change versus the random walk. Source: \texttt{results/metrics/main/leaderboard\_test.csv}.}44\label{fig:main_crps}45\end{figure}4647\begin{figure}[htbp]\centering48\includegraphics[width=\textwidth]{figures/fig_main_calibration.pdf}49\caption{Empirical coverage of the central 50\,\%, 80\,\% and 95\,\% prediction intervals by horizon (dotted line = nominal), development test 2024.}50\label{fig:main_calibration}51\end{figure}5253\begin{figure}[htbp]\centering54\includegraphics[width=\textwidth]{figures/fig_main_pit.pdf}55\caption{PIT histograms at $H=60$ for the random walk, GARCH(1,1)-$t$, LightGBM and MRPT-S (dashed line = uniform density).}56\label{fig:main_pit}57\end{figure}5859\begin{figure}[htbp]\centering60\includegraphics[width=\textwidth]{figures/fig_main_training.pdf}61\caption{Training (moving average) and validation negative log-likelihood per optimiser step, main study.}62\label{fig:main_training}63\end{figure}6465\input{tables/main_crps_scaled}66\input{tables/main_compare_crps}6768\subsection{Calibration}69Table~\ref{tab:main_calib} and Figure~\ref{fig:main_calibration} show empirical coverage of the 50/80/95\,\%70central intervals and the PIT Kolmogorov--Smirnov statistic. The random walk is over-dispersed everywhere (50\,\%71intervals cover \covfiftymainnaiverwHsixzero{}\,\% of outcomes at sixty minutes). The Student-$t$ neural models and72GARCH/EWMA are nearly nominal at 95\,\% but over-dispersed in the centre (50\,\% coverage 0.55--0.59): their73degrees of freedom and scale produce tails that are right and a body that is too wide. HAR-RV, ridge and LightGBM74are the best-calibrated models from fifteen minutes on (0.49/0.80/0.96 at sixty minutes, KS 0.008) and the worst at75five minutes (50\,\% coverage 0.37, 95\,\% coverage 0.90): a lognormal variance regression on 390-minute features76cannot react to the last five minutes. PIT histograms (Figure~\ref{fig:main_pit}) make the same point visually.77Scores and calibration thus tell one consistent story: at the shortest horizon the sequence models' ability to78read the very last bars wins; at longer horizons a well-specified heterogeneous variance regression with a79kurtosis-matched Student-$t$ is both sharper and better calibrated.8081\input{tables/main_calibration}8283\subsection{Subgroups}84Splitting the MRPT-S versus HAR-RV contrast at sixty minutes by subgroup, HAR-RV's advantage is concentrated in85low-dollar-volume stocks ($+1.3$\,\%), low-volatility instruments ($+1.0$\,\%), transport and utilities ($+1.9$\,\%),86fixed-income ETFs ($+1.8$\,\%) and the stocks without a SIC code ($+2.8$\,\%); MRPT-S is better on broad-equity,87sector and volatility-linked ETFs ($-0.3$ to $-1.1$\,\%) and statistically tied on high-dollar-volume instruments.88Afternoon origins favour HAR-RV ($+1.0$\,\%), morning origins much less ($+0.2$\,\%). On stress days (top decile of89cross-sectional $\sigma_t$, \num{100846} origins) the ordering is unchanged and every model's coverage moves by at90most one percentage point: none of the models breaks under stress, and none gains a special advantage there.9192\subsection{Economic content (Hypothesis~\ref{h5})}93The predictive medians carry no usable directional information. Their correlation with the realised sixty-minute94return is 0.003 (MRPT-S), 0.025 (MRPT-B), 0.014 (TCN) and 0.007 (MLP), the typical magnitude of the median is below95one basis point, and a sign rule that trades only when the median exceeds an assumed round-trip cost of 1 basis96point trades 17\,\% of the origins for a net return of $0.0$ bp $[-1.7, +1.8]$; at 2.5--10 bp the traded share97falls below 4\,\% and every interval contains zero. We report this null without further sensitivity analysis.9899\subsection{Zero-shot transfer and the untouched 2025 period}100\IfFileExists{sections/results_heldout.tex}{\input{sections/results_heldout}}{\emph{(Held-out and final-period results are added when the runs complete.)}}101
Files · main
Menu

Theme