SPB Git

spb/wp41_uqo

Public
6.5 KB · 73 lines latex
1\section{Data}2\label{sec:data}34\subsection{Source and connector}5All bars come from FinData~OS, a private lakehouse of FirstRate Data one-minute bars for U.S. stocks and ETFs6(6.70 and 0.91 billion rows respectively at audit time, 2000--2026). The connector discovers the catalog, the7published Arrow schema and the OpenAPI document, reads small verification pages through the paginated bars endpoint8and performs bulk retrieval through asynchronous Parquet exports pinned to a dataset version. Every call keeps the9request identifier, dataset identifier and version; every export file is schema-validated and hashed. The license10is private use: no raw bar, grid or per-origin return leaves the compute node, and the release contains code,11aggregate results and synthetic fixtures only.1213\subsection{Conventions verified on real rows}14Timestamps are bar-\emph{open} instants in UTC: the bar stamped 09:30 ET covers 09:30:00--09:30:59 and is complete15at 09:31. A regular XNYS session is 09:30--16:00 ET, i.e.\ 391 bars on a normal day (slots $0,\dots,390$) and 211 on16a 13:00 early close; the last slot is the closing print, whose volume is one to two orders of magnitude larger than17a typical minute (6.7~million against a median of 0.12~million shares for AAPL on 2024-03-08). The closing print is18excluded from context windows and targets. Extended hours (04:00--20:00 ET) are present at the source and excluded19here. Zero-volume minutes are absent at the source and are never filled: missingness is a mask and a feature.20Daylight-saving transitions are handled through the exchange calendar (first regular bar at 14:30Z on 2024-03-08 and2113:30Z on 2024-03-11). Stored bars are raw; splits are applied explicitly per session from the instrument master,22which reproduces the server-side split-adjusted series exactly (AAPL 4:1 on 2020-08-31, ratio 4.000 on prices and23volume). Cash dividends are not applied: intraday log returns within a session are invariant to the dividend factor24and any dividend adjustment is window dependent; the overnight-gap feature across an ex-dividend date is left raw.25Mergers and spin-offs are unavailable; a delisted series simply ends.2627\subsection{Point-in-time universe and coverage gate}28The selection date is 2017-12-29. Eligible instruments have a first bar at least six months before that date and a29last bar at least six months after the sample start (a documented survival requirement), an average daily dollar30volume of at least 5~million USD over December 2017 and at least fifteen daily bars in that window. From31\num{2147} candidates we draw 300 instruments proportionally over asset class (75\,\% stocks, 25\,\% ETFs),32dollar-volume tercile and daily-volatility tercile, with a round-robin over SIC divisions (stocks, from SEC33submissions) or ETF name categories inside each stratum. A first draw revealed regular-hours minute coverage34between 6\,\% and 100\,\% over 2018--2025 (median 82\,\%): a quarter of the instruments traded in fewer than 60\,\% of35regular minutes, which makes per-minute targets mostly masked. Before any model was tuned we audited the coverage of36all candidates and imposed an availability filter of at least 80\,\% coverage over the window (\num{1152} candidates37remain; Spearman correlation between dollar volume and coverage 0.76); the second draw keeps the same strata and38seed. Sixty instruments (20\,\%) are held out entirely for zero-shot evaluation. Only five delisted candidates passed39the filters because FinData's delisted one-minute archives are only partly ingested: the universe is therefore close40to a survivor sample, a limitation we carry through the paper.4142\subsection{Sample, grid and targets}43The main sample spans 2018-01-02 to 2025-12-31 (\num{2011} sessions): \num{233958776} exported rows, of which44\num{220142835} fall in regular hours (93.7\,\% coverage on average, minimum 80.3\,\%; 13.8~million extended-hours45rows excluded; zero integrity rejections and duplicates; 242 vendor-flagged rows kept with their flags). Each46instrument is stored as a dense session~$\times$~391 grid of log open, high, low, close and log volume with a47presence mask, plus past-only 5-minute (78 per day) and 30-minute (13 per day) aggregates with their own masks and a48past-20-session seasonal profile of log volume and of squared returns. An origin $(s,m)$ is session $s$ at slot $m$,49with bars $0,\dots,m$ complete; it is eligible when $m\ge 30$, $m+60$ does not cross the last tradable slot (389, or50209 on an early close), bar $m$ is present and at least 80\,\% of the next 60 slots are present. Targets are51$y_h = \log C_{m+h}-\log C_{p(m+h-1)}$, where $p(\cdot)$ is the last present slot, with mask $\ind{\text{bar } m+h52\text{ present}}$, so that the masked sum over $h\le H$ is exactly the $H$-minute log return of the last observed53bar. Derived targets are the cumulative return $R_{t,H}$, the realised variance $\sum y_h^2$, high/low excursions,54and the tail event $\abs{R_{t,H}}>2.5\,\sigma_t\sqrt{H}$, where $\sigma_t$ is the trailing root-mean-square55one-minute return in the 390-bar window (floored at $10^{-5}$), computed from the past only.5657\subsection{Features}58Every bar carries fifteen causal features: scaled return, range, upper and lower wicks, body, close location,59log volume, relative volume against the seasonal profile, missing flag, session-boundary flag, log temporal gap60(which encodes overnight and weekend gaps), minute fraction, time to close, weekday and early-close flag. Returns and61ranges are divided by $\sigma_t$ (times $\sqrt{5}$ and $\sqrt{30}$ for the aggregates), so inputs are scale free. Nine62global history-only statistics---$\log\sigma_t$, the scaled overnight gap, the calendar position, the ratios of6330-minute and intraday realised variance to $\sigma_t^2$, and the mean relative volume over 30 minutes---feed a64global token and the regime embedding. A leakage gate in the test suite perturbs every bar after the origin and65asserts that no input changes; a quarantined fixture that deliberately uses the next bar is rejected by the same test.6667\subsection{Splits}68Sessions are split chronologically: training up to 2022-12-31, validation 2023, development test 2024 and an69untouched final test 2025 evaluated once, with a one-session embargo at the start of each evaluation block.70Held-out instruments are evaluated only with models trained on the pooled instruments. Training origins are two71random eligible slots per instrument-session (a 1.3-million-example epoch is not materialised: windows are indexed72into memory-mapped grids); evaluation origins are every ten minutes from 10:00, i.e.\ 36 per normal session.73
Files · main
Menu

Theme