SPB Git

spb/wp41_uqo

Public
6.1 KB

# Data card — WP41 CHRONOS-X

# Source

FinData OS (https://findata.spboucher.ai/api/v1), a private lakehouse of FirstRate Data bars. Datasets used: market_bars.stock.1m.raw (6.70 G rows, 2000→2026-10-30) and market_bars.etf.1m.raw (0.91 G rows). Stored bars are raw (unadjusted); corporate actions come from the instrument master (/instruments/{symbol}). No fundamentals, news or quotes are used. Finnhub: not used (optional vendor, not verified).

# Connector contract (src/chronos_x/data/client.py, exports.py)

  • Discovery: GET /catalog/datasets, /catalog/schema/bars, /openapi.json (audited in docs/PHASE_0_REPORT.md).
  • Small reads: GET /market/bars (1m ≤ 31 days, ≤ 10 000 rows/page, cursor, Arrow IPC) for tests and audits.
  • Bulk: POST /exports (Parquet, dataset version pinned) per instrument for the whole sample window, polled, streamed to disk, schema-validated, sha256-hashed; manifest line with dataset version, request id, revision ids, time.
  • Preserved per call: X-Request-ID, X-Dataset-Id, X-Dataset-Version (JSONL provenance logs).
  • Rate: client token bucket (5–8 req/s), retries with jittered backoff on 429/5xx, disk quota guard (≥ 40 GB free).

# Conventions (verified on real rows, P0)

Convention Value
Timestamp timestamp_utc = bar OPEN (UTC). Bar at 09:30 ET is complete at 09:31 ET
Regular session 09:30–16:00 ET → 391 bars (slots 0..390); slot 390 = closing print, excluded from context and targets
Early close 13:00 ET → 211 bars (slot 210 = closing print)
Extended hours 04:00–20:00 ET present at the source; excluded (separate experiment, not run in v1.0)
Missing minutes absent at the source (zero volume); kept missing, masked; never forward-filled
DST handled via exchange_calendars XNYS opens/closes in UTC
Integrity rows violating OHLC inequalities, non-positive prices or negative volume are dropped and counted (0 in the pilot)
Duplicates highest revision_id kept per (instrument, timestamp) (0 duplicates in the pilot)

# Corporate-action policy (adjustments.py)

Splits applied explicitly per session from the instrument master (equals the server's adjustment=split; live test). Cash dividends are not applied: intraday log returns within a session are invariant to the dividend factor and any dividend adjustment is window dependent. The overnight gap feature across an ex-dividend date is left raw. Mergers and spin-offs are not available: a delisted series simply ends.

# Dense grid (features.py)

Per instrument: base[S, 391, 5] (log open/high/low/close, log1p volume, float32), mask[S, 391], in_session, seas[S, 391] (past-20-session mean log volume per slot, past only), b5[S, 78, 5] + m5 (5-minute buckets of slots 5j..5j+4), b30[S, 13, 5] + m30 (30-minute buckets), meta.json (session dates, slots, early closes, split factor, overnight gaps, QA counts). ≈ 16 MB per instrument-decade.

# Origins and targets (windows.py)

Origin (s, m): completed bars 0..m of session s. Eligible iff m ≥ 30, m + 60 ≤ last target slot (389 / 209), bar m present, ≥ 80 % of the next 60 slots present, s ≥ 1. Training: origins_per_session random origins per session; evaluation: stride 10 minutes (36 origins per normal session). Targets y[h] = lc[m+h] − lc[prev present ≤ m+h−1], mask = present(m+h), h = 1..60; cumulative H-minute return = Σ_{h≤H} y[h]·mask (exact log return of the last observed bar ≤ m+H); realised variance = Σ y²·mask; tail event = |cum| > 2.5·σ_t·√H.

# Features (strictly causal)

Per bar (15): scaled return, range, upper/lower wick, body, close location, log volume, relative volume vs seasonal profile, missing flag, session-boundary flag, log temporal gap, minute fraction, time to close, weekday, early-close flag. Scaling: σ_t = RMS of present same-session 1-minute returns in the 390-bar window (floor 1e-5), computed at origin only. Global (9): log σ_t, scaled overnight gap, minute fraction, time to close, weekday, early close, log RV30/σ², log RVday/σ², mean relative volume over 30 minutes. Branch B/C features use the same definitions on completed buckets with √5, √30 scaling. Leakage gate: tests/leakage/ (inputs invariant to any future bar; deliberate-future fixture rejected).

# Universe (data/manifests/universe_main.json, sha256 inside; v1 kept as universe_main_v1_precoverage.json)

  • Selection date 2017-12-29; information set: instrument master (first/last date, delisted status) + daily bars 2017-12-01..29 (ADV, daily volatility). Rules: first_date ≤ 2017-06-29; last_date ≥ 2018-07-02 (6-month survival requirement, documented bias); ADV ≥ 5 M USD; ≥ 15 daily bars in the ADV window.
  • Coverage gate (P1 audit, before any model tuning): regular-hours 1-minute coverage over 2018–2025 ≥ 80 % (results/metrics/coverage_audit_candidates.csv, 2 147 candidates audited; Spearman(ADV, coverage) = 0.76). The v1 draw had a median coverage of 82 % and a minimum of 6 %; the gate leaves 1 152 candidates (947 stocks, 205 ETFs, 5 delisted).
  • Allocation: proportional over asset class (75 % stocks / 25 % ETFs) × ADV tercile × volatility tercile; inside each stratum round-robin over SIC division (stocks, from SEC submissions; 25 "unknown") or ETF name category.
  • 300 instruments; 60 (20 %) held out entirely (zero-shot); 1 delisted instrument in the sample. Pilot = 20 pooled instruments, feasibility = 50.
  • Survivorship: only 5 delisted candidates passed the gate because FinData's delisted 1-minute archives are only partly ingested; the sample is therefore close to a survivor sample for 2018–2025 (limitation, reported in the paper).

# Splits

Chronological by session: train ≤ 2022-12-31, validation 2023, development test 2024, final untouched test 2025 (run once, configs/final.yaml). A one-session embargo at the start of each evaluation split removes overlap with the last training targets. Held-out instruments are evaluated only with models trained on pooled instruments.

# License

FirstRate Data private use; see docs/LICENSE.md. No bars, grids or per-origin returns are committed or released.

Files · main
Menu

Theme