# Data card — WP41 CHRONOS-X

## Source
FinData OS (`https://findata.spboucher.ai/api/v1`), a private lakehouse of FirstRate Data bars. Datasets used:
`market_bars.stock.1m.raw` (6.70 G rows, 2000→2026-10-30) and `market_bars.etf.1m.raw` (0.91 G rows). Stored bars are
raw (unadjusted); corporate actions come from the instrument master (`/instruments/{symbol}`). No fundamentals, news
or quotes are used. Finnhub: not used (optional vendor, not verified).

## Connector contract (`src/chronos_x/data/client.py`, `exports.py`)
- Discovery: `GET /catalog/datasets`, `/catalog/schema/bars`, `/openapi.json` (audited in `docs/PHASE_0_REPORT.md`).
- Small reads: `GET /market/bars` (1m ≤ 31 days, ≤ 10 000 rows/page, cursor, Arrow IPC) for tests and audits.
- Bulk: `POST /exports` (Parquet, dataset version pinned) per instrument for the whole sample window, polled, streamed
  to disk, schema-validated, sha256-hashed; manifest line with dataset version, request id, revision ids, time.
- Preserved per call: `X-Request-ID`, `X-Dataset-Id`, `X-Dataset-Version` (JSONL provenance logs).
- Rate: client token bucket (5–8 req/s), retries with jittered backoff on 429/5xx, disk quota guard (≥ 40 GB free).

## Conventions (verified on real rows, P0)
| Convention | Value |
|---|---|
| Timestamp | `timestamp_utc` = bar OPEN (UTC). Bar at 09:30 ET is complete at 09:31 ET |
| Regular session | 09:30–16:00 ET → 391 bars (slots 0..390); slot 390 = closing print, excluded from context and targets |
| Early close | 13:00 ET → 211 bars (slot 210 = closing print) |
| Extended hours | 04:00–20:00 ET present at the source; excluded (separate experiment, not run in v1.0) |
| Missing minutes | absent at the source (zero volume); kept missing, masked; never forward-filled |
| DST | handled via `exchange_calendars` XNYS opens/closes in UTC |
| Integrity | rows violating OHLC inequalities, non-positive prices or negative volume are dropped and counted (0 in the pilot) |
| Duplicates | highest `revision_id` kept per (instrument, timestamp) (0 duplicates in the pilot) |

## Corporate-action policy (`adjustments.py`)
Splits applied explicitly per session from the instrument master (equals the server's `adjustment=split`; live test).
Cash dividends are **not** applied: intraday log returns within a session are invariant to the dividend factor and any
dividend adjustment is window dependent. The overnight gap feature across an ex-dividend date is left raw. Mergers and
spin-offs are not available: a delisted series simply ends.

## Dense grid (`features.py`)
Per instrument: `base[S, 391, 5]` (log open/high/low/close, log1p volume, float32), `mask[S, 391]`, `in_session`,
`seas[S, 391]` (past-20-session mean log volume per slot, past only), `b5[S, 78, 5]` + `m5` (5-minute buckets of slots
5j..5j+4), `b30[S, 13, 5]` + `m30` (30-minute buckets), `meta.json` (session dates, slots, early closes, split factor,
overnight gaps, QA counts). ≈ 16 MB per instrument-decade.

## Origins and targets (`windows.py`)
Origin (s, m): completed bars 0..m of session s. Eligible iff m ≥ 30, m + 60 ≤ last target slot (389 / 209), bar m
present, ≥ 80 % of the next 60 slots present, s ≥ 1. Training: `origins_per_session` random origins per session; evaluation:
stride 10 minutes (36 origins per normal session). Targets y[h] = lc[m+h] − lc[prev present ≤ m+h−1], mask = present(m+h),
h = 1..60; cumulative H-minute return = Σ_{h≤H} y[h]·mask (exact log return of the last observed bar ≤ m+H);
realised variance = Σ y²·mask; tail event = |cum| > 2.5·σ_t·√H.

## Features (strictly causal)
Per bar (15): scaled return, range, upper/lower wick, body, close location, log volume, relative volume vs seasonal
profile, missing flag, session-boundary flag, log temporal gap, minute fraction, time to close, weekday, early-close flag.
Scaling: σ_t = RMS of present same-session 1-minute returns in the 390-bar window (floor 1e-5), computed at origin only.
Global (9): log σ_t, scaled overnight gap, minute fraction, time to close, weekday, early close, log RV30/σ², log RVday/σ²,
mean relative volume over 30 minutes. Branch B/C features use the same definitions on completed buckets with √5, √30 scaling.
Leakage gate: `tests/leakage/` (inputs invariant to any future bar; deliberate-future fixture rejected).

## Universe (`data/manifests/universe_main.json`, sha256 inside; v1 kept as `universe_main_v1_precoverage.json`)
- Selection date 2017-12-29; information set: instrument master (first/last date, delisted status) + daily bars
  2017-12-01..29 (ADV, daily volatility). Rules: first_date ≤ 2017-06-29; last_date ≥ 2018-07-02 (6-month survival
  requirement, documented bias); ADV ≥ 5 M USD; ≥ 15 daily bars in the ADV window.
- **Coverage gate (P1 audit, before any model tuning)**: regular-hours 1-minute coverage over 2018–2025 ≥ 80 %
  (`results/metrics/coverage_audit_candidates.csv`, 2 147 candidates audited; Spearman(ADV, coverage) = 0.76). The v1
  draw had a median coverage of 82 % and a minimum of 6 %; the gate leaves 1 152 candidates (947 stocks, 205 ETFs, 5 delisted).
- Allocation: proportional over asset class (75 % stocks / 25 % ETFs) × ADV tercile × volatility tercile; inside each
  stratum round-robin over SIC division (stocks, from SEC submissions; 25 "unknown") or ETF name category.
- 300 instruments; 60 (20 %) held out entirely (zero-shot); 1 delisted instrument in the sample. Pilot = 20 pooled
  instruments, feasibility = 50.
- Survivorship: only 5 delisted candidates passed the gate because FinData's delisted 1-minute archives are only partly
  ingested; the sample is therefore close to a survivor sample for 2018–2025 (limitation, reported in the paper).

## Splits
Chronological by session: train ≤ 2022-12-31, validation 2023, development test 2024, **final untouched test 2025**
(run once, `configs/final.yaml`). A one-session embargo at the start of each evaluation split removes overlap with the
last training targets. Held-out instruments are evaluated only with models trained on pooled instruments.

## License
FirstRate Data private use; see `docs/LICENSE.md`. No bars, grids or per-origin returns are committed or released.