PHASE 1 — Data report (2026-11-02)
Completed
- Connector (
data/client.py,data/exports.py): bearer auth from a mode-600 file, redaction, token-bucket rate limiting, retries with jitter, JSONL provenance (request id, dataset id/version, elapsed), Arrow page reads, asynchronous Parquet exports with schema validation, sha256 and manifest lines, disk-quota guard, resumable cache. - Calendar (
data/calendar.py): XNYS sessions in UTC, 391/211-slot grids, bar-open slot mapping (DST-safe). - QA (
data/quality.py): OHLC/price/volume integrity, duplicates, off-interval, flags, revisions, coverage. - Corporate actions (
data/adjustments.py): explicit session-based split adjustment, dividend policy (not applied). - Dense grids and past-only aggregates (
data/features.py); windows, origins, targets (data/windows.py). - PIT universe with coverage gate (
data/universe.py,scripts/build_universe.py,scripts/audit_coverage.py). - Downloads and grids (
scripts/download_pilot.py), detached node runner (scripts/node_run.sh).
Commands
.venv/bin/python scripts/audit_api.py
.venv/bin/python scripts/build_universe.py # v1 (results/logs/p1_universe.out)
.venv/bin/python scripts/audit_coverage.py --candidates --workers 4 # on M3U96b, 2 147 candidates, 335 s
.venv/bin/python scripts/build_universe.py --min-coverage 0.8 --coverage-file results/metrics/coverage_audit_candidates.csv --out universe_main_v2.json
.venv/bin/python scripts/download_pilot.py --config configs/pilot.yaml --symbols-from pilot --workers 3
.venv/bin/python scripts/download_pilot.py --config configs/main.yaml --symbols-from all --workers 3 Tests (laptop, pytest)
26 offline tests pass (calendar 3, grid/synthetic 5, leakage 5, model shapes 3, scores 6, copula 4) + 5 live contract
tests (WP41_LIVE=1 pytest -m live): schema/headers, regular-session counts on real rows (391/391/211), explicit
split == server split (AAPL 2020-08-31), export == paged bars (MSFT), grid from real rows incl. early close (SPY 2024-07-03).
Dataset coverage (measured)
| Set | Instruments | Window | Rows exported | Regular-hours rows | Coverage (mean / min) | Wall |
|---|---|---|---|---|---|---|
| Pilot (v2 universe) | 20 | 2022-09-01 → 2022-12-30 (84 sessions) | 702 674 | 626 992 | 96.0 % / 73.3 % | 24 s (3 workers) |
| Main (v2 universe) | 300 (240 pooled + 60 held out) | 2018-01-02 → 2025-12-31 (2 011 sessions) | 233 958 776 | 220 142 835 | 93.7 % / 80.3 % | 414 s download + 19 s grids |
Main cache: 3.5 GB Parquet (~/wp41-cache/bars), dataset versions 17 375 (stocks) / 5 162 (ETFs); 0 integrity drops, |
||||||
0 duplicates, 242 vendor-flagged rows (warnings, prices untouched), all revision_id = 1. 13.8 M extended-hours rows excluded from the regular grid. |
||||||
| The brief's upper bound (236 M rows) is matched by the measured 234 M exported rows (220 M regular-hours). |
Universe decision (coverage gate, taken before any model tuning)
The v1 stratified draw (ADV ≥ 5 M USD at 2017-12-29) had a regular-hours minute coverage of 6–100 % (median 82 %):
a quarter of the instruments had fewer than 60 % of minutes traded. Sparse series make per-minute targets mostly
masked and shrink origin eligibility. The audit of all 2 147 candidates (coverage_audit_candidates.csv) motivated an
availability filter: coverage ≥ 80 % over 2018–2025 (1 152 candidates remain). The v2 draw keeps the same strata and seed.
Caveat: the filter uses the full-window activity (not returns); it is an availability, not a performance, criterion, and
it tilts the sample toward liquid instruments (Spearman(ADV, coverage) = 0.76). Only 5 delisted candidates passed.
Hardware
M3U96b (M3 Ultra, 96 GB): exports and grids; API over the LAN from M3U96a. Laptop (M5 Max): tests and docs.
Blockers / risks carried forward
- Survivorship (delisted archives partial in FinData).
- Extended hours not studied (v1.0).
- Dividend ex-dates not adjusted in the overnight gap (flagged policy).
Go / no-go
GO to P2/P3. Leakage gate passed (tests/leakage), provenance gate passed (manifests + JSONL logs), numerical gate
(scores, copula) passed.