1# PHASE 1 — Data report (2026-11-02)23## Completed4- Connector (`data/client.py`, `data/exports.py`): bearer auth from a mode-600 file, redaction, token-bucket rate5 limiting, retries with jitter, JSONL provenance (request id, dataset id/version, elapsed), Arrow page reads,6 asynchronous Parquet exports with schema validation, sha256 and manifest lines, disk-quota guard, resumable cache.7- Calendar (`data/calendar.py`): XNYS sessions in UTC, 391/211-slot grids, bar-open slot mapping (DST-safe).8- QA (`data/quality.py`): OHLC/price/volume integrity, duplicates, off-interval, flags, revisions, coverage.9- Corporate actions (`data/adjustments.py`): explicit session-based split adjustment, dividend policy (not applied).10- Dense grids and past-only aggregates (`data/features.py`); windows, origins, targets (`data/windows.py`).11- PIT universe with coverage gate (`data/universe.py`, `scripts/build_universe.py`, `scripts/audit_coverage.py`).12- Downloads and grids (`scripts/download_pilot.py`), detached node runner (`scripts/node_run.sh`).1314## Commands15```bash16.venv/bin/python scripts/audit_api.py17.venv/bin/python scripts/build_universe.py # v1 (results/logs/p1_universe.out)18.venv/bin/python scripts/audit_coverage.py --candidates --workers 4 # on M3U96b, 2 147 candidates, 335 s19.venv/bin/python scripts/build_universe.py --min-coverage 0.8 --coverage-file results/metrics/coverage_audit_candidates.csv --out universe_main_v2.json20.venv/bin/python scripts/download_pilot.py --config configs/pilot.yaml --symbols-from pilot --workers 321.venv/bin/python scripts/download_pilot.py --config configs/main.yaml --symbols-from all --workers 322```2324## Tests (laptop, `pytest`)2526 offline tests pass (calendar 3, grid/synthetic 5, leakage 5, model shapes 3, scores 6, copula 4) + 5 live contract26tests (`WP41_LIVE=1 pytest -m live`): schema/headers, regular-session counts on real rows (391/391/211), explicit27split == server split (AAPL 2020-08-31), export == paged bars (MSFT), grid from real rows incl. early close (SPY 2024-07-03).2829## Dataset coverage (measured)30| Set | Instruments | Window | Rows exported | Regular-hours rows | Coverage (mean / min) | Wall |31|---|---|---|---|---|---|---|32| Pilot (v2 universe) | 20 | 2022-09-01 → 2022-12-30 (84 sessions) | 702 674 | 626 992 | 96.0 % / 73.3 % | 24 s (3 workers) |33| Main (v2 universe) | 300 (240 pooled + 60 held out) | 2018-01-02 → 2025-12-31 (2 011 sessions) | 233 958 776 | 220 142 835 | 93.7 % / 80.3 % | 414 s download + 19 s grids |34Main cache: 3.5 GB Parquet (`~/wp41-cache/bars`), dataset versions 17 375 (stocks) / 5 162 (ETFs); 0 integrity drops,350 duplicates, 242 vendor-flagged rows (warnings, prices untouched), all `revision_id = 1`. 13.8 M extended-hours rows excluded from the regular grid.36The brief's upper bound (236 M rows) is matched by the measured 234 M exported rows (220 M regular-hours).3738## Universe decision (coverage gate, taken before any model tuning)39The v1 stratified draw (ADV ≥ 5 M USD at 2017-12-29) had a regular-hours minute coverage of 6–100 % (median 82 %):40a quarter of the instruments had fewer than 60 % of minutes traded. Sparse series make per-minute targets mostly41masked and shrink origin eligibility. The audit of all 2 147 candidates (`coverage_audit_candidates.csv`) motivated an42availability filter: coverage ≥ 80 % over 2018–2025 (1 152 candidates remain). The v2 draw keeps the same strata and seed.43Caveat: the filter uses the full-window activity (not returns); it is an availability, not a performance, criterion, and44it tilts the sample toward liquid instruments (Spearman(ADV, coverage) = 0.76). Only 5 delisted candidates passed.4546## Hardware47M3U96b (M3 Ultra, 96 GB): exports and grids; API over the LAN from M3U96a. Laptop (M5 Max): tests and docs.4849## Blockers / risks carried forward50- Survivorship (delisted archives partial in FinData).51- Extended hours not studied (v1.0).52- Dividend ex-dates not adjusted in the overnight gap (flagged policy).5354## Go / no-go55GO to P2/P3. Leakage gate passed (`tests/leakage`), provenance gate passed (manifests + JSONL logs), numerical gate56(scores, copula) passed.57