# PHASE 1 — Data report (2026-11-02)

## Completed
- Connector (`data/client.py`, `data/exports.py`): bearer auth from a mode-600 file, redaction, token-bucket rate
  limiting, retries with jitter, JSONL provenance (request id, dataset id/version, elapsed), Arrow page reads,
  asynchronous Parquet exports with schema validation, sha256 and manifest lines, disk-quota guard, resumable cache.
- Calendar (`data/calendar.py`): XNYS sessions in UTC, 391/211-slot grids, bar-open slot mapping (DST-safe).
- QA (`data/quality.py`): OHLC/price/volume integrity, duplicates, off-interval, flags, revisions, coverage.
- Corporate actions (`data/adjustments.py`): explicit session-based split adjustment, dividend policy (not applied).
- Dense grids and past-only aggregates (`data/features.py`); windows, origins, targets (`data/windows.py`).
- PIT universe with coverage gate (`data/universe.py`, `scripts/build_universe.py`, `scripts/audit_coverage.py`).
- Downloads and grids (`scripts/download_pilot.py`), detached node runner (`scripts/node_run.sh`).

## Commands
```bash
.venv/bin/python scripts/audit_api.py
.venv/bin/python scripts/build_universe.py                      # v1 (results/logs/p1_universe.out)
.venv/bin/python scripts/audit_coverage.py --candidates --workers 4   # on M3U96b, 2 147 candidates, 335 s
.venv/bin/python scripts/build_universe.py --min-coverage 0.8 --coverage-file results/metrics/coverage_audit_candidates.csv --out universe_main_v2.json
.venv/bin/python scripts/download_pilot.py --config configs/pilot.yaml --symbols-from pilot --workers 3
.venv/bin/python scripts/download_pilot.py --config configs/main.yaml --symbols-from all --workers 3
```

## Tests (laptop, `pytest`)
26 offline tests pass (calendar 3, grid/synthetic 5, leakage 5, model shapes 3, scores 6, copula 4) + 5 live contract
tests (`WP41_LIVE=1 pytest -m live`): schema/headers, regular-session counts on real rows (391/391/211), explicit
split == server split (AAPL 2020-08-31), export == paged bars (MSFT), grid from real rows incl. early close (SPY 2024-07-03).

## Dataset coverage (measured)
| Set | Instruments | Window | Rows exported | Regular-hours rows | Coverage (mean / min) | Wall |
|---|---|---|---|---|---|---|
| Pilot (v2 universe) | 20 | 2022-09-01 → 2022-12-30 (84 sessions) | 702 674 | 626 992 | 96.0 % / 73.3 % | 24 s (3 workers) |
| Main (v2 universe) | 300 (240 pooled + 60 held out) | 2018-01-02 → 2025-12-31 (2 011 sessions) | 233 958 776 | 220 142 835 | 93.7 % / 80.3 % | 414 s download + 19 s grids |
Main cache: 3.5 GB Parquet (`~/wp41-cache/bars`), dataset versions 17 375 (stocks) / 5 162 (ETFs); 0 integrity drops,
0 duplicates, 242 vendor-flagged rows (warnings, prices untouched), all `revision_id = 1`. 13.8 M extended-hours rows excluded from the regular grid.
The brief's upper bound (236 M rows) is matched by the measured 234 M exported rows (220 M regular-hours).

## Universe decision (coverage gate, taken before any model tuning)
The v1 stratified draw (ADV ≥ 5 M USD at 2017-12-29) had a regular-hours minute coverage of 6–100 % (median 82 %):
a quarter of the instruments had fewer than 60 % of minutes traded. Sparse series make per-minute targets mostly
masked and shrink origin eligibility. The audit of all 2 147 candidates (`coverage_audit_candidates.csv`) motivated an
availability filter: coverage ≥ 80 % over 2018–2025 (1 152 candidates remain). The v2 draw keeps the same strata and seed.
Caveat: the filter uses the full-window activity (not returns); it is an availability, not a performance, criterion, and
it tilts the sample toward liquid instruments (Spearman(ADV, coverage) = 0.76). Only 5 delisted candidates passed.

## Hardware
M3U96b (M3 Ultra, 96 GB): exports and grids; API over the LAN from M3U96a. Laptop (M5 Max): tests and docs.

## Blockers / risks carried forward
- Survivorship (delisted archives partial in FinData).
- Extended hours not studied (v1.0).
- Dividend ex-dates not adjusted in the overnight gap (flagged policy).

## Go / no-go
GO to P2/P3. Leakage gate passed (`tests/leakage`), provenance gate passed (manifests + JSONL logs), numerical gate
(scores, copula) passed.