# docs/PROGRESS.md — phase journal (actual commands and results)

## Phase 0 — Audit (2026-11-02) — GATE PASSED
Plan: audit the UQO repository, FinData, hardware, competitors, literature and licensing; write the four audit documents.

Executed (laptop, 14:05–14:40 ET):
- Read `Recherche-UQO/README.md`, `HARMONISATION.md`, `_modele-working-paper/*`, WP 12/39/33 READMEs and `PROGRESS.md`.
  Result: WP 40 is free; conventions recorded in `docs/audit.md` §1.
- FinData: `fdq health`, `fdq datasets`, `GET /openapi.json` (saved), `GET /market/bars?format=arrow` (SPY, DST weekend),
  `GET /market/sessions` (SPY, Thanksgiving week 2024 → 211 expected bars on 11-29), `POST /exports` (SPY+QQQ Jan 2024:
  34 607 rows, 1.5 s, version 5162, SHA-256 recorded), `GET /query/views`, `GET /coverage/symbols`. Results in §2.
- Hardware: `sw_vers`, `sysctl`, `system_profiler SPDisplaysDataType`, `xcodebuild -version`, Metal device query
  program (`docs/evidence/` test), 29-node parallel SSH probe (22 reachable; files in `docs/evidence/nodes/`).
- Metal: trivial kernel compiled at runtime (34.6 ms) and executed (1 M squares, 0.029 ms GPU); `double` kernel rejected
  by both runtime and offline compilers; Metal Toolchain 27B47 downloaded (`xcodebuild -downloadComponent MetalToolchain`).
- Toolchain installs: `brew install eigen apache-arrow catch2 nlohmann-json ninja` (background), R
  `install.packages(c("highfrequency","rugarch","xts","lmtest"))` → highfrequency 1.0.3, rugarch 1.5.6, xts 0.14.3
  (lmtest failed: Java runtime missing for a dependency; not needed), `uv venv --python 3.12` + pip install of the
  Python reference stack (versions in `docs/capabilities.md`).
- Literature: 39 key references resolved on Crossref (`docs/literature/crossref_resolved.json`); preprint hits were
  re-resolved to journal versions; three DOIs verified directly (Econometrica 2008 realized kernels, SPA 2009
  pre-averaging, OUP 2010 semivariance). Still to resolve: Hansen–Lunde–Nason 2011 Econometrica MCS DOI; software
  citations (Arrow, Eigen, nanobind, Metal docs) to be added with URLs/versions.
- Legal: FirstRate private-use license (no redistribution), MIT for RAID, Metal proprietary → optional backend.

Gate memo (scope and feasibility): feasible on the laptop with the measured toolchain; FP64-on-GPU impossible → FP32
strategy; FinData provides all Tier A inputs (1-minute OHLCV bars, versions, sessions, exports); no trade/quote data →
Tier C order-flow methods excluded; cluster usable for E7 with a static bundle; 7 nodes unreachable. Decisions D-001…D-010.

## Phase 1 — Foundation (2026-11-02, 14:40–15:40 ET) — GATE PASSED
Plan: CMake/Ninja build, C++20 core types, CPU backend, data abstractions (calendar, bars, returns, aggregation),
Arrow/Parquet reader, OLS/robust covariance, realized measures, GARCH-family QML (needed by the backend contract),
Catch2 tests, nanobind Python package, CLI, docs, CI.

Executed:
- `cmake --preset macos-release && cmake --build --preset macos-release` (Apple clang 21, Eigen 5.0.1 + Accelerate,
  Arrow 26.0.0, Catch2 3.16.1, nlohmann-json 3.12). Metal sources absent → the build reports "Metal backend sources not
  present yet (Phase 3)" and continues CPU-only (`raid devices` lists the CPU only).
- `ctest --preset macos-release` → **26/26 test cases, 314 assertions** (unit: core, calendar, stats, OLS,
  realized/GARCH; reference: statsmodels 0.15 OLS/HC0/HC3/HAC/cluster at 1e−9, scipy t p-values at 1e−12,
  exchange_calendars 4.13 XNYS 2000–2026 exact session/holiday/early-close sets, numpy realized truth at 1e−12, arch 8.0
  GARCH/GJR/EGARCH/Student-t log-likelihood within 5e−4 and parameters within 5 % + 0.01, forecasts within 5 %).
- Python: `cmake -DRAID_BUILD_PYTHON=ON` builds `raid._core` (nanobind 3.1.0, Python 3.12); `pytest python/tests` →
  7 passed (OLS HC3 vs statsmodels, realized vs numpy, GARCH convergence, calendar, bar validation).
- Real-data integration (file not committed): `raid validate /tmp/raid-export-test.parquet` (SPY+QQQ 1m January 2024,
  FinData export v5162) → 21 sessions each, coverage 1.0 (391 regular bars incl. the 16:00 print), 0 duplicates,
  0 errors; `raid estimate realized … --backend cpu` → 21 sessions × 390 returns, mean RV(SPY) 2.71e−5, 0.25 ms.
- Bugs found and fixed during the phase (recorded for the paper's engineering section): thread-pool completion race
  (local mutex destroyed before the last job released it → SIGABRT under load); NLL rescaling sign; EGARCH ω
  wrongly log-transformed; `tgamma` overflow for ν > 340 (→ `lgamma`); fixture truth computed on unrounded prices.
- Environment manifest: `experiments/manifests/environment_<host>.json` via `scripts/record_environment.sh`.
- Repository: bare `gitsrv:srv/git/wp40_uqo.git` created with the SPB Git post-receive hook; `origin` set; pushed.

Gate evidence: clean build on the macOS preset, all tests green, docs (`docs/architecture/overview.md`,
`docs/methods/registry.md`, `docs/api/cpp.md`, README) written. Open items carried to later phases: `bars.aggregate`
vs FinData derived 5m (Phase 2), GARCH backcast option (Phase 4), HAR/bootstrap (Phase 4), Metal (Phase 3).

## Phase 2 — Data (2026-11-02, 15:00–15:30 ET) — GATE PASSED
Plan: secure FinData adapter (C++/libcurl), `raid data` CLI, Python access, offline + opt-in live tests, provenance,
calendar/aggregation checks against the data provider.

Executed:
- `include/raid/data/findata.hpp`, `src/data/findata.cpp`: bearer auth from env/file only, TLS, redacted logs,
  token bucket (5 req/s, burst 10), retries with backoff + `Retry-After`, Arrow-page pagination (`X-Next-Cursor`),
  31/186/3650-day chunking, async exports (POST → poll → download, `.part` rename), SHA-256 cache with `.meta.json`
  (dataset version, request ids, key fingerprint), provenance export; `raid data catalog|health|sessions|coverage|bars|
  export|fund|query|log`. Python: `raid.findata.{bars, export, health, catalog, dataset, sessions, coverage,
  fundamentals, query, read_parquet}` (GIL released).
- Offline tests (`[findata]`): chunking caps, token-bucket timing, offline refusal / missing key, synthetic Arrow IPC +
  Parquet page decoding (schema v1 metadata) → 4 passed, 2 live cases skipped. Fixture bug found: pyarrow IPC stream
  truncated when the file object was not closed → explicit `with open(...)` (recorded).
- Live tests (`RAID_LIVE=1`, key file, cache in `datasets/cache/`, AAPL 2024-11-25…29): health ok; catalog contains
  `market_bars.stock.1m.raw`; `expected_bars` 211 on 2024-11-29 == RAID calendar; bars pages (2 634 rows, 4 sessions,
  dataset version 17375) == export rows (62 933-byte Parquet, sha256 recorded) bar for bar; second call served from
  cache; RAID calendar `expected_regular_bars` == FinData `expected_bars` for every session; **aggregate 1m→5m:
  extended policy reproduces FinData's derived 5m exactly (280/280 OHLCV)**; regular policy differs only in the 16:00
  bin (FinData adds 16:01–16:04 post-close minutes) → convention documented in the registry (D-008 unchanged: the
  regular session ends with the closing print). Total 60 assertions, all passed; ≈ 25 requests.
- Python smoke: `raid.findata.bars` → realized variance per session from live bars; export + `read_parquet`
  (`zero_copy_possible = true` for single-chunk files).

Gate evidence: offline tests in `ctest`; live run transcript above; no vendor data in git (`scripts/check_no_secrets.sh`).
Open: least-privilege key for benchmarks (needs `findata create-key` on M3U96a — owner action, optional).

## Phase 3 — Numerical primitives (2026-11-02, 15:40–16:30 ET) — GATE PASSED
Plan: Metal backend with FP32 "double-float" (error-free-transform) accumulation, fused realized kernel, rolling sums,
regression sufficient statistics, batched GARCH likelihood; CPU/GPU agreement tests; measured break-even analysis.

Executed:
- `backends/metal/shaders/raid_kernels.metal` (TwoSum/Fast2Sum double-float, deterministic threadgroup tree reductions,
  chunked partial sums combined in FP64 on the CPU) and `backends/metal/host/metal_backend.mm` (device discovery,
  prebuilt `raid.metallib` with runtime-MSL fallback, pipeline cache, buffer pool by power-of-two size class, GPU
  timing from `GPUStartTime/GPUEndTime`, `-fno-fast-math` / `fastMathEnabled = NO` so the compensation is not optimised
  away). Metal has no `lgamma`/`log1p`: Student-t constants are computed in FP64 on the host and passed per parameter
  vector.
- Tests `[metal]` (6 cases, 6 411 assertions): fused and single-statistic accumulators vs CPU FP64 within 1e−6
  (Σx⁴ 1e−5) for n = 390/3 900/39 000 across chunk boundaries (4 096); bitwise determinism across runs; rolling sums
  1e−5; regression statistics 1e−6 (and the solve from GPU statistics agrees with CPU QR to 1e−5); batched GARCH NLL
  2e−5 relative (GJR + Student-t fixed ν 5e−5), argmin over a 16-point grid identical; +inf on invalid parameters.
  Full suite: `ctest` 38/38 (2 live skipped), `pytest` 7/7.
- `bench_bench_primitives` (laptop M5 Max 40-core GPU, 18 CPU threads, load ≈ 3–4 from other sessions, no thermal
  warning, 5 repetitions, shuffled order, 2 warm-ups; raw records `experiments/results/raw/bench_primitives.jsonl`
  + manifest; summary `experiments/results/bench_primitives_summary.{json,csv}`, figure
  `fig_bench_primitives_throughput.pdf`). Medians of wall-clock including FP64→FP32 conversion and buffer traffic;
  verdicts use non-overlapping bootstrap 95 % CIs of the median:
  - realized_fused: Metal wins from 39 000 elements (100 × 390: 2.4×), 2.0–3.4× vs 18-thread CPU at 3.9 M–39 M
    elements (max 3.4× at 1 000 × 39 000); ties at 390 000 for 10 × 39 000 and 100 × 3 900 shapes; CPU 1 thread is
    27–40× slower than Metal at 39 M.
  - regression_stats (k = 4): break-even ≈ 390 000 elements for a single long series (3.9×); ties for most shapes
    between 390 k and 3.9 M; 1.1–1.4× at 39 M. CPU wins below 39 000 elements.
  - rolling_sumsq (w = 22, direct O(w) kernel): **CPU wins at every size** (Metal 0.4–0.9× of CPU-MT); the GPU kernel
    is memory/arithmetic-bound by its O(w) design and the CPU running sum is O(1) per element — a negative result kept
    for the paper (a scan-based kernel is future work).
  - garch_nll: with one parameter vector per series (P = 1) the **CPU wins everywhere** (0.01–0.33×; the recursion is
    sequential and the GPU is launch-bound); with P = 64 parameter vectors per series Metal wins from 100 series
    (1.9×) up to 7.9× at 1 000 × 3 900 — batched likelihood evaluation is where the GPU pays off.
  - Device time is 5–20× smaller than wall time for the reductions: FP64→FP32 conversion and buffer fills dominate
    ("unified memory is not free computation"); E6 will add the FP32-resident ablation.
  - Shader library: prebuilt metallib loaded in 0.23 ms; first kernel call 14 ms (pipeline creation); runtime MSL
    compilation of the bundle is measured separately in E7 (toolchain-less nodes).

Gate evidence: accuracy tests and the measured break-even table above; no claim beyond the tested shapes.