1# docs/PROGRESS.md — phase journal (actual commands and results)23## Phase 0 — Audit (2026-11-02) — GATE PASSED4Plan: audit the UQO repository, FinData, hardware, competitors, literature and licensing; write the four audit documents.56Executed (laptop, 14:05–14:40 ET):7- Read `Recherche-UQO/README.md`, `HARMONISATION.md`, `_modele-working-paper/*`, WP 12/39/33 READMEs and `PROGRESS.md`.8 Result: WP 40 is free; conventions recorded in `docs/audit.md` §1.9- FinData: `fdq health`, `fdq datasets`, `GET /openapi.json` (saved), `GET /market/bars?format=arrow` (SPY, DST weekend),10 `GET /market/sessions` (SPY, Thanksgiving week 2024 → 211 expected bars on 11-29), `POST /exports` (SPY+QQQ Jan 2024:11 34 607 rows, 1.5 s, version 5162, SHA-256 recorded), `GET /query/views`, `GET /coverage/symbols`. Results in §2.12- Hardware: `sw_vers`, `sysctl`, `system_profiler SPDisplaysDataType`, `xcodebuild -version`, Metal device query13 program (`docs/evidence/` test), 29-node parallel SSH probe (22 reachable; files in `docs/evidence/nodes/`).14- Metal: trivial kernel compiled at runtime (34.6 ms) and executed (1 M squares, 0.029 ms GPU); `double` kernel rejected15 by both runtime and offline compilers; Metal Toolchain 27B47 downloaded (`xcodebuild -downloadComponent MetalToolchain`).16- Toolchain installs: `brew install eigen apache-arrow catch2 nlohmann-json ninja` (background), R17 `install.packages(c("highfrequency","rugarch","xts","lmtest"))` → highfrequency 1.0.3, rugarch 1.5.6, xts 0.14.318 (lmtest failed: Java runtime missing for a dependency; not needed), `uv venv --python 3.12` + pip install of the19 Python reference stack (versions in `docs/capabilities.md`).20- Literature: 39 key references resolved on Crossref (`docs/literature/crossref_resolved.json`); preprint hits were21 re-resolved to journal versions; three DOIs verified directly (Econometrica 2008 realized kernels, SPA 200922 pre-averaging, OUP 2010 semivariance). Still to resolve: Hansen–Lunde–Nason 2011 Econometrica MCS DOI; software23 citations (Arrow, Eigen, nanobind, Metal docs) to be added with URLs/versions.24- Legal: FirstRate private-use license (no redistribution), MIT for RAID, Metal proprietary → optional backend.2526Gate memo (scope and feasibility): feasible on the laptop with the measured toolchain; FP64-on-GPU impossible → FP3227strategy; FinData provides all Tier A inputs (1-minute OHLCV bars, versions, sessions, exports); no trade/quote data →28Tier C order-flow methods excluded; cluster usable for E7 with a static bundle; 7 nodes unreachable. Decisions D-001…D-010.2930## Phase 1 — Foundation (2026-11-02, 14:40–15:40 ET) — GATE PASSED31Plan: CMake/Ninja build, C++20 core types, CPU backend, data abstractions (calendar, bars, returns, aggregation),32Arrow/Parquet reader, OLS/robust covariance, realized measures, GARCH-family QML (needed by the backend contract),33Catch2 tests, nanobind Python package, CLI, docs, CI.3435Executed:36- `cmake --preset macos-release && cmake --build --preset macos-release` (Apple clang 21, Eigen 5.0.1 + Accelerate,37 Arrow 26.0.0, Catch2 3.16.1, nlohmann-json 3.12). Metal sources absent → the build reports "Metal backend sources not38 present yet (Phase 3)" and continues CPU-only (`raid devices` lists the CPU only).39- `ctest --preset macos-release` → **26/26 test cases, 314 assertions** (unit: core, calendar, stats, OLS,40 realized/GARCH; reference: statsmodels 0.15 OLS/HC0/HC3/HAC/cluster at 1e−9, scipy t p-values at 1e−12,41 exchange_calendars 4.13 XNYS 2000–2026 exact session/holiday/early-close sets, numpy realized truth at 1e−12, arch 8.042 GARCH/GJR/EGARCH/Student-t log-likelihood within 5e−4 and parameters within 5 % + 0.01, forecasts within 5 %).43- Python: `cmake -DRAID_BUILD_PYTHON=ON` builds `raid._core` (nanobind 3.1.0, Python 3.12); `pytest python/tests` →44 7 passed (OLS HC3 vs statsmodels, realized vs numpy, GARCH convergence, calendar, bar validation).45- Real-data integration (file not committed): `raid validate /tmp/raid-export-test.parquet` (SPY+QQQ 1m January 2024,46 FinData export v5162) → 21 sessions each, coverage 1.0 (391 regular bars incl. the 16:00 print), 0 duplicates,47 0 errors; `raid estimate realized … --backend cpu` → 21 sessions × 390 returns, mean RV(SPY) 2.71e−5, 0.25 ms.48- Bugs found and fixed during the phase (recorded for the paper's engineering section): thread-pool completion race49 (local mutex destroyed before the last job released it → SIGABRT under load); NLL rescaling sign; EGARCH ω50 wrongly log-transformed; `tgamma` overflow for ν > 340 (→ `lgamma`); fixture truth computed on unrounded prices.51- Environment manifest: `experiments/manifests/environment_<host>.json` via `scripts/record_environment.sh`.52- Repository: bare `gitsrv:srv/git/wp40_uqo.git` created with the SPB Git post-receive hook; `origin` set; pushed.5354Gate evidence: clean build on the macOS preset, all tests green, docs (`docs/architecture/overview.md`,55`docs/methods/registry.md`, `docs/api/cpp.md`, README) written. Open items carried to later phases: `bars.aggregate`56vs FinData derived 5m (Phase 2), GARCH backcast option (Phase 4), HAR/bootstrap (Phase 4), Metal (Phase 3).5758## Phase 2 — Data (2026-11-02, 15:00–15:30 ET) — GATE PASSED59Plan: secure FinData adapter (C++/libcurl), `raid data` CLI, Python access, offline + opt-in live tests, provenance,60calendar/aggregation checks against the data provider.6162Executed:63- `include/raid/data/findata.hpp`, `src/data/findata.cpp`: bearer auth from env/file only, TLS, redacted logs,64 token bucket (5 req/s, burst 10), retries with backoff + `Retry-After`, Arrow-page pagination (`X-Next-Cursor`),65 31/186/3650-day chunking, async exports (POST → poll → download, `.part` rename), SHA-256 cache with `.meta.json`66 (dataset version, request ids, key fingerprint), provenance export; `raid data catalog|health|sessions|coverage|bars|67 export|fund|query|log`. Python: `raid.findata.{bars, export, health, catalog, dataset, sessions, coverage,68 fundamentals, query, read_parquet}` (GIL released).69- Offline tests (`[findata]`): chunking caps, token-bucket timing, offline refusal / missing key, synthetic Arrow IPC +70 Parquet page decoding (schema v1 metadata) → 4 passed, 2 live cases skipped. Fixture bug found: pyarrow IPC stream71 truncated when the file object was not closed → explicit `with open(...)` (recorded).72- Live tests (`RAID_LIVE=1`, key file, cache in `datasets/cache/`, AAPL 2024-11-25…29): health ok; catalog contains73 `market_bars.stock.1m.raw`; `expected_bars` 211 on 2024-11-29 == RAID calendar; bars pages (2 634 rows, 4 sessions,74 dataset version 17375) == export rows (62 933-byte Parquet, sha256 recorded) bar for bar; second call served from75 cache; RAID calendar `expected_regular_bars` == FinData `expected_bars` for every session; **aggregate 1m→5m:76 extended policy reproduces FinData's derived 5m exactly (280/280 OHLCV)**; regular policy differs only in the 16:0077 bin (FinData adds 16:01–16:04 post-close minutes) → convention documented in the registry (D-008 unchanged: the78 regular session ends with the closing print). Total 60 assertions, all passed; ≈ 25 requests.79- Python smoke: `raid.findata.bars` → realized variance per session from live bars; export + `read_parquet`80 (`zero_copy_possible = true` for single-chunk files).8182Gate evidence: offline tests in `ctest`; live run transcript above; no vendor data in git (`scripts/check_no_secrets.sh`).83Open: least-privilege key for benchmarks (needs `findata create-key` on M3U96a — owner action, optional).8485## Phase 3 — Numerical primitives (2026-11-02, 15:40–16:30 ET) — GATE PASSED86Plan: Metal backend with FP32 "double-float" (error-free-transform) accumulation, fused realized kernel, rolling sums,87regression sufficient statistics, batched GARCH likelihood; CPU/GPU agreement tests; measured break-even analysis.8889Executed:90- `backends/metal/shaders/raid_kernels.metal` (TwoSum/Fast2Sum double-float, deterministic threadgroup tree reductions,91 chunked partial sums combined in FP64 on the CPU) and `backends/metal/host/metal_backend.mm` (device discovery,92 prebuilt `raid.metallib` with runtime-MSL fallback, pipeline cache, buffer pool by power-of-two size class, GPU93 timing from `GPUStartTime/GPUEndTime`, `-fno-fast-math` / `fastMathEnabled = NO` so the compensation is not optimised94 away). Metal has no `lgamma`/`log1p`: Student-t constants are computed in FP64 on the host and passed per parameter95 vector.96- Tests `[metal]` (6 cases, 6 411 assertions): fused and single-statistic accumulators vs CPU FP64 within 1e−697 (Σx⁴ 1e−5) for n = 390/3 900/39 000 across chunk boundaries (4 096); bitwise determinism across runs; rolling sums98 1e−5; regression statistics 1e−6 (and the solve from GPU statistics agrees with CPU QR to 1e−5); batched GARCH NLL99 2e−5 relative (GJR + Student-t fixed ν 5e−5), argmin over a 16-point grid identical; +inf on invalid parameters.100 Full suite: `ctest` 38/38 (2 live skipped), `pytest` 7/7.101- `bench_bench_primitives` (laptop M5 Max 40-core GPU, 18 CPU threads, load ≈ 3–4 from other sessions, no thermal102 warning, 5 repetitions, shuffled order, 2 warm-ups; raw records `experiments/results/raw/bench_primitives.jsonl`103 + manifest; summary `experiments/results/bench_primitives_summary.{json,csv}`, figure104 `fig_bench_primitives_throughput.pdf`). Medians of wall-clock including FP64→FP32 conversion and buffer traffic;105 verdicts use non-overlapping bootstrap 95 % CIs of the median:106 - realized_fused: Metal wins from 39 000 elements (100 × 390: 2.4×), 2.0–3.4× vs 18-thread CPU at 3.9 M–39 M107 elements (max 3.4× at 1 000 × 39 000); ties at 390 000 for 10 × 39 000 and 100 × 3 900 shapes; CPU 1 thread is108 27–40× slower than Metal at 39 M.109 - regression_stats (k = 4): break-even ≈ 390 000 elements for a single long series (3.9×); ties for most shapes110 between 390 k and 3.9 M; 1.1–1.4× at 39 M. CPU wins below 39 000 elements.111 - rolling_sumsq (w = 22, direct O(w) kernel): **CPU wins at every size** (Metal 0.4–0.9× of CPU-MT); the GPU kernel112 is memory/arithmetic-bound by its O(w) design and the CPU running sum is O(1) per element — a negative result kept113 for the paper (a scan-based kernel is future work).114 - garch_nll: with one parameter vector per series (P = 1) the **CPU wins everywhere** (0.01–0.33×; the recursion is115 sequential and the GPU is launch-bound); with P = 64 parameter vectors per series Metal wins from 100 series116 (1.9×) up to 7.9× at 1 000 × 3 900 — batched likelihood evaluation is where the GPU pays off.117 - Device time is 5–20× smaller than wall time for the reductions: FP64→FP32 conversion and buffer fills dominate118 ("unified memory is not free computation"); E6 will add the FP32-resident ablation.119 - Shader library: prebuilt metallib loaded in 0.23 ms; first kernel call 14 ms (pipeline creation); runtime MSL120 compilation of the bundle is measured separately in E7 (toolchain-less nodes).121122Gate evidence: accuracy tests and the measured break-even table above; no claim beyond the tested shapes.123124## Phase 4 — Econometrics, Tier A (2026-11-02, 16:30–17:40 ET) — GATE PASSED125Plan: HAR family, block bootstrap, R references (highfrequency, rugarch), Monte Carlo validation, gradient checks,126registry completion, Python/CLI exposure.127128Executed:129- `estimators/har.{hpp,cpp}`: HAR-RV / log-HAR / HAR-X / HARQ (causal lags, average or single targets, HAC with130 ≥ h−1 lags), `har_rolling` (causal re-estimation + QLIKE/MSE), `har_batch` (GPU sufficient statistics, FP64 solve).131 `statistics/bootstrap.{hpp,cpp}`: stationary / circular block / iid, deterministic streams, scalar and row variants.132- R references (`scripts/reference_r.R` → `datasets/fixtures/reference_r.json`, highfrequency 1.0.3, rugarch 1.5.6):133 RV/BPV/RSV±/RQ equal to 1e−10 after two documented conventions (highfrequency `rQuar` uses (M+1)/3; `rBPCov` with134 makeReturns=FALSE applies no M/(M−1) factor); `HARmodel` h=1 coefficients 1e−8 and h=5 (average target) 1e−6; HARQ135 equal once the regressor is (√RQ_t − √mean RQ)·RV_t as in highfrequency; rugarch sGARCH/gjrGARCH/eGARCH/sGARCH-std136 log-likelihoods within 2e−4 and parameters within 5 % (eGARCH alpha1/gamma1 swap documented; ν at the bound for137 Gaussian data in both packages).138- statsmodels references (`make_fixtures.py`): HAR h=1/5/22, log-HAR, HARQ with HAC → coefficients 1e−9, SE 1e−8.139- Calendar: the exchange_calendars comparison was previously **not running** (API error swallowed into a WARN) — fixed;140 the exact comparison 2000-01-03…2026-12-31 (6 790 sessions, 254 weekday holidays, 60 early closes) revealed three141 historical one-offs (early closes 2002-07-05 and 2003-12-26; full day 2002-07-03), now encoded. All sets equal.142- Monte Carlo (`tests/simulation`, printed to the test log): GARCH(1,1) n=2000, R=120: bias μ −0.0004, ω +0.0033,143 α −0.0020, β −0.0026; RMSE 0.019/0.009/0.014/0.019; 95 % QMLE coverage 0.950/0.942/0.900/0.933; 0 failures.144 HAR h=5 HAC coverage 0.895/0.945/0.910/0.905 (R=200, T=1200). RV RMSE vs √(2/M)·IV: 5.48e−5 vs 5.66e−5 (M=39),145 1.85e−5 vs 1.79e−5 (390), 5.78e−6 vs 5.66e−6 (3 900). Stationary-bootstrap mean coverage 0.89 (R=100). Numerical146 NLL gradients agree across step sizes 1e−4/1e−5/1e−6.147- Python: `raid.har_fit/har_rolling/har_batch/bootstrap` (pytest 9/9, HAR+HAC equal to statsmodels); CLI `raid148 estimate har <bars> [--horizon] [--log] [--harq] [--window --step]`.149- Full suite: `ctest` all green (see summary in RESEARCH_STATUS).150151Gate evidence: reference tables (R + Python), Monte Carlo logs, registry entries with tolerances.152153## Phase 5 — Extensions, Tier B (2026-11-02, 17:40–18:10 ET) — GATE PASSED154Selection by research value (E4 needs them): VAR(p), orthogonalised and generalized FEVD, Diebold–Yilmaz connectedness155(static + rolling), Ledoit–Wolf shrinkage. `models/var.{hpp,cpp}`. Reference: statsmodels VAR on a simulated VAR(2)156panel (`datasets/fixtures/var_panel.csv`): coefficients/Σ_u/llf/orthogonalised FEVD 1e−9, generalized FEVD (hand157computation in Python) 1e−9, DY totals 1e−7, BIC order, LW shrinkage 1e−9 (`tests/reference/test_var_reference.cpp`,158184 assertions). Rolling connectedness parallel over windows (0 failures on the fixture). Other Tier B/C items are159deliberately not implemented (registry).160161## Phase 6 — Benchmarking E1–E7 (2026-11-02, 18:10–20:40 ET) — GATE PASSED162Raw records + manifests under `experiments/results/raw/` (one JSONL per experiment, `.manifest.json` with host, versions,163load, thermal, seeds, input hashes); summaries and every paper table/figure produced by164`experiments/scripts/make_paper_assets.py` (hashes in `experiments/manifests/paper_assets.json`; prose numbers only from165`experiments/results/key_numbers.json`).166- E0: universe = seeded sample of 1 000 of the 6 046 symbols active through 2024 (14 600 coverage rows); 20 exports,167 43 346 886 bars, 631 MB, dataset version 17 375, 437 s; E0b: 1.82 M daily bars 2015–2024.168- E1 (estimator path, laptop): 926 symbols × 252 sessions = 40 M returns — CPU 18T 29–30 ms, Metal 20–28 ms169 (1.07–1.41×), ties/CPU wins ≤ 4.8 M elements; fusion gain 1.4–3.2×; max |ΔRV|/RV vs FP64 1.6e−7 (daily segments),170 1.4e−8 (yearly). Synthetic 10 000 × 1 month (85.8 M): CPU 54 ms, Metal 35 ms (1.53×).171- E2: HAR rolling (120-session window) on 100 symbols: median QLIKE HAR 0.196/0.146/0.134 vs RW 0.251/0.239/0.301172 (h = 1/5/22); HAR beats RW for 84/86/78 % of symbols. Runtime 781 series × 252 obs: per-series QR 18T 0.68 ms,173 batched CPU 3.9 ms, batched Metal 4.7 ms (tiny designs: GPU launch-bound).174- E3: 888 symbols × ~2 459 daily returns: convergence GARCH 100 %, GJR 98.4 %, EGARCH 92.8 %, GARCH-t 100 %175 (after ≤ 2 restarts); 2.9–18 ms per fit; median QLIKE of fixed-parameter 1-day forecasts vs RV 0.39–0.40 against176 0.27–0.28 for the RV random walk. GPU grid 888 × 256 parameter vectors: Metal 23 ms vs CPU 18T 117 ms (5.1×),177 argmin identical 888/888, max rel. NLL difference 2e−3 at boundary grid points.178- E4: realized covariance (5-min synchronised) N = 10/30/60/100: median condition number 81/422/5.9e3/singular (N > 78179 bins); LW shrinkage 0.16–0.33; rolling DY connectedness 133 windows in 0.5/2.3/8.9/29 ms, 0 failures.180- E5 (R = 1 000): GARCH n = 2 000 coverage 0.93–0.97, |bias| ≤ 0.005, 1 000/1 000 converged; n = 500 β bias −0.026,181 α coverage 0.90; HAR HAC coverage 0.94 (h = 1), 0.90–0.94 (h = 5); bitwise reproducibility across threads and Metal182 runs; Metal vs CPU RV 3.7e−8.183- E6: per 2.37 M-bar export file — RAID CPU 0.118 s (20 M bars/s), RAID Metal 0.125 s, Polars 0.165 s, DuckDB 0.164 s,184 pandas 2.56 s; read 0.065 s, prep 0.021 s, estimate 0.031 s (CPU) / 0.037 s (Metal). Column projection took the185 pipeline from 0.35 s to 0.12 s. Baselines: approximate early-close rule (no exchange calendar in those engines);186 bugs fixed on the way: DuckDB 2.0 API, Polars i8 overflow in hour×60.187- E7 (11 nodes, M2 → M6): all Metal tests pass with prebuilt metallib and runtime MSL; cold compile 37–49 ms (110 ms on188 the 8 GB M2), warm 0.3–0.6 ms; RV fused 1000×3900 speedup 4.0× (M2) … 1.1× (M5 Max/Pro); GARCH P=64 7.8× (M2 Max) …189 1.8× (M5 Pro); rolling 0.2–0.65× everywhere.190- Engineering findings during Phase 6 (recorded): serial tripower/post-processing diluted E1 (fixed: parallel, cbrt);191 Metal backend recreated per call through an expired weak_ptr (fixed: process singleton); E3 evaluation index bug with192 the longer daily history (fixed); per-series FP32 scaling in the GARCH kernel (max rel. error on real grids 6.8e−4 →193 2e−3 at boundary points only, argmin agreement 887 → 888/888).194195## Phase 7 — Manuscript (2026-11-02, 20:40–21:30 ET) — GATE PASSED196`paper/main.tex` + 13 sections + appendices; 30 pages; 57 BibTeX entries all resolved on Crossref (DOI content197negotiation) or hand-verified; `latexmk` exit 0, **0 undefined citations/references**; all tables `\input` from198`paper/tables/` and figures from `paper/figures/`, both generated by `make_paper_assets.py` from tracked artifacts.199Known cosmetic overfull boxes in two listings; no placeholder, no hand-typed number.200201## Phase 8 — Independent review (2026-11-02, 21:30–22:10 ET) — GATE PASSED202- Clean clone of `wp40_uqo.git` → `cmake --preset cpu-only` (Arrow/Metal off) builds, `ctest` 49/49; `macos-release`203 builds, `ctest` 55/55; `raid estimate realized … --backend metal` runs on the clone (fp32, 5 sessions); `latexmk`204 exit 0, 30 pages, 0 undefined; `make_paper_assets.py` regenerates the 9 tables byte-identical (the 3 PDF figures205 differ only by matplotlib's embedded creation timestamp).206- **Defect found and fixed by this phase**: the `.gitignore` pattern `data/` silently excluded `include/raid/data/` and207 `src/data/` from every earlier commit (the working tree built, the repository did not); now `/data/`. The raw208 benchmark records (`experiments/results/raw/`, 9.6 MB) were also ignored and are now tracked, as the charter requires.209- Numerical audit: `scripts/audit_manuscript_numbers.py` — 32/32 quantitative claims of the prose found verbatim from210 `key_numbers.json`. Secret/data scan: `scripts/check_no_secrets.sh` clean; only synthetic fixtures tracked.211- Release checklist: `docs/RELEASE-CHECKLIST.md` (gaps stated: Linux build not executed, pip wheel not built from a212 clean environment, E1–E6 need a FinData key). Tag `v0.1.0-wp40`.213214## Phase 7b — Manuscript polish (2026-11-02, 22:30–23:30 ET)215Owner request: nicer mathematics and journal-quality figures. Done: unified notation (`\DeclareMathOperator` for RV, BPV,216RQ, RSV, TriPV, RCov, Var; bold vectors/matrices; indicator and absolute-value macros), `mathtools`/`bm`; estimator217section rewritten with aligned, numbered definitions (returns, realized measures, OLS/HC/HAC/cluster covariances, HAR,218HARQ, QLIKE, GARCH/GJR/EGARCH, QML log-likelihood, QMLE sandwich, forecasts, bootstrap, VAR/GFEVD/DY, Ledoit–Wolf);219Metal section with the TwoSum error-free transformation and the input-rounding bound as equations; architecture220diagram redrawn in TikZ. Figures: new module `experiments/scripts/paper_style.py` (serif/STIX, thin spines, light221grids, panel labels "Panel A.", grayscale-safe palette, 300 dpi vector PDF) and `make_paper_figures.py` producing nine222figures from the artifacts — primitives (throughput + speedup with bootstrap CIs), E1 (throughput, speedup heat map,223fusion gain), precision vs segment length with the theoretical bound, E6 (stage breakdown + engines with CIs), E7224(speedup vs GPU cores + cold/warm compile), E2 (QLIKE distributions + runtime), E3 (convergence, persistence, QLIKE),225E5 (coverage, bias/RMSE), E4 (conditioning, shrinkage, connectedness). Manuscript 36 pages, 0 undefined, no large226overfull boxes; figure hashes appended to `experiments/manifests/paper_assets.json`.227