# docs/PROGRESS.md — phase journal (actual commands and results)

# Phase 0 — Audit (2026-11-02) — GATE PASSED

Plan: audit the UQO repository, FinData, hardware, competitors, literature and licensing; write the four audit documents.

Executed (laptop, 14:05–14:40 ET):

• Read Recherche-UQO/README.md, HARMONISATION.md, _modele-working-paper/*, WP 12/39/33 READMEs and PROGRESS.md. Result: WP 40 is free; conventions recorded in docs/audit.md §1.
• FinData: fdq health, fdq datasets, GET /openapi.json (saved), GET /market/bars?format=arrow (SPY, DST weekend), GET /market/sessions (SPY, Thanksgiving week 2024 → 211 expected bars on 11-29), POST /exports (SPY+QQQ Jan 2024: 34 607 rows, 1.5 s, version 5162, SHA-256 recorded), GET /query/views, GET /coverage/symbols. Results in §2.
• Hardware: sw_vers, sysctl, system_profiler SPDisplaysDataType, xcodebuild -version, Metal device query program (docs/evidence/ test), 29-node parallel SSH probe (22 reachable; files in docs/evidence/nodes/).
• Metal: trivial kernel compiled at runtime (34.6 ms) and executed (1 M squares, 0.029 ms GPU); double kernel rejected by both runtime and offline compilers; Metal Toolchain 27B47 downloaded (xcodebuild -downloadComponent MetalToolchain).
• Toolchain installs: brew install eigen apache-arrow catch2 nlohmann-json ninja (background), R install.packages(c("highfrequency","rugarch","xts","lmtest")) → highfrequency 1.0.3, rugarch 1.5.6, xts 0.14.3 (lmtest failed: Java runtime missing for a dependency; not needed), uv venv --python 3.12 + pip install of the Python reference stack (versions in docs/capabilities.md).
• Literature: 39 key references resolved on Crossref (docs/literature/crossref_resolved.json); preprint hits were re-resolved to journal versions; three DOIs verified directly (Econometrica 2008 realized kernels, SPA 2009 pre-averaging, OUP 2010 semivariance). Still to resolve: Hansen–Lunde–Nason 2011 Econometrica MCS DOI; software citations (Arrow, Eigen, nanobind, Metal docs) to be added with URLs/versions.
• Legal: FirstRate private-use license (no redistribution), MIT for RAID, Metal proprietary → optional backend.

Gate memo (scope and feasibility): feasible on the laptop with the measured toolchain; FP64-on-GPU impossible → FP32 strategy; FinData provides all Tier A inputs (1-minute OHLCV bars, versions, sessions, exports); no trade/quote data → Tier C order-flow methods excluded; cluster usable for E7 with a static bundle; 7 nodes unreachable. Decisions D-001…D-010.

# Phase 1 — Foundation (2026-11-02, 14:40–15:40 ET) — GATE PASSED

Plan: CMake/Ninja build, C++20 core types, CPU backend, data abstractions (calendar, bars, returns, aggregation), Arrow/Parquet reader, OLS/robust covariance, realized measures, GARCH-family QML (needed by the backend contract), Catch2 tests, nanobind Python package, CLI, docs, CI.

Executed:

• cmake --preset macos-release && cmake --build --preset macos-release (Apple clang 21, Eigen 5.0.1 + Accelerate, Arrow 26.0.0, Catch2 3.16.1, nlohmann-json 3.12). Metal sources absent → the build reports "Metal backend sources not present yet (Phase 3)" and continues CPU-only (raid devices lists the CPU only).
• ctest --preset macos-release → 26/26 test cases, 314 assertions (unit: core, calendar, stats, OLS, realized/GARCH; reference: statsmodels 0.15 OLS/HC0/HC3/HAC/cluster at 1e−9, scipy t p-values at 1e−12, exchange_calendars 4.13 XNYS 2000–2026 exact session/holiday/early-close sets, numpy realized truth at 1e−12, arch 8.0 GARCH/GJR/EGARCH/Student-t log-likelihood within 5e−4 and parameters within 5 % + 0.01, forecasts within 5 %).
• Python: cmake -DRAID_BUILD_PYTHON=ON builds raid._core (nanobind 3.1.0, Python 3.12); pytest python/tests → 7 passed (OLS HC3 vs statsmodels, realized vs numpy, GARCH convergence, calendar, bar validation).
• Real-data integration (file not committed): raid validate /tmp/raid-export-test.parquet (SPY+QQQ 1m January 2024, FinData export v5162) → 21 sessions each, coverage 1.0 (391 regular bars incl. the 16:00 print), 0 duplicates, 0 errors; raid estimate realized … --backend cpu → 21 sessions × 390 returns, mean RV(SPY) 2.71e−5, 0.25 ms.
• Bugs found and fixed during the phase (recorded for the paper's engineering section): thread-pool completion race (local mutex destroyed before the last job released it → SIGABRT under load); NLL rescaling sign; EGARCH ω wrongly log-transformed; tgamma overflow for ν > 340 (→ lgamma); fixture truth computed on unrounded prices.
• Environment manifest: experiments/manifests/environment_<host>.json via scripts/record_environment.sh.
• Repository: bare gitsrv:srv/git/wp40_uqo.git created with the SPB Git post-receive hook; origin set; pushed.

Gate evidence: clean build on the macOS preset, all tests green, docs (docs/architecture/overview.md, docs/methods/registry.md, docs/api/cpp.md, README) written. Open items carried to later phases: bars.aggregate vs FinData derived 5m (Phase 2), GARCH backcast option (Phase 4), HAR/bootstrap (Phase 4), Metal (Phase 3).

# Phase 2 — Data (2026-11-02, 15:00–15:30 ET) — GATE PASSED

Plan: secure FinData adapter (C++/libcurl), raid data CLI, Python access, offline + opt-in live tests, provenance, calendar/aggregation checks against the data provider.

Executed:

• include/raid/data/findata.hpp, src/data/findata.cpp: bearer auth from env/file only, TLS, redacted logs, token bucket (5 req/s, burst 10), retries with backoff + Retry-After, Arrow-page pagination (X-Next-Cursor), 31/186/3650-day chunking, async exports (POST → poll → download, .part rename), SHA-256 cache with .meta.json (dataset version, request ids, key fingerprint), provenance export; raid data catalog|health|sessions|coverage|bars| export|fund|query|log. Python: raid.findata.{bars, export, health, catalog, dataset, sessions, coverage, fundamentals, query, read_parquet} (GIL released).
• Offline tests ([findata]): chunking caps, token-bucket timing, offline refusal / missing key, synthetic Arrow IPC + Parquet page decoding (schema v1 metadata) → 4 passed, 2 live cases skipped. Fixture bug found: pyarrow IPC stream truncated when the file object was not closed → explicit with open(...) (recorded).
• Live tests (RAID_LIVE=1, key file, cache in datasets/cache/, AAPL 2024-11-25…29): health ok; catalog contains market_bars.stock.1m.raw; expected_bars 211 on 2024-11-29 == RAID calendar; bars pages (2 634 rows, 4 sessions, dataset version 17375) == export rows (62 933-byte Parquet, sha256 recorded) bar for bar; second call served from cache; RAID calendar expected_regular_bars == FinData expected_bars for every session; aggregate 1m→5m: extended policy reproduces FinData's derived 5m exactly (280/280 OHLCV); regular policy differs only in the 16:00 bin (FinData adds 16:01–16:04 post-close minutes) → convention documented in the registry (D-008 unchanged: the regular session ends with the closing print). Total 60 assertions, all passed; ≈ 25 requests.
• Python smoke: raid.findata.bars → realized variance per session from live bars; export + read_parquet (zero_copy_possible = true for single-chunk files).

Gate evidence: offline tests in ctest; live run transcript above; no vendor data in git (scripts/check_no_secrets.sh). Open: least-privilege key for benchmarks (needs findata create-key on M3U96a — owner action, optional).