1# docs/audit.md — Phase 0 audit (2026-11-02)23All facts below were measured on 2026-11-02 (America/Toronto) from the owner's laptop; raw probe outputs are in4`docs/evidence/` (per-node probe files, FinData OpenAPI snapshot). Nothing is assumed from older inventories.56## 1. UQO Research repository78- Root: `~/Desktop/UQO/Recherche/Recherche-UQO/` — one directory per working paper, grouped by domain9 (`immobilier/`, `finance/`, `economie/`, `ia/`, `methodes/`, `mathematiques/`), each paper is its own git repository10 with `origin = gitsrv:srv/git/wpN_uqo.git` (spbgit on M1M32). Highest existing number: **WP 39** (`wp39_uqo.git`,11 `finance/invisible-balance-sheet`, created 2026-11-02 morning). RAID is therefore **WP 40**, repository `wp40_uqo.git`12 (to be created on first push), directory `finance/wp40_raid/` as requested by the charter.13- Form standard: `HARMONISATION.md` — `paper/main.tex` with the `\WP*` macros, `sections/titlepage.tex` copied verbatim14 from `_modele-working-paper/`, natbib `[authoryear,round,semicolon]` + `aer.bst`, `\WPdate` = `Month YYYY`,15 `latexmk` Makefile, `paper/main.pdf` committed, data never committed, README with fixed section order16 (TL;DR, Status, What this paper does, Headline results, Layout, Quick start, Data, Methodology, Figures & tables,17 Limitations, Citation, Author, Notes).18- Prior volatility research to build on (not to re-do): WP 12 `finance/realized-volatility-horse-race` (HAR family,19 GARCH/GJR/EGARCH/Realized GARCH, 25 assets, 5-min subsampled RV target, QLIKE/DM/MCS; Python, `src/wp12`),20 WP 6 `intraday-volatility-connectedness` (Diebold–Yilmaz on intraday FX/fixed income), WP 15 `intraday-fragility`,21 WP 39 `invisible-balance-sheet` (first FinData OS consumer: rate limiter, export caps, duplicate last session).22 Systems-paper precedents in the same repository: WP 33 TITAN-MoE (`ia/titan-moe`: raw logs under `benchmarks/raw/`,23 `artifacts/manifest.json` with SHA-256 per artifact, hypotheses with bootstrap CIs) and WP 30.24- No existing CMake/C++ template in the repository: every prior WP is Python/LaTeX. RAID introduces the first C++/Metal25 build; it reuses only the LaTeX template, README standard and git conventions.26- Git practice: `main` branch, commit messages in English, `paper/main.pdf` tracked, SSRN uploads tracked in27 `_soumissions/`.2829## 2. FinData OS (mandatory data API) — live audit3031- Base `https://findata.spboucher.ai/api/v1`, OpenAPI 3.1 snapshot saved to `docs/evidence/findata_openapi_2026-11-02.json`32 (45 routes). `/health` → `status: ok`, Postgres ok, backup verified 9.6 h old, lake 564.65 GB curated / 303 GB free,33 cold root `/Volumes/ssd/findata` (2.98 TB free), SSD SMART verified.34- Catalog: 64 active datasets. Relevant for RAID: `market_bars.stock.1m.raw` (6.70 G rows, 2000-01-03 → 2026-10-30,35 16 195 symbols incl. `-DELISTED`), `market_bars.etf.1m.raw` (908 M rows), `market_bars.stock.5m.raw` (2.06 G),36 `market_bars.stock.1d.raw` (21 M), `market_bars.index.1m.raw` (204 M, 2008→), futures continuous/contracts, FX, crypto,37 `options.eod` (4.23 G rows), `fundamentals.*`. `market_ticks` = **unsupported** (vendor provides bars only):38 Tier C order-flow / Hawkes methods have no admissible input in this project.39- Bars endpoint (`GET /market/bars`): `interval ∈ {1m,5m,1d}`, `adjustment ∈ {raw,split,split_div}`, `format ∈40 {json,arrow,csv}`, `limit` default 5 000 (max 10 000), cursor pagination, `version` pin. Verified Arrow IPC stream41 response: `content-type: application/vnd.apache.arrow.stream`, headers `x-dataset-id`, `x-dataset-version` (5162 for42 ETF 1m), `x-request-id`; schema v1 with metadata `findata.timestamp_convention = bar_open`. SPY 2024-03-08 → 03-1143 (DST weekend): 794 + 805 bars, first bar 09:00 UTC (= 04:00 EST) on 03-08, last bar 23:59 UTC (= 19:59 EDT) on 03-11:44 DST handled at the source; extended session 04:00–19:59 ET.45- Sessions endpoint (`GET /market/sessions`): per-session `bar_count` vs `expected_bars` (391 regular; **211 on the46 2024-11-29 early close**, with extended bars until 16:59 that day). This is the calendar truth RAID must reproduce.47- Exports (`POST /exports` → 202 → poll → download): SPY+QQQ 1m January 2024 = 34 607 rows, 0.65 MB Parquet, ready in48 1.5 s, download 0.3 s; response records `snapshot.version = 5162`, `files_scanned`, `bytes`, `row_count`; Parquet49 key-value metadata carries the schema version and timestamp convention. SHA-256 of the file is recorded by RAID.50 WP 39 found: ≤ 12 years per export request (HTTP 413 beyond), last lake session may appear twice (base + delta51 layouts) → RAID deduplicates on `(instrument_id, timestamp_utc)` and reports it in the QualityReport.52- Query playground (`POST /query`): one read-only statement, 30 s, ≤ 100 000 rows; views carry partition columns53 `symbol`, `year`; `/query/views` lists them. Used only for coverage/universe selection, never for bar transport.54- Fundamentals (`GET /fundamentals/{entity}?as_of=`): PIT semantics documented; `available_at = filing_date + 1 day`55 is a proxy; no delisted entities (survivorship) — relevant only if a HAR-X design uses accounting covariates (not planned56 for the methods paper).57- Rate limit: 600 req/min per key (no `RateLimit-*` headers exposed; WP 39 observed HTTP 429 with `Retry-After`).58 RAID client: token bucket ≤ 5 req/s, exponential backoff on 429/5xx, bounded concurrency (≤ 4), honours `Retry-After`.59- License: FirstRate Data private use — no redistribution of raw bars; `license_class = frd-private`. Consequence:60 real-data experiment inputs live in an ignored cache (`datasets/cache/`), results/figures publish only aggregates;61 the repository ships synthetic fixtures only. Attribution "FirstRate Data via FinData OS" in the paper.62- API key for this project: the owner's Claude Code key (`~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt`,63 scopes read:market, read:fundamentals, query, export, admin). RAID never needs admin scope; a least-privilege key64 can be created later with `findata create-key` on M3U96a (the `POST /admin/keys` route returned 500 for WP 39).6566## 3. Hardware and toolchains (measured)6768### Laptop (primary development and benchmark machine)69Apple M5 Max, 18 CPU cores (6 P + 12 E as reported by `hw.perflevel*`), **40 GPU cores**, 48 GB unified memory, macOS 27.070(26A5425a), Xcode 27.2 (27B52), Apple clang 21.0.0, SDK 27.2, Metal 4, GPU families Apple1–Apple10, `maxBufferLength`7128 753 MB, `recommendedMaxWorkingSetSize` 38 338 MB, threadgroup max 1 024, SIMD width 32, unified memory. Metal72Toolchain (offline `metal`/`metallib`, version 32023.922) was **absent** and was downloaded during the audit73(`xcodebuild -downloadComponent MetalToolchain`). Runtime shader compilation (`newLibraryWithSource:`) works without it74(34.6 ms for a trivial kernel, first compile). **`double` is rejected by the Metal compiler** (verified: "'double' is75not supported in Metal") — all GPU numerics are FP32 (with compensated/pairwise reductions) and FP64 lives on the CPU.76Disk 1.0 TB free. Load average 2.1 (27 days uptime). Tools: cmake 4.3.2, uv 0.11.7, Python 3.14.4 system / 3.12.13 for77the project venv, R 4.6.1, pandoc 3.9, TeX Live 2026, Homebrew openblas 0.3.33, libomp 22.1.8, fmt 12.1.78Installed for RAID during the audit: Homebrew `eigen`, `apache-arrow`, `catch2`, `nlohmann-json`, `ninja`79(see `docs/capabilities.md` for versions once the install finished); R `highfrequency` 1.0.3, `rugarch` 1.5.6, `xts` 0.14.3;80Python venv `.venv` (3.12): nanobind 3.1.0, numpy 2.5.4, pyarrow 26.0.0, arch 8.0.0, statsmodels 0.15.0, scipy 1.18.1,81polars 2.1.0, duckdb 2.0.0, exchange-calendars 4.13.2.8283### MacLustr nodes (probe 2026-11-02 14:1x ET, `docs/evidence/nodes/*.txt`)8422 of 29 nodes reachable over SSH; **unreachable (timeout): m2m16, M2M32, M2M32b, M4M36, m4ma, m4mb, m4mc** — excluded85from any plan until they answer `mld discover`. All reachable nodes report Metal 4 and no thermal/performance warning.8687| Node | Chip | CPU | GPU cores | RAM | macOS | Xcode / CLT | Load 1m | Free disk | PM2 apps | Role for RAID |88|---|---|---:|---:|---:|---|---|---:|---:|---:|---|89| M1M32 | M1 Max | 10 | 24 | 32 | 26.5.1 | Xcode 26.4.1 | 0.94 | 197 Gi | gateway | E7 generation point (M1), short runs only (gateway host) |90| M2M32c | M2 Max | 12 | 30 | 32 | 26.5.1 | CLT | 1.08 | 416 Gi | 0 | E7 (M2 Max) |91| M2U64 | M2 Ultra | 24 | 60 | 64 | 26.5 | Xcode 26.6 | 1.17 | 315 Gi | apps (spb-cloud, uqo-eval…) | E7 (M2 Ultra), off-hours |92| M3BA24 / M3BA16 | M3 | 8 | 10 | 24 / 16 | 26.5.1 | CLT | 1.1–1.3 | 409–420 Gi | 0 | E7 (M3 base, laptops) |93| M3U96a | M3 Ultra | 28 | 60 | 96 | 26.6 | Xcode 26.6 | **5.96** | 281 Gi | 17 (FinData OS host) | **not used for benchmarks** (production FinData host, loaded) |94| M3U96b | M3 Ultra | 28 | 60 | 96 | 26.4 | Xcode 26.6 | 1.80 | 395 Gi | 8 | E7 (M3 Ultra), largest-panel runs |95| M4M64a / M4M64b | M4 Max | 16 | 40 | 64 | 26.5.1 | Xcode 26.6 | 1.2–1.5 | 540 / **54 Gi** | 10 / 11 | avoid (agentd / Ka hosts; M4M64b low disk) |96| M4M64c | M4 Max | 16 | 40 | 64 | 26.4 | CLT | 1.94 | 318 Gi | 0 | E7 (M4 Max) |97| M4BP48 / M4BP36 | M4 Max | 16 / 14 | 40 / 32 | 48 / 36 | 26.5.1 | CLT | 1.2 | 843 Gi | 0 | E7 alternates |98| m4mm / m4mn | M4 Pro | 12 | 16 | 48 | 26.5.1 | CLT | 1.4–1.8 | 370 Gi | 0 | E7 (M4 Pro) |99| m4md / m4ml | M4 | 10 | 10 | 16 | 26.6.2 / 26.7 | CLT | 1.6–1.8 | 180–195 Gi | 0 | E7 (M4 base, 16 GB memory-pressure case) |100| M5M36 | M5 Max | 18 | 32 | 36 | 27.0 | CLT (SDK 27.0) | 1.22 | 355 Gi | 0 | E7 (M5 Max, 32-core GPU variant) |101| m5ma | M5 Pro | 15 | 16 | 24 | 27.0 | CLT | 1.13 | 382 Gi | 0 | E7 (M5 Pro) |102| m6ma / m6mb | M6 | 12 | 12 | 24 | 27.0 | CLT | 0.9–1.2 | 395 Gi | 0 | E7 (M6 base, newest generation) |103| m2m8a / m2m8b | M2 | 8 | 10 | 8 | 26.5.1 | CLT / none | 1.5 | 196–425 Gi | 0 | 8 GB memory-pressure stress (m2m8b has no developer tools → binary bundle only) |104105Observations that shape the plan: (i) most nodes have only the Command Line Tools and no Homebrew in the non-interactive106SSH `PATH` (cmake/ninja/brew empty in the probe) → E7 ships a **self-contained arm64 bundle** (static Eigen, bundled107`.metallib` + MSL source for runtime fallback, no Arrow dependency at run time) built on the laptop with108`-mmacosx-version-min=26.0`; nothing is installed on the nodes; (ii) GPU core counts span 10 → 60, memory 8 → 96 GB:109the generation sweep E7 covers M1 Max, M2 (base/Max/Ultra), M3 (base/Ultra), M4 (base/Pro/Max), M5 (Pro/Max), M6;110(iii) M3U96a (FinData host, load 6) and M4M64a/b (Ka/agentd hosts) are excluded from timing runs.111112## 4. Competing implementations (versions verified locally)113114| Software | Version | Role in RAID validation |115|---|---|---|116| R `highfrequency` | 1.0.3 (Boudt, Kleen & Sjørup, JSS 2022) | reference for RV, BPV, semivariance, realized quarticity, rCov, HAR (`HARmodel`), BNS jump test |117| R `rugarch` | 1.5.6 (Ghalanos) | reference for GARCH(1,1), GJR, EGARCH (Gaussian / Student-t) estimates, SE, forecasts |118| Python `arch` | 8.0.0 (Sheppard) | second GARCH reference; HAR via OLS; bootstrap (stationary/circular block) |119| Python `statsmodels` | 0.15.0 | OLS/WLS, HC0–HC3, HAC (Newey–West, Bartlett), cluster-robust covariance |120| NumPy / SciPy | 2.5.4 / 1.18.1 | FP64 reference linear algebra (QR/SVD/Cholesky), optimizers |121| Polars / DuckDB / pyarrow | 2.1.0 / 2.0.0 / 26.0.0 | columnar baselines for ingestion and rolling computations |122| Eigen (C++) | Homebrew, see capabilities | CPU backend dense algebra |123| Apple Accelerate | macOS 27 | BLAS/LAPACK (`-framework Accelerate`, `ACCELERATE_NEW_LAPACK`) |124| Julia | not installed | not used (listed in the charter; optional reference, not required for gates) |125126Mathematical equivalence is established per method in `docs/methods/registry.md` (Phase 4), with the exact option127settings of each competitor (e.g. `highfrequency::rBPCov` uses the µ₁⁻² = π/2 constant and no finite-sample factor128by default; `rugarch` uses variance targeting off and sample-variance initialisation unless told otherwise).129130## 5. Licensing131132- RAID code: MIT. Dependencies: Eigen (MPL-2.0), Apache Arrow (Apache-2.0), nanobind (BSD-3), Catch2 (BSL-1.0),133 nlohmann/json (MIT), Accelerate and Metal (Apple proprietary system frameworks, linked, not redistributed).134- Metal is proprietary: the GPU backend is optional (`RAID_ENABLE_METAL`), the CPU backend builds on Linux without it.135- Data: FirstRate Data private-use via FinData OS — no raw bars in git, synthetic fixtures in CI, attribution in paper.136- Reference software licenses do not affect RAID (used only to produce validation tables).137