# ⚡ RAID — Research Architecture for Intraday Data **UQO Working Paper No. 40** — *A Metal-Accelerated C++ Framework for Reproducible High-Frequency Financial Econometrics* [-b31b1b?logo=latex&logoColor=white)](paper/main.pdf) [](CMakeLists.txt) [-000000?logo=apple&logoColor=white)](backends/metal/) [](pyproject.toml) [-4051b5)](docs/audit.md) [](https://uqo.ca) [](mailto:simon-pierre.boucher@uqo.ca) > **TL;DR** — RAID asks whether heterogeneous high-frequency econometric workloads (realized measures, rolling/batched > regressions, HAR, GARCH-family QML, block bootstrap) can be turned into batched, fused, streaming computation on Apple > Silicon **without** losing estimator accuracy, inference validity or reproducibility. It is a C++20 library with a FP64 > CPU reference backend, an optional FP32 Metal backend (Apple GPUs have no FP64), nanobind Python bindings, a CLI, a > FinData OS client, and a benchmark protocol that reports break-even points and the cases where the CPU wins. > Status: Phases 0–7 complete (2026-11-02); Phase 8 review in progress — see `RESEARCH_STATUS.md`. --- ## 📌 Status | Item | Value | |---|---| | Version | 1.0 (November 2026) — manuscript v1.0, library 0.1.0 | | SSRN | not yet submitted | | Journal | targets: J. Financial Econometrics, J. Financial Data Science, J. Computational Finance | | Code & data | 55 test cases pass on macOS 27 / Xcode 27.2 (CPU + Metal); synthetic fixtures only in git; real bars via FinData OS (private license, not redistributed); every table/figure generated from `experiments/results/` | | Ledger | `RESEARCH_STATUS.md` · `docs/PROGRESS.md` · `docs/DECISIONS.md` · `docs/risks.md` | ## 🎯 What this paper does Three falsifiable questions. **RQ1** At what data and batch sizes does Metal beat an optimized multicore CPU, once I/O, shader compilation, synchronization and transfers are counted? **RQ2** Does batched estimation raise throughput while preserving statistical coverage and convergence? **RQ3** Can an Arrow/Parquet-to-estimator pipeline materially reduce copies, memory pressure and end-to-end latency? Negative results are published. Tier A methods (returns, calendar-aware aggregation, RV/BPV/RSV/RQ/jumps/rCov, OLS/WLS/ridge with HC/HAC/cluster inference, HAR family, GARCH/GJR/EGARCH with Gaussian/Student-t QML, block bootstrap) are validated against R `highfrequency`/`rugarch` and Python `statsmodels`/`arch`; GPU results are validated against the FP64 CPU reference with per-method tolerances. ## 🏆 Headline results | Finding | Evidence | |---|---| | Metal (FP32 double-float) matches FP64 within 1e−6 on realized measures/regression statistics and 2e−5 on GARCH likelihoods; bitwise reproducible across runs and thread counts | `tests/unit/test_metal_backend.cpp`, `experiments/results/raw/e5_inference.jsonl` | | Fused realized kernel: Metal wins from ≈ 4×10⁴ elements, 2.0–3.4× vs 18-thread CPU at 4×10⁶–4×10⁷ (kernel level); 1.07–1.41× at the estimator level on 40 M real returns | `bench_primitives_summary.json`, `raw/e1_batched_rv.jsonl` | | Batched GARCH likelihood grids: 5.1× (888 series × 256 vectors) to 7.9×; single-vector recursions: CPU wins | `raw/e3_garch.jsonl`, `raw/bench_primitives.jsonl` | | Rolling sums: CPU wins at every size (negative result) | `raw/bench_primitives.jsonl` | | Parquet→estimates pipeline: 20 M bars/s (RAID CPU) vs 14 M (Polars, DuckDB) vs 0.9 M (pandas); column projection mattered more than any kernel | `raw/e6_pipeline.jsonl`, `raw/e6_baselines.jsonl` | | 11 Apple Silicon machines (M2→M6): all Metal tests pass; GPU speedup 4.0× (M2) → 1.1× (M5 Max) — largest where the CPU is weakest | `raw/e7/`, `e7_summary.json` | | Tier A reproduces highfrequency / rugarch / statsmodels / arch to documented tolerances; Monte Carlo coverage 0.90–0.97 | `tests/reference/`, `docs/methods/registry.md` | ## 🗂 Repository layout ``` wp40_raid/ ├── include/raid/{core,data,statistics,estimators,models,backend}/ # public C++20 headers ├── src/… backends/cpu/ backends/metal/{host,shaders}/ # implementations (Metal: Phase 3) ├── bindings/python/ python/raid/ cli/ # nanobind module, Python API, `raid` CLI ├── tests/{unit,property,reference,simulation,stress,integration}/ # Catch2 (reference = vs statsmodels/arch/highfrequency) ├── benchmarks/ experiments/{configs,scripts,results,manifests}/ # harness + tracked artifacts (Phase 6) ├── datasets/{synthetic,fixtures}/ # synthetic only; real data cached outside git ├── docs/ paper/ scripts/ .github/workflows/ ``` ## 🚀 Quick start ```bash brew install cmake ninja eigen apache-arrow catch2 nlohmann-json # macOS cmake --preset macos-release && cmake --build --preset macos-release # CPU + Metal + Arrow ctest --preset macos-release # unit + reference tests ./build/macos-release/raid devices ./build/macos-release/raid validate datasets/fixtures/synthetic_bars_1m.csv ./build/macos-release/raid estimate realized datasets/fixtures/synthetic_bars_1m.csv --backend auto uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -e ".[dev]" # Python package (nanobind) ``` Linux / CPU-only: `cmake --preset cpu-only && cmake --build --preset cpu-only && ctest --preset cpu-only`. ## 🗃 Data Market data come from the owner's **FinData OS** lakehouse (`https://findata.spboucher.ai/api/v1`, FirstRate Data under a private-use license): 1-minute OHLCV bars (bar-open timestamps, extended 04:00–20:00 ET, regular 09:30–16:00 with the 16:00 closing print, zero-volume minutes absent at the source), dataset versions pinned via `X-Dataset-Version`. Raw bars are never committed; `datasets/fixtures/` contains synthetic bars generated by `scripts/make_fixtures.py`. ## 🔬 Methodology See `docs/methods/registry.md` (one entry per method: equations, conventions, reference software and tolerance) and `docs/architecture/overview.md`. ## 🖼 Figures & tables inventory Generated by `experiments/scripts/make_paper_assets.py` (hashes in `experiments/manifests/paper_assets.json`): `paper/tables/tab_{primitives,e1,e2_accuracy,e2_runtime,e3_fits,e4,e5,e6,e7}.tex`, `paper/figures/fig_{primitives_throughput,e1,e7}.pdf`. ## ⚠️ Limitations Apple Metal has no FP64: GPU paths are FP32 with compensated reductions and FP64 combination on the CPU, and are only valid where the documented tolerance is met. No tick/quote data: microstructure-noise and order-flow methods are out of scope. Energy is reported only when measured. ## 📖 Citation ```bibtex @techreport{boucher2026raid, title = {RAID: A Metal-Accelerated C++ Framework for Reproducible High-Frequency Financial Econometrics}, author = {Boucher, Simon-Pierre}, institution = {Université du Québec en Outaouais, Département des sciences administratives}, type = {UQO Working Paper}, number = {40}, year = {2026} } ``` ## 👤 Author & contact Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca — Département des sciences administratives, UQO. ## 📝 Notes Directory name `wp40_raid` (charter requirement) instead of the usual slug; git remote `gitsrv:srv/git/wp40_uqo.git`.