1# CLAUDE.md — UQO Research WP4023## RAID — Research Architecture for Intraday Data4### A Metal-Accelerated C++ Framework for High-Frequency Financial Econometrics56**Mission.** Build a research-grade, open-source C++20/Metal library, Python bindings, reproducible benchmark suite, and7submission-ready LaTeX methods paper. Mandatory data API: https://findata.spboucher.ai/api/v1. Work in the existing UQO8Research repository under `wp40_raid/`, respecting its conventions. Never invent performance, results, citations,9features, or novelty.1011Repository-specific notes (added 2026-11-02 at Phase 0):12- Location: `Recherche-UQO/finance/wp40_raid/` (UQO Working Paper No. 40; git remote `gitsrv:srv/git/wp40_uqo.git`).13- Paper conventions: `Recherche-UQO/HARMONISATION.md` (title page macros, `aer` bibliography style, `Month YYYY` date).14- FinData key: `FINDATA_API_KEY` env var, or `~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt` read by the15 helper scripts; never committed, never logged.16- Development machine: the owner's laptop (Apple M5 Max, 40 GPU cores, 48 GB, macOS 27.0, Xcode 27.2). MacLustr nodes17 are used only for E7 (generation sweep) with a self-contained binary bundle; no production service is touched.18- Status ledger: `RESEARCH_STATUS.md`; decision log: `docs/DECISIONS.md`; phase journal: `docs/PROGRESS.md`.1920## 1. Scientific thesis2122RAID is not merely a faster library. Test whether heterogeneous high-frequency econometric workloads can be transformed23into batched, fused, streaming computation on Apple Silicon without sacrificing estimator accuracy, statistical24inference, or reproducibility. Three falsifiable research questions: (1) At what data and batch sizes does Metal beat25optimized multicore CPU, including I/O, shader compilation, synchronization, and transfer? (2) Does batched estimation26improve throughput while preserving statistical coverage and convergence? (3) Can an Arrow/Parquet-to-estimator pipeline27materially reduce copies, memory pressure, and end-to-end latency? Explicitly report cases where CPU wins. A long model28list alone is not a contribution.2930Target journals: Journal of Financial Econometrics, Journal of Financial Data Science, Journal of Computational Finance;31consider broader outlets only if a strong methodological advance is demonstrated. Paper title: RAID: A Metal-Accelerated32C++ Framework for Reproducible High-Frequency Financial Econometrics.3334## 2. Mandatory discovery before coding3536Audit UQO Research and its WP structure, existing CMake/Python/LaTeX templates, Git practices, and previous volatility37research. Probe available MacLustr nodes, macOS/Xcode/Metal versions, Apple GPU capabilities, RAM, disk, load, power and38thermal conditions; never assume older inventory remains valid or interfere with production services. Read FinData39API.md, DATA_DICTIONARY.md, DISCOVERY.md and live OpenAPI/catalog. Produce docs/audit.md, docs/capabilities.md,40docs/novelty_matrix.md, docs/risks.md.4142Research relevant implementations in R highfrequency/rugarch/rmgarch, Python arch/statsmodels/NumPy/SciPy/Polars, Julia,43DuckDB, Eigen, Accelerate, BLAS/LAPACK and GPU toolkits. Verify versions, mathematical equivalence and citations.44Distinguish engineering novelty from new econometric methodology. Confirm software and data licensing. Apple Metal45itself is proprietary even if RAID is open source.4647## 3. Repository structure4849```50wp40_raid/51 CLAUDE.md README.md LICENSE CITATION.cff CMakeLists.txt CMakePresets.json pyproject.toml52 include/raid/{core,data,statistics,estimators,models,backend}/53 src/{core,data,statistics,estimators,models}/54 backends/cpu/ backends/metal/{host,shaders}/55 bindings/python/ python/raid/ cli/56 tests/{unit,property,reference,simulation,stress,integration}/57 benchmarks/{micro,end_to_end,accuracy,scalability,energy}/58 experiments/{configs,scripts,results,manifests}/59 datasets/{synthetic,fixtures}/60 docs/{architecture,api,methods,validation,operations}/61 examples/{cpp,python,notebooks}/62 paper/{main.tex,sections,figures,tables,appendices,references.bib}/63 scripts/ .github/workflows/64```6566## 4. Technical architecture6768Use modern C++20, RAII, explicit buffer ownership, strong types for timestamps/frequencies, safe exceptions/errors,69structured logging, deterministic seeds, bounded memory and thread pools. CPU reference backend: multicore C++ with70vetted Eigen/Accelerate/BLAS. Metal backend: Objective-C++ bridge, Metal Shading Language kernels, device capability71discovery, reusable compute pipelines, asynchronous command buffers, buffer pools, batching, kernel fusion, chunking,72backpressure and robust CPU fallback. Python bindings via nanobind or pybind11, NumPy/Arrow interoperability, correct73lifetimes and GIL release. CMake and installable macOS ARM64 wheels; portable CPU build on Linux. CLI: raid devices,74raid data, raid estimate, raid validate, raid benchmark.7576Core types: TimeSeries, Panel, BarSeries, MarketCalendar, QualityReport, ModelSpecification, Estimate, InferenceResult,77PrecisionPolicy, ExecutionPlan, Device, Backend, ExperimentManifest. Estimators must be backend-independent and expose78coefficients, loss, uncertainty where available, convergence, warnings, runtime, backend, precision and provenance.79Never treat failed optimizations as successful.8081Optimize columnar layouts, contiguous arrays, alignment, Arrow zero-copy views where actually possible, threadgroup82occupancy, memory access, register pressure, command overhead and synchronization. Unified memory is not free83computation. Apple Metal FP64 GPU support must never be assumed; use validated FP32/mixed precision or CPU FP6484fallbacks. Use stable reductions, log-domain likelihoods, parameter transforms, Cholesky/QR/SVD as appropriate,85conditioning checks, and matrix solves rather than explicit inversion. Make deterministic behavior and nondeterministic86GPU reductions visible.8788## 5. FinData integration — mandatory8990Use https://findata.spboucher.ai/api/v1, with API key only from FINDATA_API_KEY or secure store, bearer auth, TLS,91sanitized logs, retries/backoff, bounded concurrency, rate-limit compliance, pagination, caching, checksums and92provenance. Discover current endpoints via /api/v1/openapi.json and catalog. Implement /catalog/datasets, /market/bars,93/fundamentals/{entity}?as_of=..., asynchronous /exports and polling/download, and optionally bounded read-only /query94when justified. Use Arrow IPC and Parquet for large workloads.9596Documented constraints: 1m bars at most 31 days per direct call, 5m at most 186 days, 1d at most 10 years, max 10,00097rows/page; use bulk exports for historical panels. SQL playground: read-only, 30 seconds, 100,000 rows. Default API-key98rate limit is 600 requests/minute (check current configuration). Preserve X-Dataset-Id, X-Dataset-Version, request IDs,99archive hashes and dataset snapshots.100101FinData bars are OHLCV bars, not tick trades or quotes. timestamp_utc is bar OPEN, session_date uses exchange timezone,102US equities can include 04:00–20:00 ET, regular hours 09:30–16:00, and the 16:00 bar is a closing print. Zero-volume103minutes may be absent; do not manufacture bars or silently forward-fill. Respect DST, early closes, corporate actions,104quality flags, raw/split/split_div adjustments, delisted identifiers and symbol aliases. Maintain explicit105regular/extended-session policies. Licensed datasets may not yet be ingested: check catalog and coverage, never assume106historical completeness. Fundamental PIT available_at is a proxy, and historical revisions before ingestion may be107unrecoverable. FirstRate private-use raw data must not be redistributed or included in the public repository; synthetic108fixtures only in CI. Attribute data source in paper.109110## 6. Model registry and scope111112Maintain a registry for each method: precise equations, input schema, assumptions, data frequency, parameter113restrictions, estimation method, uncertainty, supported backend, numerical tolerance, test fixtures, validation status.114Use experimental or not implemented honestly; never publish stub methods as supported.115116**Tier A — production MVP, fully validated.** Data/statistics: log and arithmetic returns, lag/lead, irregular117resampling, rolling/expanding windows, OHLCV validation, calendar-aware aggregation, realized return/variance/volatility,118rolling correlations, drawdown, descriptive moments, autocovariance and robust missing-data policies. Realized measures:119RV, bipower variation, positive/negative semivariance, realized quarticity, realized covariance/correlation, jump proxies120with explicit finite-sample constants and sampling assumptions. No claims of microstructure-noise robustness without121appropriate estimator and validation. Regression: OLS, WLS, ridge, rolling/batched OLS, fixed effects, HC robust SE,122HAC/Newey-West, cluster-robust covariance, stable QR/SVD CPU reference, GPU batched sufficient-statistic accumulation123when advantageous. HAR: HAR-RV, HAR-logRV, HAR-X and HARQ with explicit definitions, causal lag construction, forecast124horizons and robust inference. Volatility: Gaussian/Student-t GARCH(1,1), GJR-GARCH(1,1), EGARCH(1,1), parameter125constraints, stable likelihood, initialization, gradients or validated numerical derivatives, optimizer convergence, SE126and forecasting. GPU primitives: parallel reductions, rolling sums/squares, covariance, batched matrix products, GARCH127likelihood, Monte Carlo and bootstrap replicas, batched HAR fitting. Measure precision and speed for every kernel.128129**Tier B — research extensions, only after Tier A passes.** Realized GARCH, HEAVY, GARCH-MIDAS, MIDAS, FIGARCH where130stable, ARMA/ARIMA, VAR, BVAR, VECM, DCC-GARCH, multivariate realized covariance, local projections, Granger tests,131generalized FEVD, Diebold–Yilmaz connectedness, rolling networks, factor models, PCA, shrinkage covariance, portfolio132risk/forecasting, quantile regression, state-space/Kalman filtering, change-point detection.133134**Tier C — experimental and data-dependent.** Pre-averaging, multi-scale RV, realized kernels, BNS and other jump135tests, lead-lag methods, high-dimensional covariance, stochastic volatility, sparse VAR, Bayesian MCMC, particle136filtering, regime switching, graph volatility models, Monte Carlo/quasi-Monte Carlo, Hawkes only if actual event137timestamps are separately available, and order-flow/order-book methods only with independently licensed trade/quote138data. Never infer true order-book imbalance or transaction-level microstructure from 1-minute OHLCV.139140## 7. Correctness, numerical validation and inference141142Every implemented method needs: (a) independent CPU FP64 reference, (b) analytical or simulation truth where possible,143(c) comparison to established software with identical definitions, (d) GPU/CPU agreement within documented144method-specific tolerance, (e) adversarial cases. Test zero/missing volume, holidays, DST, 16:00 print, split days,145irregular timestamps, extreme returns, singular matrices, near-unit-root GARCH, nonconvergence, insufficient samples,146memory pressure, cancellation and device fallback. Check gradients via finite differences where applicable. Separate147approximation error from floating-point error. Monte Carlo evaluation: bias, RMSE, empirical size, power, confidence148interval coverage, forecast loss, parameter convergence and failure rate. Use block/bootstrap schemes appropriate for149dependent financial data. Track random seed and parallel RNG streams. Set tolerances per algorithm and precision.150151## 8. Publication-grade benchmark protocol152153Compare RAID CPU single-thread, RAID multicore CPU, RAID Metal, and mathematically equivalent established competitors.154Record versions, build flags, machine, GPU, OS, clocks/power mode when measurable, memory, compiler, cache, precision and155convergence tolerance. Test series counts 1/10/100/1000/10000 and lengths 390/3900/39000/390000 where feasible. Include156sparse/irregular bars, 1m/5m, synthetic and explicitly identified real panels. Workloads: realized measures, rolling157covariance, HAR, GARCH, VAR, bootstrap, Monte Carlo, plus full FinData export → Arrow/Parquet → preprocessing → fit →158output. Report cold and warm cache, first shader compilation and cached compilation, synchronization, allocation and159end-to-end transfer. Use warmups, randomized benchmark order, multiple repetitions, medians and uncertainty intervals.160Metrics: wall-clock, fits/sec, bars/sec, peak RSS, buffer allocations, numerical error, convergence rate, energy only if161measured with a valid instrument. Speedup = T_reference / T_RAID for equivalent computations. Never extrapolate untested162throughput. Show break-even charts and CPU-wins cases. Include ablation of batching, kernel fusion, memory reuse,163precision, and I/O strategy.164165## 9. Research experiments166167E1 batched realized variance on thousands of equities, scaling versus panel width and window length. E2 HAR-RV/HAR-X168rolling estimation and 1/5/22-day forecasts, accuracy and runtime. E3 GARCH-family panel estimation, likelihood169accuracy, convergence, SE and forecast quality. E4 realized covariance and dynamic network connectedness with170memory/conditioning constraints. E5 bootstrap and Monte Carlo inference with coverage and reproducibility. E6 streaming171FinData Arrow/Parquet ingestion and end-to-end throughput. E7 stress tests across available Apple Silicon generations172and CPU-only fallback. Strict chronological splits and point-in-time handling for forecasting. Separate performance173benchmarking from predictive-model selection. Publish negative results.174175## 10. Paper structure176177Abstract; introduction and scientific gap; related econometric and systems literature; RAID architecture and API;178mathematical estimator definitions; Metal execution and precision strategy; statistical correctness; experimental setup;179microbenchmarks; end-to-end benchmarks; financial applications; threats to validity; reproducibility/licensing;180conclusion. Appendices: kernel pseudocode, optimizer details, finite-sample conventions, complete benchmark181configuration, reference-validation tables, numerical tolerance tables, API provenance and hardware inventory. Real182verified BibTeX references, journal-ready tables/figures, zero unresolved citations. Never claim peer review, novelty, or183superior accuracy without evidence.184185## 11. Execution phases and gates186187Phase 0 Audit → Phase 1 Foundation → Phase 2 Data → Phase 3 Numerical primitives → Phase 4 Econometrics → Phase 5188Extensions → Phase 6 Benchmarking → Phase 7 Manuscript → Phase 8 Independent review. Each gate is recorded in189`RESEARCH_STATUS.md` and `docs/PROGRESS.md` with the actual commands and results.190191## 12. Deliverables and release criteria192193Compiled C++ library, Metal kernels, Python wheel, CLI, public synthetic examples, API documentation, method registry,194tests, CI, benchmark harness, reproducibility manifests, versioned release, CITATION.cff, license and a complete195PDF/LaTeX paper. Minimal Python and C++ examples with `backend="auto|cpu|metal"` and clear CPU fallback.196`RESEARCH_STATUS.md` lists what is implemented, verified, experimental, blocked and planned. No public release until:197no secrets in git; no redistribution of raw FirstRate data; Tier A methods pass reference tests; Metal unsupported cases198fall back safely; performance claims are reproduced; all paper tables/figures derive from tracked artifacts; a clean199environment can build and run the project.200201## 13. Agent operating rules202203Work sequentially. At each phase: write a concise plan, implement, execute tests, record actual commands/results,204update RESEARCH_STATUS.md, and only advance when the gate passes. When blocked, document the blocker and the next safe205action rather than pretending success. Never rewrite unrelated UQO Research code, overwrite datasets, alter live206infrastructure, leak credentials, or launch unbounded vendor downloads. Prioritize statistical validity and a207publishable scientific contribution over breadth and marketing language.208