# CLAUDE.md — UQO Research WP40

# RAID — Research Architecture for Intraday Data

# A Metal-Accelerated C++ Framework for High-Frequency Financial Econometrics

Mission. Build a research-grade, open-source C++20/Metal library, Python bindings, reproducible benchmark suite, and submission-ready LaTeX methods paper. Mandatory data API: https://findata.spboucher.ai/api/v1. Work in the existing UQO Research repository under wp40_raid/, respecting its conventions. Never invent performance, results, citations, features, or novelty.

Repository-specific notes (added 2026-11-02 at Phase 0):

Location: Recherche-UQO/finance/wp40_raid/ (UQO Working Paper No. 40; git remote gitsrv:srv/git/wp40_uqo.git).
Paper conventions: Recherche-UQO/HARMONISATION.md (title page macros, aer bibliography style, Month YYYY date).
FinData key: FINDATA_API_KEY env var, or ~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt read by the helper scripts; never committed, never logged.
Development machine: the owner's laptop (Apple M5 Max, 40 GPU cores, 48 GB, macOS 27.0, Xcode 27.2). MacLustr nodes are used only for E7 (generation sweep) with a self-contained binary bundle; no production service is touched.
Status ledger: RESEARCH_STATUS.md; decision log: docs/DECISIONS.md; phase journal: docs/PROGRESS.md.

# 1. Scientific thesis

RAID is not merely a faster library. Test whether heterogeneous high-frequency econometric workloads can be transformed into batched, fused, streaming computation on Apple Silicon without sacrificing estimator accuracy, statistical inference, or reproducibility. Three falsifiable research questions: (1) At what data and batch sizes does Metal beat optimized multicore CPU, including I/O, shader compilation, synchronization, and transfer? (2) Does batched estimation improve throughput while preserving statistical coverage and convergence? (3) Can an Arrow/Parquet-to-estimator pipeline materially reduce copies, memory pressure, and end-to-end latency? Explicitly report cases where CPU wins. A long model list alone is not a contribution.

Target journals: Journal of Financial Econometrics, Journal of Financial Data Science, Journal of Computational Finance; consider broader outlets only if a strong methodological advance is demonstrated. Paper title: RAID: A Metal-Accelerated C++ Framework for Reproducible High-Frequency Financial Econometrics.

# 2. Mandatory discovery before coding

Audit UQO Research and its WP structure, existing CMake/Python/LaTeX templates, Git practices, and previous volatility research. Probe available MacLustr nodes, macOS/Xcode/Metal versions, Apple GPU capabilities, RAM, disk, load, power and thermal conditions; never assume older inventory remains valid or interfere with production services. Read FinData API.md, DATA_DICTIONARY.md, DISCOVERY.md and live OpenAPI/catalog. Produce docs/audit.md, docs/capabilities.md, docs/novelty_matrix.md, docs/risks.md.

Research relevant implementations in R highfrequency/rugarch/rmgarch, Python arch/statsmodels/NumPy/SciPy/Polars, Julia, DuckDB, Eigen, Accelerate, BLAS/LAPACK and GPU toolkits. Verify versions, mathematical equivalence and citations. Distinguish engineering novelty from new econometric methodology. Confirm software and data licensing. Apple Metal itself is proprietary even if RAID is open source.

# 3. Repository structure

text

wp40_raid/
  CLAUDE.md README.md LICENSE CITATION.cff CMakeLists.txt CMakePresets.json pyproject.toml
  include/raid/{core,data,statistics,estimators,models,backend}/
  src/{core,data,statistics,estimators,models}/
  backends/cpu/ backends/metal/{host,shaders}/
  bindings/python/ python/raid/ cli/
  tests/{unit,property,reference,simulation,stress,integration}/
  benchmarks/{micro,end_to_end,accuracy,scalability,energy}/
  experiments/{configs,scripts,results,manifests}/
  datasets/{synthetic,fixtures}/
  docs/{architecture,api,methods,validation,operations}/
  examples/{cpp,python,notebooks}/
  paper/{main.tex,sections,figures,tables,appendices,references.bib}/
  scripts/ .github/workflows/

# 4. Technical architecture

Use modern C++20, RAII, explicit buffer ownership, strong types for timestamps/frequencies, safe exceptions/errors, structured logging, deterministic seeds, bounded memory and thread pools. CPU reference backend: multicore C++ with vetted Eigen/Accelerate/BLAS. Metal backend: Objective-C++ bridge, Metal Shading Language kernels, device capability discovery, reusable compute pipelines, asynchronous command buffers, buffer pools, batching, kernel fusion, chunking, backpressure and robust CPU fallback. Python bindings via nanobind or pybind11, NumPy/Arrow interoperability, correct lifetimes and GIL release. CMake and installable macOS ARM64 wheels; portable CPU build on Linux. CLI: raid devices, raid data, raid estimate, raid validate, raid benchmark.

Core types: TimeSeries, Panel, BarSeries, MarketCalendar, QualityReport, ModelSpecification, Estimate, InferenceResult, PrecisionPolicy, ExecutionPlan, Device, Backend, ExperimentManifest. Estimators must be backend-independent and expose coefficients, loss, uncertainty where available, convergence, warnings, runtime, backend, precision and provenance. Never treat failed optimizations as successful.

Optimize columnar layouts, contiguous arrays, alignment, Arrow zero-copy views where actually possible, threadgroup occupancy, memory access, register pressure, command overhead and synchronization. Unified memory is not free computation. Apple Metal FP64 GPU support must never be assumed; use validated FP32/mixed precision or CPU FP64 fallbacks. Use stable reductions, log-domain likelihoods, parameter transforms, Cholesky/QR/SVD as appropriate, conditioning checks, and matrix solves rather than explicit inversion. Make deterministic behavior and nondeterministic GPU reductions visible.

# 5. FinData integration — mandatory

Use https://findata.spboucher.ai/api/v1, with API key only from FINDATA_API_KEY or secure store, bearer auth, TLS, sanitized logs, retries/backoff, bounded concurrency, rate-limit compliance, pagination, caching, checksums and provenance. Discover current endpoints via /api/v1/openapi.json and catalog. Implement /catalog/datasets, /market/bars, /fundamentals/{entity}?as_of=..., asynchronous /exports and polling/download, and optionally bounded read-only /query when justified. Use Arrow IPC and Parquet for large workloads.

Documented constraints: 1m bars at most 31 days per direct call, 5m at most 186 days, 1d at most 10 years, max 10,000 rows/page; use bulk exports for historical panels. SQL playground: read-only, 30 seconds, 100,000 rows. Default API-key rate limit is 600 requests/minute (check current configuration). Preserve X-Dataset-Id, X-Dataset-Version, request IDs, archive hashes and dataset snapshots.

FinData bars are OHLCV bars, not tick trades or quotes. timestamp_utc is bar OPEN, session_date uses exchange timezone, US equities can include 04:00–20:00 ET, regular hours 09:30–16:00, and the 16:00 bar is a closing print. Zero-volume minutes may be absent; do not manufacture bars or silently forward-fill. Respect DST, early closes, corporate actions, quality flags, raw/split/split_div adjustments, delisted identifiers and symbol aliases. Maintain explicit regular/extended-session policies. Licensed datasets may not yet be ingested: check catalog and coverage, never assume historical completeness. Fundamental PIT available_at is a proxy, and historical revisions before ingestion may be unrecoverable. FirstRate private-use raw data must not be redistributed or included in the public repository; synthetic fixtures only in CI. Attribute data source in paper.

# 6. Model registry and scope

Maintain a registry for each method: precise equations, input schema, assumptions, data frequency, parameter restrictions, estimation method, uncertainty, supported backend, numerical tolerance, test fixtures, validation status. Use experimental or not implemented honestly; never publish stub methods as supported.

Tier A — production MVP, fully validated. Data/statistics: log and arithmetic returns, lag/lead, irregular resampling, rolling/expanding windows, OHLCV validation, calendar-aware aggregation, realized return/variance/volatility, rolling correlations, drawdown, descriptive moments, autocovariance and robust missing-data policies. Realized measures: RV, bipower variation, positive/negative semivariance, realized quarticity, realized covariance/correlation, jump proxies with explicit finite-sample constants and sampling assumptions. No claims of microstructure-noise robustness without appropriate estimator and validation. Regression: OLS, WLS, ridge, rolling/batched OLS, fixed effects, HC robust SE, HAC/Newey-West, cluster-robust covariance, stable QR/SVD CPU reference, GPU batched sufficient-statistic accumulation when advantageous. HAR: HAR-RV, HAR-logRV, HAR-X and HARQ with explicit definitions, causal lag construction, forecast horizons and robust inference. Volatility: Gaussian/Student-t GARCH(1,1), GJR-GARCH(1,1), EGARCH(1,1), parameter constraints, stable likelihood, initialization, gradients or validated numerical derivatives, optimizer convergence, SE and forecasting. GPU primitives: parallel reductions, rolling sums/squares, covariance, batched matrix products, GARCH likelihood, Monte Carlo and bootstrap replicas, batched HAR fitting. Measure precision and speed for every kernel.

Tier B — research extensions, only after Tier A passes. Realized GARCH, HEAVY, GARCH-MIDAS, MIDAS, FIGARCH where stable, ARMA/ARIMA, VAR, BVAR, VECM, DCC-GARCH, multivariate realized covariance, local projections, Granger tests, generalized FEVD, Diebold–Yilmaz connectedness, rolling networks, factor models, PCA, shrinkage covariance, portfolio risk/forecasting, quantile regression, state-space/Kalman filtering, change-point detection.

Tier C — experimental and data-dependent. Pre-averaging, multi-scale RV, realized kernels, BNS and other jump tests, lead-lag methods, high-dimensional covariance, stochastic volatility, sparse VAR, Bayesian MCMC, particle filtering, regime switching, graph volatility models, Monte Carlo/quasi-Monte Carlo, Hawkes only if actual event timestamps are separately available, and order-flow/order-book methods only with independently licensed trade/quote data. Never infer true order-book imbalance or transaction-level microstructure from 1-minute OHLCV.

# 7. Correctness, numerical validation and inference

Every implemented method needs: (a) independent CPU FP64 reference, (b) analytical or simulation truth where possible, (c) comparison to established software with identical definitions, (d) GPU/CPU agreement within documented method-specific tolerance, (e) adversarial cases. Test zero/missing volume, holidays, DST, 16:00 print, split days, irregular timestamps, extreme returns, singular matrices, near-unit-root GARCH, nonconvergence, insufficient samples, memory pressure, cancellation and device fallback. Check gradients via finite differences where applicable. Separate approximation error from floating-point error. Monte Carlo evaluation: bias, RMSE, empirical size, power, confidence interval coverage, forecast loss, parameter convergence and failure rate. Use block/bootstrap schemes appropriate for dependent financial data. Track random seed and parallel RNG streams. Set tolerances per algorithm and precision.

# 8. Publication-grade benchmark protocol

Compare RAID CPU single-thread, RAID multicore CPU, RAID Metal, and mathematically equivalent established competitors. Record versions, build flags, machine, GPU, OS, clocks/power mode when measurable, memory, compiler, cache, precision and convergence tolerance. Test series counts 1/10/100/1000/10000 and lengths 390/3900/39000/390000 where feasible. Include sparse/irregular bars, 1m/5m, synthetic and explicitly identified real panels. Workloads: realized measures, rolling covariance, HAR, GARCH, VAR, bootstrap, Monte Carlo, plus full FinData export → Arrow/Parquet → preprocessing → fit → output. Report cold and warm cache, first shader compilation and cached compilation, synchronization, allocation and end-to-end transfer. Use warmups, randomized benchmark order, multiple repetitions, medians and uncertainty intervals. Metrics: wall-clock, fits/sec, bars/sec, peak RSS, buffer allocations, numerical error, convergence rate, energy only if measured with a valid instrument. Speedup = T_reference / T_RAID for equivalent computations. Never extrapolate untested throughput. Show break-even charts and CPU-wins cases. Include ablation of batching, kernel fusion, memory reuse, precision, and I/O strategy.

# 9. Research experiments

E1 batched realized variance on thousands of equities, scaling versus panel width and window length. E2 HAR-RV/HAR-X rolling estimation and 1/5/22-day forecasts, accuracy and runtime. E3 GARCH-family panel estimation, likelihood accuracy, convergence, SE and forecast quality. E4 realized covariance and dynamic network connectedness with memory/conditioning constraints. E5 bootstrap and Monte Carlo inference with coverage and reproducibility. E6 streaming FinData Arrow/Parquet ingestion and end-to-end throughput. E7 stress tests across available Apple Silicon generations and CPU-only fallback. Strict chronological splits and point-in-time handling for forecasting. Separate performance benchmarking from predictive-model selection. Publish negative results.

# 10. Paper structure

Abstract; introduction and scientific gap; related econometric and systems literature; RAID architecture and API; mathematical estimator definitions; Metal execution and precision strategy; statistical correctness; experimental setup; microbenchmarks; end-to-end benchmarks; financial applications; threats to validity; reproducibility/licensing; conclusion. Appendices: kernel pseudocode, optimizer details, finite-sample conventions, complete benchmark configuration, reference-validation tables, numerical tolerance tables, API provenance and hardware inventory. Real verified BibTeX references, journal-ready tables/figures, zero unresolved citations. Never claim peer review, novelty, or superior accuracy without evidence.

# 11. Execution phases and gates

Phase 0 Audit → Phase 1 Foundation → Phase 2 Data → Phase 3 Numerical primitives → Phase 4 Econometrics → Phase 5 Extensions → Phase 6 Benchmarking → Phase 7 Manuscript → Phase 8 Independent review. Each gate is recorded in RESEARCH_STATUS.md and docs/PROGRESS.md with the actual commands and results.

# 12. Deliverables and release criteria

Compiled C++ library, Metal kernels, Python wheel, CLI, public synthetic examples, API documentation, method registry, tests, CI, benchmark harness, reproducibility manifests, versioned release, CITATION.cff, license and a complete PDF/LaTeX paper. Minimal Python and C++ examples with backend="auto|cpu|metal" and clear CPU fallback. RESEARCH_STATUS.md lists what is implemented, verified, experimental, blocked and planned. No public release until: no secrets in git; no redistribution of raw FirstRate data; Tier A methods pass reference tests; Metal unsupported cases fall back safely; performance claims are reproduced; all paper tables/figures derive from tracked artifacts; a clean environment can build and run the project.

# 13. Agent operating rules

Work sequentially. At each phase: write a concise plan, implement, execute tests, record actual commands/results, update RESEARCH_STATUS.md, and only advance when the gate passes. When blocked, document the blocker and the next safe action rather than pretending success. Never rewrite unrelated UQO Research code, overwrite datasets, alter live infrastructure, leak credentials, or launch unbounded vendor downloads. Prioritize statistical validity and a publishable scientific contribution over breadth and marketing language.

spb/wp40_uqo