| |
1 | +# CLAUDE.md — UQO Research WP40 |
| |
2 | + |
| |
3 | +## RAID — Research Architecture for Intraday Data |
| |
4 | +### A Metal-Accelerated C++ Framework for High-Frequency Financial Econometrics |
| |
5 | + |
| |
6 | +**Mission.** Build a research-grade, open-source C++20/Metal library, Python bindings, reproducible benchmark suite, and |
| |
7 | +submission-ready LaTeX methods paper. Mandatory data API: https://findata.spboucher.ai/api/v1. Work in the existing UQO |
| |
8 | +Research repository under `wp40_raid/`, respecting its conventions. Never invent performance, results, citations, |
| |
9 | +features, or novelty. |
| |
10 | + |
| |
11 | +Repository-specific notes (added 2026-11-02 at Phase 0): |
| |
12 | +- Location: `Recherche-UQO/finance/wp40_raid/` (UQO Working Paper No. 40; git remote `gitsrv:srv/git/wp40_uqo.git`). |
| |
13 | +- Paper conventions: `Recherche-UQO/HARMONISATION.md` (title page macros, `aer` bibliography style, `Month YYYY` date). |
| |
14 | +- FinData key: `FINDATA_API_KEY` env var, or `~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt` read by the |
| |
15 | + helper scripts; never committed, never logged. |
| |
16 | +- Development machine: the owner's laptop (Apple M5 Max, 40 GPU cores, 48 GB, macOS 27.0, Xcode 27.2). MacLustr nodes |
| |
17 | + are used only for E7 (generation sweep) with a self-contained binary bundle; no production service is touched. |
| |
18 | +- Status ledger: `RESEARCH_STATUS.md`; decision log: `docs/DECISIONS.md`; phase journal: `docs/PROGRESS.md`. |
| |
19 | + |
| |
20 | +## 1. Scientific thesis |
| |
21 | + |
| |
22 | +RAID is not merely a faster library. Test whether heterogeneous high-frequency econometric workloads can be transformed |
| |
23 | +into batched, fused, streaming computation on Apple Silicon without sacrificing estimator accuracy, statistical |
| |
24 | +inference, or reproducibility. Three falsifiable research questions: (1) At what data and batch sizes does Metal beat |
| |
25 | +optimized multicore CPU, including I/O, shader compilation, synchronization, and transfer? (2) Does batched estimation |
| |
26 | +improve throughput while preserving statistical coverage and convergence? (3) Can an Arrow/Parquet-to-estimator pipeline |
| |
27 | +materially reduce copies, memory pressure, and end-to-end latency? Explicitly report cases where CPU wins. A long model |
| |
28 | +list alone is not a contribution. |
| |
29 | + |
| |
30 | +Target journals: Journal of Financial Econometrics, Journal of Financial Data Science, Journal of Computational Finance; |
| |
31 | +consider broader outlets only if a strong methodological advance is demonstrated. Paper title: RAID: A Metal-Accelerated |
| |
32 | +C++ Framework for Reproducible High-Frequency Financial Econometrics. |
| |
33 | + |
| |
34 | +## 2. Mandatory discovery before coding |
| |
35 | + |
| |
36 | +Audit UQO Research and its WP structure, existing CMake/Python/LaTeX templates, Git practices, and previous volatility |
| |
37 | +research. Probe available MacLustr nodes, macOS/Xcode/Metal versions, Apple GPU capabilities, RAM, disk, load, power and |
| |
38 | +thermal conditions; never assume older inventory remains valid or interfere with production services. Read FinData |
| |
39 | +API.md, DATA_DICTIONARY.md, DISCOVERY.md and live OpenAPI/catalog. Produce docs/audit.md, docs/capabilities.md, |
| |
40 | +docs/novelty_matrix.md, docs/risks.md. |
| |
41 | + |
| |
42 | +Research relevant implementations in R highfrequency/rugarch/rmgarch, Python arch/statsmodels/NumPy/SciPy/Polars, Julia, |
| |
43 | +DuckDB, Eigen, Accelerate, BLAS/LAPACK and GPU toolkits. Verify versions, mathematical equivalence and citations. |
| |
44 | +Distinguish engineering novelty from new econometric methodology. Confirm software and data licensing. Apple Metal |
| |
45 | +itself is proprietary even if RAID is open source. |
| |
46 | + |
| |
47 | +## 3. Repository structure |
| |
48 | + |
| |
49 | +``` |
| |
50 | +wp40_raid/ |
| |
51 | + CLAUDE.md README.md LICENSE CITATION.cff CMakeLists.txt CMakePresets.json pyproject.toml |
| |
52 | + include/raid/{core,data,statistics,estimators,models,backend}/ |
| |
53 | + src/{core,data,statistics,estimators,models}/ |
| |
54 | + backends/cpu/ backends/metal/{host,shaders}/ |
| |
55 | + bindings/python/ python/raid/ cli/ |
| |
56 | + tests/{unit,property,reference,simulation,stress,integration}/ |
| |
57 | + benchmarks/{micro,end_to_end,accuracy,scalability,energy}/ |
| |
58 | + experiments/{configs,scripts,results,manifests}/ |
| |
59 | + datasets/{synthetic,fixtures}/ |
| |
60 | + docs/{architecture,api,methods,validation,operations}/ |
| |
61 | + examples/{cpp,python,notebooks}/ |
| |
62 | + paper/{main.tex,sections,figures,tables,appendices,references.bib}/ |
| |
63 | + scripts/ .github/workflows/ |
| |
64 | +``` |
| |
65 | + |
| |
66 | +## 4. Technical architecture |
| |
67 | + |
| |
68 | +Use modern C++20, RAII, explicit buffer ownership, strong types for timestamps/frequencies, safe exceptions/errors, |
| |
69 | +structured logging, deterministic seeds, bounded memory and thread pools. CPU reference backend: multicore C++ with |
| |
70 | +vetted Eigen/Accelerate/BLAS. Metal backend: Objective-C++ bridge, Metal Shading Language kernels, device capability |
| |
71 | +discovery, reusable compute pipelines, asynchronous command buffers, buffer pools, batching, kernel fusion, chunking, |
| |
72 | +backpressure and robust CPU fallback. Python bindings via nanobind or pybind11, NumPy/Arrow interoperability, correct |
| |
73 | +lifetimes and GIL release. CMake and installable macOS ARM64 wheels; portable CPU build on Linux. CLI: raid devices, |
| |
74 | +raid data, raid estimate, raid validate, raid benchmark. |
| |
75 | + |
| |
76 | +Core types: TimeSeries, Panel, BarSeries, MarketCalendar, QualityReport, ModelSpecification, Estimate, InferenceResult, |
| |
77 | +PrecisionPolicy, ExecutionPlan, Device, Backend, ExperimentManifest. Estimators must be backend-independent and expose |
| |
78 | +coefficients, loss, uncertainty where available, convergence, warnings, runtime, backend, precision and provenance. |
| |
79 | +Never treat failed optimizations as successful. |
| |
80 | + |
| |
81 | +Optimize columnar layouts, contiguous arrays, alignment, Arrow zero-copy views where actually possible, threadgroup |
| |
82 | +occupancy, memory access, register pressure, command overhead and synchronization. Unified memory is not free |
| |
83 | +computation. Apple Metal FP64 GPU support must never be assumed; use validated FP32/mixed precision or CPU FP64 |
| |
84 | +fallbacks. Use stable reductions, log-domain likelihoods, parameter transforms, Cholesky/QR/SVD as appropriate, |
| |
85 | +conditioning checks, and matrix solves rather than explicit inversion. Make deterministic behavior and nondeterministic |
| |
86 | +GPU reductions visible. |
| |
87 | + |
| |
88 | +## 5. FinData integration — mandatory |
| |
89 | + |
| |
90 | +Use https://findata.spboucher.ai/api/v1, with API key only from FINDATA_API_KEY or secure store, bearer auth, TLS, |
| |
91 | +sanitized logs, retries/backoff, bounded concurrency, rate-limit compliance, pagination, caching, checksums and |
| |
92 | +provenance. Discover current endpoints via /api/v1/openapi.json and catalog. Implement /catalog/datasets, /market/bars, |
| |
93 | +/fundamentals/{entity}?as_of=..., asynchronous /exports and polling/download, and optionally bounded read-only /query |
| |
94 | +when justified. Use Arrow IPC and Parquet for large workloads. |
| |
95 | + |
| |
96 | +Documented constraints: 1m bars at most 31 days per direct call, 5m at most 186 days, 1d at most 10 years, max 10,000 |
| |
97 | +rows/page; use bulk exports for historical panels. SQL playground: read-only, 30 seconds, 100,000 rows. Default API-key |
| |
98 | +rate limit is 600 requests/minute (check current configuration). Preserve X-Dataset-Id, X-Dataset-Version, request IDs, |
| |
99 | +archive hashes and dataset snapshots. |
| |
100 | + |
| |
101 | +FinData bars are OHLCV bars, not tick trades or quotes. timestamp_utc is bar OPEN, session_date uses exchange timezone, |
| |
102 | +US equities can include 04:00–20:00 ET, regular hours 09:30–16:00, and the 16:00 bar is a closing print. Zero-volume |
| |
103 | +minutes may be absent; do not manufacture bars or silently forward-fill. Respect DST, early closes, corporate actions, |
| |
104 | +quality flags, raw/split/split_div adjustments, delisted identifiers and symbol aliases. Maintain explicit |
| |
105 | +regular/extended-session policies. Licensed datasets may not yet be ingested: check catalog and coverage, never assume |
| |
106 | +historical completeness. Fundamental PIT available_at is a proxy, and historical revisions before ingestion may be |
| |
107 | +unrecoverable. FirstRate private-use raw data must not be redistributed or included in the public repository; synthetic |
| |
108 | +fixtures only in CI. Attribute data source in paper. |
| |
109 | + |
| |
110 | +## 6. Model registry and scope |
| |
111 | + |
| |
112 | +Maintain a registry for each method: precise equations, input schema, assumptions, data frequency, parameter |
| |
113 | +restrictions, estimation method, uncertainty, supported backend, numerical tolerance, test fixtures, validation status. |
| |
114 | +Use experimental or not implemented honestly; never publish stub methods as supported. |
| |
115 | + |
| |
116 | +**Tier A — production MVP, fully validated.** Data/statistics: log and arithmetic returns, lag/lead, irregular |
| |
117 | +resampling, rolling/expanding windows, OHLCV validation, calendar-aware aggregation, realized return/variance/volatility, |
| |
118 | +rolling correlations, drawdown, descriptive moments, autocovariance and robust missing-data policies. Realized measures: |
| |
119 | +RV, bipower variation, positive/negative semivariance, realized quarticity, realized covariance/correlation, jump proxies |
| |
120 | +with explicit finite-sample constants and sampling assumptions. No claims of microstructure-noise robustness without |
| |
121 | +appropriate estimator and validation. Regression: OLS, WLS, ridge, rolling/batched OLS, fixed effects, HC robust SE, |
| |
122 | +HAC/Newey-West, cluster-robust covariance, stable QR/SVD CPU reference, GPU batched sufficient-statistic accumulation |
| |
123 | +when advantageous. HAR: HAR-RV, HAR-logRV, HAR-X and HARQ with explicit definitions, causal lag construction, forecast |
| |
124 | +horizons and robust inference. Volatility: Gaussian/Student-t GARCH(1,1), GJR-GARCH(1,1), EGARCH(1,1), parameter |
| |
125 | +constraints, stable likelihood, initialization, gradients or validated numerical derivatives, optimizer convergence, SE |
| |
126 | +and forecasting. GPU primitives: parallel reductions, rolling sums/squares, covariance, batched matrix products, GARCH |
| |
127 | +likelihood, Monte Carlo and bootstrap replicas, batched HAR fitting. Measure precision and speed for every kernel. |
| |
128 | + |
| |
129 | +**Tier B — research extensions, only after Tier A passes.** Realized GARCH, HEAVY, GARCH-MIDAS, MIDAS, FIGARCH where |
| |
130 | +stable, ARMA/ARIMA, VAR, BVAR, VECM, DCC-GARCH, multivariate realized covariance, local projections, Granger tests, |
| |
131 | +generalized FEVD, Diebold–Yilmaz connectedness, rolling networks, factor models, PCA, shrinkage covariance, portfolio |
| |
132 | +risk/forecasting, quantile regression, state-space/Kalman filtering, change-point detection. |
| |
133 | + |
| |
134 | +**Tier C — experimental and data-dependent.** Pre-averaging, multi-scale RV, realized kernels, BNS and other jump |
| |
135 | +tests, lead-lag methods, high-dimensional covariance, stochastic volatility, sparse VAR, Bayesian MCMC, particle |
| |
136 | +filtering, regime switching, graph volatility models, Monte Carlo/quasi-Monte Carlo, Hawkes only if actual event |
| |
137 | +timestamps are separately available, and order-flow/order-book methods only with independently licensed trade/quote |
| |
138 | +data. Never infer true order-book imbalance or transaction-level microstructure from 1-minute OHLCV. |
| |
139 | + |
| |
140 | +## 7. Correctness, numerical validation and inference |
| |
141 | + |
| |
142 | +Every implemented method needs: (a) independent CPU FP64 reference, (b) analytical or simulation truth where possible, |
| |
143 | +(c) comparison to established software with identical definitions, (d) GPU/CPU agreement within documented |
| |
144 | +method-specific tolerance, (e) adversarial cases. Test zero/missing volume, holidays, DST, 16:00 print, split days, |
| |
145 | +irregular timestamps, extreme returns, singular matrices, near-unit-root GARCH, nonconvergence, insufficient samples, |
| |
146 | +memory pressure, cancellation and device fallback. Check gradients via finite differences where applicable. Separate |
| |
147 | +approximation error from floating-point error. Monte Carlo evaluation: bias, RMSE, empirical size, power, confidence |
| |
148 | +interval coverage, forecast loss, parameter convergence and failure rate. Use block/bootstrap schemes appropriate for |
| |
149 | +dependent financial data. Track random seed and parallel RNG streams. Set tolerances per algorithm and precision. |
| |
150 | + |
| |
151 | +## 8. Publication-grade benchmark protocol |
| |
152 | + |
| |
153 | +Compare RAID CPU single-thread, RAID multicore CPU, RAID Metal, and mathematically equivalent established competitors. |
| |
154 | +Record versions, build flags, machine, GPU, OS, clocks/power mode when measurable, memory, compiler, cache, precision and |
| |
155 | +convergence tolerance. Test series counts 1/10/100/1000/10000 and lengths 390/3900/39000/390000 where feasible. Include |
| |
156 | +sparse/irregular bars, 1m/5m, synthetic and explicitly identified real panels. Workloads: realized measures, rolling |
| |
157 | +covariance, HAR, GARCH, VAR, bootstrap, Monte Carlo, plus full FinData export → Arrow/Parquet → preprocessing → fit → |
| |
158 | +output. Report cold and warm cache, first shader compilation and cached compilation, synchronization, allocation and |
| |
159 | +end-to-end transfer. Use warmups, randomized benchmark order, multiple repetitions, medians and uncertainty intervals. |
| |
160 | +Metrics: wall-clock, fits/sec, bars/sec, peak RSS, buffer allocations, numerical error, convergence rate, energy only if |
| |
161 | +measured with a valid instrument. Speedup = T_reference / T_RAID for equivalent computations. Never extrapolate untested |
| |
162 | +throughput. Show break-even charts and CPU-wins cases. Include ablation of batching, kernel fusion, memory reuse, |
| |
163 | +precision, and I/O strategy. |
| |
164 | + |
| |
165 | +## 9. Research experiments |
| |
166 | + |
| |
167 | +E1 batched realized variance on thousands of equities, scaling versus panel width and window length. E2 HAR-RV/HAR-X |
| |
168 | +rolling estimation and 1/5/22-day forecasts, accuracy and runtime. E3 GARCH-family panel estimation, likelihood |
| |
169 | +accuracy, convergence, SE and forecast quality. E4 realized covariance and dynamic network connectedness with |
| |
170 | +memory/conditioning constraints. E5 bootstrap and Monte Carlo inference with coverage and reproducibility. E6 streaming |
| |
171 | +FinData Arrow/Parquet ingestion and end-to-end throughput. E7 stress tests across available Apple Silicon generations |
| |
172 | +and CPU-only fallback. Strict chronological splits and point-in-time handling for forecasting. Separate performance |
| |
173 | +benchmarking from predictive-model selection. Publish negative results. |
| |
174 | + |
| |
175 | +## 10. Paper structure |
| |
176 | + |
| |
177 | +Abstract; introduction and scientific gap; related econometric and systems literature; RAID architecture and API; |
| |
178 | +mathematical estimator definitions; Metal execution and precision strategy; statistical correctness; experimental setup; |
| |
179 | +microbenchmarks; end-to-end benchmarks; financial applications; threats to validity; reproducibility/licensing; |
| |
180 | +conclusion. Appendices: kernel pseudocode, optimizer details, finite-sample conventions, complete benchmark |
| |
181 | +configuration, reference-validation tables, numerical tolerance tables, API provenance and hardware inventory. Real |
| |
182 | +verified BibTeX references, journal-ready tables/figures, zero unresolved citations. Never claim peer review, novelty, or |
| |
183 | +superior accuracy without evidence. |
| |
184 | + |
| |
185 | +## 11. Execution phases and gates |
| |
186 | + |
| |
187 | +Phase 0 Audit → Phase 1 Foundation → Phase 2 Data → Phase 3 Numerical primitives → Phase 4 Econometrics → Phase 5 |
| |
188 | +Extensions → Phase 6 Benchmarking → Phase 7 Manuscript → Phase 8 Independent review. Each gate is recorded in |
| |
189 | +`RESEARCH_STATUS.md` and `docs/PROGRESS.md` with the actual commands and results. |
| |
190 | + |
| |
191 | +## 12. Deliverables and release criteria |
| |
192 | + |
| |
193 | +Compiled C++ library, Metal kernels, Python wheel, CLI, public synthetic examples, API documentation, method registry, |
| |
194 | +tests, CI, benchmark harness, reproducibility manifests, versioned release, CITATION.cff, license and a complete |
| |
195 | +PDF/LaTeX paper. Minimal Python and C++ examples with `backend="auto|cpu|metal"` and clear CPU fallback. |
| |
196 | +`RESEARCH_STATUS.md` lists what is implemented, verified, experimental, blocked and planned. No public release until: |
| |
197 | +no secrets in git; no redistribution of raw FirstRate data; Tier A methods pass reference tests; Metal unsupported cases |
| |
198 | +fall back safely; performance claims are reproduced; all paper tables/figures derive from tracked artifacts; a clean |
| |
199 | +environment can build and run the project. |
| |
200 | + |
| |
201 | +## 13. Agent operating rules |
| |
202 | + |
| |
203 | +Work sequentially. At each phase: write a concise plan, implement, execute tests, record actual commands/results, |
| |
204 | +update RESEARCH_STATUS.md, and only advance when the gate passes. When blocked, document the blocker and the next safe |
| |
205 | +action rather than pretending success. Never rewrite unrelated UQO Research code, overwrite datasets, alter live |
| |
206 | +infrastructure, leak credentials, or launch unbounded vendor downloads. Prioritize statistical validity and a |
| |
207 | +publishable scientific contribution over breadth and marketing language. |