|
1 |
+# CLAUDE.md — UQO Research WP40 |
|
2 |
+ |
|
3 |
+## RAID — Research Architecture for Intraday Data |
|
4 |
+### A Metal-Accelerated C++ Framework for High-Frequency Financial Econometrics |
|
5 |
+ |
|
6 |
+**Mission.** Build a research-grade, open-source C++20/Metal library, Python bindings, reproducible benchmark suite, and |
|
7 |
+submission-ready LaTeX methods paper. Mandatory data API: https://findata.spboucher.ai/api/v1. Work in the existing UQO |
|
8 |
+Research repository under `wp40_raid/`, respecting its conventions. Never invent performance, results, citations, |
|
9 |
+features, or novelty. |
|
10 |
+ |
|
11 |
+Repository-specific notes (added 2026-11-02 at Phase 0): |
|
12 |
+- Location: `Recherche-UQO/finance/wp40_raid/` (UQO Working Paper No. 40; git remote `gitsrv:srv/git/wp40_uqo.git`). |
|
13 |
+- Paper conventions: `Recherche-UQO/HARMONISATION.md` (title page macros, `aer` bibliography style, `Month YYYY` date). |
|
14 |
+- FinData key: `FINDATA_API_KEY` env var, or `~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt` read by the |
|
15 |
+ helper scripts; never committed, never logged. |
|
16 |
+- Development machine: the owner's laptop (Apple M5 Max, 40 GPU cores, 48 GB, macOS 27.0, Xcode 27.2). MacLustr nodes |
|
17 |
+ are used only for E7 (generation sweep) with a self-contained binary bundle; no production service is touched. |
|
18 |
+- Status ledger: `RESEARCH_STATUS.md`; decision log: `docs/DECISIONS.md`; phase journal: `docs/PROGRESS.md`. |
|
19 |
+ |
|
20 |
+## 1. Scientific thesis |
|
21 |
+ |
|
22 |
+RAID is not merely a faster library. Test whether heterogeneous high-frequency econometric workloads can be transformed |
|
23 |
+into batched, fused, streaming computation on Apple Silicon without sacrificing estimator accuracy, statistical |
|
24 |
+inference, or reproducibility. Three falsifiable research questions: (1) At what data and batch sizes does Metal beat |
|
25 |
+optimized multicore CPU, including I/O, shader compilation, synchronization, and transfer? (2) Does batched estimation |
|
26 |
+improve throughput while preserving statistical coverage and convergence? (3) Can an Arrow/Parquet-to-estimator pipeline |
|
27 |
+materially reduce copies, memory pressure, and end-to-end latency? Explicitly report cases where CPU wins. A long model |
|
28 |
+list alone is not a contribution. |
|
29 |
+ |
|
30 |
+Target journals: Journal of Financial Econometrics, Journal of Financial Data Science, Journal of Computational Finance; |
|
31 |
+consider broader outlets only if a strong methodological advance is demonstrated. Paper title: RAID: A Metal-Accelerated |
|
32 |
+C++ Framework for Reproducible High-Frequency Financial Econometrics. |
|
33 |
+ |
|
34 |
+## 2. Mandatory discovery before coding |
|
35 |
+ |
|
36 |
+Audit UQO Research and its WP structure, existing CMake/Python/LaTeX templates, Git practices, and previous volatility |
|
37 |
+research. Probe available MacLustr nodes, macOS/Xcode/Metal versions, Apple GPU capabilities, RAM, disk, load, power and |
|
38 |
+thermal conditions; never assume older inventory remains valid or interfere with production services. Read FinData |
|
39 |
+API.md, DATA_DICTIONARY.md, DISCOVERY.md and live OpenAPI/catalog. Produce docs/audit.md, docs/capabilities.md, |
|
40 |
+docs/novelty_matrix.md, docs/risks.md. |
|
41 |
+ |
|
42 |
+Research relevant implementations in R highfrequency/rugarch/rmgarch, Python arch/statsmodels/NumPy/SciPy/Polars, Julia, |
|
43 |
+DuckDB, Eigen, Accelerate, BLAS/LAPACK and GPU toolkits. Verify versions, mathematical equivalence and citations. |
|
44 |
+Distinguish engineering novelty from new econometric methodology. Confirm software and data licensing. Apple Metal |
|
45 |
+itself is proprietary even if RAID is open source. |
|
46 |
+ |
|
47 |
+## 3. Repository structure |
|
48 |
+ |
|
49 |
+``` |
|
50 |
+wp40_raid/ |
|
51 |
+ CLAUDE.md README.md LICENSE CITATION.cff CMakeLists.txt CMakePresets.json pyproject.toml |
|
52 |
+ include/raid/{core,data,statistics,estimators,models,backend}/ |
|
53 |
+ src/{core,data,statistics,estimators,models}/ |
|
54 |
+ backends/cpu/ backends/metal/{host,shaders}/ |
|
55 |
+ bindings/python/ python/raid/ cli/ |
|
56 |
+ tests/{unit,property,reference,simulation,stress,integration}/ |
|
57 |
+ benchmarks/{micro,end_to_end,accuracy,scalability,energy}/ |
|
58 |
+ experiments/{configs,scripts,results,manifests}/ |
|
59 |
+ datasets/{synthetic,fixtures}/ |
|
60 |
+ docs/{architecture,api,methods,validation,operations}/ |
|
61 |
+ examples/{cpp,python,notebooks}/ |
|
62 |
+ paper/{main.tex,sections,figures,tables,appendices,references.bib}/ |
|
63 |
+ scripts/ .github/workflows/ |
|
64 |
+``` |
|
65 |
+ |
|
66 |
+## 4. Technical architecture |
|
67 |
+ |
|
68 |
+Use modern C++20, RAII, explicit buffer ownership, strong types for timestamps/frequencies, safe exceptions/errors, |
|
69 |
+structured logging, deterministic seeds, bounded memory and thread pools. CPU reference backend: multicore C++ with |
|
70 |
+vetted Eigen/Accelerate/BLAS. Metal backend: Objective-C++ bridge, Metal Shading Language kernels, device capability |
|
71 |
+discovery, reusable compute pipelines, asynchronous command buffers, buffer pools, batching, kernel fusion, chunking, |
|
72 |
+backpressure and robust CPU fallback. Python bindings via nanobind or pybind11, NumPy/Arrow interoperability, correct |
|
73 |
+lifetimes and GIL release. CMake and installable macOS ARM64 wheels; portable CPU build on Linux. CLI: raid devices, |
|
74 |
+raid data, raid estimate, raid validate, raid benchmark. |
|
75 |
+ |
|
76 |
+Core types: TimeSeries, Panel, BarSeries, MarketCalendar, QualityReport, ModelSpecification, Estimate, InferenceResult, |
|
77 |
+PrecisionPolicy, ExecutionPlan, Device, Backend, ExperimentManifest. Estimators must be backend-independent and expose |
|
78 |
+coefficients, loss, uncertainty where available, convergence, warnings, runtime, backend, precision and provenance. |
|
79 |
+Never treat failed optimizations as successful. |
|
80 |
+ |
|
81 |
+Optimize columnar layouts, contiguous arrays, alignment, Arrow zero-copy views where actually possible, threadgroup |
|
82 |
+occupancy, memory access, register pressure, command overhead and synchronization. Unified memory is not free |
|
83 |
+computation. Apple Metal FP64 GPU support must never be assumed; use validated FP32/mixed precision or CPU FP64 |
|
84 |
+fallbacks. Use stable reductions, log-domain likelihoods, parameter transforms, Cholesky/QR/SVD as appropriate, |
|
85 |
+conditioning checks, and matrix solves rather than explicit inversion. Make deterministic behavior and nondeterministic |
|
86 |
+GPU reductions visible. |
|
87 |
+ |
|
88 |
+## 5. FinData integration — mandatory |
|
89 |
+ |
|
90 |
+Use https://findata.spboucher.ai/api/v1, with API key only from FINDATA_API_KEY or secure store, bearer auth, TLS, |
|
91 |
+sanitized logs, retries/backoff, bounded concurrency, rate-limit compliance, pagination, caching, checksums and |
|
92 |
+provenance. Discover current endpoints via /api/v1/openapi.json and catalog. Implement /catalog/datasets, /market/bars, |
|
93 |
+/fundamentals/{entity}?as_of=..., asynchronous /exports and polling/download, and optionally bounded read-only /query |
|
94 |
+when justified. Use Arrow IPC and Parquet for large workloads. |
|
95 |
+ |
|
96 |
+Documented constraints: 1m bars at most 31 days per direct call, 5m at most 186 days, 1d at most 10 years, max 10,000 |
|
97 |
+rows/page; use bulk exports for historical panels. SQL playground: read-only, 30 seconds, 100,000 rows. Default API-key |
|
98 |
+rate limit is 600 requests/minute (check current configuration). Preserve X-Dataset-Id, X-Dataset-Version, request IDs, |
|
99 |
+archive hashes and dataset snapshots. |
|
100 |
+ |
|
101 |
+FinData bars are OHLCV bars, not tick trades or quotes. timestamp_utc is bar OPEN, session_date uses exchange timezone, |
|
102 |
+US equities can include 04:00–20:00 ET, regular hours 09:30–16:00, and the 16:00 bar is a closing print. Zero-volume |
|
103 |
+minutes may be absent; do not manufacture bars or silently forward-fill. Respect DST, early closes, corporate actions, |
|
104 |
+quality flags, raw/split/split_div adjustments, delisted identifiers and symbol aliases. Maintain explicit |
|
105 |
+regular/extended-session policies. Licensed datasets may not yet be ingested: check catalog and coverage, never assume |
|
106 |
+historical completeness. Fundamental PIT available_at is a proxy, and historical revisions before ingestion may be |
|
107 |
+unrecoverable. FirstRate private-use raw data must not be redistributed or included in the public repository; synthetic |
|
108 |
+fixtures only in CI. Attribute data source in paper. |
|
109 |
+ |
|
110 |
+## 6. Model registry and scope |
|
111 |
+ |
|
112 |
+Maintain a registry for each method: precise equations, input schema, assumptions, data frequency, parameter |
|
113 |
+restrictions, estimation method, uncertainty, supported backend, numerical tolerance, test fixtures, validation status. |
|
114 |
+Use experimental or not implemented honestly; never publish stub methods as supported. |
|
115 |
+ |
|
116 |
+**Tier A — production MVP, fully validated.** Data/statistics: log and arithmetic returns, lag/lead, irregular |
|
117 |
+resampling, rolling/expanding windows, OHLCV validation, calendar-aware aggregation, realized return/variance/volatility, |
|
118 |
+rolling correlations, drawdown, descriptive moments, autocovariance and robust missing-data policies. Realized measures: |
|
119 |
+RV, bipower variation, positive/negative semivariance, realized quarticity, realized covariance/correlation, jump proxies |
|
120 |
+with explicit finite-sample constants and sampling assumptions. No claims of microstructure-noise robustness without |
|
121 |
+appropriate estimator and validation. Regression: OLS, WLS, ridge, rolling/batched OLS, fixed effects, HC robust SE, |
|
122 |
+HAC/Newey-West, cluster-robust covariance, stable QR/SVD CPU reference, GPU batched sufficient-statistic accumulation |
|
123 |
+when advantageous. HAR: HAR-RV, HAR-logRV, HAR-X and HARQ with explicit definitions, causal lag construction, forecast |
|
124 |
+horizons and robust inference. Volatility: Gaussian/Student-t GARCH(1,1), GJR-GARCH(1,1), EGARCH(1,1), parameter |
|
125 |
+constraints, stable likelihood, initialization, gradients or validated numerical derivatives, optimizer convergence, SE |
|
126 |
+and forecasting. GPU primitives: parallel reductions, rolling sums/squares, covariance, batched matrix products, GARCH |
|
127 |
+likelihood, Monte Carlo and bootstrap replicas, batched HAR fitting. Measure precision and speed for every kernel. |
|
128 |
+ |
|
129 |
+**Tier B — research extensions, only after Tier A passes.** Realized GARCH, HEAVY, GARCH-MIDAS, MIDAS, FIGARCH where |
|
130 |
+stable, ARMA/ARIMA, VAR, BVAR, VECM, DCC-GARCH, multivariate realized covariance, local projections, Granger tests, |
|
131 |
+generalized FEVD, Diebold–Yilmaz connectedness, rolling networks, factor models, PCA, shrinkage covariance, portfolio |
|
132 |
+risk/forecasting, quantile regression, state-space/Kalman filtering, change-point detection. |
|
133 |
+ |
|
134 |
+**Tier C — experimental and data-dependent.** Pre-averaging, multi-scale RV, realized kernels, BNS and other jump |
|
135 |
+tests, lead-lag methods, high-dimensional covariance, stochastic volatility, sparse VAR, Bayesian MCMC, particle |
|
136 |
+filtering, regime switching, graph volatility models, Monte Carlo/quasi-Monte Carlo, Hawkes only if actual event |
|
137 |
+timestamps are separately available, and order-flow/order-book methods only with independently licensed trade/quote |
|
138 |
+data. Never infer true order-book imbalance or transaction-level microstructure from 1-minute OHLCV. |
|
139 |
+ |
|
140 |
+## 7. Correctness, numerical validation and inference |
|
141 |
+ |
|
142 |
+Every implemented method needs: (a) independent CPU FP64 reference, (b) analytical or simulation truth where possible, |
|
143 |
+(c) comparison to established software with identical definitions, (d) GPU/CPU agreement within documented |
|
144 |
+method-specific tolerance, (e) adversarial cases. Test zero/missing volume, holidays, DST, 16:00 print, split days, |
|
145 |
+irregular timestamps, extreme returns, singular matrices, near-unit-root GARCH, nonconvergence, insufficient samples, |
|
146 |
+memory pressure, cancellation and device fallback. Check gradients via finite differences where applicable. Separate |
|
147 |
+approximation error from floating-point error. Monte Carlo evaluation: bias, RMSE, empirical size, power, confidence |
|
148 |
+interval coverage, forecast loss, parameter convergence and failure rate. Use block/bootstrap schemes appropriate for |
|
149 |
+dependent financial data. Track random seed and parallel RNG streams. Set tolerances per algorithm and precision. |
|
150 |
+ |
|
151 |
+## 8. Publication-grade benchmark protocol |
|
152 |
+ |
|
153 |
+Compare RAID CPU single-thread, RAID multicore CPU, RAID Metal, and mathematically equivalent established competitors. |
|
154 |
+Record versions, build flags, machine, GPU, OS, clocks/power mode when measurable, memory, compiler, cache, precision and |
|
155 |
+convergence tolerance. Test series counts 1/10/100/1000/10000 and lengths 390/3900/39000/390000 where feasible. Include |
|
156 |
+sparse/irregular bars, 1m/5m, synthetic and explicitly identified real panels. Workloads: realized measures, rolling |
|
157 |
+covariance, HAR, GARCH, VAR, bootstrap, Monte Carlo, plus full FinData export → Arrow/Parquet → preprocessing → fit → |
|
158 |
+output. Report cold and warm cache, first shader compilation and cached compilation, synchronization, allocation and |
|
159 |
+end-to-end transfer. Use warmups, randomized benchmark order, multiple repetitions, medians and uncertainty intervals. |
|
160 |
+Metrics: wall-clock, fits/sec, bars/sec, peak RSS, buffer allocations, numerical error, convergence rate, energy only if |
|
161 |
+measured with a valid instrument. Speedup = T_reference / T_RAID for equivalent computations. Never extrapolate untested |
|
162 |
+throughput. Show break-even charts and CPU-wins cases. Include ablation of batching, kernel fusion, memory reuse, |
|
163 |
+precision, and I/O strategy. |
|
164 |
+ |
|
165 |
+## 9. Research experiments |
|
166 |
+ |
|
167 |
+E1 batched realized variance on thousands of equities, scaling versus panel width and window length. E2 HAR-RV/HAR-X |
|
168 |
+rolling estimation and 1/5/22-day forecasts, accuracy and runtime. E3 GARCH-family panel estimation, likelihood |
|
169 |
+accuracy, convergence, SE and forecast quality. E4 realized covariance and dynamic network connectedness with |
|
170 |
+memory/conditioning constraints. E5 bootstrap and Monte Carlo inference with coverage and reproducibility. E6 streaming |
|
171 |
+FinData Arrow/Parquet ingestion and end-to-end throughput. E7 stress tests across available Apple Silicon generations |
|
172 |
+and CPU-only fallback. Strict chronological splits and point-in-time handling for forecasting. Separate performance |
|
173 |
+benchmarking from predictive-model selection. Publish negative results. |
|
174 |
+ |
|
175 |
+## 10. Paper structure |
|
176 |
+ |
|
177 |
+Abstract; introduction and scientific gap; related econometric and systems literature; RAID architecture and API; |
|
178 |
+mathematical estimator definitions; Metal execution and precision strategy; statistical correctness; experimental setup; |
|
179 |
+microbenchmarks; end-to-end benchmarks; financial applications; threats to validity; reproducibility/licensing; |
|
180 |
+conclusion. Appendices: kernel pseudocode, optimizer details, finite-sample conventions, complete benchmark |
|
181 |
+configuration, reference-validation tables, numerical tolerance tables, API provenance and hardware inventory. Real |
|
182 |
+verified BibTeX references, journal-ready tables/figures, zero unresolved citations. Never claim peer review, novelty, or |
|
183 |
+superior accuracy without evidence. |
|
184 |
+ |
|
185 |
+## 11. Execution phases and gates |
|
186 |
+ |
|
187 |
+Phase 0 Audit → Phase 1 Foundation → Phase 2 Data → Phase 3 Numerical primitives → Phase 4 Econometrics → Phase 5 |
|
188 |
+Extensions → Phase 6 Benchmarking → Phase 7 Manuscript → Phase 8 Independent review. Each gate is recorded in |
|
189 |
+`RESEARCH_STATUS.md` and `docs/PROGRESS.md` with the actual commands and results. |
|
190 |
+ |
|
191 |
+## 12. Deliverables and release criteria |
|
192 |
+ |
|
193 |
+Compiled C++ library, Metal kernels, Python wheel, CLI, public synthetic examples, API documentation, method registry, |
|
194 |
+tests, CI, benchmark harness, reproducibility manifests, versioned release, CITATION.cff, license and a complete |
|
195 |
+PDF/LaTeX paper. Minimal Python and C++ examples with `backend="auto|cpu|metal"` and clear CPU fallback. |
|
196 |
+`RESEARCH_STATUS.md` lists what is implemented, verified, experimental, blocked and planned. No public release until: |
|
197 |
+no secrets in git; no redistribution of raw FirstRate data; Tier A methods pass reference tests; Metal unsupported cases |
|
198 |
+fall back safely; performance claims are reproduced; all paper tables/figures derive from tracked artifacts; a clean |
|
199 |
+environment can build and run the project. |
|
200 |
+ |
|
201 |
+## 13. Agent operating rules |
|
202 |
+ |
|
203 |
+Work sequentially. At each phase: write a concise plan, implement, execute tests, record actual commands/results, |
|
204 |
+update RESEARCH_STATUS.md, and only advance when the gate passes. When blocked, document the blocker and the next safe |
|
205 |
+action rather than pretending success. Never rewrite unrelated UQO Research code, overwrite datasets, alter live |
|
206 |
+infrastructure, leak credentials, or launch unbounded vendor downloads. Prioritize statistical validity and a |
|
207 |
+publishable scientific contribution over breadth and marketing language. |