docs/novelty_matrix.md — what exists, what RAID adds, what is not new (Phase 0, 2026-11-02)
Legend — E engineering novelty (new implementation strategy, measured), M methodological novelty (new estimator or inference result), = equivalent to existing software (RAID must match it), × not offered. Claims marked E are only valid once the corresponding experiment (E1–E7) produces the number; nothing in this table is a result.
| Capability | R highfrequency 1.0.3 | R rugarch 1.5.6 | Python arch 8.0 | statsmodels 0.15 | Polars / DuckDB | RAID (planned) | Novelty class |
|---|---|---|---|---|---|---|---|
| RV, BPV, semivariance, quarticity, rCov from bars | yes (rCov, rBPCov, rSVar, rQuar) |
× | × | × | by hand (SQL/expr) | CPU FP64 + Metal FP32 batched over thousands of series | E (batched GPU, fused rolling) ; = definitions |
| Calendar-aware aggregation, early closes, DST, 16:00 print | partial (aggregateTS, user calendars) |
× | × | × | manual | explicit MarketCalendar from XNYS rules, FinData sessions as truth | E (correctness tooling), not M |
| HAR-RV / log-HAR / HAR-X / HARQ | HARmodel (HAR-RV, HARQ, HAR-J/CJ) |
× | OLS by hand | OLS + HAC | × | batched rolling HAR with HAC/HC inference, GPU sufficient statistics | E (batched, GPU), = definitions (Corsi 2009; Bollerslev–Patton–Quaedvlieg 2016) |
| OLS/WLS/ridge, HC0–3, HAC (Newey–West), cluster-robust | × | × | OLS + HAC in arch.unitroot/covariance |
yes | × | QR/SVD CPU reference; GPU batched X'X/X'y accumulation | = inference; E batched accumulation |
| GARCH(1,1), GJR, EGARCH, Gaussian / Student-t | × | yes (ugarchfit) |
yes (arch_model) |
× | × | CPU reference optimizer; GPU batched likelihood evaluation across series / parameter proposals | E (batched likelihood on Metal), = models |
| Realized GARCH, HEAVY, DCC, GARCH-MIDAS | HEAVY (HEAVYmodel) |
realized GARCH, rmgarch DCC | HEAVY ×, DCC × | × | × | Tier B after Tier A | = (if done) |
| Diebold–Yilmaz connectedness, generalized FEVD | × | × | × | VAR + FEVD (orthogonalized; generalized by hand) | × | Tier B batched rolling VAR + GFEVD | E (rolling batched), = method (WP 6 precedent) |
| Stationary / circular block bootstrap | × | × | yes (arch.bootstrap) |
× | × | GPU replica generation + batched statistic evaluation | E |
| Pre-averaging, realized kernels, multi-scale RV | yes (rKernelCov, rAVGCov, rTSCov) |
× | × | × | × | Tier C, only if validated against highfrequency; 1-min bars limit their relevance | = at best |
| Arrow/Parquet → estimator zero-copy | × (xts) | × | × (pandas) | × | native columnar | Arrow C++ views over FinData exports feeding CPU/GPU without copies where layout allows | E (measured copies / RSS, E6) |
| Python bindings with GIL release, Arrow/NumPy interop | — | — | — | — | — | nanobind | engineering, standard practice |
| Device discovery, CPU fallback, precision policy, provenance in every estimate | — | — | — | — | — | yes | E (reproducibility tooling) |
What the paper can legitimately claim (if the experiments support it)
- A measured break-even analysis Metal vs multicore CPU for realized measures, rolling regressions, batched GARCH likelihoods and bootstrap replicas on Apple Silicon, including shader compilation, synchronization and transfer, with explicit CPU-wins regions (RQ1).
- Evidence that batching across series preserves estimator accuracy and inference (coverage, convergence rates identical to the FP64 reference within tolerance) while raising throughput (RQ2).
- Evidence on copies, peak RSS and end-to-end latency of an Arrow/Parquet-to-estimator pipeline vs pandas/Polars/DuckDB baselines (RQ3).
- A validated, open C++20 library whose Tier A estimators reproduce highfrequency / rugarch / arch / statsmodels to documented tolerances (reference tables in the appendix).
What the paper must not claim
- New econometric estimators or new asymptotic results (none planned; M = none).
- Microstructure-noise robustness from 1-minute OHLCV bars (no trades/quotes; realized kernels/pre-averaging on 1-min bars are a sampling-frequency exercise, not noise correction at tick level).
- Order-flow or order-book content (not inferable from bars).
- Energy savings unless
powermetrics(root) or an external meter is actually used;powermetricsrequires sudo and is optional in E7 (documented as "measured" only when run). - GPU "speedups" without the equivalent CPU multicore implementation and the same convergence tolerance.
Prior work positioning (verified DOIs in docs/literature/crossref_resolved.json)
Realized measures: Andersen, Bollerslev, Diebold & Labys (2003, Econometrica); Barndorff-Nielsen & Shephard (2004, 2006, JFEC); Barndorff-Nielsen, Kinnebrock & Shephard (2010, OUP volume); Andersen, Bollerslev & Diebold (2007, REStat); Huang & Tauchen (2005, JFEC); Liu, Patton & Sheppard (2015, J. Econometrics). Noise-robust: Zhang, Mykland & Aït-Sahalia (2005, JASA); Barndorff-Nielsen, Hansen, Lunde & Shephard (2008, Econometrica); Jacod et al. (2009, SPA). HAR: Corsi (2009, JFEC); Bollerslev, Patton & Quaedvlieg (2016, J. Econometrics); Patton & Sheppard (2015, REStat). GARCH: Engle (1982); Bollerslev (1986); Nelson (1991); Glosten, Jagannathan & Runkle (1993); Bollerslev & Wooldridge (1992, Econometric Reviews); Fiorentini, Calzolari & Panattoni (1996, JAE, analytic derivatives); Hansen & Lunde (2005, JAE); Hansen, Huang & Shek (2012, JAE, Realized GARCH); Noureldin, Shephard & Sheppard (2012, JAE, HEAVY); Engle (2002, JBES, DCC); Engle, Ghysels & Sohn (2013, REStat, GARCH-MIDAS). Inference: White (1980); Newey & West (1987); Cameron, Gelbach & Miller (2011, JBES); Politis & Romano (1994, JASA); Diebold & Mariano (1995/2002, JBES); Hansen, Lunde & Nason (2003 OBES; 2011 Econometrica MCS — the 2011 DOI is still to be resolved); Patton (2011, J. Econometrics). Connectedness: Diebold & Yilmaz (2012, IJF; 2014, J. Econometrics); Pesaran & Shin (1998, Econ. Letters). Software: Boudt, Kleen & Sjørup (2022, JSS); Ghalanos (rugarch, CRAN). Computing: Aldrich, Fernández-Villaverde, Gallant & Rubio-Ramírez (2011, JEDC, GPU in economics); Kahan (1965, CACM); Blackford et al. (2002, ACM TOMS, BLAS). Systems references still to add with verified sources: Apache Arrow/Parquet documentation, Apple Metal documentation, nanobind, Eigen.