SPB Git

spb/wp46_uqo

Public

WP46: verification appendix, test-count macro, run-manifest dedupe

Simon-Pierre Boucher committed (Nov 9, 2026) parent 905b677 Browse files

4 changed files +50 −3

modified paper/appendices/verification.tex +42 −1
@@ -1 +1,42 @@
1 −% appendix verification placeholder
1 +\section{Simulation and replication verification}\label{app:verification}
2 +
3 +\paragraph{Test suite.} The replication package ships \NTests{} automated tests in four layers: unit tests of the standardized
4 +densities (log-densities equal to arch's to $10^{-15}$ for the normal, Student-$t$, GED and skew-$t$; simulated moments and
5 +expected shortfalls within sampling error), of the recursions (the custom GARCH(1,1) reproduces arch's
6 +\texttt{compute\_variance} to $10^{-9}$ and its analytic forecasts to $10^{-15}$), of nesting (every custom asymmetric,
7 +component and in-mean model attains at least the GARCH(1,1) likelihood), of admissibility, of the loss functions and realized
8 +measures (units: 390 one-minute returns of 0.1\% give $\RV^{1m}=3.9$ percent$^2$), of the calendars (official XNYS early
9 +closes; futures and FX windows), and of the HAR direct targets; integration tests of the rolling driver (re-estimation cadence,
10 +missing-input handling, no look-ahead: perturbing all returns and realized measures after the last origin leaves every
11 +forecast of a job unchanged, for a Tier~A and a Tier~B model); seeded Monte Carlo recovery tests for GJR (median over twelve
12 +replications within 0.02--0.03 of the truth), NAGARCH, GARCH-M, QGARCH, Realized GARCH, HEAVY and MS-GARCH (regime
13 +persistence and variance separation recovered); and replication tests against R packages on a shared synthetic fixture
14 +(3{,}000 observations of a GJR-$t$ process with a noisy realized measure).
15 +
16 +\paragraph{Cross-package replication.} Table~\ref{tab:crosscheck} reports the maximized log-likelihoods of our implementations
17 +and of \texttt{rugarch} 1.5-6 \citep{galanos2011rugarch} on the fixture. For every architecture available in both, our optimum
18 +is within 0.3 log-likelihood units of \texttt{rugarch}'s (and above it in all but one case, which reflects optimizer
19 +differences, not specification differences); GARCH(1,1)-$t$ parameters agree to three decimals ($\alpha$ 0.0767 vs 0.0766,
20 +$\beta$ 0.8979 vs 0.8979, $\nu$ 6.92 vs 6.92), NAGARCH's asymmetry parameter to three decimals (0.3397 vs 0.3398) and the
21 +component model's parameters to two. FIGARCH is the exception: arch's truncated ARCH($\infty$) representation with its
22 +sufficient positivity constraints yields a likelihood 1.47 units below \texttt{rugarch}'s; we keep the arch implementation
23 +(version-locked, tested) and flag the difference. \texttt{rugarch}'s Realized GARCH did not converge on the fixture, so that
24 +model is verified by Monte Carlo recovery only. The GAS-$t$ model is checked against the \texttt{GAS} package of
25 +\citet{ardia2019generalized}, the GARCH-X return equation against \texttt{rugarch} with an external variance regressor, and the
26 +two-regime MS-GARCH likelihood and 22-step variance forecasts against \texttt{MSGARCH} \citep{ardia2019markov}; the
27 +corresponding numbers are reported in the table when the R fits converged.
28 +
29 +\begin{table}[tp]\centering\footnotesize
30 +\caption{Cross-package verification on the shared fixture (maximized log-likelihoods; differences = ours $-$ R).}\label{tab:crosscheck}
31 +\input{tables/tab_crosscheck}
32 +\end{table}
33 +
34 +\paragraph{Known implementation caveats.} (i) arch uses $\kappa=1/2$ in the GJR multi-step recursion for every innovation law
35 +(exact for symmetric laws; the skew-$t$ GJR forecasts at $h>1$ are therefore approximate). (ii) No stationarity flag is
36 +computed for TGARCH, AVGARCH and APARCH (their conditions involve $\E(|z|-\gamma z)^{\delta}$); convergence is their only
37 +failure criterion. (iii) EGARCH in arch centres $|z|$ at $\sqrt{2/\pi}$ for all laws and restricts $\beta\in[0,1)$. (iv) arch
38 +emits a scale warning, without rescaling, when the return variance is outside $[0.1, 10^4]$; percent returns keep every
39 +instrument inside this range. (v) Simulation forecasts use the distribution generator seeded by the origin index (amendment
40 +A-01; the first pass of four simulation-based specifications with non-Gaussian innovations was re-run after the seed was
41 +found missing). (vi) $\kappa$ is held at $1/2$ inside the optimizer of the custom models (it only enters the first-observation
42 +initialization of the custom GJR, which is not a scored model) and computed exactly once after each fit (A-04).
modified scripts/build_paper.py +6 −0
@@ -50,6 +50,12 @@ def main() -> None:
50 50 macros = {}
51 51 # --- headline numbers
52 52 macros["NInstruments"] = len(per_symbol)
53 + import subprocess
54 + try:
55 + col = subprocess.run([str(ROOT / ".venv" / "bin" / "python"), "-m", "pytest", "--collect-only", "-q", "tests"], capture_output=True, text=True, cwd=ROOT, timeout=300).stdout
56 + macros["NTests"] = sum(1 for l in col.splitlines() if "::" in l)
57 + except Exception: # noqa: BLE001
58 + macros["NTests"] = "--"
53 59 macros["NSpecs"] = len(specs_all)
54 60 macros["NModels"] = len({s.split("__")[0] for s in specs_all})
55 61 macros["NOriginsTotal"] = sum(r["horizons"]["1"]["n_origins"] for r in per_symbol.values())
modified src/wp46/reporting/figures.py +1 −1
@@ -243,7 +243,7 @@ def lineage(path=FIG / "fig_lineage.pdf") -> Path:
243 243 n_rows = sum(r.get("row_count") or 0 for v in im.values() for r in v.get("minute", []))
244 244 n_panels = sum(1 for v in pm.values() if "rows" in v)
245 245 runs = ROOT / "results" / "forecasts" / "main" / "_runs.jsonl"
246 − n_jobs = sum(1 for _ in open(runs)) if runs.exists() else 0
246 + n_jobs = len({(json.loads(l)["symbol"], json.loads(l)["spec_id"]) for l in open(runs) if "wall_s" in l}) if runs.exists() else 0
247 247 boxes = [("FinData OS\n(FirstRate Data, versioned snapshots)", f"{len(im)} instruments\n{n_files} export files, SHA-256\n{n_rows/1e6:.0f} M one-minute bars"),
248 248 ("Daily panels", f"{n_panels} panels\nreturns, overnight split,\n7 realized measures, validity flags"),
249 249 ("Pre-registered protocol", "universe, periods, models,\nlosses, bootstrap, regimes\n(git tag prereg-frozen)"),
modified tests/replication/rugarch_crosscheck.R +1 −1
@@ -41,7 +41,7 @@ if (requireNamespace("GAS", quietly = TRUE)) {
41 41 }
42 42 if (requireNamespace("MSGARCH", quietly = TRUE)) {
43 43 suppressPackageStartupMessages(library(MSGARCH))
44 − ms <- CreateSpec(variance.spec = list(model = c("sGARCH", "sGARCH")), distribution.spec = list(distribution = c("norm", "norm")), switch.spec = list(do.mix = FALSE, K = 2))
44 + ms <- CreateSpec(variance.spec = list(model = c("sGARCH", "sGARCH")), distribution.spec = list(distribution = c("norm", "norm")), switch.spec = list(do.mix = FALSE))
45 45 mf <- tryCatch(FitML(ms, data = r - mean(r)), error = function(e) NULL)
46 46 out2$msgarch2_n <- tryCatch(if (!is.null(mf)) { pr <- predict(mf, nahead = 22, do.return.draw = FALSE)
47 47 list(converged = TRUE, loglik = as.numeric(mf$loglik), par = as.list(mf$par), vol_forecast = as.numeric(pr$vol), mean_removed = mean(r)) } else list(converged = FALSE),
48 48
Menu

Theme