spb/wp7_uqo

1<!--2Author: Simon-Pierre Boucher3Contact: contact@spboucher.ai4-->56# CHANGES — Restructuring report (2026-08-05)78> **Note (2026-10-17).** Superseded for all multi-day results by the v1.2 revision (`REVIEW-REPORT.md`, `CHANGELOG-REVISION.md`): the 5-day targets used in v1.0/v1.1 were trailing, not forward, windows. The v1.1 result CSVs are preserved in `results/_pre-revision-20261017/`.91011Source: `~/Desktop/UQO/UQO_WP/hf-wp7-spb20260519` (preserved verbatim in `_old/`).12Target: this repository (`~/Desktop/wp7_uqo`). **No original result or number was changed13anywhere.** Revision 1.1 (§6) *adds* new robustness analyses and references on the author's14request and corrects one unsupported sentence, as documented there. Companion document:15`AUDIT.md` (pre-restructuring audit, reproducibility map, and verification evidence).1617---1819## 1. What was moved / renamed2021| Original | New location | Notes |22|---|---|---|23| `data/*.parquet` (5 files) | `data/processed/` | unchanged bytes; raw data documented in `data/raw/README.md` |24| `results/*.csv` (33 files) | `results/` | unchanged bytes; **+2 regenerated files** (see §4) |25| `scripts/01…11_*.py` | `scripts/` (same names/numbers) | fully refactored (see §2) |26| `wp7/main.tex` | `paper/main.tex` + `paper/preamble.tex` | preamble split out; BibTeX enabled |27| `wp7/sections/*.tex` | `paper/sections/` | editorial rewrite (see §3) |28| `wp7/sections/references.tex` | `paper/references.bib` | manual `thebibliography` (37 entries) → BibTeX; file deleted from sections |29| `wp7/appendix/appendix.tex` | `paper/appendix/appendix.tex` | + lead-in prose, cross-referenced from the body |30| `wp7/uq_logo.jpg`, `Makefile`, `.latexmkrc` | `paper/` | `.latexmkrc` now runs bibtex (`$bibtex_use = 2`) |31| `WP7_Options_Implied_Information_Content.tex` (root, 1,066 lines) | `_old/` only | older monolithic duplicate of the same paper — dead file |32| `wp7/main.{aux,log,out,toc,fls,fdb_latexmk}` | not migrated | build artifacts (regenerable; `.gitignore`d) |33| `.DS_Store`, `.claude/` | not migrated | noise / session config |34| — (new) | `README.md`, `AUDIT.md`, `CHANGES.md`, `requirements.txt`, `pyproject.toml`, `Makefile`, `.gitignore`, `_verify/` | repository infrastructure |35| — (new) | `scripts/12_make_figures.py`, `figures/fig_*.{png,pdf}` | supplementary figures (the paper remains tables-only) |3637## 2. Code refactoring (results-preserving)3839All statistical logic now lives in a small library, `src/wp7/`, defined exactly once:4041- **`config.py`** — every path (env-overridable via `WP7_RAW_DATA_DIR`, `WP7_RESULTS_DIR`),42  ticker universes, feature sets, subperiods, crisis dates, VIX-regime bins. Previously these43  were duplicated across up to 6 scripts with hard-coded `parent.parent.parent` paths.44- **`data_io.py`** — parquet loaders + raw-DuckDB access with a dedicated45  `RawDataUnavailableError` so raw-dependent steps skip gracefully instead of crashing.46- **`econometrics.py`** — `winsorize`, standardization, OLS via `lstsq`, HC1 / Newey-West /47  double-clustered (CGM 2011) inference, Fama-MacBeth, quantile regression (IRLS),48  Granger F-test, bivariate VAR + IRF. Previously copy-pasted 6–15× across scripts.49- **`forecasting.py`** — OLS forecasting, rolling out-of-sample evaluation, Diebold-Mariano.50- **`portfolio.py`** — quintile sorts, double sorts, performance statistics.5152Scripts keep their original numbers and filenames; each has a docstring stating purpose,53inputs and outputs, and the required author header (`Author: Simon-Pierre Boucher /54Contact: contact@spboucher.ai` — as `#`/`%` comments in Python/LaTeX, as an HTML comment in55Markdown, since `<!-- -->` is not valid syntax in Python or LaTeX, and a literal `@` cannot56appear in a BibTeX comment).5758Deliberate preservation of historical quirks (documented in-code):5960- Script 02's `panel_ols` keeps its pandas (ddof = 1) standardization, zero-filled missing61  z-scores, and the ad-hoc normal-tail "p_value" approximation — these define the published62  numbers.63- All sandwich estimators keep the original `n/(n−k−1)` small-sample factor and64  `lstsq`-based estimation (no statsmodels), so results reproduce to the bit.6566Genuine fixes (behavior-preserving or bug-fixing, flagged in `AUDIT.md`):6768- **Memory bug fixed** (`10_robustness.py`): `X.T @ np.diag(e²) @ X` and the IRLS weight69  matrix materialized an *n×n* dense matrix (≈97 GB → the original run of sections D–F70  died; two result CSVs were missing). Rewritten with broadcasting — mathematically71  identical, O(nk) memory.72- **Dead code removed** (`04`): an unused DuckDB query (`const_returns`) whose result was73  never read.74- Bare `except:` → targeted exceptions; `RES.mkdir` centralized; no other logic touched.7576## 3. Paper rewrite (editorial only)7778- **Structure**: `main.tex` (metadata + skeleton) / `preamble.tex` / one file per section /79  `appendix/` — the original section split was kept; the root-level monolithic draft was80  retired to `_old/`.81- **Bibliography**: manual `thebibliography` → `references.bib` with natbib + `apalike`82  (BibTeX). All 37 cite keys unchanged; entry metadata transcribed faithfully. The PDF now83  renders the reference list from BibTeX — no visual regression.84- **Prose**: light editorial pass over introduction, literature, discussion, conclusion85  (flow, voice consistency, hyphenation, article usage). Data/methodology/results/robustness86  needed only targeted touches. **Every number, table value, citation, and claim is87  unchanged.**88- **Cross-referencing completed**: the four appendix tables were never referenced from the89  body; the results and methodology sections now point to90  `app:crises`, `app:features`, `app:importance`, `app:fevd`, and each appendix section91  gained a one-sentence lead-in.92- **Build**: `latexmk` (pdflatex + bibtex) compiles clean — exit 0, **0 undefined references,93  0 undefined citations, 24 pages**. `make paper` from the repo root.9495## 4. Verification of reproduction (summary — details in `AUDIT.md` §7)9697Re-run environment: Python 3.13, numpy 2.4.4, pandas 3.0.2 (Apple Silicon). Regenerated CSVs98written to `_verify/results/` and compared with `_verify/compare_results.py`:99100- **10 files byte-identical**; **15 files numerically equal** (max relative difference101  ≤ 2×10⁻¹⁰ — floating-point noise across BLAS/numpy versions).102- **1 explained deviation**: `rq3_stress_prediction.csv` t-statistics only. The regression103  contains an exact linear dependency (`corr_divergence = implied_corr − realized_corr`),104  making HC1 t-stats numerically degenerate; the significant `corr_ratio` t-stats move by105  ≤ 0.18 with identical significance conclusions. Original CSV kept as authoritative.106- **2 files regenerated** that were missing from the original archive (the §2 memory bug):107  `robustness_quantile_regression.csv`, `robustness_ticker_r2.csv` — added to `results/`.108- Raw-dependent outputs (rq4 Greeks decay, rq5, VIX-regime tables, options quality) cannot109  be re-run (raw stores gone) — original CSVs shipped unchanged.110111## 5. Items requiring your review1121131. **Ticker-level R² claim (robustness §4.4) is not reproducible.** Paper: mean 5.9 %,114   median 4.7 %, IQR [2.0 %, 8.2 %]. Regenerated from the surviving panel with the script's115   own specification: **mean 14.0 %, median 13.2 %, IQR [10.0 %, 17.6 %]** (69 tickers). The116   text was left as-is per the no-content-change rule; the qualitative conclusion (results117   not driven by outliers) holds under either set. Consider updating the paragraph or118   documenting the original specification.1192. **Quantile-regression "significance" (robustness §4.4).** The code computes coefficients120   only — no standard errors or t-statistics were ever produced — so "significant across the121   return distribution" has no supporting statistic. The regenerated coefficients are122   consistently signed across τ (kurtosis +, PC ratio −), supporting a weaker "stable sign"123   statement.1243. **RQ3 t-statistics are fragile by construction** (exact collinearity among the four125   correlation regressors). Consider dropping `corr_divergence` or `implied_corr`/126   `realized_corr` from the specification in a future revision.1274. **Raw data loss.** The four DuckDB stores are gone; `data/processed/` is now the only128   copy of the analysis data. Recommend an off-machine backup of this repository.1295. **Possible bibliography metadata issue** (`ni2009does`): "Does option trading convey130   stock price information?" is listed for both Ni-Pearson-Poteshman-White (2009, JF) and131   Hu (2014, JFE). Both entries were transcribed as the author wrote them, but the Ni et132   al. title/venue/pages may need checking against the published record.1336. **Panel F/H, regime, and rq5 tables** are frozen (raw-dependent) — any future data134   refresh must rebuild the raw stores first.135136## 6. Revision 1.1 — extended robustness analyses and references (added on request)137138On top of the results-preserving restructure, the paper was upgraded with **new** analyses139(all computable from the surviving processed panel) and eight classical references:140141- **New script** `scripts/13_extended_robustness.py` → six new result files142  (`results/extended_*.csv`):143  - *Winsorization sensitivity* — 5-day return regression under cutoffs144    {none, 0.5%, 1%, 2.5%, 5%}: R² ranges 0.034–0.062; key predictors significant throughout.145  - *Newey-West lag sensitivity* — HAC t-stats at 5/10/22 lags: 9/10 significant predictors146    at every lag (kurtosis t: 25.6 → 22.1).147  - *Spearman information coefficients* — daily cross-sectional rank correlations (stocks):148    kurtosis IC +0.098 (t=16.8, 64.3% positive days); PC volume ratio −0.059 (t=−13.8).149  - *Decile sorts* — D10−D1 spreads widen (kurtosis 44.0% → 52.1% ann.) with Sharpe ratios150    essentially unchanged.151  - *Leave-one-year-out* — 5D-return R² ∈ [0.047, 0.057]; HAR+IV RV R² ∈ [0.414, 0.492]152    (2020 exclusion is the only visible mover).153  - *Placebo* — features permuted within ticker (seed 42, 10 draws): R² collapses from154    0.0529 to ≤ 0.0010.155- **Paper**: four new subsections in `sections/robustness.tex` (§6.5–6.8) with three new156  tables + one two-panel table; abstract, introduction and methodology updated accordingly.157  The unsupported quantile-regression "significance" sentence (see §5.2) was reworded to the158  supported claim (stable sign/magnitude across τ) with a \citep{koenker1978regression}.159- **New references (8)**: White (1980); Newey & West (1987); Koenker & Bassett (1978);160  Petersen (2009); Fama & French (2008); Welch & Goyal (2008); Campbell & Thompson (2008);161  Harvey, Liu & Zhu (2016). `references.bib` now has 45 entries, all cited.162- `portfolio_sort()` was generalized to arbitrary quantile counts (`LS_<n>_1` key);163  script 09 was re-verified **byte-identical** after the change.164- Compiled PDF: 29 pages, 0 undefined references/citations.165166### Additional review item discovered during this revision1677. **The "HC1" column of the original Table `tab:robust_se` matches no surviving output.**168   For implied kurtosis it shows 25.87, whereas the surviving CSVs give: pooled HC1169   (rq1, All|5-Day) 40.07, NW(5) 25.64, double-clustered 9.02, with-controls 21.95. The170   column likely comes from an earlier data vintage. The table was left unchanged (its NW171   and DC columns match the CSVs exactly); the new Table "Winsorization and HAC-lag172   sensitivity" reports the reproducible HC1 value (40.07) explicitly.173174## 7. Author headers175176Every code file (Python, LaTeX, BibTeX, Makefile, requirements, Markdown docs) carries the177requested header — `Author: Simon-Pierre Boucher / Contact: contact@spboucher.ai` — in the178comment syntax valid for its language.179