spb/wp7_uqo Public

1<!--2Author: Simon-Pierre Boucher3Contact: contact@spboucher.ai4-->56# CHANGES — Restructuring report (2026-08-05)78Source: `~/Desktop/UQO/UQO_WP/hf-wp7-spb20260519` (preserved verbatim in `_old/`).9Target: this repository (`~/Desktop/wp7_uqo`). **No original result or number was changed10anywhere.** Revision 1.1 (§6) *adds* new robustness analyses and references on the author's11request and corrects one unsupported sentence, as documented there. Companion document:12`AUDIT.md` (pre-restructuring audit, reproducibility map, and verification evidence).1314---1516## 1. What was moved / renamed1718| Original | New location | Notes |19|---|---|---|20| `data/*.parquet` (5 files) | `data/processed/` | unchanged bytes; raw data documented in `data/raw/README.md` |21| `results/*.csv` (33 files) | `results/` | unchanged bytes; **+2 regenerated files** (see §4) |22| `scripts/01…11_*.py` | `scripts/` (same names/numbers) | fully refactored (see §2) |23| `wp7/main.tex` | `paper/main.tex` + `paper/preamble.tex` | preamble split out; BibTeX enabled |24| `wp7/sections/*.tex` | `paper/sections/` | editorial rewrite (see §3) |25| `wp7/sections/references.tex` | `paper/references.bib` | manual `thebibliography` (37 entries) → BibTeX; file deleted from sections |26| `wp7/appendix/appendix.tex` | `paper/appendix/appendix.tex` | + lead-in prose, cross-referenced from the body |27| `wp7/uq_logo.jpg`, `Makefile`, `.latexmkrc` | `paper/` | `.latexmkrc` now runs bibtex (`$bibtex_use = 2`) |28| `WP7_Options_Implied_Information_Content.tex` (root, 1,066 lines) | `_old/` only | older monolithic duplicate of the same paper — dead file |29| `wp7/main.{aux,log,out,toc,fls,fdb_latexmk}` | not migrated | build artifacts (regenerable; `.gitignore`d) |30| `.DS_Store`, `.claude/` | not migrated | noise / session config |31| — (new) | `README.md`, `AUDIT.md`, `CHANGES.md`, `requirements.txt`, `pyproject.toml`, `Makefile`, `.gitignore`, `_verify/` | repository infrastructure |32| — (new) | `scripts/12_make_figures.py`, `figures/fig_*.{png,pdf}` | supplementary figures (the paper remains tables-only) |3334## 2. Code refactoring (results-preserving)3536All statistical logic now lives in a small library, `src/wp7/`, defined exactly once:3738- **`config.py`** — every path (env-overridable via `WP7_RAW_DATA_DIR`, `WP7_RESULTS_DIR`),39  ticker universes, feature sets, subperiods, crisis dates, VIX-regime bins. Previously these40  were duplicated across up to 6 scripts with hard-coded `parent.parent.parent` paths.41- **`data_io.py`** — parquet loaders + raw-DuckDB access with a dedicated42  `RawDataUnavailableError` so raw-dependent steps skip gracefully instead of crashing.43- **`econometrics.py`** — `winsorize`, standardization, OLS via `lstsq`, HC1 / Newey-West /44  double-clustered (CGM 2011) inference, Fama-MacBeth, quantile regression (IRLS),45  Granger F-test, bivariate VAR + IRF. Previously copy-pasted 6–15× across scripts.46- **`forecasting.py`** — OLS forecasting, rolling out-of-sample evaluation, Diebold-Mariano.47- **`portfolio.py`** — quintile sorts, double sorts, performance statistics.4849Scripts keep their original numbers and filenames; each has a docstring stating purpose,50inputs and outputs, and the required author header (`Author: Simon-Pierre Boucher /51Contact: contact@spboucher.ai` — as `#`/`%` comments in Python/LaTeX, as an HTML comment in52Markdown, since `<!-- -->` is not valid syntax in Python or LaTeX, and a literal `@` cannot53appear in a BibTeX comment).5455Deliberate preservation of historical quirks (documented in-code):5657- Script 02's `panel_ols` keeps its pandas (ddof = 1) standardization, zero-filled missing58  z-scores, and the ad-hoc normal-tail "p_value" approximation — these define the published59  numbers.60- All sandwich estimators keep the original `n/(n−k−1)` small-sample factor and61  `lstsq`-based estimation (no statsmodels), so results reproduce to the bit.6263Genuine fixes (behavior-preserving or bug-fixing, flagged in `AUDIT.md`):6465- **Memory bug fixed** (`10_robustness.py`): `X.T @ np.diag(e²) @ X` and the IRLS weight66  matrix materialized an *n×n* dense matrix (≈97 GB → the original run of sections D–F67  died; two result CSVs were missing). Rewritten with broadcasting — mathematically68  identical, O(nk) memory.69- **Dead code removed** (`04`): an unused DuckDB query (`const_returns`) whose result was70  never read.71- Bare `except:` → targeted exceptions; `RES.mkdir` centralized; no other logic touched.7273## 3. Paper rewrite (editorial only)7475- **Structure**: `main.tex` (metadata + skeleton) / `preamble.tex` / one file per section /76  `appendix/` — the original section split was kept; the root-level monolithic draft was77  retired to `_old/`.78- **Bibliography**: manual `thebibliography` → `references.bib` with natbib + `apalike`79  (BibTeX). All 37 cite keys unchanged; entry metadata transcribed faithfully. The PDF now80  renders the reference list from BibTeX — no visual regression.81- **Prose**: light editorial pass over introduction, literature, discussion, conclusion82  (flow, voice consistency, hyphenation, article usage). Data/methodology/results/robustness83  needed only targeted touches. **Every number, table value, citation, and claim is84  unchanged.**85- **Cross-referencing completed**: the four appendix tables were never referenced from the86  body; the results and methodology sections now point to87  `app:crises`, `app:features`, `app:importance`, `app:fevd`, and each appendix section88  gained a one-sentence lead-in.89- **Build**: `latexmk` (pdflatex + bibtex) compiles clean — exit 0, **0 undefined references,90  0 undefined citations, 24 pages**. `make paper` from the repo root.9192## 4. Verification of reproduction (summary — details in `AUDIT.md` §7)9394Re-run environment: Python 3.13, numpy 2.4.4, pandas 3.0.2 (Apple Silicon). Regenerated CSVs95written to `_verify/results/` and compared with `_verify/compare_results.py`:9697- **10 files byte-identical**; **15 files numerically equal** (max relative difference98  ≤ 2×10⁻¹⁰ — floating-point noise across BLAS/numpy versions).99- **1 explained deviation**: `rq3_stress_prediction.csv` t-statistics only. The regression100  contains an exact linear dependency (`corr_divergence = implied_corr − realized_corr`),101  making HC1 t-stats numerically degenerate; the significant `corr_ratio` t-stats move by102  ≤ 0.18 with identical significance conclusions. Original CSV kept as authoritative.103- **2 files regenerated** that were missing from the original archive (the §2 memory bug):104  `robustness_quantile_regression.csv`, `robustness_ticker_r2.csv` — added to `results/`.105- Raw-dependent outputs (rq4 Greeks decay, rq5, VIX-regime tables, options quality) cannot106  be re-run (raw stores gone) — original CSVs shipped unchanged.107108## 5. Items requiring your review1091101. **Ticker-level R² claim (robustness §4.4) is not reproducible.** Paper: mean 5.9 %,111   median 4.7 %, IQR [2.0 %, 8.2 %]. Regenerated from the surviving panel with the script's112   own specification: **mean 14.0 %, median 13.2 %, IQR [10.0 %, 17.6 %]** (69 tickers). The113   text was left as-is per the no-content-change rule; the qualitative conclusion (results114   not driven by outliers) holds under either set. Consider updating the paragraph or115   documenting the original specification.1162. **Quantile-regression "significance" (robustness §4.4).** The code computes coefficients117   only — no standard errors or t-statistics were ever produced — so "significant across the118   return distribution" has no supporting statistic. The regenerated coefficients are119   consistently signed across τ (kurtosis +, PC ratio −), supporting a weaker "stable sign"120   statement.1213. **RQ3 t-statistics are fragile by construction** (exact collinearity among the four122   correlation regressors). Consider dropping `corr_divergence` or `implied_corr`/123   `realized_corr` from the specification in a future revision.1244. **Raw data loss.** The four DuckDB stores are gone; `data/processed/` is now the only125   copy of the analysis data. Recommend an off-machine backup of this repository.1265. **Possible bibliography metadata issue** (`ni2009does`): "Does option trading convey127   stock price information?" is listed for both Ni-Pearson-Poteshman-White (2009, JF) and128   Hu (2014, JFE). Both entries were transcribed as the author wrote them, but the Ni et129   al. title/venue/pages may need checking against the published record.1306. **Panel F/H, regime, and rq5 tables** are frozen (raw-dependent) — any future data131   refresh must rebuild the raw stores first.132133## 6. Revision 1.1 — extended robustness analyses and references (added on request)134135On top of the results-preserving restructure, the paper was upgraded with **new** analyses136(all computable from the surviving processed panel) and eight classical references:137138- **New script** `scripts/13_extended_robustness.py` → six new result files139  (`results/extended_*.csv`):140  - *Winsorization sensitivity* — 5-day return regression under cutoffs141    {none, 0.5%, 1%, 2.5%, 5%}: R² ranges 0.034–0.062; key predictors significant throughout.142  - *Newey-West lag sensitivity* — HAC t-stats at 5/10/22 lags: 9/10 significant predictors143    at every lag (kurtosis t: 25.6 → 22.1).144  - *Spearman information coefficients* — daily cross-sectional rank correlations (stocks):145    kurtosis IC +0.098 (t=16.8, 64.3% positive days); PC volume ratio −0.059 (t=−13.8).146  - *Decile sorts* — D10−D1 spreads widen (kurtosis 44.0% → 52.1% ann.) with Sharpe ratios147    essentially unchanged.148  - *Leave-one-year-out* — 5D-return R² ∈ [0.047, 0.057]; HAR+IV RV R² ∈ [0.414, 0.492]149    (2020 exclusion is the only visible mover).150  - *Placebo* — features permuted within ticker (seed 42, 10 draws): R² collapses from151    0.0529 to ≤ 0.0010.152- **Paper**: four new subsections in `sections/robustness.tex` (§6.5–6.8) with three new153  tables + one two-panel table; abstract, introduction and methodology updated accordingly.154  The unsupported quantile-regression "significance" sentence (see §5.2) was reworded to the155  supported claim (stable sign/magnitude across τ) with a \citep{koenker1978regression}.156- **New references (8)**: White (1980); Newey & West (1987); Koenker & Bassett (1978);157  Petersen (2009); Fama & French (2008); Welch & Goyal (2008); Campbell & Thompson (2008);158  Harvey, Liu & Zhu (2016). `references.bib` now has 45 entries, all cited.159- `portfolio_sort()` was generalized to arbitrary quantile counts (`LS_<n>_1` key);160  script 09 was re-verified **byte-identical** after the change.161- Compiled PDF: 29 pages, 0 undefined references/citations.162163### Additional review item discovered during this revision1647. **The "HC1" column of the original Table `tab:robust_se` matches no surviving output.**165   For implied kurtosis it shows 25.87, whereas the surviving CSVs give: pooled HC1166   (rq1, All|5-Day) 40.07, NW(5) 25.64, double-clustered 9.02, with-controls 21.95. The167   column likely comes from an earlier data vintage. The table was left unchanged (its NW168   and DC columns match the CSVs exactly); the new Table "Winsorization and HAC-lag169   sensitivity" reports the reproducible HC1 value (40.07) explicitly.170171## 7. Author headers172173Every code file (Python, LaTeX, BibTeX, Makefile, requirements, Markdown docs) carries the174requested header — `Author: Simon-Pierre Boucher / Contact: contact@spboucher.ai` — in the175comment syntax valid for its language.176