# CHANGES — Restructuring report (2026-08-05) Source: `~/Desktop/UQO/UQO_WP/hf-wp7-spb20260519` (preserved verbatim in `_old/`). Target: this repository (`~/Desktop/wp7_uqo`). **No original result or number was changed anywhere.** Revision 1.1 (§6) *adds* new robustness analyses and references on the author's request and corrects one unsupported sentence, as documented there. Companion document: `AUDIT.md` (pre-restructuring audit, reproducibility map, and verification evidence). --- ## 1. What was moved / renamed | Original | New location | Notes | |---|---|---| | `data/*.parquet` (5 files) | `data/processed/` | unchanged bytes; raw data documented in `data/raw/README.md` | | `results/*.csv` (33 files) | `results/` | unchanged bytes; **+2 regenerated files** (see §4) | | `scripts/01…11_*.py` | `scripts/` (same names/numbers) | fully refactored (see §2) | | `wp7/main.tex` | `paper/main.tex` + `paper/preamble.tex` | preamble split out; BibTeX enabled | | `wp7/sections/*.tex` | `paper/sections/` | editorial rewrite (see §3) | | `wp7/sections/references.tex` | `paper/references.bib` | manual `thebibliography` (37 entries) → BibTeX; file deleted from sections | | `wp7/appendix/appendix.tex` | `paper/appendix/appendix.tex` | + lead-in prose, cross-referenced from the body | | `wp7/uq_logo.jpg`, `Makefile`, `.latexmkrc` | `paper/` | `.latexmkrc` now runs bibtex (`$bibtex_use = 2`) | | `WP7_Options_Implied_Information_Content.tex` (root, 1,066 lines) | `_old/` only | older monolithic duplicate of the same paper — dead file | | `wp7/main.{aux,log,out,toc,fls,fdb_latexmk}` | not migrated | build artifacts (regenerable; `.gitignore`d) | | `.DS_Store`, `.claude/` | not migrated | noise / session config | | — (new) | `README.md`, `AUDIT.md`, `CHANGES.md`, `requirements.txt`, `pyproject.toml`, `Makefile`, `.gitignore`, `_verify/` | repository infrastructure | | — (new) | `scripts/12_make_figures.py`, `figures/fig_*.{png,pdf}` | supplementary figures (the paper remains tables-only) | ## 2. Code refactoring (results-preserving) All statistical logic now lives in a small library, `src/wp7/`, defined exactly once: - **`config.py`** — every path (env-overridable via `WP7_RAW_DATA_DIR`, `WP7_RESULTS_DIR`), ticker universes, feature sets, subperiods, crisis dates, VIX-regime bins. Previously these were duplicated across up to 6 scripts with hard-coded `parent.parent.parent` paths. - **`data_io.py`** — parquet loaders + raw-DuckDB access with a dedicated `RawDataUnavailableError` so raw-dependent steps skip gracefully instead of crashing. - **`econometrics.py`** — `winsorize`, standardization, OLS via `lstsq`, HC1 / Newey-West / double-clustered (CGM 2011) inference, Fama-MacBeth, quantile regression (IRLS), Granger F-test, bivariate VAR + IRF. Previously copy-pasted 6–15× across scripts. - **`forecasting.py`** — OLS forecasting, rolling out-of-sample evaluation, Diebold-Mariano. - **`portfolio.py`** — quintile sorts, double sorts, performance statistics. Scripts keep their original numbers and filenames; each has a docstring stating purpose, inputs and outputs, and the required author header (`Author: Simon-Pierre Boucher / Contact: contact@spboucher.ai` — as `#`/`%` comments in Python/LaTeX, as an HTML comment in Markdown, since `` is not valid syntax in Python or LaTeX, and a literal `@` cannot appear in a BibTeX comment). Deliberate preservation of historical quirks (documented in-code): - Script 02's `panel_ols` keeps its pandas (ddof = 1) standardization, zero-filled missing z-scores, and the ad-hoc normal-tail "p_value" approximation — these define the published numbers. - All sandwich estimators keep the original `n/(n−k−1)` small-sample factor and `lstsq`-based estimation (no statsmodels), so results reproduce to the bit. Genuine fixes (behavior-preserving or bug-fixing, flagged in `AUDIT.md`): - **Memory bug fixed** (`10_robustness.py`): `X.T @ np.diag(e²) @ X` and the IRLS weight matrix materialized an *n×n* dense matrix (≈97 GB → the original run of sections D–F died; two result CSVs were missing). Rewritten with broadcasting — mathematically identical, O(nk) memory. - **Dead code removed** (`04`): an unused DuckDB query (`const_returns`) whose result was never read. - Bare `except:` → targeted exceptions; `RES.mkdir` centralized; no other logic touched. ## 3. Paper rewrite (editorial only) - **Structure**: `main.tex` (metadata + skeleton) / `preamble.tex` / one file per section / `appendix/` — the original section split was kept; the root-level monolithic draft was retired to `_old/`. - **Bibliography**: manual `thebibliography` → `references.bib` with natbib + `apalike` (BibTeX). All 37 cite keys unchanged; entry metadata transcribed faithfully. The PDF now renders the reference list from BibTeX — no visual regression. - **Prose**: light editorial pass over introduction, literature, discussion, conclusion (flow, voice consistency, hyphenation, article usage). Data/methodology/results/robustness needed only targeted touches. **Every number, table value, citation, and claim is unchanged.** - **Cross-referencing completed**: the four appendix tables were never referenced from the body; the results and methodology sections now point to `app:crises`, `app:features`, `app:importance`, `app:fevd`, and each appendix section gained a one-sentence lead-in. - **Build**: `latexmk` (pdflatex + bibtex) compiles clean — exit 0, **0 undefined references, 0 undefined citations, 24 pages**. `make paper` from the repo root. ## 4. Verification of reproduction (summary — details in `AUDIT.md` §7) Re-run environment: Python 3.13, numpy 2.4.4, pandas 3.0.2 (Apple Silicon). Regenerated CSVs written to `_verify/results/` and compared with `_verify/compare_results.py`: - **10 files byte-identical**; **15 files numerically equal** (max relative difference ≤ 2×10⁻¹⁰ — floating-point noise across BLAS/numpy versions). - **1 explained deviation**: `rq3_stress_prediction.csv` t-statistics only. The regression contains an exact linear dependency (`corr_divergence = implied_corr − realized_corr`), making HC1 t-stats numerically degenerate; the significant `corr_ratio` t-stats move by ≤ 0.18 with identical significance conclusions. Original CSV kept as authoritative. - **2 files regenerated** that were missing from the original archive (the §2 memory bug): `robustness_quantile_regression.csv`, `robustness_ticker_r2.csv` — added to `results/`. - Raw-dependent outputs (rq4 Greeks decay, rq5, VIX-regime tables, options quality) cannot be re-run (raw stores gone) — original CSVs shipped unchanged. ## 5. Items requiring your review 1. **Ticker-level R² claim (robustness §4.4) is not reproducible.** Paper: mean 5.9 %, median 4.7 %, IQR [2.0 %, 8.2 %]. Regenerated from the surviving panel with the script's own specification: **mean 14.0 %, median 13.2 %, IQR [10.0 %, 17.6 %]** (69 tickers). The text was left as-is per the no-content-change rule; the qualitative conclusion (results not driven by outliers) holds under either set. Consider updating the paragraph or documenting the original specification. 2. **Quantile-regression "significance" (robustness §4.4).** The code computes coefficients only — no standard errors or t-statistics were ever produced — so "significant across the return distribution" has no supporting statistic. The regenerated coefficients are consistently signed across τ (kurtosis +, PC ratio −), supporting a weaker "stable sign" statement. 3. **RQ3 t-statistics are fragile by construction** (exact collinearity among the four correlation regressors). Consider dropping `corr_divergence` or `implied_corr`/ `realized_corr` from the specification in a future revision. 4. **Raw data loss.** The four DuckDB stores are gone; `data/processed/` is now the only copy of the analysis data. Recommend an off-machine backup of this repository. 5. **Possible bibliography metadata issue** (`ni2009does`): "Does option trading convey stock price information?" is listed for both Ni-Pearson-Poteshman-White (2009, JF) and Hu (2014, JFE). Both entries were transcribed as the author wrote them, but the Ni et al. title/venue/pages may need checking against the published record. 6. **Panel F/H, regime, and rq5 tables** are frozen (raw-dependent) — any future data refresh must rebuild the raw stores first. ## 6. Revision 1.1 — extended robustness analyses and references (added on request) On top of the results-preserving restructure, the paper was upgraded with **new** analyses (all computable from the surviving processed panel) and eight classical references: - **New script** `scripts/13_extended_robustness.py` → six new result files (`results/extended_*.csv`): - *Winsorization sensitivity* — 5-day return regression under cutoffs {none, 0.5%, 1%, 2.5%, 5%}: R² ranges 0.034–0.062; key predictors significant throughout. - *Newey-West lag sensitivity* — HAC t-stats at 5/10/22 lags: 9/10 significant predictors at every lag (kurtosis t: 25.6 → 22.1). - *Spearman information coefficients* — daily cross-sectional rank correlations (stocks): kurtosis IC +0.098 (t=16.8, 64.3% positive days); PC volume ratio −0.059 (t=−13.8). - *Decile sorts* — D10−D1 spreads widen (kurtosis 44.0% → 52.1% ann.) with Sharpe ratios essentially unchanged. - *Leave-one-year-out* — 5D-return R² ∈ [0.047, 0.057]; HAR+IV RV R² ∈ [0.414, 0.492] (2020 exclusion is the only visible mover). - *Placebo* — features permuted within ticker (seed 42, 10 draws): R² collapses from 0.0529 to ≤ 0.0010. - **Paper**: four new subsections in `sections/robustness.tex` (§6.5–6.8) with three new tables + one two-panel table; abstract, introduction and methodology updated accordingly. The unsupported quantile-regression "significance" sentence (see §5.2) was reworded to the supported claim (stable sign/magnitude across τ) with a \citep{koenker1978regression}. - **New references (8)**: White (1980); Newey & West (1987); Koenker & Bassett (1978); Petersen (2009); Fama & French (2008); Welch & Goyal (2008); Campbell & Thompson (2008); Harvey, Liu & Zhu (2016). `references.bib` now has 45 entries, all cited. - `portfolio_sort()` was generalized to arbitrary quantile counts (`LS__1` key); script 09 was re-verified **byte-identical** after the change. - Compiled PDF: 29 pages, 0 undefined references/citations. ### Additional review item discovered during this revision 7. **The "HC1" column of the original Table `tab:robust_se` matches no surviving output.** For implied kurtosis it shows 25.87, whereas the surviving CSVs give: pooled HC1 (rq1, All|5-Day) 40.07, NW(5) 25.64, double-clustered 9.02, with-controls 21.95. The column likely comes from an earlier data vintage. The table was left unchanged (its NW and DC columns match the CSVs exactly); the new Table "Winsorization and HAC-lag sensitivity" reports the reproducible HC1 value (40.07) explicitly. ## 7. Author headers Every code file (Python, LaTeX, BibTeX, Makefile, requirements, Markdown docs) carries the requested header — `Author: Simon-Pierre Boucher / Contact: contact@spboucher.ai` — in the comment syntax valid for its language.