# CHANGES — Restructuring report (2026-08-05)

Source: ~/Desktop/UQO/UQO_WP/hf-wp7-spb20260519 (preserved verbatim in _old/). Target: this repository (~/Desktop/wp7_uqo). No original result or number was changed anywhere. Revision 1.1 (§6) adds new robustness analyses and references on the author's request and corrects one unsupported sentence, as documented there. Companion document: AUDIT.md (pre-restructuring audit, reproducibility map, and verification evidence).

# 1. What was moved / renamed

# 2. Code refactoring (results-preserving)

All statistical logic now lives in a small library, src/wp7/, defined exactly once:

• config.py — every path (env-overridable via WP7_RAW_DATA_DIR, WP7_RESULTS_DIR), ticker universes, feature sets, subperiods, crisis dates, VIX-regime bins. Previously these were duplicated across up to 6 scripts with hard-coded parent.parent.parent paths.
• data_io.py — parquet loaders + raw-DuckDB access with a dedicated RawDataUnavailableError so raw-dependent steps skip gracefully instead of crashing.
• econometrics.py — winsorize, standardization, OLS via lstsq, HC1 / Newey-West / double-clustered (CGM 2011) inference, Fama-MacBeth, quantile regression (IRLS), Granger F-test, bivariate VAR + IRF. Previously copy-pasted 6–15× across scripts.
• forecasting.py — OLS forecasting, rolling out-of-sample evaluation, Diebold-Mariano.
• portfolio.py — quintile sorts, double sorts, performance statistics.

Scripts keep their original numbers and filenames; each has a docstring stating purpose, inputs and outputs, and the required author header (Author: Simon-Pierre Boucher / Contact: contact@spboucher.ai — as #/% comments in Python/LaTeX, as an HTML comment in Markdown, since <!-- --> is not valid syntax in Python or LaTeX, and a literal @ cannot appear in a BibTeX comment).

Deliberate preservation of historical quirks (documented in-code):

• Script 02's panel_ols keeps its pandas (ddof = 1) standardization, zero-filled missing z-scores, and the ad-hoc normal-tail "p_value" approximation — these define the published numbers.
• All sandwich estimators keep the original n/(n−k−1) small-sample factor and lstsq-based estimation (no statsmodels), so results reproduce to the bit.

Genuine fixes (behavior-preserving or bug-fixing, flagged in AUDIT.md):

• Memory bug fixed (10_robustness.py): X.T @ np.diag(e²) @ X and the IRLS weight matrix materialized an n×n dense matrix (≈97 GB → the original run of sections D–F died; two result CSVs were missing). Rewritten with broadcasting — mathematically identical, O(nk) memory.
• Dead code removed (04): an unused DuckDB query (const_returns) whose result was never read.
• Bare except: → targeted exceptions; RES.mkdir centralized; no other logic touched.

# 3. Paper rewrite (editorial only)

• Structure: main.tex (metadata + skeleton) / preamble.tex / one file per section / appendix/ — the original section split was kept; the root-level monolithic draft was retired to _old/.
• Bibliography: manual thebibliography → references.bib with natbib + apalike (BibTeX). All 37 cite keys unchanged; entry metadata transcribed faithfully. The PDF now renders the reference list from BibTeX — no visual regression.
• Prose: light editorial pass over introduction, literature, discussion, conclusion (flow, voice consistency, hyphenation, article usage). Data/methodology/results/robustness needed only targeted touches. Every number, table value, citation, and claim is unchanged.
• Cross-referencing completed: the four appendix tables were never referenced from the body; the results and methodology sections now point to app:crises, app:features, app:importance, app:fevd, and each appendix section gained a one-sentence lead-in.
• Build: latexmk (pdflatex + bibtex) compiles clean — exit 0, 0 undefined references, 0 undefined citations, 24 pages. make paper from the repo root.

# 4. Verification of reproduction (summary — details in AUDIT.md §7)

Re-run environment: Python 3.13, numpy 2.4.4, pandas 3.0.2 (Apple Silicon). Regenerated CSVs written to _verify/results/ and compared with _verify/compare_results.py:

• 10 files byte-identical; 15 files numerically equal (max relative difference ≤ 2×10⁻¹⁰ — floating-point noise across BLAS/numpy versions).
• 1 explained deviation: rq3_stress_prediction.csv t-statistics only. The regression contains an exact linear dependency (corr_divergence = implied_corr − realized_corr), making HC1 t-stats numerically degenerate; the significant corr_ratio t-stats move by ≤ 0.18 with identical significance conclusions. Original CSV kept as authoritative.
• 2 files regenerated that were missing from the original archive (the §2 memory bug): robustness_quantile_regression.csv, robustness_ticker_r2.csv — added to results/.
• Raw-dependent outputs (rq4 Greeks decay, rq5, VIX-regime tables, options quality) cannot be re-run (raw stores gone) — original CSVs shipped unchanged.

# 5. Items requiring your review

1. Ticker-level R² claim (robustness §4.4) is not reproducible. Paper: mean 5.9 %, median 4.7 %, IQR [2.0 %, 8.2 %]. Regenerated from the surviving panel with the script's own specification: mean 14.0 %, median 13.2 %, IQR [10.0 %, 17.6 %] (69 tickers). The text was left as-is per the no-content-change rule; the qualitative conclusion (results not driven by outliers) holds under either set. Consider updating the paragraph or documenting the original specification.
2. Quantile-regression "significance" (robustness §4.4). The code computes coefficients only — no standard errors or t-statistics were ever produced — so "significant across the return distribution" has no supporting statistic. The regenerated coefficients are consistently signed across τ (kurtosis +, PC ratio −), supporting a weaker "stable sign" statement.
3. RQ3 t-statistics are fragile by construction (exact collinearity among the four correlation regressors). Consider dropping corr_divergence or implied_corr/ realized_corr from the specification in a future revision.
4. Raw data loss. The four DuckDB stores are gone; data/processed/ is now the only copy of the analysis data. Recommend an off-machine backup of this repository.
5. Possible bibliography metadata issue (ni2009does): "Does option trading convey stock price information?" is listed for both Ni-Pearson-Poteshman-White (2009, JF) and Hu (2014, JFE). Both entries were transcribed as the author wrote them, but the Ni et al. title/venue/pages may need checking against the published record.
6. Panel F/H, regime, and rq5 tables are frozen (raw-dependent) — any future data refresh must rebuild the raw stores first.

# 6. Revision 1.1 — extended robustness analyses and references (added on request)

On top of the results-preserving restructure, the paper was upgraded with new analyses (all computable from the surviving processed panel) and eight classical references:

• New script scripts/13_extended_robustness.py → six new result files (results/extended_*.csv):
  • Winsorization sensitivity — 5-day return regression under cutoffs {none, 0.5%, 1%, 2.5%, 5%}: R² ranges 0.034–0.062; key predictors significant throughout.
  • Newey-West lag sensitivity — HAC t-stats at 5/10/22 lags: 9/10 significant predictors at every lag (kurtosis t: 25.6 → 22.1).
  • Spearman information coefficients — daily cross-sectional rank correlations (stocks): kurtosis IC +0.098 (t=16.8, 64.3% positive days); PC volume ratio −0.059 (t=−13.8).
  • Decile sorts — D10−D1 spreads widen (kurtosis 44.0% → 52.1% ann.) with Sharpe ratios essentially unchanged.
  • Leave-one-year-out — 5D-return R² ∈ [0.047, 0.057]; HAR+IV RV R² ∈ [0.414, 0.492] (2020 exclusion is the only visible mover).
  • Placebo — features permuted within ticker (seed 42, 10 draws): R² collapses from 0.0529 to ≤ 0.0010.
• Paper: four new subsections in sections/robustness.tex (§6.5–6.8) with three new tables + one two-panel table; abstract, introduction and methodology updated accordingly. The unsupported quantile-regression "significance" sentence (see §5.2) was reworded to the supported claim (stable sign/magnitude across τ) with a \citep{koenker1978regression}.
• New references (8): White (1980); Newey & West (1987); Koenker & Bassett (1978); Petersen (2009); Fama & French (2008); Welch & Goyal (2008); Campbell & Thompson (2008); Harvey, Liu & Zhu (2016). references.bib now has 45 entries, all cited.
• portfolio_sort() was generalized to arbitrary quantile counts (LS_<n>_1 key); script 09 was re-verified byte-identical after the change.
• Compiled PDF: 29 pages, 0 undefined references/citations.

# Additional review item discovered during this revision

7. The "HC1" column of the original Table tab:robust_se matches no surviving output. For implied kurtosis it shows 25.87, whereas the surviving CSVs give: pooled HC1 (rq1, All|5-Day) 40.07, NW(5) 25.64, double-clustered 9.02, with-controls 21.95. The column likely comes from an earlier data vintage. The table was left unchanged (its NW and DC columns match the CSVs exactly); the new Table "Winsorization and HAC-lag sensitivity" reports the reproducible HC1 value (40.07) explicitly.

# 7. Author headers

Every code file (Python, LaTeX, BibTeX, Makefile, requirements, Markdown docs) carries the requested header — Author: Simon-Pierre Boucher / Contact: contact@spboucher.ai — in the comment syntax valid for its language.