spb/wp7_uqo Public

1<!--2Author: Simon-Pierre Boucher3Contact: contact@spboucher.ai4-->56# AUDIT — WP7 "Options-Implied Information Content" (source: `UQO/UQO_WP/hf-wp7-spb20260519`)78Audit performed on 2026-08-05, **before any restructuring**. The full original tree is preserved9verbatim in [`_old/`](_old/).1011---1213## 1. Project overview1415Working Paper No. 7 (UQO): *The Options-Implied Information Content for Cross-Asset Return and16Volatility Prediction: Evidence from 3.8 Billion Option Contracts*. Five research questions (RQ1–RQ5)17answered with 11 Python scripts, 5 derived Parquet datasets, 33 CSV result tables, and a sectioned18LaTeX paper. **The paper contains no figures — it is tables-only by design** (the only19`\includegraphics` is the UQO logo on the title page). The original `figures/` directory was empty.2021## 2. Scripts (original `scripts/`, 2,814 lines total)2223| Script | Purpose | Inputs | Outputs |24|---|---|---|---|25| `01_extract_data.py` | Extract option-implied features + realized vol from raw DuckDB stores, merge | `options.duckdb`, `stock_5min.duckdb`, `etf_5min.duckdb`, `index_5min.duckdb` (external) | `options_features.parquet`, `realized_vol.parquet`, `merged_options_rv.parquet` |26| `02_rq1_return_predictability.py` | RQ1: pooled OLS (HC1) + Fama-MacBeth panel regressions | `merged_options_rv.parquet` | `rq1_regression_results.csv`, `rq1_meta.csv` |27| `03_rq2_rv_forecasting.py` | RQ2: HAR-RV vs IV-surface models, rolling OOS, Diebold-Mariano | `merged_options_rv.parquet` | `rq2_insample.csv`, `rq2_oos_results.csv`, `rq2_diebold_mariano.csv` |28| `04_rq3_correlation_divergence.py` | RQ3: implied vs realized correlation, stress prediction, crisis windows | `options.duckdb`, `stock_5min.duckdb`, `index_5min.duckdb`, `realized_vol.parquet` | `correlation_divergence.parquet`, `rq3_stress_prediction.csv`, `rq3_crisis_analysis.csv` |29| `05_rq4_greeks_decay_magnets.py` | RQ4: Greeks info decay by DTE + max-OI price-magnet test | `options.duckdb`, `stock_5min.duckdb`, `realized_vol.parquet` | `rq4_greeks_decay.csv`, `rq4_price_magnet.csv`, `price_magnet_data.parquet` |30| `06_rq5_ml_rv_forecast.py` | RQ5: RF/GBM on SPX surface vs VIX vs HAR-RV | `options.duckdb`, `index_5min.duckdb` | `rq5_model_comparison.csv`, `rq5_feature_importance.csv` |31| `07_descriptive_stats.py` | Descriptive statistics (Panels A–H) | `merged_options_rv.parquet`, `index_5min.duckdb` (VIX), `options.duckdb` (quality) | 8 `descriptive_*.csv` |32| `08_subperiod_regime.py` | Subperiods, VIX regimes, rolling R², pre/post COVID | `merged_options_rv.parquet`, `index_5min.duckdb` (VIX) | `subperiod_results.csv`, `regime_results.csv`, `rolling_r2.csv`, `pre_post_covid.csv` |33| `09_portfolio_sorts.py` | Quintile sorts, double sorts, transaction-cost analysis | `merged_options_rv.parquet` | `portfolio_sort_results.csv`, `double_sort_iv_skew.csv`, `transaction_cost_analysis.csv` |34| `10_robustness.py` | Newey-West, double-clustered SE, controls, quantile reg., ticker R² | `merged_options_rv.parquet`, `index_5min.duckdb` (VIX, section E) | `robustness_newey_west.csv`, `robustness_double_clustered.csv`, `robustness_with_controls.csv`, (`robustness_quantile_regression.csv`, `robustness_ticker_r2.csv` — **missing, see §6.2**) |35| `11_granger_var.py` | Granger causality, bivariate VAR(5), IRF, FEVD | `merged_options_rv.parquet` | `granger_causality.csv`, `var_results.csv`, `irf_results.csv`, `fevd_results.csv` |3637### Code-quality observations (fixed in the refactor, without changing any computation)38- Heavy duplication: `winsorize()` re-defined in 6 scripts; the OLS + HC1 t-stat block copy-pasted in39  7 places; standardize-then-`lstsq` pattern repeated ~15 times.40- Hard-coded ticker lists duplicated across scripts (stocks/ETFs/indices exclusion lists appear in41  02, 07, 09 with identical content).42- Paths built with `Path(__file__).parent.parent.parent` — the raw-database location is implicit and43  non-configurable.44- `warnings.filterwarnings('ignore')` everywhere; bare `except:` clauses in several hot loops.45- The p-value column in `02` (`rq1_regression_results.csv`) uses an *ad-hoc normal-tail46  approximation* (`0.5·exp(−t²/2)·√(2/π)`), not an exact two-sided normal/t p-value. Kept as-is47  (results preservation) but documented in the code.4849## 3. Data5051| File | Size | Status |52|---|---|---|53| `data/merged_options_rv.parquet` | 56 MB | **Derived** (script 01). Master analysis panel: 264,383 ticker-days × 69 tickers, 2010–2025 |54| `data/options_features.parquet` | 24 MB | Derived (script 01, intermediate) |55| `data/realized_vol.parquet` | 49 MB | Derived (script 01, intermediate; consumed by 04, 05) |56| `data/correlation_divergence.parquet` | 351 KB | Derived (script 04). Contains VIX close series as a column |57| `data/price_magnet_data.parquet` | 319 KB | Derived (script 05) |5859**Raw data:** the four raw DuckDB stores (`options.duckdb` ≈ 3.83 B rows, `stock_5min.duckdb`,60`etf_5min.duckdb`, `index_5min.duckdb`) lived **outside the project** (two directories up) and61**no longer exist on this machine**. Consequence: scripts 01, 04, 05, 06 and parts of 07, 08, 1062cannot be re-executed end-to-end. All *derived* datasets and result CSVs survive, so every63parquet-only analysis remains reproducible (see §6.1).6465## 4. Results (33 CSVs)6667All 33 CSVs in `results/` are outputs of the scripts above (mapping in §2). No orphan results.68Two CSVs that `10_robustness.py` is coded to write are **absent** (see §6.2).6970## 5. LaTeX7172- `wp7/` — **current paper**: `main.tex` (clean preamble, metadata macros) + `sections/titlepage,73  introduction, literature, data, methodology, results, robustness, discussion, conclusion,74  references` + `appendix/appendix.tex` + `Makefile`/`.latexmkrc` + `uq_logo.jpg`. Compiles with75  latexmk (last built 2026-06-13, `main.pdf` 300 KB). Bibliography is a **manual76  `thebibliography`** (37 entries; exactly matches the 37 cited keys — no missing/unused refs).77- `WP7_Options_Implied_Information_Content.tex` (root, 1,066 lines) — **older monolithic draft** of78  the same paper (identical section structure). Superseded by `wp7/`. → kept only in `_old/`,79  treated as dead file.80- Build artifacts (`main.aux/.log/.out/.toc/.fls/.fdb_latexmk`) — regenerable, not migrated.81- `.DS_Store`, `.claude/settings.local.json` — noise/session config, not migrated.8283## 6. Flags — items requiring the author's attention8485### 6.1 Raw data gone → partial reproducibility (blocking for full pipeline)86The raw DuckDB stores are not on this machine. The pipeline is therefore reproducible **from the87processed parquets onward** only. In the new repo, every raw-dependent script checks for the raw88stores under a configurable `WP7_RAW_DATA_DIR` and exits with a clear message when absent.89Re-runnable today: RQ1 (02), RQ2 (03), portfolio sorts (09), Granger/VAR (11), robustness A–D & F90(10), descriptives A–E & G (07), subperiods A/C/D (08), and the regression stages of RQ3 (3E–3F,91from `correlation_divergence.parquet`).9293### 6.2 `10_robustness.py` — memory bug; two result files missing; paper cites them94- Section D (quantile regression) builds `np.diag(weights)` on the full sample95  (~110k obs → a 110k×110k dense matrix ≈ **97 GB**). The original run almost certainly died there96  (MemoryError), which explains why `robustness_quantile_regression.csv` and97  `robustness_ticker_r2.csv` (written later in the script) are missing from `results/`.98- Yet `sections/robustness.tex` cites those results (quantile-regression significance; ticker-level99  R²: mean 5.9 %, median 4.7 %, IQR [2.0 %, 8.2 %]). **These numbers had no surviving CSV.**100- In the refactor the weighting was rewritten with broadcasting (`X * w[:,None]`) — mathematically101  identical, memory-safe. Sections D and F were re-run to regenerate the two missing CSVs.102- **Verification outcome — DISCREPANCY (author review required):**103  - *Ticker-level R² (robustness.tex §4.4):* the paper quotes mean 5.9 %, median 4.7 %,104    IQR [2.0 %, 8.2 %]. Regenerating section F from the surviving panel with the script's exact105    specification yields **mean 14.0 %, median 13.2 %, IQR [10.0 %, 17.6 %]** (69 tickers).106    The paper's figures cannot be reproduced from the surviving data; they may come from an107    earlier data vintage or a different specification. The paper text was left unchanged.108    (Qualitatively the claim survives either way — panel results are not driven by outliers.)109  - *Quantile regressions (robustness.tex §4.4):* the paper states the predictors "are110    significant across the return distribution", but the quantile-regression code computes111    **coefficients only — no standard errors or t-statistics exist** in any output. The112    regenerated coefficients are consistently signed across τ ∈ [0.10, 0.90] (implied kurtosis113    positive, PC volume ratio negative), which supports stability but not a significance claim.114    Paper text left unchanged; flagged for the author.115  - *Rolling-window stats (robustness.tex §4.4):* verified exact — paper (mean 8.0 %, std 4.5 %,116    range [2.1 %, 17.9 %] for 5-day returns; 41.2 %, std 10.7 % for 1-day RV) matches117    `rolling_r2.csv` to the digit.118119### 6.3 Paper claims not fully backed by a surviving results file120- Robustness §"VIX < 30" subsample (script 10, section E) prints to console only — no CSV was ever121  written. Not quoted numerically in the paper (prose only), so nothing to reconcile.122- Diebold-Mariano: the paper quotes 4 tickers (AAPL, AMD, CAT, COST); `rq2_diebold_mariano.csv`123  contains the full set — verified consistent.124125### 6.4 Minor inconsistencies (documentation-level, no action taken on results)126- `04` computes a `const_returns` DuckDB query whose result is **never used** (dead code; realized127  correlations actually come from daily returns in `realized_vol.parquet`). Removed in refactor —128  no output touched.129- Sample-size figures in the paper (e.g., N = 264,383) match `merged_options_rv.parquet` (verified130  by row count) and `descriptive_by_group.csv`.131132## 7. Verification of reproduction (Phase 2c results)133134Method: the refactored scripts were re-run with `WP7_RESULTS_DIR` pointed at a scratch directory135(`_verify/results/`, kept in the repo for audit), then compared against the originals with136`_verify/compare_results.py` (byte comparison first, then numeric with `rtol=1e-9`).137Environment: Python 3.13 / numpy 2.4.4 / pandas 3.0.2 (Apple Silicon).138139**Outcome — 28 files compared, all conform; 1 explained deviation; 2 files newly generated:**140141| Status | Files |142|---|---|143| Byte-identical | 10 (all descriptives A–E/G, portfolio sorts, double sort, transaction costs, rq1_meta, rq3_crisis, rq4_price_magnet) |144| Numerically equal (max rel. diff ≤ 2×10⁻¹⁰, i.e. floating-point noise) | 15 (rq1 regressions, rq2 in-sample/OOS/DM, granger, VAR/IRF/FEVD, subperiods, rolling R², pre/post-COVID, NW, double-clustered, controls) |145| Deviation, explained | `rq3_stress_prediction.csv` — only the `t_stat` column. Root cause: the design matrix contains an **exact linear dependency** (`corr_divergence = implied_corr − realized_corr`), so `X'X` is singular and HC1 t-stats are numerically degenerate. The economically meaningful `corr_ratio` t-stats move from (4.07, 8.30, 3.91) to (4.07, 8.22, 3.73) across BLAS/numpy versions — same significance everywhere; the near-zero t-stats (≈10⁻⁷) on the collinear regressors are pure noise. Original CSV kept as authoritative. |146| Newly generated (missing from original archive) | `robustness_quantile_regression.csv`, `robustness_ticker_r2.csv` — see §6.2 |147148Raw-dependent outputs (rq4_greeks_decay, rq5_*, descriptive_vix_regimes,149descriptive_options_quality, regime_results) could not be re-run (§6.1); the original CSVs are150shipped unchanged in `results/`.151152**No original CSV and no number in the paper was modified.**153