# AUDIT — WP7 "Options-Implied Information Content" (source: `UQO/UQO_WP/hf-wp7-spb20260519`) Audit performed on 2026-08-05, **before any restructuring**. The full original tree is preserved verbatim in [`_old/`](_old/). --- ## 1. Project overview Working Paper No. 7 (UQO): *The Options-Implied Information Content for Cross-Asset Return and Volatility Prediction: Evidence from 3.8 Billion Option Contracts*. Five research questions (RQ1–RQ5) answered with 11 Python scripts, 5 derived Parquet datasets, 33 CSV result tables, and a sectioned LaTeX paper. **The paper contains no figures — it is tables-only by design** (the only `\includegraphics` is the UQO logo on the title page). The original `figures/` directory was empty. ## 2. Scripts (original `scripts/`, 2,814 lines total) | Script | Purpose | Inputs | Outputs | |---|---|---|---| | `01_extract_data.py` | Extract option-implied features + realized vol from raw DuckDB stores, merge | `options.duckdb`, `stock_5min.duckdb`, `etf_5min.duckdb`, `index_5min.duckdb` (external) | `options_features.parquet`, `realized_vol.parquet`, `merged_options_rv.parquet` | | `02_rq1_return_predictability.py` | RQ1: pooled OLS (HC1) + Fama-MacBeth panel regressions | `merged_options_rv.parquet` | `rq1_regression_results.csv`, `rq1_meta.csv` | | `03_rq2_rv_forecasting.py` | RQ2: HAR-RV vs IV-surface models, rolling OOS, Diebold-Mariano | `merged_options_rv.parquet` | `rq2_insample.csv`, `rq2_oos_results.csv`, `rq2_diebold_mariano.csv` | | `04_rq3_correlation_divergence.py` | RQ3: implied vs realized correlation, stress prediction, crisis windows | `options.duckdb`, `stock_5min.duckdb`, `index_5min.duckdb`, `realized_vol.parquet` | `correlation_divergence.parquet`, `rq3_stress_prediction.csv`, `rq3_crisis_analysis.csv` | | `05_rq4_greeks_decay_magnets.py` | RQ4: Greeks info decay by DTE + max-OI price-magnet test | `options.duckdb`, `stock_5min.duckdb`, `realized_vol.parquet` | `rq4_greeks_decay.csv`, `rq4_price_magnet.csv`, `price_magnet_data.parquet` | | `06_rq5_ml_rv_forecast.py` | RQ5: RF/GBM on SPX surface vs VIX vs HAR-RV | `options.duckdb`, `index_5min.duckdb` | `rq5_model_comparison.csv`, `rq5_feature_importance.csv` | | `07_descriptive_stats.py` | Descriptive statistics (Panels A–H) | `merged_options_rv.parquet`, `index_5min.duckdb` (VIX), `options.duckdb` (quality) | 8 `descriptive_*.csv` | | `08_subperiod_regime.py` | Subperiods, VIX regimes, rolling R², pre/post COVID | `merged_options_rv.parquet`, `index_5min.duckdb` (VIX) | `subperiod_results.csv`, `regime_results.csv`, `rolling_r2.csv`, `pre_post_covid.csv` | | `09_portfolio_sorts.py` | Quintile sorts, double sorts, transaction-cost analysis | `merged_options_rv.parquet` | `portfolio_sort_results.csv`, `double_sort_iv_skew.csv`, `transaction_cost_analysis.csv` | | `10_robustness.py` | Newey-West, double-clustered SE, controls, quantile reg., ticker R² | `merged_options_rv.parquet`, `index_5min.duckdb` (VIX, section E) | `robustness_newey_west.csv`, `robustness_double_clustered.csv`, `robustness_with_controls.csv`, (`robustness_quantile_regression.csv`, `robustness_ticker_r2.csv` — **missing, see §6.2**) | | `11_granger_var.py` | Granger causality, bivariate VAR(5), IRF, FEVD | `merged_options_rv.parquet` | `granger_causality.csv`, `var_results.csv`, `irf_results.csv`, `fevd_results.csv` | ### Code-quality observations (fixed in the refactor, without changing any computation) - Heavy duplication: `winsorize()` re-defined in 6 scripts; the OLS + HC1 t-stat block copy-pasted in 7 places; standardize-then-`lstsq` pattern repeated ~15 times. - Hard-coded ticker lists duplicated across scripts (stocks/ETFs/indices exclusion lists appear in 02, 07, 09 with identical content). - Paths built with `Path(__file__).parent.parent.parent` — the raw-database location is implicit and non-configurable. - `warnings.filterwarnings('ignore')` everywhere; bare `except:` clauses in several hot loops. - The p-value column in `02` (`rq1_regression_results.csv`) uses an *ad-hoc normal-tail approximation* (`0.5·exp(−t²/2)·√(2/π)`), not an exact two-sided normal/t p-value. Kept as-is (results preservation) but documented in the code. ## 3. Data | File | Size | Status | |---|---|---| | `data/merged_options_rv.parquet` | 56 MB | **Derived** (script 01). Master analysis panel: 264,383 ticker-days × 69 tickers, 2010–2025 | | `data/options_features.parquet` | 24 MB | Derived (script 01, intermediate) | | `data/realized_vol.parquet` | 49 MB | Derived (script 01, intermediate; consumed by 04, 05) | | `data/correlation_divergence.parquet` | 351 KB | Derived (script 04). Contains VIX close series as a column | | `data/price_magnet_data.parquet` | 319 KB | Derived (script 05) | **Raw data:** the four raw DuckDB stores (`options.duckdb` ≈ 3.83 B rows, `stock_5min.duckdb`, `etf_5min.duckdb`, `index_5min.duckdb`) lived **outside the project** (two directories up) and **no longer exist on this machine**. Consequence: scripts 01, 04, 05, 06 and parts of 07, 08, 10 cannot be re-executed end-to-end. All *derived* datasets and result CSVs survive, so every parquet-only analysis remains reproducible (see §6.1). ## 4. Results (33 CSVs) All 33 CSVs in `results/` are outputs of the scripts above (mapping in §2). No orphan results. Two CSVs that `10_robustness.py` is coded to write are **absent** (see §6.2). ## 5. LaTeX - `wp7/` — **current paper**: `main.tex` (clean preamble, metadata macros) + `sections/titlepage, introduction, literature, data, methodology, results, robustness, discussion, conclusion, references` + `appendix/appendix.tex` + `Makefile`/`.latexmkrc` + `uq_logo.jpg`. Compiles with latexmk (last built 2026-06-13, `main.pdf` 300 KB). Bibliography is a **manual `thebibliography`** (37 entries; exactly matches the 37 cited keys — no missing/unused refs). - `WP7_Options_Implied_Information_Content.tex` (root, 1,066 lines) — **older monolithic draft** of the same paper (identical section structure). Superseded by `wp7/`. → kept only in `_old/`, treated as dead file. - Build artifacts (`main.aux/.log/.out/.toc/.fls/.fdb_latexmk`) — regenerable, not migrated. - `.DS_Store`, `.claude/settings.local.json` — noise/session config, not migrated. ## 6. Flags — items requiring the author's attention ### 6.1 Raw data gone → partial reproducibility (blocking for full pipeline) The raw DuckDB stores are not on this machine. The pipeline is therefore reproducible **from the processed parquets onward** only. In the new repo, every raw-dependent script checks for the raw stores under a configurable `WP7_RAW_DATA_DIR` and exits with a clear message when absent. Re-runnable today: RQ1 (02), RQ2 (03), portfolio sorts (09), Granger/VAR (11), robustness A–D & F (10), descriptives A–E & G (07), subperiods A/C/D (08), and the regression stages of RQ3 (3E–3F, from `correlation_divergence.parquet`). ### 6.2 `10_robustness.py` — memory bug; two result files missing; paper cites them - Section D (quantile regression) builds `np.diag(weights)` on the full sample (~110k obs → a 110k×110k dense matrix ≈ **97 GB**). The original run almost certainly died there (MemoryError), which explains why `robustness_quantile_regression.csv` and `robustness_ticker_r2.csv` (written later in the script) are missing from `results/`. - Yet `sections/robustness.tex` cites those results (quantile-regression significance; ticker-level R²: mean 5.9 %, median 4.7 %, IQR [2.0 %, 8.2 %]). **These numbers had no surviving CSV.** - In the refactor the weighting was rewritten with broadcasting (`X * w[:,None]`) — mathematically identical, memory-safe. Sections D and F were re-run to regenerate the two missing CSVs. - **Verification outcome — DISCREPANCY (author review required):** - *Ticker-level R² (robustness.tex §4.4):* the paper quotes mean 5.9 %, median 4.7 %, IQR [2.0 %, 8.2 %]. Regenerating section F from the surviving panel with the script's exact specification yields **mean 14.0 %, median 13.2 %, IQR [10.0 %, 17.6 %]** (69 tickers). The paper's figures cannot be reproduced from the surviving data; they may come from an earlier data vintage or a different specification. The paper text was left unchanged. (Qualitatively the claim survives either way — panel results are not driven by outliers.) - *Quantile regressions (robustness.tex §4.4):* the paper states the predictors "are significant across the return distribution", but the quantile-regression code computes **coefficients only — no standard errors or t-statistics exist** in any output. The regenerated coefficients are consistently signed across τ ∈ [0.10, 0.90] (implied kurtosis positive, PC volume ratio negative), which supports stability but not a significance claim. Paper text left unchanged; flagged for the author. - *Rolling-window stats (robustness.tex §4.4):* verified exact — paper (mean 8.0 %, std 4.5 %, range [2.1 %, 17.9 %] for 5-day returns; 41.2 %, std 10.7 % for 1-day RV) matches `rolling_r2.csv` to the digit. ### 6.3 Paper claims not fully backed by a surviving results file - Robustness §"VIX < 30" subsample (script 10, section E) prints to console only — no CSV was ever written. Not quoted numerically in the paper (prose only), so nothing to reconcile. - Diebold-Mariano: the paper quotes 4 tickers (AAPL, AMD, CAT, COST); `rq2_diebold_mariano.csv` contains the full set — verified consistent. ### 6.4 Minor inconsistencies (documentation-level, no action taken on results) - `04` computes a `const_returns` DuckDB query whose result is **never used** (dead code; realized correlations actually come from daily returns in `realized_vol.parquet`). Removed in refactor — no output touched. - Sample-size figures in the paper (e.g., N = 264,383) match `merged_options_rv.parquet` (verified by row count) and `descriptive_by_group.csv`. ## 7. Verification of reproduction (Phase 2c results) Method: the refactored scripts were re-run with `WP7_RESULTS_DIR` pointed at a scratch directory (`_verify/results/`, kept in the repo for audit), then compared against the originals with `_verify/compare_results.py` (byte comparison first, then numeric with `rtol=1e-9`). Environment: Python 3.13 / numpy 2.4.4 / pandas 3.0.2 (Apple Silicon). **Outcome — 28 files compared, all conform; 1 explained deviation; 2 files newly generated:** | Status | Files | |---|---| | Byte-identical | 10 (all descriptives A–E/G, portfolio sorts, double sort, transaction costs, rq1_meta, rq3_crisis, rq4_price_magnet) | | Numerically equal (max rel. diff ≤ 2×10⁻¹⁰, i.e. floating-point noise) | 15 (rq1 regressions, rq2 in-sample/OOS/DM, granger, VAR/IRF/FEVD, subperiods, rolling R², pre/post-COVID, NW, double-clustered, controls) | | Deviation, explained | `rq3_stress_prediction.csv` — only the `t_stat` column. Root cause: the design matrix contains an **exact linear dependency** (`corr_divergence = implied_corr − realized_corr`), so `X'X` is singular and HC1 t-stats are numerically degenerate. The economically meaningful `corr_ratio` t-stats move from (4.07, 8.30, 3.91) to (4.07, 8.22, 3.73) across BLAS/numpy versions — same significance everywhere; the near-zero t-stats (≈10⁻⁷) on the collinear regressors are pure noise. Original CSV kept as authoritative. | | Newly generated (missing from original archive) | `robustness_quantile_regression.csv`, `robustness_ticker_r2.csv` — see §6.2 | Raw-dependent outputs (rq4_greeks_decay, rq5_*, descriptive_vix_regimes, descriptive_options_quality, regime_results) could not be re-run (§6.1); the original CSVs are shipped unchanged in `results/`. **No original CSV and no number in the paper was modified.**