SPB Git

spb/wp7_uqo Public

UQO Working Paper No. 7 — Options-implied information for cross-asset return and volatility prediction: evidence from 3.8B option contracts.

Python 66.5% TeX 32.7% Makefile 0.8%
11.6 KB

# AUDIT — WP7 "Options-Implied Information Content" (source: UQO/UQO_WP/hf-wp7-spb20260519)

Audit performed on 2026-08-05, before any restructuring. The full original tree is preserved verbatim in _old/.


# 1. Project overview

Working Paper No. 7 (UQO): The Options-Implied Information Content for Cross-Asset Return and Volatility Prediction: Evidence from 3.8 Billion Option Contracts. Five research questions (RQ1–RQ5) answered with 11 Python scripts, 5 derived Parquet datasets, 33 CSV result tables, and a sectioned LaTeX paper. The paper contains no figures — it is tables-only by design (the only \includegraphics is the UQO logo on the title page). The original figures/ directory was empty.

# 2. Scripts (original scripts/, 2,814 lines total)

Script Purpose Inputs Outputs
01_extract_data.py Extract option-implied features + realized vol from raw DuckDB stores, merge options.duckdb, stock_5min.duckdb, etf_5min.duckdb, index_5min.duckdb (external) options_features.parquet, realized_vol.parquet, merged_options_rv.parquet
02_rq1_return_predictability.py RQ1: pooled OLS (HC1) + Fama-MacBeth panel regressions merged_options_rv.parquet rq1_regression_results.csv, rq1_meta.csv
03_rq2_rv_forecasting.py RQ2: HAR-RV vs IV-surface models, rolling OOS, Diebold-Mariano merged_options_rv.parquet rq2_insample.csv, rq2_oos_results.csv, rq2_diebold_mariano.csv
04_rq3_correlation_divergence.py RQ3: implied vs realized correlation, stress prediction, crisis windows options.duckdb, stock_5min.duckdb, index_5min.duckdb, realized_vol.parquet correlation_divergence.parquet, rq3_stress_prediction.csv, rq3_crisis_analysis.csv
05_rq4_greeks_decay_magnets.py RQ4: Greeks info decay by DTE + max-OI price-magnet test options.duckdb, stock_5min.duckdb, realized_vol.parquet rq4_greeks_decay.csv, rq4_price_magnet.csv, price_magnet_data.parquet
06_rq5_ml_rv_forecast.py RQ5: RF/GBM on SPX surface vs VIX vs HAR-RV options.duckdb, index_5min.duckdb rq5_model_comparison.csv, rq5_feature_importance.csv
07_descriptive_stats.py Descriptive statistics (Panels A–H) merged_options_rv.parquet, index_5min.duckdb (VIX), options.duckdb (quality) 8 descriptive_*.csv
08_subperiod_regime.py Subperiods, VIX regimes, rolling R², pre/post COVID merged_options_rv.parquet, index_5min.duckdb (VIX) subperiod_results.csv, regime_results.csv, rolling_r2.csv, pre_post_covid.csv
09_portfolio_sorts.py Quintile sorts, double sorts, transaction-cost analysis merged_options_rv.parquet portfolio_sort_results.csv, double_sort_iv_skew.csv, transaction_cost_analysis.csv
10_robustness.py Newey-West, double-clustered SE, controls, quantile reg., ticker R² merged_options_rv.parquet, index_5min.duckdb (VIX, section E) robustness_newey_west.csv, robustness_double_clustered.csv, robustness_with_controls.csv, (robustness_quantile_regression.csv, robustness_ticker_r2.csvmissing, see §6.2)
11_granger_var.py Granger causality, bivariate VAR(5), IRF, FEVD merged_options_rv.parquet granger_causality.csv, var_results.csv, irf_results.csv, fevd_results.csv

# Code-quality observations (fixed in the refactor, without changing any computation)

  • Heavy duplication: winsorize() re-defined in 6 scripts; the OLS + HC1 t-stat block copy-pasted in 7 places; standardize-then-lstsq pattern repeated ~15 times.
  • Hard-coded ticker lists duplicated across scripts (stocks/ETFs/indices exclusion lists appear in 02, 07, 09 with identical content).
  • Paths built with Path(__file__).parent.parent.parent — the raw-database location is implicit and non-configurable.
  • warnings.filterwarnings('ignore') everywhere; bare except: clauses in several hot loops.
  • The p-value column in 02 (rq1_regression_results.csv) uses an ad-hoc normal-tail approximation (0.5·exp(−t²/2)·√(2/π)), not an exact two-sided normal/t p-value. Kept as-is (results preservation) but documented in the code.

# 3. Data

File Size Status
data/merged_options_rv.parquet 56 MB Derived (script 01). Master analysis panel: 264,383 ticker-days × 69 tickers, 2010–2025
data/options_features.parquet 24 MB Derived (script 01, intermediate)
data/realized_vol.parquet 49 MB Derived (script 01, intermediate; consumed by 04, 05)
data/correlation_divergence.parquet 351 KB Derived (script 04). Contains VIX close series as a column
data/price_magnet_data.parquet 319 KB Derived (script 05)

Raw data: the four raw DuckDB stores (options.duckdb ≈ 3.83 B rows, stock_5min.duckdb, etf_5min.duckdb, index_5min.duckdb) lived outside the project (two directories up) and no longer exist on this machine. Consequence: scripts 01, 04, 05, 06 and parts of 07, 08, 10 cannot be re-executed end-to-end. All derived datasets and result CSVs survive, so every parquet-only analysis remains reproducible (see §6.1).

# 4. Results (33 CSVs)

All 33 CSVs in results/ are outputs of the scripts above (mapping in §2). No orphan results. Two CSVs that 10_robustness.py is coded to write are absent (see §6.2).

# 5. LaTeX

  • wp7/current paper: main.tex (clean preamble, metadata macros) + sections/titlepage, introduction, literature, data, methodology, results, robustness, discussion, conclusion, references + appendix/appendix.tex + Makefile/.latexmkrc + uq_logo.jpg. Compiles with latexmk (last built 2026-06-13, main.pdf 300 KB). Bibliography is a manual thebibliography (37 entries; exactly matches the 37 cited keys — no missing/unused refs).
  • WP7_Options_Implied_Information_Content.tex (root, 1,066 lines) — older monolithic draft of the same paper (identical section structure). Superseded by wp7/. → kept only in _old/, treated as dead file.
  • Build artifacts (main.aux/.log/.out/.toc/.fls/.fdb_latexmk) — regenerable, not migrated.
  • .DS_Store, .claude/settings.local.json — noise/session config, not migrated.

# 6. Flags — items requiring the author's attention

# 6.1 Raw data gone → partial reproducibility (blocking for full pipeline)

The raw DuckDB stores are not on this machine. The pipeline is therefore reproducible from the processed parquets onward only. In the new repo, every raw-dependent script checks for the raw stores under a configurable WP7_RAW_DATA_DIR and exits with a clear message when absent. Re-runnable today: RQ1 (02), RQ2 (03), portfolio sorts (09), Granger/VAR (11), robustness A–D & F (10), descriptives A–E & G (07), subperiods A/C/D (08), and the regression stages of RQ3 (3E–3F, from correlation_divergence.parquet).

# 6.2 10_robustness.py — memory bug; two result files missing; paper cites them

  • Section D (quantile regression) builds np.diag(weights) on the full sample (~110k obs → a 110k×110k dense matrix ≈ 97 GB). The original run almost certainly died there (MemoryError), which explains why robustness_quantile_regression.csv and robustness_ticker_r2.csv (written later in the script) are missing from results/.
  • Yet sections/robustness.tex cites those results (quantile-regression significance; ticker-level R²: mean 5.9 %, median 4.7 %, IQR [2.0 %, 8.2 %]). These numbers had no surviving CSV.
  • In the refactor the weighting was rewritten with broadcasting (X * w[:,None]) — mathematically identical, memory-safe. Sections D and F were re-run to regenerate the two missing CSVs.
  • Verification outcome — DISCREPANCY (author review required):
    • Ticker-level R² (robustness.tex §4.4): the paper quotes mean 5.9 %, median 4.7 %, IQR [2.0 %, 8.2 %]. Regenerating section F from the surviving panel with the script's exact specification yields mean 14.0 %, median 13.2 %, IQR [10.0 %, 17.6 %] (69 tickers). The paper's figures cannot be reproduced from the surviving data; they may come from an earlier data vintage or a different specification. The paper text was left unchanged. (Qualitatively the claim survives either way — panel results are not driven by outliers.)
    • Quantile regressions (robustness.tex §4.4): the paper states the predictors "are significant across the return distribution", but the quantile-regression code computes coefficients only — no standard errors or t-statistics exist in any output. The regenerated coefficients are consistently signed across τ ∈ [0.10, 0.90] (implied kurtosis positive, PC volume ratio negative), which supports stability but not a significance claim. Paper text left unchanged; flagged for the author.
    • Rolling-window stats (robustness.tex §4.4): verified exact — paper (mean 8.0 %, std 4.5 %, range [2.1 %, 17.9 %] for 5-day returns; 41.2 %, std 10.7 % for 1-day RV) matches rolling_r2.csv to the digit.

# 6.3 Paper claims not fully backed by a surviving results file

  • Robustness §"VIX < 30" subsample (script 10, section E) prints to console only — no CSV was ever written. Not quoted numerically in the paper (prose only), so nothing to reconcile.
  • Diebold-Mariano: the paper quotes 4 tickers (AAPL, AMD, CAT, COST); rq2_diebold_mariano.csv contains the full set — verified consistent.

# 6.4 Minor inconsistencies (documentation-level, no action taken on results)

  • 04 computes a const_returns DuckDB query whose result is never used (dead code; realized correlations actually come from daily returns in realized_vol.parquet). Removed in refactor — no output touched.
  • Sample-size figures in the paper (e.g., N = 264,383) match merged_options_rv.parquet (verified by row count) and descriptive_by_group.csv.

# 7. Verification of reproduction (Phase 2c results)

Method: the refactored scripts were re-run with WP7_RESULTS_DIR pointed at a scratch directory (_verify/results/, kept in the repo for audit), then compared against the originals with _verify/compare_results.py (byte comparison first, then numeric with rtol=1e-9). Environment: Python 3.13 / numpy 2.4.4 / pandas 3.0.2 (Apple Silicon).

Outcome — 28 files compared, all conform; 1 explained deviation; 2 files newly generated:

Status Files
Byte-identical 10 (all descriptives A–E/G, portfolio sorts, double sort, transaction costs, rq1_meta, rq3_crisis, rq4_price_magnet)
Numerically equal (max rel. diff ≤ 2×10⁻¹⁰, i.e. floating-point noise) 15 (rq1 regressions, rq2 in-sample/OOS/DM, granger, VAR/IRF/FEVD, subperiods, rolling R², pre/post-COVID, NW, double-clustered, controls)
Deviation, explained rq3_stress_prediction.csv — only the t_stat column. Root cause: the design matrix contains an exact linear dependency (corr_divergence = implied_corr − realized_corr), so X'X is singular and HC1 t-stats are numerically degenerate. The economically meaningful corr_ratio t-stats move from (4.07, 8.30, 3.91) to (4.07, 8.22, 3.73) across BLAS/numpy versions — same significance everywhere; the near-zero t-stats (≈10⁻⁷) on the collinear regressors are pure noise. Original CSV kept as authoritative.
Newly generated (missing from original archive) robustness_quantile_regression.csv, robustness_ticker_r2.csv — see §6.2

Raw-dependent outputs (rq4_greeks_decay, rq5_*, descriptive_vix_regimes, descriptive_options_quality, regime_results) could not be re-run (§6.1); the original CSVs are shipped unchanged in results/.

No original CSV and no number in the paper was modified.