SPB Git

spb/anomaly-atlas Public License

Systematic discovery & rigorous validation of statistical anomalies in open HF market data (hfmarketdata.io) — pre-registered, artifact-null-driven, fully reproducible. Live atlas: www.anomaly-atlas.io

quant-researchmarket-microstructureanomaliesreproducibilityduckdbapple-silicon
Python 61.4% JavaScript 28.7% CSS 8.6% Shell 0.7% Makefile 0.5%
5.5 KB
View source Raw Blame History

# project: anomaly-atlas document: Phase 1 notes — multiple testing, data snooping, backtest overfitting author: Simon-Pierre Boucher contact: contact@spboucher.ai data_source: hfmarketdata.io created: 2026-08-12 status: reviewed

# The statistics of not fooling yourself (§4.2 — most important)

Phase 1 literature notes. Citations verified via OpenAlex, accessed 2026-08-12. This literature IS our methodology (Phase 9): every technique below maps to a concrete step in expF.

# 1. The formal machinery

  • White, H. (2000). A Reality Check for Data Snooping. Econometrica 68(5), 1097–1126. https://doi.org/10.1111/1468-0262.00152 — Bootstrap test of "does the BEST rule in my searched universe beat the benchmark?", correcting for the search itself. The core of expF.
  • Hansen, P. R. (2005). A Test for Superior Predictive Ability. Journal of Business & Economic Statistics 23(4), 365–380. https://doi.org/10.1198/073500105000000063 — SPA: studentized, less sensitive to irrelevant alternatives in the universe than White RC. Preferred variant.
  • Romano, J. P. & Wolf, M. (2005). Stepwise Multiple Testing as Formalized Data Snooping. Econometrica 73(4), 1237–1282. https://doi.org/10.1111/j.1468-0262.2005.00615.x — StepM: identifies which rules beat the benchmark with FWE control — not just whether the best one does.
  • Benjamini, Y. & Hochberg, Y. (1995). Controlling the False Discovery Rate. JRSS-B 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x — FDR: the right error rate for scans (expC–E), where we tolerate a known fraction of false leads into the next stage.
  • Bailey, D. H. & López de Prado, M. (2014). The Deflated Sharpe Ratio. Journal of Portfolio Management 40(5), 94–107. https://doi.org/10.3905/jpm.2014.40.5.094 — DSR: deflate the best Sharpe by the number of trials + non-normality. Requires honest trial counting — our LOG's hypothesis budget is exactly that input.
  • Bailey, Borwein, López de Prado & Zhu (2014). Pseudo-Mathematics and Financial Charlatanism. Notices of the AMS 61(5), 458–471. https://doi.org/10.1090/noti1105 — With enough configurations, a stellar in-sample backtest is guaranteed. The quantitative case for pre-registration.
  • Bailey, Borwein, López de Prado & Zhu (2016). The probability of backtest overfitting. Journal of Computational Finance 20(4), 39–69. https://doi.org/10.21314/jcf.2016.322 — PBO via combinatorially symmetric cross-validation (CSCV) — the basis of CPCV; candidate for expH's split design.

# 2. Applied to finance's own literature (the base rates)

  • Harvey, C. R., Liu, Y. & Zhu, H. (2016). …and the Cross-Section of Expected Returns. Review of Financial Studies 29(1), 5–68. https://doi.org/10.1093/rfs/hhv059 — 300+ published factors; with multiple testing, a new "discovery" needs t > 3.0.
  • Harvey, C. R. (2017). Presidential Address: The Scientific Outlook in Financial Economics. Journal of Finance 72(4), 1399–1440. https://doi.org/10.1111/jofi.12530 — p-hacking incentives; minimum Bayes factors; Bayesianized p-values.
  • Chordia, T., Goyal, A. & Saretto, A. (2020). Anomalies and False Rejections. Review of Financial Studies 33(5), 2134–2179. https://doi.org/10.1093/rfs/hhaa018 — 2M+ simulated strategies → multiple-testing-safe hurdles t ≈ 3.4–3.8; at classic thresholds ~45 % of "anomalies" are false.
  • Harvey, C. R. & Liu, Y. (2020). False (and Missed) Discoveries in Financial Economics. Journal of Finance 75(5), 2503–2553. https://doi.org/10.1111/jofi.12951 — double bootstrap to pick t-hurdles for a target FDR, balancing Type I and Type II.
  • Giglio, S., Liao, Y. & Xiu, D. (2021). Thousands of Alpha Tests. Review of Financial Studies 34(7), 3456–3496. https://doi.org/10.1093/rfs/hhaa111 — alpha multiple-testing robust to omitted factors/missing data, wild bootstrap.
  • Hou, K., Xue & Zhang (2020). Replicating Anomalies. Review of Financial Studies 33(5), 2019–2133. https://doi.org/10.1093/rfs/hhy131 — 452 anomalies replicated with microcap-robust methods: ~65 % fail at |t|≥1.96.
  • Jensen, T. I., Kelly, B. & Pedersen, L. H. (2023). Is There a Replication Crisis in Finance? Journal of Finance 78(5), 2465–2518. https://doi.org/10.1111/jofi.13249 — the counterpoint: Bayesian hierarchical pooling says most factor themes replicate, incl. in 93 countries. Read together with Hou et al.: the answer depends on the unit of analysis (theme vs individual signal) and the prior.
  • Sullivan, R., Timmermann, A. & White, H. (2001). Dangers of data mining: the case of calendar effects in stock returns. Journal of Econometrics 105(1), 249–286. https://doi.org/10.1016/s0304-4076(01)00077-x — the full calendar-rule universe under the Reality Check: calendar effects largely vanish. The single most important prior for expE.

# 3. Design consequences for anomaly-atlas

  1. Error-rate ladder: FDR (BH) for the Level-0 scans → SPA/StepM against the artifact-null benchmark for Level-1 promotion → DSR with the logged trial count for anything Sharpe-like.
  2. t-hurdle: adopt t ≥ 3 (Harvey–Liu–Zhu) as the floor, and report the Chordia–Goyal–Saretto 3.4–3.8 band alongside.
  3. Trial counting is a first-class artifact: the LOG's hypothesis budget (charter §12) is the DSR/PBO input; silent hypothesis-space expansion invalidates the correction — hence append-only logging.
  4. expF deliverable: the survival curve (naive → FDR → SPA → DSR → costs → OOS) is finding E of the charter, whatever survives.