---
project: anomaly-atlas
document: Phase 1 notes — multiple testing, data snooping, backtest overfitting
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
data_source: hfmarketdata.io
created: 2026-08-12
status: reviewed
---

# The statistics of not fooling yourself (§4.2 — most important)

*Phase 1 literature notes. Citations verified via OpenAlex, accessed
2026-08-12. This literature IS our methodology (Phase 9): every technique
below maps to a concrete step in expF.*

## 1. The formal machinery

- White, H. (2000). A Reality Check for Data Snooping. *Econometrica* 68(5), 1097–1126. https://doi.org/10.1111/1468-0262.00152 — Bootstrap test of "does the BEST rule in my searched universe beat the benchmark?", correcting for the search itself. **The core of expF.**
- Hansen, P. R. (2005). A Test for Superior Predictive Ability. *Journal of Business & Economic Statistics* 23(4), 365–380. https://doi.org/10.1198/073500105000000063 — SPA: studentized, less sensitive to irrelevant alternatives in the universe than White RC. Preferred variant.
- Romano, J. P. & Wolf, M. (2005). Stepwise Multiple Testing as Formalized Data Snooping. *Econometrica* 73(4), 1237–1282. https://doi.org/10.1111/j.1468-0262.2005.00615.x — StepM: identifies *which* rules beat the benchmark with FWE control — not just whether the best one does.
- Benjamini, Y. & Hochberg, Y. (1995). Controlling the False Discovery Rate. *JRSS-B* 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x — FDR: the right error rate for *scans* (expC–E), where we tolerate a known fraction of false leads into the next stage.
- Bailey, D. H. & López de Prado, M. (2014). The Deflated Sharpe Ratio. *Journal of Portfolio Management* 40(5), 94–107. https://doi.org/10.3905/jpm.2014.40.5.094 — DSR: deflate the best Sharpe by the number of trials + non-normality. Requires *honest trial counting* — our LOG's hypothesis budget is exactly that input.
- Bailey, Borwein, López de Prado & Zhu (2014). Pseudo-Mathematics and Financial Charlatanism. *Notices of the AMS* 61(5), 458–471. https://doi.org/10.1090/noti1105 — With enough configurations, a stellar in-sample backtest is *guaranteed*. The quantitative case for pre-registration.
- Bailey, Borwein, López de Prado & Zhu (2016). The probability of backtest overfitting. *Journal of Computational Finance* 20(4), 39–69. https://doi.org/10.21314/jcf.2016.322 — PBO via combinatorially symmetric cross-validation (CSCV) — the basis of CPCV; candidate for expH's split design.

## 2. Applied to finance's own literature (the base rates)

- Harvey, C. R., Liu, Y. & Zhu, H. (2016). …and the Cross-Section of Expected Returns. *Review of Financial Studies* 29(1), 5–68. https://doi.org/10.1093/rfs/hhv059 — 300+ published factors; with multiple testing, a new "discovery" needs **t > 3.0**.
- Harvey, C. R. (2017). Presidential Address: The Scientific Outlook in Financial Economics. *Journal of Finance* 72(4), 1399–1440. https://doi.org/10.1111/jofi.12530 — p-hacking incentives; minimum Bayes factors; Bayesianized p-values.
- Chordia, T., Goyal, A. & Saretto, A. (2020). Anomalies and False Rejections. *Review of Financial Studies* 33(5), 2134–2179. https://doi.org/10.1093/rfs/hhaa018 — 2M+ simulated strategies → multiple-testing-safe hurdles **t ≈ 3.4–3.8**; at classic thresholds ~45 % of "anomalies" are false.
- Harvey, C. R. & Liu, Y. (2020). False (and Missed) Discoveries in Financial Economics. *Journal of Finance* 75(5), 2503–2553. https://doi.org/10.1111/jofi.12951 — double bootstrap to pick t-hurdles for a target FDR, balancing Type I *and* Type II.
- Giglio, S., Liao, Y. & Xiu, D. (2021). Thousands of Alpha Tests. *Review of Financial Studies* 34(7), 3456–3496. https://doi.org/10.1093/rfs/hhaa111 — alpha multiple-testing robust to omitted factors/missing data, wild bootstrap.
- Hou, K., Xue & Zhang (2020). Replicating Anomalies. *Review of Financial Studies* 33(5), 2019–2133. https://doi.org/10.1093/rfs/hhy131 — 452 anomalies replicated with microcap-robust methods: **~65 % fail** at |t|≥1.96.
- Jensen, T. I., Kelly, B. & Pedersen, L. H. (2023). Is There a Replication Crisis in Finance? *Journal of Finance* 78(5), 2465–2518. https://doi.org/10.1111/jofi.13249 — the counterpoint: Bayesian hierarchical pooling says most factor *themes* replicate, incl. in 93 countries. Read together with Hou et al.: the answer depends on the unit of analysis (theme vs individual signal) and the prior.
- Sullivan, R., Timmermann, A. & White, H. (2001). Dangers of data mining: the case of calendar effects in stock returns. *Journal of Econometrics* 105(1), 249–286. https://doi.org/10.1016/s0304-4076(01)00077-x — the full calendar-rule universe under the Reality Check: **calendar effects largely vanish**. The single most important prior for expE.

## 3. Design consequences for anomaly-atlas

1. **Error-rate ladder**: FDR (BH) for the Level-0 scans → SPA/StepM against
   the artifact-null benchmark for Level-1 promotion → DSR with the logged
   trial count for anything Sharpe-like.
2. **t-hurdle**: adopt t ≥ 3 (Harvey–Liu–Zhu) as the *floor*, and report the
   Chordia–Goyal–Saretto 3.4–3.8 band alongside.
3. **Trial counting is a first-class artifact**: the LOG's hypothesis budget
   (charter §12) is the DSR/PBO input; silent hypothesis-space expansion
   invalidates the correction — hence append-only logging.
4. **expF deliverable**: the survival curve (naive → FDR → SPA → DSR →
   costs → OOS) is finding E of the charter, whatever survives.