---
project: modelmap
document: Phase 1 notes — §4.8 Statistics and epistemology of interpretability
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
website: https://modelmap.io
created: 2026-08-12
status: draft
---

# §4.8 — Statistics and epistemology of interpretability

## B1. The BERT interpretability illusion (dataset sensitivity)
- **Key paper:** Bolukbasi et al. 2021, https://arxiv.org/abs/2104.07143 — the same neuron looks monosemantic on one dataset and encodes a different "concept" on another; top-activating-example analysis systematically over-reads.
- **Consequence (already in charter):** every unit-level claim needs ≥2 prompt sets; atlas entries report per-dataset agreement.
- **Status:** Established.

## B2. Subspace-patching illusions — Makelov vs Wu (unresolved)
- **Key papers:** Makelov, Lange, Nanda 2023, https://arxiv.org/abs/2311.17030 (subspace patching can activate **dormant parallel pathways** — interventional success ≠ correct localization) vs Wu et al. reply 2024, https://arxiv.org/abs/2401.12631 (the illusion criterion also rejects intuitively correct explanations).
- **Common ground:** interchange-intervention success alone under-determines mechanism. **Rule: Level-3 labels record which intervention family and which basis; DAS-style optimized subspaces get a distinct flag.**
- **Status:** Contested (productive open debate).

## B3. Illusions from simplified proxies (SVD, clustering, low-rank views)
- **Key paper:** Friedman et al., ICML 2024, https://arxiv.org/abs/2312.03656 — simplifications matching the model in-distribution diverge OOD; the proxy captures ID behavior, not mechanism.
- **Direct warning for expE:** SVD/effective-rank maps are hypotheses, not mechanisms — **every geometry map gets an OOD promptset check.**
- **Status:** Established.

## B4. Methodological sensitivity of activation patching
- **Key papers:** Zhang & Nanda, ICLR 2024, https://arxiv.org/abs/2309.16042 ; Heimersheim & Nanda 2024, https://arxiv.org/abs/2404.15255
- **Substance:** results flip with corruption type (Gaussian vs symmetric counterfactual), metric (prob vs logit-diff vs KL), window, position. ROME-style Gaussian corruption inflates mid-layer MLP localization. **The closest thing to a pre-registration standard: fix corruption + metric before scanning — methodology.md pins these per map type.**
- **Status:** Established.

## B5. Faithfulness vs plausibility
- **Key paper:** Jacovi & Goldberg, ACL 2020, https://arxiv.org/abs/2004.03685 — the vocabulary underlying modelmap's confidence taxonomy; plausibility (human appeal) must never borrow causal authority. Visualization design (§4.10) must respect this firewall.
- **Status:** Established (canonical).

## B6. Probing controls, selectivity
- **Key papers:** Hewitt & Liang, EMNLP 2019, https://arxiv.org/abs/1909.03368 ; Belinkov, CL 2022, https://arxiv.org/abs/2102.12452
- **Substance:** the mandatory-controls doctrine expA implements — shuffled-label nulls, capacity-matched probes, multi-dataset replication, selectivity over raw accuracy.
- **Status:** Established.

## B7. Seed sensitivity of SAEs — **the noise-floor number**
- **Key paper:** Paulo & Belrose 2025, https://arxiv.org/abs/2501.16615 — SAEs differing only in seed share as little as **~30% of features** (131k latents, Llama-3-8B); TopK more seed-dependent than ReLU+L1.
- **Consequence:** cross-seed overlap is the noise floor that cross-*model* universality claims must beat. Combined with 1–5% neuron universality (Gurnee), it makes modelmap's replication-rate metric first-class. **Replicating at 0.5–1B scale is feasible and would be a strong early result.**
- **Status:** Established (multi-LLM, multi-architecture).

## B8. Multiple comparisons — interpretability as statistical inference
- **Key sources:** *The Dead Salmons of AI Interpretability* (2025), https://arxiv.org/abs/2512.18792 — attribution, probing, SAEs and causal analyses yield plausible "explanations" of **randomly initialized networks**; reframe interpretability as statistical inference (uncertainty, explicit alternatives, multiplicity control). Historical anchor: Bennett et al. 2009, https://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
- **Consequence:** random-init baselines non-negotiable (charter §8.4); **add per-map FDR control to src/modelmap/stats.**
- **Status:** Emerging position; underlying statistics established.

## B9. Ground-truth benchmarks
- **Tracr** (compiled transformers, exact circuits): https://arxiv.org/abs/2301.05062 , code https://github.com/google-deepmind/tracr (JAX, CPU-runnable on Mac)
- **InterpBench** (17 semi-synthetic transformers via Strict IIT, NeurIPS 2024): https://arxiv.org/abs/2407.14494
- **RAVEL** (attribute disentanglement, intervention-based): https://arxiv.org/abs/2402.17700
- **MIB — Mechanistic Interpretability Benchmark** (ICML 2025): https://arxiv.org/abs/2504.13151 — two tracks (circuit localization; causal-variable localization), 4 tasks × 5 models; attribution/mask-optimization best for circuits; supervised DAS beats SAEs on causal variables; BlackboxNLP 2025 shared task https://arxiv.org/html/2510.06811v1
- **Consequence:** **validate every modelmap pipeline on Tracr/InterpBench first (tiny, CPU/MPS-feasible), report recovery rates, then apply to real models.** MIB pairs are within Mac budget.
- **Status:** Established, rapidly standardizing.

## B10. Hypothesis testing for circuits
- **Key papers:** Shi et al., NeurIPS 2024, https://arxiv.org/abs/2410.13032 (code https://github.com/blei-lab/circuitry) — formal equivalence/independence/minimality tests; published circuits pass only partially, Tracr circuits pass fully. Hanna et al. 2024, https://arxiv.org/abs/2403.17806 (faithfulness > overlap).
- **Consequence:** **run circuitry-style tests on every promoted circuit; report pass/fail per criterion in confidence.md — a ready-made harness for the causal confirmation rate.**
- **Status:** Established methodology.

## B11. Causal abstraction — the formal framework and its limit
- **Key papers:** Geiger et al., JMLR 2025, https://arxiv.org/abs/2301.04709 — unifies patching/scrubbing/circuits/SAEs/steering as causal abstraction; **Non-Linear Representation Dilemma (2025): https://arxiv.org/abs/2507.08802 — with arbitrary nonlinear featurizers, any model "abstracts" any algorithm (accuracy ~1 even for random networks); causal abstraction is only meaningful jointly with a constrained featurizer class.**
- **Consequence:** confidence.md records the featurizer class (linear / orthogonal / DAS-optimized) for every causal claim.
- **Status:** Framework established; sufficiency contested.

## B12. Calls for rigor (2023–2026)
- Saphra & Wiegreffe, *Mechanistic?* (BlackboxNLP 2024): https://arxiv.org/abs/2410.09087 — four senses of "mechanistic"; state which one is used (modelmap: the narrow causal one).
- Casper, *The Engineer's Interpretability Sequence* (2023): https://www.alignmentforum.org/s/a6ne2ve5uturEEQK7
- Sharkey et al., *Open Problems in Mechanistic Interpretability* (2025): https://arxiv.org/abs/2501.16496 — lists validation/method-evaluation gaps modelmap targets.
- *Mechanistic Interpretability Needs Philosophy* (2025): https://arxiv.org/abs/2506.18852 ; *Make Mechanistic Interpretability Auditable* (ACL 2026): https://arxiv.org/pdf/2606.00033 — auditability guidelines are close cousins of modelmap's provenance/confidence schema.
- **Status:** Converging consensus: ground-truth benchmarks + statistical controls + causal verification — exactly the charter's bar.

## Synthesis for modelmap
1. Noise floors first (expA): selectivity, ≥2 datasets, ≥3–5 seeds, random-init baselines, FDR control.
2. Pipeline validation on Tracr/InterpBench before any real-model map.
3. confidence.md schema additions: intervention family, basis/featurizer class, ablation-scheme curves, per-dataset agreement.
4. The 30%-SAE-seed-overlap and 1–5%-neuron-universality numbers are the published calibration points our replication-rate metric extends.