spb/modelmap Public License

1---2project: modelmap3document: Phase 1 notes — §4.8 Statistics and epistemology of interpretability4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128status: draft9---1011# §4.8 — Statistics and epistemology of interpretability1213## B1. The BERT interpretability illusion (dataset sensitivity)14- **Key paper:** Bolukbasi et al. 2021, https://arxiv.org/abs/2104.07143 — the same neuron looks monosemantic on one dataset and encodes a different "concept" on another; top-activating-example analysis systematically over-reads.15- **Consequence (already in charter):** every unit-level claim needs ≥2 prompt sets; atlas entries report per-dataset agreement.16- **Status:** Established.1718## B2. Subspace-patching illusions — Makelov vs Wu (unresolved)19- **Key papers:** Makelov, Lange, Nanda 2023, https://arxiv.org/abs/2311.17030 (subspace patching can activate **dormant parallel pathways** — interventional success ≠ correct localization) vs Wu et al. reply 2024, https://arxiv.org/abs/2401.12631 (the illusion criterion also rejects intuitively correct explanations).20- **Common ground:** interchange-intervention success alone under-determines mechanism. **Rule: Level-3 labels record which intervention family and which basis; DAS-style optimized subspaces get a distinct flag.**21- **Status:** Contested (productive open debate).2223## B3. Illusions from simplified proxies (SVD, clustering, low-rank views)24- **Key paper:** Friedman et al., ICML 2024, https://arxiv.org/abs/2312.03656 — simplifications matching the model in-distribution diverge OOD; the proxy captures ID behavior, not mechanism.25- **Direct warning for expE:** SVD/effective-rank maps are hypotheses, not mechanisms — **every geometry map gets an OOD promptset check.**26- **Status:** Established.2728## B4. Methodological sensitivity of activation patching29- **Key papers:** Zhang & Nanda, ICLR 2024, https://arxiv.org/abs/2309.16042 ; Heimersheim & Nanda 2024, https://arxiv.org/abs/2404.1525530- **Substance:** results flip with corruption type (Gaussian vs symmetric counterfactual), metric (prob vs logit-diff vs KL), window, position. ROME-style Gaussian corruption inflates mid-layer MLP localization. **The closest thing to a pre-registration standard: fix corruption + metric before scanning — methodology.md pins these per map type.**31- **Status:** Established.3233## B5. Faithfulness vs plausibility34- **Key paper:** Jacovi & Goldberg, ACL 2020, https://arxiv.org/abs/2004.03685 — the vocabulary underlying modelmap's confidence taxonomy; plausibility (human appeal) must never borrow causal authority. Visualization design (§4.10) must respect this firewall.35- **Status:** Established (canonical).3637## B6. Probing controls, selectivity38- **Key papers:** Hewitt & Liang, EMNLP 2019, https://arxiv.org/abs/1909.03368 ; Belinkov, CL 2022, https://arxiv.org/abs/2102.1245239- **Substance:** the mandatory-controls doctrine expA implements — shuffled-label nulls, capacity-matched probes, multi-dataset replication, selectivity over raw accuracy.40- **Status:** Established.4142## B7. Seed sensitivity of SAEs — **the noise-floor number**43- **Key paper:** Paulo & Belrose 2025, https://arxiv.org/abs/2501.16615 — SAEs differing only in seed share as little as **~30% of features** (131k latents, Llama-3-8B); TopK more seed-dependent than ReLU+L1.44- **Consequence:** cross-seed overlap is the noise floor that cross-*model* universality claims must beat. Combined with 1–5% neuron universality (Gurnee), it makes modelmap's replication-rate metric first-class. **Replicating at 0.5–1B scale is feasible and would be a strong early result.**45- **Status:** Established (multi-LLM, multi-architecture).4647## B8. Multiple comparisons — interpretability as statistical inference48- **Key sources:** *The Dead Salmons of AI Interpretability* (2025), https://arxiv.org/abs/2512.18792 — attribution, probing, SAEs and causal analyses yield plausible "explanations" of **randomly initialized networks**; reframe interpretability as statistical inference (uncertainty, explicit alternatives, multiplicity control). Historical anchor: Bennett et al. 2009, https://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf49- **Consequence:** random-init baselines non-negotiable (charter §8.4); **add per-map FDR control to src/modelmap/stats.**50- **Status:** Emerging position; underlying statistics established.5152## B9. Ground-truth benchmarks53- **Tracr** (compiled transformers, exact circuits): https://arxiv.org/abs/2301.05062 , code https://github.com/google-deepmind/tracr (JAX, CPU-runnable on Mac)54- **InterpBench** (17 semi-synthetic transformers via Strict IIT, NeurIPS 2024): https://arxiv.org/abs/2407.1449455- **RAVEL** (attribute disentanglement, intervention-based): https://arxiv.org/abs/2402.1770056- **MIB — Mechanistic Interpretability Benchmark** (ICML 2025): https://arxiv.org/abs/2504.13151 — two tracks (circuit localization; causal-variable localization), 4 tasks × 5 models; attribution/mask-optimization best for circuits; supervised DAS beats SAEs on causal variables; BlackboxNLP 2025 shared task https://arxiv.org/html/2510.06811v157- **Consequence:** **validate every modelmap pipeline on Tracr/InterpBench first (tiny, CPU/MPS-feasible), report recovery rates, then apply to real models.** MIB pairs are within Mac budget.58- **Status:** Established, rapidly standardizing.5960## B10. Hypothesis testing for circuits61- **Key papers:** Shi et al., NeurIPS 2024, https://arxiv.org/abs/2410.13032 (code https://github.com/blei-lab/circuitry) — formal equivalence/independence/minimality tests; published circuits pass only partially, Tracr circuits pass fully. Hanna et al. 2024, https://arxiv.org/abs/2403.17806 (faithfulness > overlap).62- **Consequence:** **run circuitry-style tests on every promoted circuit; report pass/fail per criterion in confidence.md — a ready-made harness for the causal confirmation rate.**63- **Status:** Established methodology.6465## B11. Causal abstraction — the formal framework and its limit66- **Key papers:** Geiger et al., JMLR 2025, https://arxiv.org/abs/2301.04709 — unifies patching/scrubbing/circuits/SAEs/steering as causal abstraction; **Non-Linear Representation Dilemma (2025): https://arxiv.org/abs/2507.08802 — with arbitrary nonlinear featurizers, any model "abstracts" any algorithm (accuracy ~1 even for random networks); causal abstraction is only meaningful jointly with a constrained featurizer class.**67- **Consequence:** confidence.md records the featurizer class (linear / orthogonal / DAS-optimized) for every causal claim.68- **Status:** Framework established; sufficiency contested.6970## B12. Calls for rigor (2023–2026)71- Saphra & Wiegreffe, *Mechanistic?* (BlackboxNLP 2024): https://arxiv.org/abs/2410.09087 — four senses of "mechanistic"; state which one is used (modelmap: the narrow causal one).72- Casper, *The Engineer's Interpretability Sequence* (2023): https://www.alignmentforum.org/s/a6ne2ve5uturEEQK773- Sharkey et al., *Open Problems in Mechanistic Interpretability* (2025): https://arxiv.org/abs/2501.16496 — lists validation/method-evaluation gaps modelmap targets.74- *Mechanistic Interpretability Needs Philosophy* (2025): https://arxiv.org/abs/2506.18852 ; *Make Mechanistic Interpretability Auditable* (ACL 2026): https://arxiv.org/pdf/2606.00033 — auditability guidelines are close cousins of modelmap's provenance/confidence schema.75- **Status:** Converging consensus: ground-truth benchmarks + statistical controls + causal verification — exactly the charter's bar.7677## Synthesis for modelmap781. Noise floors first (expA): selectivity, ≥2 datasets, ≥3–5 seeds, random-init baselines, FDR control.792. Pipeline validation on Tracr/InterpBench before any real-model map.803. confidence.md schema additions: intervention family, basis/featurizer class, ablation-scheme curves, per-dataset agreement.814. The 30%-SAE-seed-overlap and 1–5%-neuron-universality numbers are the published calibration points our replication-rate metric extends.82