spb/modelmap Public License

1---2project: modelmap3document: Phase 1 notes — §4.2 Causal / interventional methods4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128status: draft9---1011# §4.2 — Causal / interventional methods1213## B1. Activation patching (causal mediation / interchange intervention)14- **Key papers:** origins: Vig et al., NeurIPS 2020, https://arxiv.org/abs/2004.12265 ; best practices: Zhang & Nanda, ICLR 2024, https://arxiv.org/abs/2309.16042 ; Heimersheim & Nanda 2024, https://arxiv.org/pdf/2404.1525515- **Reveals:** which components causally mediate a behavior (clean/corrupted runs; noising vs denoising directions differ in meaning).16- **Cost:** 2 forwards per (component, prompt-pair); exhaustive sweeps O(#components). Time-heavy, memory-light — well-suited to Macs 0.5B–8B; 14B with patience/quantization. Hook overhead MLX vs MPS = Experiment H.17- **Implementations:** TransformerLens (MPS opt-in), NNsight https://github.com/ndif-team/nnsight , pyvene, baukit.18- **Failure modes:** metric (logit-diff vs prob) and corruption type (Gaussian vs symmetric token swap) change localization conclusions; backup/hydra compensation; dormant-pathway subspace illusions; multi-mediator interactions (*Curse of Multiple Mediators*, 2026, https://arxiv.org/html/2606.27510v1).19- **Epistemic status:** Established core method; hyperparameter sensitivity well-documented. **methodology.md must pin corruption method + metric per map type, before scanning.**2021## B2. Path patching22- **Key papers:** Wang et al. (IOI), ICLR 2023, https://arxiv.org/abs/2211.00593 ; Goldowsky-Dill et al. 2023, https://arxiv.org/abs/2304.05969 ; accelerated: APP 2025, https://arxiv.org/html/2511.0544223- **Reveals:** causal importance of specific *edges/paths*, not just nodes.24- **Cost:** combinatorially heavier than node patching. Practical ≤3B–7B on Mac; 14B painful.25- **Failure modes:** metric sensitivity + hypothesis-space explosion; faithfulness not guaranteed (Miller et al. 2024, https://arxiv.org/pdf/2407.08734).26- **Epistemic status:** Established; IOI replicated (even in Mamba, https://arxiv.org/html/2407.14008v1) but faithfulness swings with ablation details.2728## B3. Attribution patching (AtP/AtP*) and gradient-based screening29- **Key papers:** Nanda 2023, https://www.neelnanda.io/mechanistic-interpretability/attribution-patching ; Syed, Rager, Conmy, BlackboxNLP 2024, https://aclanthology.org/2024.blackboxnlp-1.25.pdf ; AtP*: Kramár et al. 2024, https://arxiv.org/abs/2403.00745 ; EAP-IG: Hanna et al., COLM 2024, https://arxiv.org/abs/2403.1780630- **Reveals:** first-order approximation of every patch effect in ~2 forwards + 1 backward; AtP* fixes false negatives (QK softmax linearization + GradDrop).31- **Cost:** cheapest per-component causal estimate, but backward ≈ 2–3× weight memory: ≤3B fp16 on 16–32 GB, ≤7–8B on 64 GB. **Gradients through 4-bit quantized models are awkward — a real Mac constraint. MLX autodiff is the native path worth benchmarking (Experiment H).**32- **Failure modes:** bad approximation for large activations (residual, MLP0, saturated attention); systematic false negatives (quantified in AtP*); second-order failures (*When Attribution Patching Lies*, 2026, https://arxiv.org/pdf/2606.09899); LRP alternative RelP (2025, https://arxiv.org/html/2508.21258v1).33- **Epistemic status:** Established as *screening*; never a substitute for real patching on final claims.3435## B4. Causal tracing / ROME-style locating36- **Key papers:** Meng et al., NeurIPS 2022, https://arxiv.org/abs/2202.05262 (code https://rome.baulab.info/) ; **critique:** Hase et al., NeurIPS 2023, https://arxiv.org/abs/2301.0421337- **Reveals:** where factual associations appear "stored" (noise subject tokens, restore states; mid-layer MLPs at last subject token).38- **THE central cautionary tale for modelmap:** Hase et al. — edit success essentially uncorrelated with tracing-identified layers (ρ≈−0.13 at GPT-J layer 6); editing works almost anywhere; **localization ≠ editability**. Gaussian noising also distorts activations off-distribution.39- **Epistemic status:** Contested / debunked-in-part (the locating→editing inference); the tracing observation itself replicates.4041## B5. Ablations (zero, mean, resample)42- **Key analyses:** Zhang & Nanda (B1); Miller et al. 2024 (ablation choice flips faithfulness scores).43- **Reveals:** necessity (not sufficiency) of a component.44- **Cost:** cheapest intervention class; 1 forward per ablation. **Ideal for Experiment B (localization vs diffusion).**45- **Failure modes:** zero-ablation is off-distribution (can create signal); mean-ablation destroys within-distribution variance; **resample > mean > zero** is the emerging norm; hydra effects mask importance; unit scans → multiple-comparison inflation.46- **Epistemic status:** Established, with distribution-dependence caveats.4748## B6. Causal scrubbing49- **Key sources:** Chan et al. (Redwood) 2022, https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing ; pitfalls: https://www.alignmentforum.org/posts/DFarDnQjMnjsKvW8s/practical-pitfalls-of-causal-scrubbing50- **Reveals:** quantitative faithfulness score for a *full* interpretability hypothesis via all claimed-behavior-preserving resample ablations.51- **Cost:** expensive; fine ≤1B on Mac, research-grade beyond.52- **Note:** Redwood's rust_circuit is archived; **no maintained macOS-friendly implementation exists — a genuine gap.**53- **Epistemic status:** Established conceptually, little-used; succeeded in spirit by circuit-faithfulness metrics.5455## B7. DAS / interchange intervention accuracy / pyvene56- **Key papers:** Geiger et al., CLeaR 2024, https://arxiv.org/abs/2303.02536 ; Boundless DAS: Wu et al., NeurIPS 2023, https://arxiv.org/abs/2305.08809 ; pyvene: https://arxiv.org/abs/2403.07809 , https://github.com/stanfordnlp/pyvene57- **Reveals:** whether a high-level causal model is implemented in learned subspaces (train a rotation; IIA = fraction of counterfactuals matched).58- **Cost:** moderate–high; Boundless DAS ran on 7B — Mac-feasible ≤7B (gradients only through rotation + partial graph).59- **The headline dispute:** Makelov, Lange, Nanda 2023, https://arxiv.org/abs/2311.17030 (trained subspaces can work via *dormant causally-disconnected pathways*) vs Wu, Geiger et al. reply 2024, https://www.alphaxiv.org/abs/2401.12631 . Genuinely unresolved — **trained-subspace claims need convergent evidence and a distinct flag in confidence.md**.60- **Epistemic status:** Framework established; subspace-illusion question contested. RAVEL provides standardized evaluation.6162## B8. Steering vectors (ActAdd, CAA, ITI)63- **Key papers:** ActAdd: Turner et al. 2023, https://arxiv.org/abs/2308.10248 ; CAA: Rimsky et al., ACL 2024, https://arxiv.org/abs/2312.06681 ; ITI: Li et al., NeurIPS 2023, https://arxiv.org/pdf/2306.0334164- **Reveals:** behaviors as linear residual-stream directions — sufficiency-style causal evidence.65- **Cost:** very cheap; among the most Mac-friendly causal techniques. **repeng + llama.cpp applies control vectors to GGUF quantized models.**66- **Critiques (the 2024–2026 evaluation wave):** Tan et al., NeurIPS 2024, https://arxiv.org/abs/2407.12404 (steerability input-heterogeneous, sometimes anti-steers); *Steering off Course*, ACL 2025, https://aclanthology.org/2025.acl-long.974/ (fails to transfer across 36 models); AxBench 2025, https://arxiv.org/abs/2501.17148 (prompting/finetuning beat representation steering); non-identifiability (2026, https://arxiv.org/pdf/2602.06801); geometric predictors of failure (2026, https://arxiv.org/pdf/2602.17881).67- **Epistemic status:** Steering *can* work — established; reliability contested. **Always report against prompting baselines + random-direction controls. Opportunity: publish steerability maps *with variance*.**6869## Cross-cutting 2024–2026 signals701. Faithfulness (not circuit overlap) became the standard comparison metric (EAP-IG) — then shown non-robust itself (Miller et al.).712. Patchscopes unifies lens-style readouts with interventions.723. Field syntheses: *Open Problems in Mechanistic Interpretability* (Sharkey et al. 2025, https://arxiv.org/abs/2501.16496); Bereska & Gavves review 2024, https://arxiv.org/pdf/2404.14082 ; *Dead Salmons* (2025, https://arxiv.org/pdf/2512.18792) — null-model controls, aligned with charter §8.4.734. Tooling: NNsight/NDIF paper https://arxiv.org/abs/2407.14561 benchmarks NNsight vs TransformerLens/pyvene/baukit. TransformerLens MPS remains opt-in. **Plan: baukit-style raw hooks on MPS + native MLX hooks, benchmark both (Experiment H). No mainstream interpretability library targets MLX — publishable gap.**74