spb/modelmap Public License

1---2project: modelmap3document: Phase 1 notes — §4.4 Circuits and computational graphs4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128status: draft9---1011# §4.4 — Circuits and computational graphs1213Cost classes: **L** (laptop-trivial), **M** (hours + tens of GB), **H** (multi-day / >64 GB / cluster-typical).1415## B1. Induction heads16- **Key paper:** Olsson et al. (Anthropic) 2022, https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html17- **Claims:** Two-head circuit (previous-token + induction head) implements [A][B]…[A]→[B] copying; forms in a phase change coincident with in-context learning.18- **Type:** Both — pattern-based detection plus ablations ("medium-confidence" causal per the paper itself).19- **Cost:** L. Prefix-matching scores on any model that fits memory; **classic first map layer for modelmap** (cheap, replicable, cross-model).20- **Implementation:** TransformerLens https://github.com/TransformerLensOrg/TransformerLens (MPS works, occasional dtype/op fallbacks); trivially portable to MLX hooks.21- **Failure modes:** "Induction head" is behaviorally defined — heads heterogeneous and multifunctional; ICL attribution contested for large models.22- **Epistemic status:** Established (most-replicated finding in mech interp).2324## B2. IOI circuit and attention-head taxonomies25- **Key papers:** IOI: Wang et al. 2022, https://arxiv.org/abs/2211.00593 ; Copy suppression: McDougall et al. 2023, https://arxiv.org/abs/2310.04625 ; Successor heads: Gould et al. 2023, https://arxiv.org/abs/2312.0923026- **Claims:** 26-head circuit computes indirect-object identification in GPT-2-small; established faithfulness/completeness/minimality criteria. Copy suppression explains "negative heads" and self-repair. Successor heads recur across families 31M–12B on shared "mod-10" numeric features.27- **Type:** Causal (path patching, ablations) + observational weight analysis.28- **Cost:** L–M. GPT-2-small path patching laptop-trivial; head-taxonomy scans across a 7B model are M on 32–64 GB.29- **Failure modes:** Backup heads / self-repair make ablation-based importance unstable (**Hydra effect** — McGrath et al. 2023, https://arxiv.org/abs/2307.15771); IOI only ~87% faithful, node-level; taxonomy labels are task-distribution-relative.30- **Epistemic status:** Established (IOI heavily replicated cross-model); taxonomy generality contested at margins.3132## B3. MLP layers as key–value memories33- **Key papers:** Geva et al. 2021, https://arxiv.org/abs/2012.14913 ; Geva et al. 2022, https://arxiv.org/abs/2203.1468034- **Claims:** MLP input weights = pattern detectors (keys), output weights = vocabulary-space updates (values); predictions built by additive sub-updates.35- **Cost:** L — weight-only plus light forward passes; **pairs naturally with Experiment E (weight-space cartography) and logit lens**.36- **Implementation:** simple to reimplement; baukit https://github.com/davidbau/baukit for hooks. MPS/MLX trivial.37- **Failure modes:** Vocabulary projection misleads in early/mid layers; neuron-level keys polysemantic; "memory" metaphor overclaims vs editing critiques (Hase et al. 2023, https://arxiv.org/abs/2301.04213).38- **Epistemic status:** Established as a lens; literal KV-memory reading contested.3940## B4. Activation patching / attribution patching41- **Key papers:** causal tracing: Meng et al. 2022, https://arxiv.org/abs/2202.05262 ; attribution patching: Nanda 2023, https://www.neelnanda.io/mechanistic-interpretability/attribution-patching ; best practices: Heimersheim & Nanda 2024, https://arxiv.org/abs/2404.1525542- **Claims:** Swapping activations between clean/corrupted runs localizes causally necessary components; attribution patching approximates all components in 2 forward + 1 backward pass.43- **Cost:** Patching scans M at 7B (thousands of forwards); attribution patching L–M — **the key trick making circuit scans Mac-feasible**, but needs gradients (MLX autodiff or PyTorch-MPS backward; memory ≈ 2–3× inference).44- **Implementations:** TransformerLens, nnsight https://github.com/ndif-team/nnsight , pyvene.45- **Failure modes:** Corrupted-baseline choice changes conclusions (noising vs denoising, zero vs mean vs resample); gradient approximation fails near zero-gradient (fixed by IG); localization ≠ editability.46- **Epistemic status:** Established core method; interpretation pitfalls well-documented.4748## B5. Attribution graphs / circuit tracing (Anthropic 2025)49- **Key papers:** Ameisen, Lindsey et al., *Circuit Tracing*, 2025, https://transformer-circuits.pub/2025/attribution-graphs/methods.html ; Lindsey et al., *On the Biology of a Large Language Model*, 2025, https://transformer-circuits.pub/2025/attribution-graphs/biology.html ; QK follow-up: https://transformer-circuits.pub/2025/attention-qk/index.html50- **Claims:** Replace MLPs with cross-layer transcoders → per-prompt linear attribution graph (nodes = features/errors/tokens, edges = direct causal contributions). Revealed planning-in-poetry, multilingual shared circuits, multi-step reasoning, unfaithful CoT, refusal mechanics.51- **Type:** Hybrid — graph construction is attributional through a lossy proxy model; hypotheses validated with steering/suppression interventions (the modelmap Level-2→Level-3 pattern).52- **Cost:** With pretrained CLTs (Gemma-2-2B, Llama-3.2-1B, Qwen3-4B): M — community reports ~15–40 GB for Gemma-2-2B graphs → 64 GB Mac plausible, 16 GB not. Training own CLTs for 7B: H.53- **Implementations:** circuit-tracer https://github.com/safety-research/circuit-tracer (announcement: https://www.anthropic.com/research/open-source-circuit-tracing); Neuronpedia hosts interactive graphs; frontend https://github.com/anthropics/attribution-graphs-frontend . CUDA-oriented; **MPS port is a concrete modelmap engineering task**.54- **Failure modes:** Error nodes are unexplained dark matter; attention patterns frozen (QK side has substantial interference); replacement-model faithfulness varies by prompt; per-prompt graphs need aggregation to become "maps"; many graphs unreadable.55- **Epistemic status:** Established methodology with candid limitation reporting; individual findings Level 2–3.5657## B6. Automated circuit discovery: ACDC → EAP → EAP-IG → edge pruning58- **Key papers:** ACDC: Conmy et al., NeurIPS 2023, https://arxiv.org/abs/2304.14997 ; EAP: Syed, Rager, Conmy 2023, https://arxiv.org/abs/2310.10348 ; EAP-IG: Hanna et al., COLM 2024, https://arxiv.org/abs/2403.17806 ; Edge pruning: Bhaskar et al., NeurIPS 2024, https://arxiv.org/abs/2406.1677859- **Claims:** ACDC: greedy edge removal recovers known circuits. EAP: gradient attribution scores all edges in ~3 passes. EAP-IG: integrated gradients fix zero-gradient failure; circuits must be selected by *faithfulness*, not node-overlap. Edge pruning: continuous masks; half-size circuits at equal faithfulness; scaled to CodeLlama-13B.60- **Cost:** ACDC M–H (slow beyond GPT-2). **EAP/EAP-IG: L–M — the Mac-feasible default for 0.5B–8B.** Edge pruning M–H (~1 GPU-day class; marginal on 64 GB Mac, feasible on M3U96a-class nodes).61- **Implementations:** https://github.com/ArthurConmy/Automatic-Circuit-Discovery ; https://github.com/hannamw/EAP-IG ; https://github.com/princeton-nlp/Edge-Pruning . PyTorch; MPS-workable (edge pruning assumes CUDA-scale memory).62- **Epistemic status:** Established tooling; "which method is right" contested — see B7.6364## B7. Faithfulness metrics for circuits — and their critiques65- **Key papers:** Miller, Chughtai, Saunders 2024, https://arxiv.org/abs/2407.08734 ; Shi et al. 2024, https://arxiv.org/abs/2410.13032 ; subspace-patching illusion: Makelov, Lange, Nanda 2023, https://arxiv.org/abs/2311.1703066- **Substance:** Measured faithfulness swings wildly with ablation type (zero/mean/resample), direction, token positions, and metric (logit diff vs KL) — the same circuit can look faithful or unfaithful by experimenter choice. Shi et al. formalize equivalence/independence/minimality tests; published circuits pass only partially. Subspace patching can activate *dormant* pathways → illusory localization.67- **Cost:** L–M at ≤2B — running a battery of ablation variants is exactly the charter §8.4 multi-control discipline.68- **Implementations:** auto-circuit https://github.com/UFO-101/auto-circuit (efficient multi-ablation); https://github.com/blei-lab/circuitry69- **Consequence for modelmap:** **report faithfulness curves across ablation schemes, never single numbers.**70- **Epistemic status:** Critiques established; no consensus null for "faithful enough".7172## B8. Sparse feature circuits73- **Key paper:** Marks et al., ICLR 2025, https://arxiv.org/abs/2403.1964774- **Claims:** Circuits over SAE features are far more interpretable than head/neuron circuits; SHIFT removes spurious signals; unsupervised discovery at scale.75- **Type:** Causal — IG attribution patching validated by ablation.76- **Cost:** M on Pythia-70M/Gemma-2-2B with public SAEs (Mac-feasible); needs SAEs for every stream, so capture cost multiplies.77- **Implementation:** https://github.com/saprmarks/feature-circuits (dictionary_learning + nnsight). MPS-workable.78- **Failure modes:** Inherits SAE pathologies — circuits over non-replicating features inherit their fragility.79- **Epistemic status:** Established method, actively extended; robustness contested.8081## B9. Weight-sparse transformers (OpenAI 2025) — circuits by construction82- **Key paper:** Gao, Rajaram et al. 2025, https://arxiv.org/abs/2511.13653 (toolkit: https://github.com/openai/circuit_sparsity)83- **Claims:** Training at ~1/1000 weight density yields compact, human-legible, causally-verifiable task circuits.84- **Relevance:** applies only to models trained this way — useful to modelmap as a **planted-ground-truth benchmark** for validating discovery methods, not for mapping existing checkpoints.85- **Epistemic status:** Established for tiny models; scaling unresolved.8687## 2024–2026 developments88- Mar 2025: attribution-graph diptych reorients field from head-level to feature-level circuits; May 2025 circuit-tracer open-sourced; Aug 2025 Neuronpedia "Circuits Research Landscape" https://www.neuronpedia.org/graph/info89- Nov–Dec 2025: OpenAI weight-sparse circuits; 2026 follow-up on interpretable individual parameters (https://arxiv.org/abs/2607.02964)90- 2026: ADAG — automated attribution-graph description (https://arxiv.org/abs/2604.07615); Circuit Insights — interpretability beyond activations toward weights (https://arxiv.org/abs/2510.14936); Anthropic Circuits Updates May/June 2026 apply features+graphs to safety auditing (https://transformer-circuits.pub/2026/june-update/index.html)9192## Implications for modelmap931. Induction-head and head-taxonomy maps are the cheapest replicable cross-model map type — good Experiment B/D substrate.942. EAP-IG is the default Mac-feasible circuit-discovery method; ACDC only for validation on tiny models.953. Faithfulness must be published as curves across ablation schemes (charter §8.4 alignment).964. An MPS/MLX port of circuit-tracer (or a minimal reimplementation) is a concrete high-value engineering contribution.975. The Hydra effect / self-repair is a first-class threat to any ablation-based localization map — Experiment C must include backup-head detection.98