SPB Git

spb/modelmap Public License

Internal cartography of local LLMs on Apple Silicon — registered, gated, negative-first. Public atlas at modelmap.io.

🔗 https://modelmap.io
interpretabilityllmapple-siliconmlxreproducibility
Python 66.3% JavaScript 24.5% CSS 8.1% Shell 0.7%
11.2 KB
View source Raw Blame History

# project: modelmap document: Phase 1 notes — §4.4 Circuits and computational graphs author: Simon-Pierre Boucher contact: contact@spboucher.ai website: https://modelmap.io created: 2026-08-12 status: draft

# §4.4 — Circuits and computational graphs

Cost classes: L (laptop-trivial), M (hours + tens of GB), H (multi-day / >64 GB / cluster-typical).

# B1. Induction heads

  • Key paper: Olsson et al. (Anthropic) 2022, https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html
  • Claims: Two-head circuit (previous-token + induction head) implements [A][B]…[A]→[B] copying; forms in a phase change coincident with in-context learning.
  • Type: Both — pattern-based detection plus ablations ("medium-confidence" causal per the paper itself).
  • Cost: L. Prefix-matching scores on any model that fits memory; classic first map layer for modelmap (cheap, replicable, cross-model).
  • Implementation: TransformerLens https://github.com/TransformerLensOrg/TransformerLens (MPS works, occasional dtype/op fallbacks); trivially portable to MLX hooks.
  • Failure modes: "Induction head" is behaviorally defined — heads heterogeneous and multifunctional; ICL attribution contested for large models.
  • Epistemic status: Established (most-replicated finding in mech interp).

# B2. IOI circuit and attention-head taxonomies

  • Key papers: IOI: Wang et al. 2022, https://arxiv.org/abs/2211.00593 ; Copy suppression: McDougall et al. 2023, https://arxiv.org/abs/2310.04625 ; Successor heads: Gould et al. 2023, https://arxiv.org/abs/2312.09230
  • Claims: 26-head circuit computes indirect-object identification in GPT-2-small; established faithfulness/completeness/minimality criteria. Copy suppression explains "negative heads" and self-repair. Successor heads recur across families 31M–12B on shared "mod-10" numeric features.
  • Type: Causal (path patching, ablations) + observational weight analysis.
  • Cost: L–M. GPT-2-small path patching laptop-trivial; head-taxonomy scans across a 7B model are M on 32–64 GB.
  • Failure modes: Backup heads / self-repair make ablation-based importance unstable (Hydra effect — McGrath et al. 2023, https://arxiv.org/abs/2307.15771); IOI only ~87% faithful, node-level; taxonomy labels are task-distribution-relative.
  • Epistemic status: Established (IOI heavily replicated cross-model); taxonomy generality contested at margins.

# B3. MLP layers as key–value memories

  • Key papers: Geva et al. 2021, https://arxiv.org/abs/2012.14913 ; Geva et al. 2022, https://arxiv.org/abs/2203.14680
  • Claims: MLP input weights = pattern detectors (keys), output weights = vocabulary-space updates (values); predictions built by additive sub-updates.
  • Cost: L — weight-only plus light forward passes; pairs naturally with Experiment E (weight-space cartography) and logit lens.
  • Implementation: simple to reimplement; baukit https://github.com/davidbau/baukit for hooks. MPS/MLX trivial.
  • Failure modes: Vocabulary projection misleads in early/mid layers; neuron-level keys polysemantic; "memory" metaphor overclaims vs editing critiques (Hase et al. 2023, https://arxiv.org/abs/2301.04213).
  • Epistemic status: Established as a lens; literal KV-memory reading contested.

# B4. Activation patching / attribution patching

  • Key papers: causal tracing: Meng et al. 2022, https://arxiv.org/abs/2202.05262 ; attribution patching: Nanda 2023, https://www.neelnanda.io/mechanistic-interpretability/attribution-patching ; best practices: Heimersheim & Nanda 2024, https://arxiv.org/abs/2404.15255
  • Claims: Swapping activations between clean/corrupted runs localizes causally necessary components; attribution patching approximates all components in 2 forward + 1 backward pass.
  • Cost: Patching scans M at 7B (thousands of forwards); attribution patching L–M — the key trick making circuit scans Mac-feasible, but needs gradients (MLX autodiff or PyTorch-MPS backward; memory ≈ 2–3× inference).
  • Implementations: TransformerLens, nnsight https://github.com/ndif-team/nnsight , pyvene.
  • Failure modes: Corrupted-baseline choice changes conclusions (noising vs denoising, zero vs mean vs resample); gradient approximation fails near zero-gradient (fixed by IG); localization ≠ editability.
  • Epistemic status: Established core method; interpretation pitfalls well-documented.

# B5. Attribution graphs / circuit tracing (Anthropic 2025)

# B6. Automated circuit discovery: ACDC → EAP → EAP-IG → edge pruning

# B7. Faithfulness metrics for circuits — and their critiques

  • Key papers: Miller, Chughtai, Saunders 2024, https://arxiv.org/abs/2407.08734 ; Shi et al. 2024, https://arxiv.org/abs/2410.13032 ; subspace-patching illusion: Makelov, Lange, Nanda 2023, https://arxiv.org/abs/2311.17030
  • Substance: Measured faithfulness swings wildly with ablation type (zero/mean/resample), direction, token positions, and metric (logit diff vs KL) — the same circuit can look faithful or unfaithful by experimenter choice. Shi et al. formalize equivalence/independence/minimality tests; published circuits pass only partially. Subspace patching can activate dormant pathways → illusory localization.
  • Cost: L–M at ≤2B — running a battery of ablation variants is exactly the charter §8.4 multi-control discipline.
  • Implementations: auto-circuit https://github.com/UFO-101/auto-circuit (efficient multi-ablation); https://github.com/blei-lab/circuitry
  • Consequence for modelmap: report faithfulness curves across ablation schemes, never single numbers.
  • Epistemic status: Critiques established; no consensus null for "faithful enough".

# B8. Sparse feature circuits

  • Key paper: Marks et al., ICLR 2025, https://arxiv.org/abs/2403.19647
  • Claims: Circuits over SAE features are far more interpretable than head/neuron circuits; SHIFT removes spurious signals; unsupervised discovery at scale.
  • Type: Causal — IG attribution patching validated by ablation.
  • Cost: M on Pythia-70M/Gemma-2-2B with public SAEs (Mac-feasible); needs SAEs for every stream, so capture cost multiplies.
  • Implementation: https://github.com/saprmarks/feature-circuits (dictionary_learning + nnsight). MPS-workable.
  • Failure modes: Inherits SAE pathologies — circuits over non-replicating features inherit their fragility.
  • Epistemic status: Established method, actively extended; robustness contested.

# B9. Weight-sparse transformers (OpenAI 2025) — circuits by construction

  • Key paper: Gao, Rajaram et al. 2025, https://arxiv.org/abs/2511.13653 (toolkit: https://github.com/openai/circuit_sparsity)
  • Claims: Training at ~1/1000 weight density yields compact, human-legible, causally-verifiable task circuits.
  • Relevance: applies only to models trained this way — useful to modelmap as a planted-ground-truth benchmark for validating discovery methods, not for mapping existing checkpoints.
  • Epistemic status: Established for tiny models; scaling unresolved.

# 2024–2026 developments

# Implications for modelmap

  1. Induction-head and head-taxonomy maps are the cheapest replicable cross-model map type — good Experiment B/D substrate.
  2. EAP-IG is the default Mac-feasible circuit-discovery method; ACDC only for validation on tiny models.
  3. Faithfulness must be published as curves across ablation schemes (charter §8.4 alignment).
  4. An MPS/MLX port of circuit-tracer (or a minimal reimplementation) is a concrete high-value engineering contribution.
  5. The Hydra effect / self-repair is a first-class threat to any ablation-based localization map — Experiment C must include backup-head detection.