---
project: modelmap
document: Phase 1 notes — §4.3 Feature decomposition (SAEs, superposition, dictionary methods)
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
website: https://modelmap.io
created: 2026-08-12
status: draft
---

# §4.3 — Feature decomposition

Cost classes used below: **L** (laptop-trivial, minutes), **M** (hours + tens of GB
activations), **H** (multi-day / >64 GB working sets / cluster-typical). Apple Silicon
status as of 2026-08-12: almost all tooling is PyTorch and thus MPS-capable in
principle but CUDA-assumed in practice — expect `device="mps"` patching, float64
fallbacks, no flash-attn.

## A1. Superposition theory (Toy Models of Superposition)
- **Key paper:** Elhage, Hume, Olsson, … Olah (Anthropic), *Toy Models of Superposition*, 2022. https://transformer-circuits.pub/2022/toy_model/index.html
- **Claims:** Networks represent more features than dimensions as non-orthogonal directions; occurrence sparsity governs superposition; predicts polysemantic neurons and phase transitions in toy ReLU models.
- **Type:** Theoretical/constructive (toy models with known ground truth).
- **Cost:** L. Toy models train in seconds on any Mac; ideal for MLX reimplementation and as planted-ground-truth testbeds.
- **Implementation:** Anthropic notebooks; many replications (ARENA, TransformerLens-based). Trivial on MPS/MLX.
- **Failure modes / critiques:** Toy results extrapolate uncertainly to real LLMs; "features as linear directions" is an assumption, not a theorem (multi-dimensional/circular features — Engels et al. 2024, https://arxiv.org/abs/2405.14860); degree of superposition in real models is inferred, not measured directly.
- **Epistemic status:** Established as theory of toy models; contested as a complete account of real-LLM representation.

## A2. Dictionary learning precursors
- **Key papers:** Yun, Chen, Olshausen, LeCun 2021, https://arxiv.org/abs/2103.15949 ; Sharkey, Braun, Millidge 2022 (interim report), https://www.lesswrong.com/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition ; Cunningham, Ewart, Riggs, Huben, Sharkey 2023, https://arxiv.org/abs/2309.08600
- **Claims:** Activations decompose into a sparse nonnegative combination of an overcomplete dictionary of interpretable directions.
- **Type:** Observational (Cunningham et al. add ablation checks).
- **Cost:** L–M at Pythia-70M/410M scale.
- **Status:** Established as lineage; methods superseded by SAELens-era tooling. Classical sparse coding (ISTA/OMP) too slow at LLM scale; early SAEs had dead latents and shrinkage.

## A3. ReLU SAEs — Towards / Scaling Monosemanticity
- **Key papers:** Bricken et al. 2023, https://transformer-circuits.pub/2023/monosemantic-features/index.html ; Templeton et al. 2024, https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
- **Claims:** L1-penalized overcomplete autoencoders on MLP/residual activations recover largely monosemantic features; scale to frontier models; features can steer behavior.
- **Type:** Primarily observational; steering demos are weak causal evidence about behavior, not about decomposition correctness.
- **Cost — THE cost driver for modelmap:** competitive SAEs need ~100M–2B tokens of activations; at d_model=2048 fp32, one hook point = 8 KB/token → 500M tokens ≈ 4 TB if materialized (hence streaming/regenerating activations + shuffle buffer; bf16 halves it). On a 32–64 GB Mac: SAEs on 0.5B–2B models at 4k–65k width feasible (M, hours–days); 7B–14B residual SAEs at ≥131k width are H, marginal.
- **Implementations:** SAELens https://github.com/jbloomAus/SAELens (PyTorch; official tutorial cites "~2 hours on an M3 Max" — MPS works); dictionary_learning https://github.com/saprmarks/dictionary_learning (device-agnostic, MPS-workable); EleutherAI sparsify https://github.com/EleutherAI/sparsify (top-k recipe, CUDA-oriented, MPS untested); MLX-native: **mlxterp** https://github.com/coairesearch/mlxterp (young, purpose-built for Apple Silicon — novelty-check target).
- **Failure modes:** Shrinkage from L1; dead latents; substantial reconstruction error ("dark matter"); interpretability of a feature ≠ its causal role.
- **Epistemic status:** Established as a decomposition method; contested as a route to faithful "true features".

## A4. SAE architecture variants (Gated, TopK, JumpReLU, BatchTopK, Matryoshka)
- **Key papers:** Gated: Rajamanoharan et al. 2024, https://arxiv.org/abs/2404.16014 ; TopK: Gao et al. (OpenAI) 2024, https://arxiv.org/abs/2406.04093 ; JumpReLU: Rajamanoharan et al. 2024, https://arxiv.org/abs/2407.14435 ; BatchTopK: Bussmann, Leask, Nanda 2024, https://arxiv.org/abs/2412.06410 ; Matryoshka: Bussmann et al. 2025, https://arxiv.org/abs/2503.17547
- **Claims:** Fix shrinkage (Gated); direct sparsity control + clean scaling laws (TopK/BatchTopK); Pareto-improve fidelity-vs-sparsity (JumpReLU); reduce feature absorption via nested dictionaries (Matryoshka).
- **Cost:** Same class as A3; TopK/BatchTopK slightly cheaper to tune (no L1 sweep). All implementable in MLX.
- **Implementations:** All in SAELens and SAEBench's suite (200+ open SAEs, 7 architectures); dictionary_learning implements most; sparsify has TopK/transcoders. JumpReLU straight-through estimators port fine to MPS.
- **Failure modes:** SAEBench showed proxy-metric gains (loss recovered, L0) do not reliably transfer to downstream utility; Matryoshka *underperforms* proxies while winning on disentanglement — proxy metrics are unreliable selection criteria.
- **Epistemic status:** Established engineering improvements; ranking among them contested and metric-dependent.

## A5. Transcoders, skip transcoders, cross-layer transcoders (CLTs), crosscoders
- **Key papers:** Dunefsky, Chlenski, Nanda 2024, https://arxiv.org/abs/2406.11944 ; Paulo, Shabalin, Belrose 2025 (skip transcoders), https://arxiv.org/abs/2501.18823 ; Lindsey, Templeton et al. 2024 (crosscoders), https://transformer-circuits.pub/2024/crosscoders/index.html
- **Claims:** Transcoders sparsely approximate an MLP's input→output map, enabling feature-to-feature weight analysis and circuit tracing; skip transcoders Pareto-dominate SAEs on interpretability+fidelity; crosscoders learn features shared across layers (cross-layer superposition) or across models (base-vs-instruct diffing).
- **Cost:** Like SAEs per layer (M for ≤2B on a Mac); CLTs/crosscoders read many layers at once — capture cost multiplies by layer count, pushing 7B+ toward H on 64 GB.
- **Implementations:** sparsify; EleutherAI clt-training https://github.com/EleutherAI/clt-training ; dictionary_learning; crosscoder diffing replication https://github.com/ckkissane/crosscoder-model-diff-replication . PyTorch; MPS untested but no exotic kernels.
- **Failure modes:** Transcoders only cover MLPs (attention needs separate treatment — cf. Anthropic 2025 QK-tracing); replacement-model error nodes can dominate; crosscoder diffing latents can be training-asymmetry artifacts.
- **Epistemic status:** Established and ascendant (they power attribution graphs); details contested.

## A6. Polysemanticity, feature splitting, feature absorption
- **Key papers:** Bricken et al. 2023 (splitting); Chanin et al. 2024, *A is for Absorption*, https://arxiv.org/abs/2409.14507
- **Claims:** As dictionary width grows, features split hierarchically; "absorption" makes a general latent silently fail on tokens claimed by more specific child latents — latents are not reliable classifiers of their nominal concept.
- **Cost:** L–M (evaluation on existing SAEs, e.g. Gemma Scope, on any Mac).
- **Implementation:** SAEBench includes absorption metrics — https://github.com/adamkarvonen/SAEBench . MPS-workable.
- **Epistemic status:** Established phenomenon; Matryoshka SAEs partially mitigate.

## A7. Automated interpretability (auto-interp) and its failure modes
- **Key papers:** Bills et al. (OpenAI) 2023, https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html ; Paulo, Mallen, Juang, Belrose 2024, https://arxiv.org/abs/2410.13928 ; critique: Huang et al. 2023, https://arxiv.org/abs/2309.10312
- **Claims:** An explainer LLM labels each latent from top-activating examples; scorers (simulation, detection, fuzzing, intervention scoring) grade at scale.
- **Cost:** Dominated by LLM inference; with a local MLX-served model (e.g. the maclustr llm endpoint) it is M-class.
- **Implementations:** EleutherAI Delphi https://github.com/EleutherAI/delphi (local models OK); Neuronpedia pipeline https://www.neuronpedia.org
- **Failure modes:** High-scoring explanations had low precision/recall under targeted tests (Huang et al.); top-activation bias; explanations describe *activation*, not downstream causal role.
- **Epistemic status:** Established as tooling; explanation quality claims debunked-in-part — **never publish auto-interp labels above Level 1 without intervention scoring**.

## A8. SAE evaluation: SAEBench, RAVEL, the metrics problem
- **Key papers:** SAEBench: Karvonen et al. 2025, https://arxiv.org/abs/2503.09532 (https://www.neuronpedia.org/sae-bench/info) ; RAVEL: Huang et al., ACL 2024, https://arxiv.org/abs/2402.17700 ; reliability critique: Chanin et al. 2026, https://arxiv.org/abs/2605.18229
- **Claims:** SAEBench: 8 metrics over 200+ open SAEs — key finding: proxy metrics ≠ downstream utility. RAVEL: counterfactual-intervention benchmark; distributed methods (DAS/MDAS) beat neurons and SAE latents.
- **Cost:** L–M — SAEBench on Gemma-2-2B SAEs fits a 32 GB Mac; strong candidate to port as modelmap's SAE gate.
- **Implementations:** https://github.com/adamkarvonen/SAEBench ; https://github.com/explanare/ravel . MPS-workable with patching.
- **Failure modes:** Benchmark rankings unstable across tasks/seeds (Chanin 2026); Goodharting.
- **Epistemic status:** Established best practice; individual metrics contested.

## A9. The 2024–2026 "SAEs disappointing?" debate; seed/dataset dependence; universality
- **Key sources:** GDM deprioritization: Smith et al., Mar 2025, https://deepmindsafetyresearch.medium.com/negative-results-for-sparse-autoencoders-on-downstream-tasks-and-deprioritising-sae-research-6cadcfc125b9 ; sparse probing negative: Kantamneni et al. 2025, https://arxiv.org/abs/2502.16681 ; AxBench: Wu et al. 2025, https://arxiv.org/abs/2501.17148 ; dataset dependence: Kissane et al. 2024, https://www.alignmentforum.org/posts/rtp6n7Z23uJpEH7od/saes-are-highly-dataset-dependent-a-case-study-on-the ; seed dependence: Paulo & Belrose 2025, https://arxiv.org/abs/2501.16615 ; reframing: Movva et al. 2025, https://arxiv.org/abs/2506.23845 ; universality: Lan et al. 2024, https://arxiv.org/abs/2410.06981 ; Gemma Scope: https://arxiv.org/abs/2408.05147
- **Substance:** SAE probes underperform plain linear probes OOD (GDM, deprioritized); SAE probes lose to logistic regression across 100+ datasets (Kantamneni); prompting/finetuning beat SAE steering, difference-in-means beats SAE dictionaries (AxBench). Counterpoints: dataset-matched SAEs close much of the gap (Kissane — chat-trained SAE recovers the refusal direction Pile-trained SAEs miss); **only ~30% of latents replicate across seeds at 131k width on Llama-3-8B (Paulo & Belrose) — a direct replication-rate result modelmap must reproduce**; nevertheless significant cross-model SAE-space similarity exists (Lan). Emerging 2025–2026 position: SAEs are hypothesis-generation/discovery tools, not optimal actuators or probes for known concepts. Anthropic doubled down via transcoder-based attribution graphs and continues SAE-based safety auditing (Circuits Updates, June 2026).
- **Epistemic status:** Negative downstream results established and replicated; "SAEs are useless" is an overreach — utility-for-discovery contested, exactly what modelmap's replication/causal-confirmation metrics can quantify.
- **Mac note:** every experiment in this debate (probing vs SAE probing, steering baselines, seed-replication Jaccard) is L–M on ≤2B models with public SAEs — a realistic early modelmap experiment family.

## Implications for modelmap
1. SAE training cost is the capture-pipeline design driver (Experiment H must measure streaming activation capture throughput on MLX vs MPS).
2. Seed replication (~30% latent overlap) is the noise floor Experiment A/D must beat or report.
3. SAEBench (or a subset) should be ported as the SAE gate; never select SAEs on proxy metrics alone.
4. mlxterp is direct prior art for the MLX capture layer — study and cite; differentiate via provenance/replication discipline.