# project: modelmap document: Phase 1 notes — §4.3 Feature decomposition (SAEs, superposition, dictionary methods) author: Simon-Pierre Boucher contact: contact@spboucher.ai website: https://modelmap.io created: 2026-08-12 status: draft

# §4.3 — Feature decomposition

Cost classes used below: L (laptop-trivial, minutes), M (hours + tens of GB activations), H (multi-day / >64 GB working sets / cluster-typical). Apple Silicon status as of 2026-08-12: almost all tooling is PyTorch and thus MPS-capable in principle but CUDA-assumed in practice — expect device="mps" patching, float64 fallbacks, no flash-attn.

# A1. Superposition theory (Toy Models of Superposition)

• Key paper: Elhage, Hume, Olsson, … Olah (Anthropic), Toy Models of Superposition, 2022. https://transformer-circuits.pub/2022/toy_model/index.html
• Claims: Networks represent more features than dimensions as non-orthogonal directions; occurrence sparsity governs superposition; predicts polysemantic neurons and phase transitions in toy ReLU models.
• Type: Theoretical/constructive (toy models with known ground truth).
• Cost: L. Toy models train in seconds on any Mac; ideal for MLX reimplementation and as planted-ground-truth testbeds.
• Implementation: Anthropic notebooks; many replications (ARENA, TransformerLens-based). Trivial on MPS/MLX.
• Failure modes / critiques: Toy results extrapolate uncertainly to real LLMs; "features as linear directions" is an assumption, not a theorem (multi-dimensional/circular features — Engels et al. 2024, https://arxiv.org/abs/2405.14860); degree of superposition in real models is inferred, not measured directly.
• Epistemic status: Established as theory of toy models; contested as a complete account of real-LLM representation.

# A2. Dictionary learning precursors

• Key papers: Yun, Chen, Olshausen, LeCun 2021, https://arxiv.org/abs/2103.15949 ; Sharkey, Braun, Millidge 2022 (interim report), https://www.lesswrong.com/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition ; Cunningham, Ewart, Riggs, Huben, Sharkey 2023, https://arxiv.org/abs/2309.08600
• Claims: Activations decompose into a sparse nonnegative combination of an overcomplete dictionary of interpretable directions.
• Type: Observational (Cunningham et al. add ablation checks).
• Cost: L–M at Pythia-70M/410M scale.
• Status: Established as lineage; methods superseded by SAELens-era tooling. Classical sparse coding (ISTA/OMP) too slow at LLM scale; early SAEs had dead latents and shrinkage.

# A3. ReLU SAEs — Towards / Scaling Monosemanticity

• Key papers: Bricken et al. 2023, https://transformer-circuits.pub/2023/monosemantic-features/index.html ; Templeton et al. 2024, https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
• Claims: L1-penalized overcomplete autoencoders on MLP/residual activations recover largely monosemantic features; scale to frontier models; features can steer behavior.
• Type: Primarily observational; steering demos are weak causal evidence about behavior, not about decomposition correctness.
• Cost — THE cost driver for modelmap: competitive SAEs need ~100M–2B tokens of activations; at d_model=2048 fp32, one hook point = 8 KB/token → 500M tokens ≈ 4 TB if materialized (hence streaming/regenerating activations + shuffle buffer; bf16 halves it). On a 32–64 GB Mac: SAEs on 0.5B–2B models at 4k–65k width feasible (M, hours–days); 7B–14B residual SAEs at ≥131k width are H, marginal.
• Implementations: SAELens https://github.com/jbloomAus/SAELens (PyTorch; official tutorial cites "~2 hours on an M3 Max" — MPS works); dictionary_learning https://github.com/saprmarks/dictionary_learning (device-agnostic, MPS-workable); EleutherAI sparsify https://github.com/EleutherAI/sparsify (top-k recipe, CUDA-oriented, MPS untested); MLX-native: mlxterp https://github.com/coairesearch/mlxterp (young, purpose-built for Apple Silicon — novelty-check target).
• Failure modes: Shrinkage from L1; dead latents; substantial reconstruction error ("dark matter"); interpretability of a feature ≠ its causal role.
• Epistemic status: Established as a decomposition method; contested as a route to faithful "true features".

# A4. SAE architecture variants (Gated, TopK, JumpReLU, BatchTopK, Matryoshka)

• Key papers: Gated: Rajamanoharan et al. 2024, https://arxiv.org/abs/2404.16014 ; TopK: Gao et al. (OpenAI) 2024, https://arxiv.org/abs/2406.04093 ; JumpReLU: Rajamanoharan et al. 2024, https://arxiv.org/abs/2407.14435 ; BatchTopK: Bussmann, Leask, Nanda 2024, https://arxiv.org/abs/2412.06410 ; Matryoshka: Bussmann et al. 2025, https://arxiv.org/abs/2503.17547
• Claims: Fix shrinkage (Gated); direct sparsity control + clean scaling laws (TopK/BatchTopK); Pareto-improve fidelity-vs-sparsity (JumpReLU); reduce feature absorption via nested dictionaries (Matryoshka).
• Cost: Same class as A3; TopK/BatchTopK slightly cheaper to tune (no L1 sweep). All implementable in MLX.
• Implementations: All in SAELens and SAEBench's suite (200+ open SAEs, 7 architectures); dictionary_learning implements most; sparsify has TopK/transcoders. JumpReLU straight-through estimators port fine to MPS.
• Failure modes: SAEBench showed proxy-metric gains (loss recovered, L0) do not reliably transfer to downstream utility; Matryoshka underperforms proxies while winning on disentanglement — proxy metrics are unreliable selection criteria.
• Epistemic status: Established engineering improvements; ranking among them contested and metric-dependent.

# A5. Transcoders, skip transcoders, cross-layer transcoders (CLTs), crosscoders

• Key papers: Dunefsky, Chlenski, Nanda 2024, https://arxiv.org/abs/2406.11944 ; Paulo, Shabalin, Belrose 2025 (skip transcoders), https://arxiv.org/abs/2501.18823 ; Lindsey, Templeton et al. 2024 (crosscoders), https://transformer-circuits.pub/2024/crosscoders/index.html
• Claims: Transcoders sparsely approximate an MLP's input→output map, enabling feature-to-feature weight analysis and circuit tracing; skip transcoders Pareto-dominate SAEs on interpretability+fidelity; crosscoders learn features shared across layers (cross-layer superposition) or across models (base-vs-instruct diffing).
• Cost: Like SAEs per layer (M for ≤2B on a Mac); CLTs/crosscoders read many layers at once — capture cost multiplies by layer count, pushing 7B+ toward H on 64 GB.
• Implementations: sparsify; EleutherAI clt-training https://github.com/EleutherAI/clt-training ; dictionary_learning; crosscoder diffing replication https://github.com/ckkissane/crosscoder-model-diff-replication . PyTorch; MPS untested but no exotic kernels.
• Failure modes: Transcoders only cover MLPs (attention needs separate treatment — cf. Anthropic 2025 QK-tracing); replacement-model error nodes can dominate; crosscoder diffing latents can be training-asymmetry artifacts.
• Epistemic status: Established and ascendant (they power attribution graphs); details contested.

# A6. Polysemanticity, feature splitting, feature absorption

• Key papers: Bricken et al. 2023 (splitting); Chanin et al. 2024, A is for Absorption, https://arxiv.org/abs/2409.14507
• Claims: As dictionary width grows, features split hierarchically; "absorption" makes a general latent silently fail on tokens claimed by more specific child latents — latents are not reliable classifiers of their nominal concept.
• Cost: L–M (evaluation on existing SAEs, e.g. Gemma Scope, on any Mac).
• Implementation: SAEBench includes absorption metrics — https://github.com/adamkarvonen/SAEBench . MPS-workable.
• Epistemic status: Established phenomenon; Matryoshka SAEs partially mitigate.

# A7. Automated interpretability (auto-interp) and its failure modes

• Key papers: Bills et al. (OpenAI) 2023, https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html ; Paulo, Mallen, Juang, Belrose 2024, https://arxiv.org/abs/2410.13928 ; critique: Huang et al. 2023, https://arxiv.org/abs/2309.10312
• Claims: An explainer LLM labels each latent from top-activating examples; scorers (simulation, detection, fuzzing, intervention scoring) grade at scale.
• Cost: Dominated by LLM inference; with a local MLX-served model (e.g. the maclustr llm endpoint) it is M-class.
• Implementations: EleutherAI Delphi https://github.com/EleutherAI/delphi (local models OK); Neuronpedia pipeline https://www.neuronpedia.org
• Failure modes: High-scoring explanations had low precision/recall under targeted tests (Huang et al.); top-activation bias; explanations describe activation, not downstream causal role.
• Epistemic status: Established as tooling; explanation quality claims debunked-in-part — never publish auto-interp labels above Level 1 without intervention scoring.

# A8. SAE evaluation: SAEBench, RAVEL, the metrics problem

• Key papers: SAEBench: Karvonen et al. 2025, https://arxiv.org/abs/2503.09532 (https://www.neuronpedia.org/sae-bench/info) ; RAVEL: Huang et al., ACL 2024, https://arxiv.org/abs/2402.17700 ; reliability critique: Chanin et al. 2026, https://arxiv.org/abs/2605.18229
• Claims: SAEBench: 8 metrics over 200+ open SAEs — key finding: proxy metrics ≠ downstream utility. RAVEL: counterfactual-intervention benchmark; distributed methods (DAS/MDAS) beat neurons and SAE latents.
• Cost: L–M — SAEBench on Gemma-2-2B SAEs fits a 32 GB Mac; strong candidate to port as modelmap's SAE gate.
• Implementations: https://github.com/adamkarvonen/SAEBench ; https://github.com/explanare/ravel . MPS-workable with patching.
• Failure modes: Benchmark rankings unstable across tasks/seeds (Chanin 2026); Goodharting.
• Epistemic status: Established best practice; individual metrics contested.

# A9. The 2024–2026 "SAEs disappointing?" debate; seed/dataset dependence; universality

• Key sources: GDM deprioritization: Smith et al., Mar 2025, https://deepmindsafetyresearch.medium.com/negative-results-for-sparse-autoencoders-on-downstream-tasks-and-deprioritising-sae-research-6cadcfc125b9 ; sparse probing negative: Kantamneni et al. 2025, https://arxiv.org/abs/2502.16681 ; AxBench: Wu et al. 2025, https://arxiv.org/abs/2501.17148 ; dataset dependence: Kissane et al. 2024, https://www.alignmentforum.org/posts/rtp6n7Z23uJpEH7od/saes-are-highly-dataset-dependent-a-case-study-on-the ; seed dependence: Paulo & Belrose 2025, https://arxiv.org/abs/2501.16615 ; reframing: Movva et al. 2025, https://arxiv.org/abs/2506.23845 ; universality: Lan et al. 2024, https://arxiv.org/abs/2410.06981 ; Gemma Scope: https://arxiv.org/abs/2408.05147
• Substance: SAE probes underperform plain linear probes OOD (GDM, deprioritized); SAE probes lose to logistic regression across 100+ datasets (Kantamneni); prompting/finetuning beat SAE steering, difference-in-means beats SAE dictionaries (AxBench). Counterpoints: dataset-matched SAEs close much of the gap (Kissane — chat-trained SAE recovers the refusal direction Pile-trained SAEs miss); only ~30% of latents replicate across seeds at 131k width on Llama-3-8B (Paulo & Belrose) — a direct replication-rate result modelmap must reproduce; nevertheless significant cross-model SAE-space similarity exists (Lan). Emerging 2025–2026 position: SAEs are hypothesis-generation/discovery tools, not optimal actuators or probes for known concepts. Anthropic doubled down via transcoder-based attribution graphs and continues SAE-based safety auditing (Circuits Updates, June 2026).
• Epistemic status: Negative downstream results established and replicated; "SAEs are useless" is an overreach — utility-for-discovery contested, exactly what modelmap's replication/causal-confirmation metrics can quantify.
• Mac note: every experiment in this debate (probing vs SAE probing, steering baselines, seed-replication Jaccard) is L–M on ≤2B models with public SAEs — a realistic early modelmap experiment family.

# Implications for modelmap

1. SAE training cost is the capture-pipeline design driver (Experiment H must measure streaming activation capture throughput on MLX vs MPS).
2. Seed replication (~30% latent overlap) is the noise floor Experiment A/D must beat or report.
3. SAEBench (or a subset) should be ported as the SAE gate; never select SAEs on proxy metrics alone.
4. mlxterp is direct prior art for the MLX capture layer — study and cite; differentiate via provenance/replication discipline.