spb/modelmap Public License

1---2project: modelmap3document: State of the art — technique taxonomy (Phase 2)4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128modified: 2026-08-129status: reviewed10---1112# State of the art — what can currently be measured inside an open-weight LLM1314Phase 2 deliverable (charter §5). Built from the Phase 1 sweep15(`research/notes/4.1–4.10`, `research/bibliography.md`, ~200 sources accessed162026-08-12). Each technique carries the charter's eleven-field block, condensed.17Citations live in the notes and bibliography; this document is the synthesis.1819**Cost classes.** L = laptop-trivial (minutes, fits any 16 GB Mac).20M = hours + tens of GB of activations (32–64 GB Mac, careful engineering).21H = multi-day / >64 GB working sets / cluster-typical (out of local scope or marginal).2223**Mac feasibility bands** (unified memory; fp16 unless stated): 16 GB → ≤3B fp1624or ≤8B 4-bit; 32 GB → ≤8B fp16 or ≤14B 4-bit; 64 GB → ≤14B fp16 or ~32B 4-bit.25Backward passes ≈ 2–3× forward memory. All feasibility judgments are **estimated**26from published numbers, not yet measured — Experiment H converts them to measured.2728**Epistemic flags.** ✅ established · ⚠️ contested · ❌ debunked-in-part.2930---3132## Family 1 — Observational probing3334### 1.1 Linear probes ✅(as decodability)/⚠️(as "use")35- **Reveals:** per-layer linear decodability of a property (language, domain, truth, syntax).36- **Observational/causal:** observational. **Access:** activations.37- **Cost:** L. **Mac:** yes, 0.5B–14B; storage is the binding cost (stream to disk).38- **Failure modes:** probe learns the task; decodable-but-unused; seed/dataset sensitivity; structure found in random models.39- **Reproducibility:** good *with* controls; historically often uncontrolled.40- **Implementation:** any framework; TransformerLens (MPS opt-in), baukit (device-agnostic); trivial in MLX.41- **Main limitation:** decodability ≠ causality — ceiling is Level 1 without intervention.42- **Extension opportunity:** probe maps with selectivity/MDL + FDR control as the atlas's *baseline layer*; probe-drift across quantization levels (unoccupied).4344### 1.2 Logit lens ✅(diagnostic)/❌(quantitative)45- **Reveals:** layer-by-layer refinement of next-token prediction.46- **Observational/causal:** observational. **Access:** activations + unembedding.47- **Cost:** L. **Mac:** every runnable size; trivial in MLX.48- **Failure modes:** family-dependent validity (fails on BLOOM/OPT/some Llamas); early layers unreadable.49- **Reproducibility:** deterministic; validity varies by family.50- **Main limitation:** intermediate residual bases differ from the final one.51- **Extension:** use only as sanity layer; superseded by 1.3/1.4.5253### 1.3 Tuned lens ✅54- **Reveals:** less-biased per-layer prediction trajectories via trained affine translators.55- **Observational/causal:** observational. **Access:** activations; gradients w.r.t. lens only.56- **Cost:** L–M (train lenses once per checkpoint). **Mac:** yes ≤14B.57- **Failure modes:** translator capacity objection (mitigated: affine-only).58- **Reproducibility:** good (validated to 20B).59- **Main limitation:** must be retrained per checkpoint *and per quantization level*.60- **Extension:** tuned-lens drift as a cheap quantization-deformation signal (expF) — not in the literature.6162### 1.4 Patchscopes / early decoding (DoLa, Future Lens) ✅(tools)/⚠️(DoLa mechanism)63- **Reveals:** verbalized readouts of hidden states; multi-token-ahead information.64- **Observational/causal:** observational readout implemented via patching.65- **Cost:** L–M (≈2× inference). **Mac:** yes ≤14B.66- **Failure modes:** readouts inherit model confabulation; DoLa's layer story unreliable across families.67- **Main limitation:** qualitative; hard to aggregate into maps.68- **Extension:** structured Patchscopes corpora as *labels* for probe maps (combination not systematically tried).6970### 1.5 Representation reading (RepE) ⚠️71- **Reveals:** concept directions via PCA on stimulus-pair activation differences.72- **Cost:** L. **Mac:** very friendly (repeng: CPU/MPS, exports GGUF control vectors).73- **Failure modes:** direction conflates stimulus correlates; single-direction assumption; weak evaluation (AxBench: prompting beats most methods).74- **Main limitation:** unreliable without baselines.75- **Extension:** reading-vector *stability maps* (across seeds of stimulus sets) — fits our replication-first program.7677### 1.6 Concept erasure (INLP → LEACE) ✅(LEACE guarantee)/⚠️(amnesic inference)78- **Reveals:** behavioral dependence on a property, by provably-minimal linear removal.79- **Observational/causal:** interventional on representations (bridge to Family 2).80- **Cost:** L–M (LEACE closed-form). **Mac:** yes ≤14B.81- **Failure modes:** nonlinear residue can survive or grow; erasure-operator choice changes conclusions; INLP rank damage.82- **Main limitation:** linear-only guarantee.83- **Extension:** erasure-based capability maps (erase concept → measure per-task damage) as a *causal upgrade of probe maps* — partially tried (amnesic probing), never as a systematic atlas layer.8485**Meta (1.x): the controls doctrine.** Selectivity (Hewitt & Liang), MDL (Voita &86Titov), shuffled labels, capacity-matched probes, random-init baselines, ≥2 datasets,87≥3–5 seeds. Non-negotiable; encoded in Experiment A.8889---9091## Family 2 — Causal / interventional methods9293### 2.1 Activation patching ✅ (with protocol pinned)94- **Reveals:** components causally mediating a behavior (single-mediator, input-relative).95- **Access:** activations (read/write hooks). **Cost:** L–M (2 forwards × components × pairs).96- **Mac:** yes 0.5B–8B; 14B slow. Memory-light, time-heavy — good Mac fit.97- **Failure modes:** conclusions flip with corruption type and metric; hydra/backup compensation; multi-mediator interactions missed.98- **Reproducibility:** good on classic tasks when protocol fixed.99- **Implementation:** TransformerLens/NNsight/pyvene/baukit; all MPS-workable; MLX = wrap modules.100- **Main limitation:** O(components) sweeps; single-mediator blindness.101- **Extension:** pre-registered patching protocols per map type (methodology.md); patched-maps across quantization levels (unoccupied).102103### 2.2 Path patching ✅104- **Reveals:** causal importance of edges/paths (the IOI instrument).105- **Cost:** M (combinatorial). **Mac:** ≤3B–7B practical.106- **Failure modes:** hypothesis-space explosion + all of 2.1's.107- **Main limitation:** does not scale to full-model edge maps locally.108- **Extension:** use only to verify circuits proposed by 2.3/4.5 (screen-then-verify).109110### 2.3 Attribution patching (AtP*, EAP, EAP-IG) ✅ (as screening)111- **Reveals:** first-order estimate of every patch effect in ~3 passes.112- **Access:** activations **and gradients**. **Cost:** L–M.113- **Mac:** ≤3B fp16 @16–32 GB, ≤8B @64 GB; **gradients through 4-bit models are the open constraint; MLX autodiff is the native path to benchmark (expH).**114- **Failure modes:** fails near-zero gradients (fixed by IG), saturated attention, residual stream; documented false negatives; second-order lies (2026).115- **Reproducibility:** good; public reimplementations replicate.116- **Main limitation:** an *estimator* — final claims need real patching.117- **Extension:** the default Mac circuit-scanner; calibration study (AtP error vs true patching) per model family is publishable.118119### 2.4 Causal tracing (ROME-style) ⚠️/❌ (locating→editing inference)120- **Reveals:** where restored states recover corrupted behavior.121- **Cost:** M per fact corpus. **Mac:** yes ≤8B.122- **Failure modes:** **Hase et al.: tracing does not predict editability**; Gaussian noising is off-distribution and inflates mid-layer MLP localization.123- **Main limitation:** interpretation, not mechanics.124- **Extension:** rerun with symmetric-token corruption + resample baselines; treat as one voter in method-agreement maps, never alone.125126### 2.5 Ablations (zero / mean / resample) ✅127- **Reveals:** necessity of components. **Cost:** L. **Mac:** all sizes; ideal for expB.128- **Failure modes:** zero/mean off-distribution; hydra effect; multiple comparisons.129- **Norm:** resample > mean > zero.130- **Main limitation:** necessity only; distribution-relative.131- **Extension:** ablation-scheme *curves* (all three schemes) as the atlas standard — no one publishes this systematically.132133### 2.6 Causal scrubbing ✅(concept)/little-used134- **Reveals:** quantitative faithfulness of a full hypothesis graph.135- **Cost:** M–H. **Mac:** ≤1B realistically.136- **Failure modes:** unfaithful hypotheses can score well; combinatorial resample structure.137- **Implementation:** rust_circuit archived — **no maintained macOS-friendly implementation exists (gap).**138- **Extension:** a minimal MLX scrubbing kernel for micro-models would be new tooling.139140### 2.7 DAS / interchange interventions (pyvene) ⚠️ (subspace-illusion debate unresolved)141- **Reveals:** whether a high-level causal model lives in learned subspaces (IIA).142- **Access:** activations + gradients (rotation trained). **Cost:** M. **Mac:** ≤7B.143- **Failure modes:** dormant-pathway illusion (Makelov) vs over-strict criterion (Wu reply); with unconstrained featurizers, causal abstraction is vacuous (Non-Linear Representation Dilemma).144- **Main limitation:** optimized bases need convergent evidence.145- **Extension:** confidence.md records featurizer class; DAS results always paired with natural-basis patching.146147### 2.8 Steering vectors (ActAdd/CAA/ITI) ✅(existence)/⚠️(reliability)148- **Reveals:** linear sufficiency of behavior directions.149- **Cost:** L. **Mac:** the most Mac-friendly causal technique; **works on GGUF via llama.cpp control vectors.**150- **Failure modes:** input-heterogeneous, anti-steering, OOD-brittle, non-identifiable; loses to prompting baselines (AxBench).151- **Main limitation:** evaluation, not mechanism.152- **Extension:** *steerability maps with variance and baselines* per model/quantization — the literature reports means, not distributions.153154---155156## Family 3 — Feature decomposition157158### 3.1 Superposition theory ✅(toy)/⚠️(real models)159- **Reveals:** why polysemanticity exists; predicts feature geometry under sparsity.160- **Cost:** L (toy models in seconds). **Mac:** trivially; ideal MLX testbed with planted ground truth.161- **Main limitation:** linear-feature assumption is not a theorem (multi-dimensional features exist).162- **Extension:** planted-superposition testbeds to validate our SAE pipeline before real models (validation-first rule).163164### 3.2 Sparse autoencoders (ReLU → TopK/JumpReLU/BatchTopK/Matryoshka) ✅(method)/⚠️(faithful "true features")165- **Reveals:** overcomplete sparse dictionaries of activation directions.166- **Access:** activations at scale. **Cost:** **the cost driver** — 10⁸–10⁹ tokens/layer; 500M tokens ≈ 4 TB fp16 if materialized → streaming mandatory.167- **Mac:** M for 0.5B–2B models at 4k–65k width (SAELens documents an M3 Max run); H for 7B+ at ≥131k width.168- **Failure modes:** ~30% seed replication (the noise floor); dataset dependence; absorption/splitting; proxy metrics ≠ downstream utility; dark-matter reconstruction error.169- **Reproducibility:** the weak point — quantified and low.170- **Implementation:** SAELens (MPS works), dictionary_learning, sparsify (no-cache streaming, CUDA-leaning), mlxterp (MLX, embryonic).171- **Main limitation:** decompositions are pragmatic, not canonical.172- **Extension:** seed-replication rate as a *published per-SAE metric*; buffer-shuffle vs full-shuffle replication study (unpublished); SAE feature drift under quantization (one 2026 paper — extend it).173174### 3.3 Transcoders / CLTs / crosscoders ✅(ascendant)175- **Reveals:** sparse input→output MLP maps (analyzable feature-to-feature weights); cross-layer features; cross-model diffs.176- **Cost:** M (≤2B) to H (7B+, cross-layer multiplies capture). **Mac:** ≤2B pairs realistic.177- **Failure modes:** MLP-only coverage; error nodes dominate on some prompts; diffing latents can be sparsity artifacts (fixed by BatchTopK + latent scaling).178- **Main limitation:** analyses describe a *replacement model*.179- **Extension:** an MPS/MLX transcoder recipe at 0.5–2B with replication reporting.180181### 3.4 Automated interpretability (auto-interp) ✅(tooling)/❌(label quality claims)182- **Reveals:** natural-language labels for latents, at scale.183- **Cost:** M, dominated by LLM inference — **local via the maclustr LLM endpoint.**184- **Failure modes:** low precision/recall under targeted tests; top-activation bias; metrics fail to distinguish trained from random transformers.185- **Rule:** never above Level 1 without intervention scoring.186- **Extension:** intervention-scored labels only; label *stability* across explainer models as an uncertainty channel.187188### 3.5 SAE evaluation (SAEBench, RAVEL, MIB causal-variable track) ✅(best practice)/⚠️(metric stability)189- **Reveals:** which decompositions are worth anything (proxy + downstream + intervention metrics).190- **Cost:** L–M with public SAEs (Gemma Scope on 2B fits 16 GB). **Mac:** yes.191- **Failure modes:** benchmark rankings unstable across tasks/seeds; Goodharting.192- **Extension:** port a SAEBench subset as modelmap's SAE gate; add our replication-rate metric to it.193194---195196## Family 4 — Circuits and computational graphs197198### 4.1 Induction heads ✅ (most replicated finding in the field)199- **Reveals:** the [A][B]…[A]→[B] two-head circuit; cross-architecture universality.200- **Cost:** L (one forward with attention capture). **Mac:** to 32B 4-bit via MLX.201- **Main limitation:** behaviorally defined; heads are multifunctional.202- **Extension:** **the calibration landmark for any cross-model coordinate system** (if alignment can't match induction heads, it fails); first atlas map type.203204### 4.2 IOI circuit & head taxonomies ✅ (details ⚠️)205- **Reveals:** name movers, S-inhibition, copy suppression, successor heads; faithfulness criteria.206- **Cost:** L–M. **Mac:** GPT-2-scale trivial; 7B head scans M.207- **Failure modes:** hydra/backup heads destabilize ablation importance; ~87% faithfulness; task-distribution-relative labels.208- **Extension:** head-taxonomy maps across sizes/quantizations with stability scores (expD substrate).209210### 4.3 MLP key–value memory view ✅(lens)/⚠️(literal reading)211- **Reveals:** input weights as pattern keys, output weights as vocabulary updates.212- **Cost:** L, near-weight-only. **Mac:** trivial.213- **Extension:** pairs with Family 5 as a weight-first map layer; vocabulary-projection maps with early-layer validity flags.214215### 4.4 Attribution graphs / circuit tracing (CLT-based) ✅ (candid limits)216- **Reveals:** per-prompt causal graphs over features (planning, multi-hop, refusal mechanics).217- **Cost:** M with pretrained CLTs (Gemma-2-2B graphs ≈ 15–40 GB → 64 GB Mac plausible, 16 GB not); H to train CLTs at 7B.218- **Failure modes:** error nodes unexplained; frozen attention (QK interference); per-prompt graphs need aggregation to become maps; many graphs unreadable.219- **Implementation:** circuit-tracer (CUDA-first) — **MPS/MLX port is a concrete high-value engineering task.**220- **Extension:** graph *aggregation across prompt corpora* with replication stats — the "map" layer Anthropic hasn't published.221222### 4.5 Automated circuit discovery (ACDC → EAP-IG → edge pruning) ✅ (method choice ⚠️)223- **Reveals:** minimal subgraphs reproducing behavior under faithfulness objectives.224- **Cost:** ACDC M–H; **EAP-IG L–M (the Mac default)**; edge pruning M–H (scaled to 13B on cluster-class compute — fits M3U96a/b).225- **Failure modes:** inherits patching baselines; greedy/independence assumptions; metric overfitting.226- **Extension:** circuit discovery under *multiple ablation schemes* reported as curves.227228### 4.6 Circuit hypothesis testing (circuitry; faithfulness critiques) ✅229- **Reveals:** pass/fail on equivalence/independence/minimality; robustness of faithfulness itself.230- **Cost:** L–M ≤2B. **Mac:** yes.231- **Rule adopted:** every promoted circuit runs the battery; results in confidence.md.232233### 4.7 Sparse feature circuits ✅(method)/⚠️(robustness)234- **Reveals:** circuits over SAE/transcoder features (more interpretable than heads); SHIFT debiasing.235- **Cost:** M with public SAEs. **Mac:** Pythia/Gemma-2-2B scale yes.236- **Failure modes:** inherits SAE seed-fragility — circuits over non-replicating features are sandcastles.237- **Extension:** feature-circuit replication rate across SAE seeds — directly quantifies the sandcastle problem; not in the literature.238239### 4.8 Weight-sparse transformers (circuits by construction) ✅(tiny models)240- **Relevance:** planted-ground-truth benchmark for validating discovery pipelines, not for mapping existing checkpoints.241242---243244## Family 5 — Weight-space cartography (zero/low forward-pass)245246### 5.1 SVD spectra & LASER ✅(phenomenon)/⚠️(interpretation)247- **Reveals:** rank structure per matrix; behaviorally-loaded high-order components.248- **Cost:** L (SVD of 4096×11008 = seconds on Accelerate). **Mac:** fully native.249- **Failure modes:** (layer, matrix, rank) sweeps invite multiple comparisons; ID-only proxies diverge OOD (Friedman illusion).250- **Extension:** per-matrix spectral maps as the atlas's cheapest layer; **test whether they predict A–D properties (expE — the unoccupied gap).**251252### 5.2 Heavy-tailed spectral diagnostics (WeightWatcher) ✅(trends)/⚠️(fine grain)253- **Reveals:** training quality per layer from ESD power-law fits, zero data.254- **Cost:** L; pip-installable, runs on Apple Silicon today.255- **Failure modes:** fit-range sensitivity; Simpson's paradox documented.256- **Extension:** alpha profiles vs probe/localization maps on the same checkpoints — the direct expE experiment.257258### 5.3 Outliers: super weights, massive activations, sinks ✅259- **Reveals:** a handful of weight-visible scalars with catastrophic causal load; the weight→activation bridge.260- **Cost:** L (scan) + minutes (verify by zeroing). **Mac:** yes (MLX 4-bit PPL checks).261- **Extension:** super-weight/outlier maps per model+quantization; **outlier-aware controls for all similarity metrics** (CKA is dominated by these dims).262263### 5.4 Representation/weight similarity (CKA/SVCCA/Procrustes) ✅(tool)/❌(standalone evidence)264- **Reveals:** cross-layer/cross-model geometry alignment.265- **Cost:** L on cached activations. **Mac:** yes.266- **Failure modes:** outlier domination; manipulability; metric disagreement (ReSi); saturation on LLM pairs; estimator drift.267- **Rule:** ≥2 metrics + statistical tests + outlier-stripped variants + random/width-matched nulls.268269### 5.5 Layer redundancy / depth maps ✅270- **Reveals:** deletable contiguous deep blocks (angular similarity).271- **Cost:** L (small calibration set). **Failure modes:** perplexity survives while reasoning dies.272- **Extension:** redundancy maps as a *target* for weight-only predictors; feeds localvm working-set questions (Q5).273274### 5.6 Stitching ⚠️ | 5.7 Git re-basin ⚠️(transformers) | 5.8 Mode connectivity ✅/⚠️275- Stitching: protocol-sensitive; LLM literature thin; tokenizer collision cross-family.276- Re-basin: **fails on transformers**; the LLM-relevant analogue (cross-size alignment) is open = Experiment G.277- Mode connectivity: cheap base↔instruct interpolation profiles are the Mac-friendly artifact.278279### 5.9 Task arithmetic / delta maps ✅(engineering)/⚠️(interpretation)280- **Reveals:** θ_ft − θ_base as sparse, composable task vectors (90%+ droppable).281- **Cost:** L (state-dict math). **Mac:** 7–14B fine.282- **Extension:** **systematic base-vs-instruct delta atlas (per-matrix norms, delta rank, delta spectra) — no published atlas exists; ideal cheap modelmap product.**283284### 5.10 Crosscoder model diffing ⚠️ + weight-side monitoring (2025) — see 3.3285- The field is converging on weight-first diffing (Watch the Weights; weight-diff interpretation) — adjacent to, not overlapping, our pre-screen program.286287### 5.11 MoE router structure ✅(negative result known)288- "Expert = topic" debunked; router-weight geometry is an open, cheap target. Mixtral-class needs 4-bit on 64 GB.289290### 5.12 Weight-space learning ✅(field)/nascent for LLMs291- Model-zoo regime breaks at LLM scale; borrow representations (per-matrix statistics vectors), not the zoo methodology.292293---294295## Family 6 — Knowledge localization and editing296297### 6.1 ROME/MEMIT/PMET/EMMET/AlphaEdit ✅(mechanics)/❌(localization inference)298- **Reveals:** facts are *rewritable* via mid-layer MLP updates; AlphaEdit's null-space projection tames sequential collapse.299- **Cost:** M (covariance stats = hours of forwards; edits cheap). **Mac:** feasible ≤8B, **but all repos are CUDA-first — an MLX editing port is an open tooling contribution.**300- **Failure modes:** localization ≠ editability (Hase); sequential collapse; ripple-effect failures (in-context editing beats parametric); general-ability damage.301- **Extension:** editing as *perturbation probe* (measure what else moves), never as localization evidence.302303### 6.2 Knowledge neurons ❌ (one-fact-one-neuron)304- Degenerate/redundant neurons; token-pattern confound. Keep only as a historical baseline for expB nulls.305306### 6.3 Factual recall pipeline (Geva) + attribute lens ✅(scaffold)/⚠️(details)307- **Reveals:** subject enrichment (MLPs) → relation propagation → attribute extraction (attention heads); ~48% of relations linearly decodable.308- **Cost:** L–M. **Mac:** ≤7B yes.309- **Extension:** the concrete map types for Q1: *enrichment maps* (MLP) and *extraction maps* (heads), with the additivity caveat (multiple mechanisms sum).310311### 6.4 Multilingual knowledge sharing ✅(qualitative)/⚠️(mechanism)312- **Reveals:** partially language-agnostic substrate, English-hub behavior, poor cross-lingual edit transfer.313- **Mac:** ≤8B multilingual models fine. **Extension:** FR/EN localization atlas (expB) has clean prior art to extend and a personal-fit corpus advantage.314315---316317## Cross-cutting instruments318319### 7.1 Cross-model comparison320Universal neurons (1–5% across seeds — methodology template: correlate → taxonomize →321ablate); induction-head landmark; Tigges consistency across training/scale (the expG322band is 70M–2.8B); SAE feature matching (moderate overlap, seed-noise-limited);323platonic convergence deflated to *local/topological* alignment (2026); Rosetta-neuron324mining untested on LLMs (candidate coordinate system); **tokenizer alignment is325unsolved — within-family comparison is the safe default; cross-family maps must326declare their convention in the schema.**327328### 7.2 Ground-truth validation & statistics329Tracr / InterpBench / MIB as pipeline validators (all Mac-runnable); circuitry330hypothesis tests; causal abstraction as the Level-3 formal spine *plus* featurizer-class331disclosure; Dead-Salmons doctrine (random-init nulls, FDR control, uncertainty332quantification) — adopted into `src/modelmap/stats` requirements.333334### 7.3 Quantization × internals (the thin shelf)335Five shallow papers; effects "inconsistent in direction"; one SAE-drift study (2026).336Nothing on probe drift, circuit survival, Q2/Q3, GGUF-runtime capture, or337weight-geometry-predicts-quantization-sensitivity. **The single clearest open field338this survey found.**339340### 7.4 Systems substrate (macOS)341Split regime: probing/patching corpora (≤10⁶ tokens) → cache everything; SAE corpora →342stream, never store. mmap likely wins on Apple NVMe (unmeasured); APFS clones = free343store snapshots for provenance; MLX = only Python path to quantized-model capture;344llama.cpp eval-callback = only path into *actually-quantized* compute (unwrapped);345MPS hooks work but sync/fallback costs unmeasured. All quantitative claims here are346expH's to measure.347348---349350## Overlaps and redundancies (what is secretly the same thing)3513521. **Patching ≡ interchange intervention ≡ causal mediation** — one method, three353   vocabularies (mech-interp, causal-abstraction, NLP-fairness). DAS adds a learned354   basis; Patchscopes adds a verbal readout; causal tracing adds Gaussian noise (its355   weakest link).3562. **Tuned lens ≈ per-layer linear probes onto vocabulary** — a probe family, not a357   separate epistemic category; inherits all probe pitfalls.3583. **LEACE ≈ steering's inverse** — erasure removes a direction, steering adds one;359   both test linear-direction causality and should share controls (random directions).3604. **SAE / transcoder / crosscoder / CLT = one estimator family** (sparse dictionary361   on activations) differing in domain (layer, MLP map, model pair, layer stack);362   they share the seed-replication pathology, so replication findings transfer.3635. **Faithfulness metrics ≈ causal scrubbing lite** — circuit faithfulness under364   resample ablation is scrubbing's score without the hypothesis-graph machinery.3656. **Layer-redundancy maps ≈ cross-layer CKA** (angular similarity is a similarity366   metric) — inherits CKA's outlier pathology; strip sink dimensions first.367368## Combinations already tried (do not claim as new)369370- Probes + causal verification: amnesic probing; RAVEL; MIB causal-variable track.371- SAE features + circuits: sparse feature circuits; attribution graphs.372- SAE + steering: AxBench (negative); feature steering in Scaling Monosemanticity.373- Weight diffs + interpretation: Watch the Weights; Learning to Interpret Weight Differences (2025).374- Patching + automated search: ACDC/EAP/EAP-IG/edge pruning.375- Editing + localization: ROME→Hase (the canonical failed combination).376- Atlas + API hosting: Neuronpedia (without provenance contracts).377378## Ideas that seem novel but are known (novelty traps)379380- "Label every neuron/feature with an LLM" → Transluce, OpenAI 2023, Delphi. Known, and known-unreliable.381- "Compare models with CKA to find shared structure" → large literature *and* a debunking literature.382- "Find the layers where facts live and edit them" → ROME line + Hase refutation.383- "A browsable atlas of features" → Neuronpedia/Neuroscope. The *provenance/confidence* layer is the gap, not the browser.384- "Predict quality from weight spectra" → WeightWatcher (2019–2021).385- "Mine correlated units across models" → Rosetta Neurons (vision) — LLM transfer is open but the method is not new.386- "MLX interpretability library" → mlxterp exists (embryonic) — extend or differentiate, cite regardless.387388## Cost frontier summary (estimated; expH converts to measured)389390| Map type | 16 GB | 32 GB | 64 GB |391|---|---|---|---|392| Weight-only maps (spectra, outliers, deltas) | ≤14B | ≤32B | ≤70B (disk-bound) |393| Probe / lens / erasure maps | ≤3B fp16, 8B 4-bit | ≤8B fp16 | ≤14B fp16 |394| Patching / ablation scans | ≤3B | ≤8B | ≤14B (slow) |395| Attribution patching (gradients) | ≤1.5B | ≤3B fp16 | ≤8B fp16 |396| SAE/transcoder training | ≤0.5B narrow | ≤2B | ≤3B (7B marginal) |397| Attribution graphs (pretrained CLTs) | no | Gemma-2-2B tight | ≤4B |398| Editing (with covariance precompute) | ≤3B | ≤8B | ≤8B+ |399400## Conclusions feeding Phase 34014021. The unoccupied high-value squares: **quantization × internal maps**, **weight-only403   pre-screens validated against activation-level ground truth**, **replication-rate404   reporting as a first-class artifact metric**, **quantized-runtime capture405   (llama.cpp/MLX)**, **provenance-first atlas schema**.4062. The methodological spine is buyable off the shelf: Tracr/InterpBench validation,407   circuitry tests, SAEBench subset, controls doctrine — integrate, don't reinvent.4083. Everything promoted to the atlas needs: pinned patching protocol, ablation-scheme409   curves, ≥2 similarity metrics with nulls, featurizer-class disclosure, seed410   replication rate, per-dataset agreement. These become the map-card schema fields.411