---
project: modelmap
document: Research gaps — candidate directions (Phase 3)
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
website: https://modelmap.io
created: 2026-08-12
modified: 2026-08-12
status: reviewed
---

# Research gaps — 24 candidate directions

Phase 3 deliverable (charter §6). Each gap carries the six-field block:
**(1) existing work · (2) what it does not do · (3) why it matters ·
(4) why it could work · (5) biggest failure risk · (6) smallest falsifying
experiment runnable on a 16–64 GB Mac.** Gaps marked ⭐ have direct synergy
with the ranked shortlist criteria (Phase 4). Sources: `state_of_the_art.md`,
notes §4.1–4.10.

---

## Cluster A — Quantization × internals (the thin shelf)

### G01 ⭐ Probe-drift atlas across quantization levels
1. **Existing:** ~5 shallow papers; effects "inconsistent in direction"; no controls.
2. **Missing:** probe/selectivity drift FP16→Q8→Q4→Q2 with shuffled-label controls, seeds, multiple datasets, on the same checkpoint.
3. **Matters:** local models are almost always quantized; if probe maps built at FP16 don't transfer, every published FP16-only map is misleading for real deployments.
4. **Could work:** MLX quantizes natively; probes are L-cost; drift = probe-accuracy delta + direction rotation, both well-defined.
5. **Failure risk:** drift may be smaller than the seed/dataset noise floor → "no measurable effect" (itself publishable, Level 1).
6. **Falsifier:** Qwen3-0.6B FP16 vs Q4: 10 probes × 5 seeds × 2 datasets; if |drift| < noise floor for all probes, the "quantization deforms decodability" hypothesis dies at this scale.

### G02 ⭐ Circuit survival under quantization
1. **Existing:** circuit discovery at FP16 only; one 2026 SAE-drift paper; zero circuit-level studies.
2. **Missing:** does the induction/IOI/EAP-IG circuit found at FP16 survive at Q4/Q2 (same nodes? same faithfulness)?
3. **Matters:** directly tests whether interpretability results transfer to the models people actually run; a cheap universality probe.
4. **Could work:** induction heads are L-cost to detect; EAP-IG runs ≤3B; node-set Jaccard + faithfulness curves are clean metrics.
5. **Failure risk:** gradient-based EAP through quantized weights is awkward — may need dequantized-weight gradients as approximation (documented caveat).
6. **Falsifier:** GPT-2-small / Qwen3-0.6B: induction-head set at FP16 vs Q4 (MLX). If Jaccard ≈ 1.0 and faithfulness curves overlap within CIs, "quantization deforms circuits" is falsified at this granularity.

### G03 ⭐ Weight-geometry predicts quantization sensitivity
1. **Existing:** quantization methods use activation stats (imatrix, AWQ) for *calibration*; WeightWatcher predicts behavioral quality.
2. **Missing:** per-matrix weight statistics (spectra, alpha, kurtosis, outlier counts) predicting *where* internal drift (G01/G02) concentrates.
3. **Matters:** a weight-only pre-screen for "which layers will break when you quantize" — instant value for local-model users; bridges expE and expF.
4. **Could work:** outlier theory (super weights, LLM.int8) already links weight statistics to quantization fragility at the extremes.
5. **Failure risk:** drift may be diffuse/global rather than matrix-localized → nothing to predict.
6. **Falsifier:** rank matrices by weight-stat fragility score; correlate with measured per-layer drift from G01. Spearman ρ < 0.3 across 2 models kills it.

### G04 Tuned-lens translation distance as a deformation metric
1. **Existing:** tuned lens per checkpoint; never across quantization levels.
2. **Missing:** retrain lenses per Q level; use translator distance (‖A_q − A_fp‖, rotation angle) as a cheap per-layer deformation index.
3. **Matters:** a continuous, per-layer scalar of internal deformation, far cheaper than SAE retraining.
4. **Could work:** lenses are affine, small, fast to train; distances are well-defined.
5. **Failure risk:** lens training variance may swamp the signal (lens seed-sensitivity is undocumented — measure first).
6. **Falsifier:** train 5-seed lens ensembles at FP16 and Q4 on a 0.5B model; if between-Q distance ≤ within-Q seed distance at every layer, the metric is dead.

### G05 Quantized-runtime capture (llama.cpp/GGUF toolchain)
1. **Existing:** llama.cpp `eval-callback` dumps tensors; imatrix computes activation stats; nothing wraps either for interpretability.
2. **Missing:** a capture exporter (eval-callback → mmap store with token bookkeeping) enabling probes/ablation on *actually-quantized* Metal compute, not simulated quantization.
3. **Matters:** MLX/PyTorch "Q4" simulates quantized weights in fp arithmetic; the deployed runtime differs (kernels, dequant paths). Only this measures reality.
4. **Could work:** the callback exists, is C-level, works on Metal; the exporter is engineering, not research risk.
5. **Failure risk:** per-op dump overhead may be 10–100× — could limit corpora to 10⁴ tokens (still enough for probes).
6. **Falsifier:** build the exporter; if probe accuracy on llama.cpp-Q4 activations ≠ probe accuracy on MLX-simulated-Q4 beyond CI, simulation-based studies (incl. our own G01) are systematically biased — either way we learn something publishable.

## Cluster B — Weight-only cartography as pre-screen

### G06 ⭐ Weight-only statistics → activation-level ground truth
1. **Existing:** WeightWatcher (weights→behavioral quality); super weights (one phenomenon); weight-space learning (global properties, tiny models).
2. **Missing:** the systematic study — per-matrix weight features (spectra, effective rank, alpha, outliers, norms) predicting probe accuracy, localization concentration, ablation damage on the *same* checkpoints.
3. **Matters:** if it works, any model gets a free first map in CPU-minutes, and expensive activation methods get targeted where they matter (the accuracy/cost frontier, Q4).
4. **Could work:** MLP-KV theory + LASER + super-weight results all say weight structure carries functional signal; nobody has regressed it against activation maps.
5. **Failure risk:** weight statistics may only predict *global* quality, not *where* things live (Friedman illusion: ID-proxy ≠ mechanism).
6. **Falsifier:** on Pythia-410M: 20 weight features per matrix vs per-layer probe accuracy for 10 properties. If cross-validated R² < 0.2 for all properties, weight-only pre-screens die (a Level-1 negative worth publishing).

### G07 Base↔instruct delta atlas
1. **Existing:** task arithmetic, DARE sparsity, crosscoder diffing (activation-side), Watch-the-Weights (monitoring).
2. **Missing:** a systematic *atlas* of instruct deltas: per-matrix delta norms, delta effective rank, delta spectra across ≥3 families, versioned and comparable.
3. **Matters:** "what does instruction-tuning touch, structurally?" is a question every fine-tuner has; pure weight arithmetic = free at 7–14B.
4. **Could work:** deltas are known-sparse and low-rank-ish; the descriptive statistics are trivially computable; cross-family comparison is the novelty.
5. **Failure risk:** descriptive-only risk — without behavioral correlation it's stamp collecting (mitigate: correlate delta concentration with steering/refusal behavior deltas).
6. **Falsifier:** if delta statistics are indistinguishable across families/matrices (uniform noise), there is no structure to map. Qwen3 base/instruct pair, one afternoon.

### G08 Super-weight / outlier cartography across families and quantization
1. **Existing:** super weights identified on ~10 models (Apple 2024); sinks/massive activations literature.
2. **Missing:** a versioned catalog: outlier coordinates per model+quantization, causal verification (zero → PPL), do quantization schemes preserve them, do they collide with LoRA/edit targets?
3. **Matters:** single weight-visible scalars with catastrophic causal load are the strongest known weight→function bridge; atlas-perfect objects (small, verifiable, comparable).
4. **Could work:** detection is near-free; verification is minutes per model in MLX.
5. **Failure risk:** low — the phenomenon replicates; risk is *shallowness* (a catalog without new science). Mitigate by testing preservation under Q2 and interaction with G03.
6. **Falsifier:** if super-weight coordinates are not stable across quantization of the same checkpoint, the "weight-visible" premise weakens (worth knowing either way).

### G09 Cheap maps → pruning/redundancy tolerance
1. **Existing:** angular-similarity depth pruning (ShortGPT/Gromov); WeightWatcher alphas.
2. **Missing:** do weight-only features predict which layers are deletable *before* running the calibration pass; does redundancy predict capability-specific damage (reasoning vs perplexity)?
3. **Matters:** pruning tolerance is Q5 utility; also feeds localvm (skip-loading redundant blocks).
4. **Could work:** redundancy is measurable ground truth at L-cost; regression setup identical to G06.
5. **Failure risk:** redundancy may be purely activation-geometric (invisible in weights).
6. **Falsifier:** same protocol as G06 with layer-deletion damage as target; R² < 0.2 kills it.

## Cluster C — Replication and method-agreement as first-class science

### G10 ⭐ Noise-floor atlas (seed/dataset variance of every map type)
1. **Existing:** isolated numbers — ~30% SAE seed overlap, 1–5% universal neurons, probe-control literature.
2. **Missing:** a systematic noise-floor table: for each map type (probe, head-importance, feature, circuit), the distribution of map-to-map distance under seed/data resampling, on fixed checkpoints.
3. **Matters:** it is the denominator of every claim in the atlas; the field cites the SAE number precisely because nothing else exists.
4. **Could work:** replication is embarrassingly parallel (MacLustr cluster!); metrics (Jaccard, rank correlation) are standard.
5. **Failure risk:** none scientific — cost only. This is expA/expD formalized.
6. **Falsifier:** n/a (it *is* the calibration); the interesting outcome is which map types have floors low enough to build on.

### G11 Method-agreement scoring (map = intersection of techniques)
1. **Existing:** MIB compares methods on benchmarks; pairwise agreement studies are ad hoc.
2. **Missing:** agreement *as the map*: publish per-unit agreement across probe/erasure/patching (Level-2 = method-robust made operational and quantitative).
3. **Matters:** operationalizes our confidence taxonomy; agreement rate itself calibrates how much any single method over-claims.
4. **Could work:** all three methods are L–M cost; agreement statistics are simple; charter already demands ≥2 techniques for Level 2.
5. **Failure risk:** agreement near chance everywhere (charter §17 failure criterion) — would force a pivot but is a Level-1 publishable negative.
6. **Falsifier:** language-identity capability on a 1B bilingual model: probe-top-k vs LEACE-damage vs ablation-damage unit sets. Cohen's κ ≤ 0 across methods = methods measure different things; the "map" concept needs redefinition.

### G12 Correlational→causal survival rate as a published number
1. **Existing:** anecdotes (Hase; IOI ~87%); no systematic conversion rates per technique.
2. **Missing:** run the pipeline correlational-finding → causal test at scale and publish survival rates per (technique, capability, model).
3. **Matters:** the single number that tells the field how much to discount correlational maps; calibrates all our Level assignments (expC).
4. **Could work:** we generate correlational findings in bulk anyway (G10/G11); verification is patching/ablation at M-cost.
5. **Failure risk:** survival rate may depend heavily on thresholds — pre-register thresholds to avoid garden-of-forking-paths.
6. **Falsifier:** n/a (measurement); risk-case is rates so threshold-sensitive they're meaningless — detectable by sensitivity analysis.

### G13 Ablation-scheme robustness curves as the artifact standard
1. **Existing:** Miller et al. showed faithfulness isn't robust; recommendations exist; nobody *publishes* curves.
2. **Missing:** every importance/faithfulness claim shipped as a curve over {zero, mean, resample} × {noising, denoising} instead of one number.
3. **Matters:** turns a known critique into a schema requirement; scheme-sensitivity itself becomes a stability metric per map.
4. **Could work:** auto-circuit already implements efficient multi-scheme ablation; cost multiplier ≈ 6× on L–M experiments.
5. **Failure risk:** none scientific; adoption/curation cost only.
6. **Falsifier:** n/a (standard); informative outcome = which map types are scheme-stable.

## Cluster D — Cross-model coordinate systems

### G14 ⭐ Landmark-anchored cross-model alignment
1. **Existing:** induction heads universal; universal neurons (seeds); SAE feature matching (noisy); PRH deflated to local alignment.
2. **Missing:** a coordinate system *built from verified landmarks* (induction heads, successor heads, super weights, entropy neurons) rather than global similarity metrics; alignment quality scored by landmark correspondence.
3. **Matters:** Q3 directly; global metrics are debunked-in-part — landmarks are the surviving signal.
4. **Could work:** landmarks are L-cost to detect and causally verifiable; within-family (shared tokenizer) removes the alignment confound.
5. **Failure risk:** too few landmarks (~dozens) for a useful coordinate system → coordinates only near landmarks.
6. **Falsifier:** Qwen3 0.6B vs 1.7B: detect landmarks independently, test whether landmark-anchored layer mapping beats depth-proportional mapping at predicting held-out landmark positions. No improvement = anchoring adds nothing.

### G15 Rosetta-mining on LLM pairs
1. **Existing:** Rosetta Neurons (vision, 8 models); Gurnee universal neurons (same architecture, different seeds).
2. **Missing:** correlation-mining of shared units across *different-size* LLMs of one family over a shared corpus.
3. **Matters:** direct, assumption-light candidate for Q3; complements G14 (dense where G14 is sparse).
4. **Could work:** method is simple (normalized cross-correlation over shared stimuli); within-family tokenizers align.
5. **Failure risk:** neuron-level correspondence may be mostly absent above the 1–5% universal set (the honest prior).
6. **Falsifier:** Pythia-160M vs 410M over 10M shared tokens: if matched-unit rate ≈ shuffled-baseline rate, neuron-level cross-scale correspondence is dead; move to feature-level (SAE) matching.

### G16 Cross-scale capability emergence maps (expG formalized)
1. **Existing:** Tigges (circuits consistent across training/scale, Pythia ≤2.8B); Lieberum (70B case study).
2. **Missing:** capability-resolved emergence maps: for each capability (code, FR, arithmetic), at which size does localized structure appear, and does its *location* (relative depth) stay put?
3. **Matters:** turns "emergence" folklore into mapped, versioned claims; the 70M–2.8B band is fully Mac-feasible.
4. **Could work:** Pythia/Qwen ladders exist with shared tokenizers; probes + ablation at each size are L–M.
5. **Failure risk:** capability thresholds may be corpus-artifacts; needs behavioral gating first.
6. **Falsifier:** if relative-depth location of a capability's probe peak wanders arbitrarily across sizes (no correlation), cross-scale "location" is meaningless and the atlas should never draw it.

## Cluster E — Localization science (Q1)

### G17 Formal localization-vs-diffusion index
1. **Existing:** ad hoc top-k ablation curves; "X is localized" claims without a shared measure.
2. **Missing:** a standard concentration index per (capability, granularity): area-under-damage-curve for top-k units, Gini of importance mass, with nulls and CIs — comparable across models and map types.
3. **Matters:** Q1's answer must be a *number with a distribution*, not an adjective; enables cross-capability comparison (code vs FR vs facts).
4. **Could work:** damage curves are cheap (ablations, L); index properties testable on planted Tracr circuits (known ground truth: fully localized).
5. **Failure risk:** hydra/backup compensation corrupts single-unit damage curves → must use resample ablation + joint ablations; index may be granularity-dependent (report per granularity).
6. **Falsifier:** on Tracr models the index must read ≈1 (fully localized); on random-init models ≈0. If it can't separate these, the index is broken before touching real models.

### G18 Multilingual (FR/EN) localization atlas
1. **Existing:** language-agnostic substrate results; multilingual editing transfer studies; no versioned atlas, nothing FR-focused.
2. **Missing:** enrichment/extraction maps per language on one multilingual model, causally verified, with overlap coefficients (shared vs language-specific units).
3. **Matters:** clean Q1 instance with personal corpus advantage (native FR promptsets); editing-transfer literature provides validation targets.
4. **Could work:** Geva pipeline replicates multilingually (2025); 7–8B multilingual models fit a 64 GB Mac.
5. **Failure risk:** FR/EN too close (shared script/vocabulary) — add a distant-language control (e.g. zh) to calibrate.
6. **Falsifier:** if FR-specific unit sets under resample ablation damage EN equally (no selectivity), language localization at unit granularity is falsified for this model class.

### G19 Editing as perturbation probe (not localization evidence)
1. **Existing:** ROME→Hase established the negative; editing evaluation critiques.
2. **Missing:** using edits *correctly*: edit at layer L, measure the full ripple field (what else moved: probes, features, behavior) as a *response map* of the network.
3. **Matters:** turns the field's failed inference into a valid instrument — perturbation-response mapping is standard physics practice.
4. **Could work:** EasyEdit-class edits are cheap once covariance stats exist; response = pre/post map deltas we already compute.
5. **Failure risk:** sequential-collapse contamination — use fresh checkpoints per edit; MLX port effort (CUDA-first repos).
6. **Falsifier:** if response maps are indistinguishable across edit layers (uniform ripple), edits carry no spatial information at all — strengthening Hase into a stronger negative.

## Cluster F — Systems and tooling (Q4)

### G20 ⭐ Capture cost frontier on Apple Silicon (expH formalized)
1. **Existing:** zero published macOS numbers (hook overhead MLX vs MPS, mmap vs zarr vs safetensors on APFS, streaming SAE throughput).
2. **Missing:** the measured "what can you map on which Mac" table; every feasibility estimate in state_of_the_art.md is unverified.
3. **Matters:** gates everything else; is itself a publishable atlas artifact; extends localvm's SSD results to the capture workload.
4. **Could work:** benchmarking is deterministic engineering; localvm harness patterns reusable.
5. **Failure risk:** none scientific; risk is version-fragility (pin MLX/PyTorch versions in manifests).
6. **Falsifier:** n/a (measurement). Success = published table with CIs; the localvm SSD ceiling (13.1 GB/s ≥256 KiB QD≥4) predicts mmap wins — testable.

### G21 Buffer-shuffle vs full-shuffle SAE replication
1. **Existing:** Anthropic full-shuffles on disk; SAELens/sparsify buffer-shuffle; effect on feature replication unpublished.
2. **Missing:** does Mac-feasible streaming (small shuffle buffer) degrade seed-replication or feature quality vs full shuffle?
3. **Matters:** determines whether local SAE training is *scientifically* viable or systematically biased; informs everyone training SAEs on one GPU.
4. **Could work:** at 0.5B scale both regimes are runnable locally (full shuffle of a 10⁸-token store fits on SSD).
5. **Failure risk:** effect may interact with width/architecture — fix one recipe (TopK) first.
6. **Falsifier:** train 5+5 seeds (buffer vs full shuffle) on Pythia-160M L6; if replication-rate distributions are indistinguishable, buffer streaming is vindicated (positive result for the field).

### G22 MLX-native capture/intervention layer
1. **Existing:** mlxterp (embryonic, no store, no provenance); TransformerLens MPS opt-in; NNsight remote philosophy.
2. **Missing:** a hardened MLX capture+intervention layer with streaming mmap store, provenance embedding, quantized-model support — the substrate for all our experiments.
3. **Matters:** without it every experiment pays PyTorch-MPS tax and can't touch quantized models pythonically.
4. **Could work:** MLX modules are plain Python (wrap-based hooks); laziness makes retention cheap; mlxterp proves the pattern.
5. **Failure risk:** MLX API churn; mitigation = thin layer + pinned versions + upstream issues.
6. **Falsifier:** n/a (tooling); acceptance test = capture overhead <2× plain inference at 3B (measured via G20 harness).

## Cluster G — Atlas methodology and utility

### G23 ⭐ Map cards: machine-readable provenance + confidence schema
1. **Existing:** datasheets/model cards (data/models); Gemma Scope versioning (no confidence); Neuronpedia hosting (no reproducibility contract).
2. **Missing:** a schema where every map ships {commit, config, model hash+quant, promptset checksums, hardware manifest, seeds, controls run, replication rate, ablation curves, confidence level} and a regeneration command — enforced by tooling (publish.py refuses incomplete cards).
3. **Matters:** the charter's core deliverable; §4.10 verified nothing like it exists; it is what makes negative results publishable.
4. **Could work:** schema design + enforcement is engineering; fields all fall out of Phase 2 conclusions.
5. **Failure risk:** over-engineering before content exists — co-evolve with the first real maps (expA output = first map card).
6. **Falsifier:** n/a (standard); external test = a third party regenerates a map from its card alone.

### G24 Working-set maps for out-of-core execution (localvm bridge, Q5)
1. **Existing:** localvm measures SSD/execution side; MoE/routing literature; contextual-sparsity literature (Deja Vu-style) is CUDA-centric and predictor-focused.
2. **Missing:** *maps* of token-conditional weight relevance (which blocks matter for which input regime), built with interpretability instruments (ablation, attribution) and validated as *prefetch predictions* on localvm's engine.
3. **Matters:** breakthrough criterion E (charter §16): interpretability that materially improves systems decisions — the strongest cross-project payoff.
4. **Could work:** per-block ablation damage conditioned on input domain is measurable at L–M; localvm provides the consumer and the metric (bytes/token saved at fixed quality).
5. **Failure risk:** relevance may be too input-specific (no stable per-domain working sets) — itself a decisive result for localvm's architecture choices.
6. **Falsifier:** on a 3B model: domain-conditioned block-ablation maps (code vs FR prose). If domain-conditional working sets are no smaller than global ones (<10% reduction), per-domain prefetching dies for this model class.

---

## Coverage check against charter §6 example categories

quantization-aware maps (G01–G05) · weight-only pre-screens (G06–G09) ·
cross-model coordinates (G14–G16) · systems-property prediction / localvm
bridge (G09, G24) · method-agreement (G11) · longitudinal/cross-scale (G16) ·
multilingual atlases (G18) · local SAE recipes (G21–G22) · scaled-down causal
verification (G12, G17, G19) · uncertainty-first schema (G13, G23).
All example categories are covered; G05, G08, G10, G19 fall outside them.

## Note on independence

The 24 gaps reduce to roughly five falsification kernels (drift-vs-noise-floor;
weight→activation regression; agreement/survival rates; landmark alignment;
domain-conditioned ablation) — deliberately: shared kernels mean shared
harnesses, and one experiment often feeds several gaps. Phase 4 ranks the
gaps, not the kernels.