spb/modelmap Public License

1---2project: modelmap3document: Candidate ranking (Phase 4)4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128modified: 2026-08-129status: reviewed10---1112# Candidate ranking — 24 gaps scored on ten axes1314Phase 4 deliverable (charter §7). Every gap from `research_gaps.md` scored 1–1015per axis. **Convention: 10 is always favorable** — for *Implementation16complexity*, 10 = simple; for *Risk*, 10 = low risk. Total = sum (max 100).17Scores are prior judgments to be revised as evidence lands; the reasoning18column is the commitment, the numbers are its summary.1920Axes: **SV** scientific value · **ES** epistemic soundness · **NV** novelty ·21**FA** feasibility on Apple Silicon · **RP** reproducibility potential ·22**GN** cross-model generality · **AV** public atlas value · **IC** implementation23complexity (10=simple) · **SY** synergy with localvm · **RK** risk (10=low).2425## Score table (sorted by total)2627| Gap | SV | ES | NV | FA | RP | GN | AV | IC | SY | RK | **Total** |28|---|---|---|---|---|---|---|---|---|---|---|---|29| G23 map cards schema | 7 | 8 | 9 | 10 | 10 | 10 | 10 | 8 | 6 | 9 | **87** |30| G20 capture cost frontier (expH) | 6 | 9 | 9 | 10 | 9 | 7 | 9 | 7 | 10 | 10 | **86** |31| G17 localization index | 9 | 9 | 8 | 9 | 9 | 9 | 9 | 7 | 6 | 8 | **83** |32| G10 noise-floor atlas (expA/D) | 9 | 8 | 8 | 8 | 10 | 9 | 9 | 7 | 5 | 9 | **82** |33| G06 weight→activation regression | 10 | 7 | 9 | 9 | 9 | 8 | 9 | 7 | 8 | 5 | **81** |34| G08 super-weight cartography | 6 | 9 | 6 | 10 | 10 | 8 | 8 | 9 | 5 | 9 | **80** |35| G01 probe-drift across quantization | 8 | 7 | 8 | 9 | 9 | 8 | 9 | 8 | 5 | 8 | **79** |36| G11 method-agreement maps | 9 | 9 | 8 | 8 | 9 | 8 | 9 | 6 | 5 | 7 | **78** |37| G03 weight-geometry → quant sensitivity | 9 | 6 | 9 | 9 | 8 | 8 | 8 | 7 | 6 | 6 | **76** |38| G12 correlational→causal survival rate | 9 | 10 | 8 | 7 | 8 | 8 | 9 | 6 | 4 | 7 | **76** |39| G13 ablation-scheme curves standard | 6 | 8 | 6 | 9 | 9 | 9 | 8 | 8 | 3 | 9 | **75** |40| G07 base↔instruct delta atlas | 6 | 4 | 7 | 10 | 10 | 8 | 8 | 9 | 4 | 8 | **74** |41| G24 working-set maps (localvm bridge) | 9 | 8 | 9 | 7 | 7 | 6 | 8 | 5 | 10 | 5 | **74** |42| G02 circuit survival under quantization | 8 | 8 | 9 | 7 | 8 | 8 | 8 | 6 | 4 | 7 | **73** |43| G09 cheap maps → pruning tolerance | 7 | 7 | 7 | 8 | 8 | 7 | 7 | 7 | 9 | 6 | **73** |44| G14 landmark-anchored alignment | 8 | 8 | 8 | 8 | 8 | 9 | 8 | 6 | 4 | 6 | **73** |45| G21 shuffle-regime SAE replication | 8 | 8 | 9 | 7 | 9 | 7 | 7 | 6 | 5 | 7 | **73** |46| G05 quantized-runtime capture (llama.cpp) | 7 | 7 | 9 | 7 | 8 | 9 | 7 | 4 | 7 | 6 | **71** |47| G16 cross-scale emergence maps | 8 | 7 | 7 | 8 | 8 | 8 | 8 | 6 | 4 | 6 | **70** |48| G22 MLX capture layer | 5 | 6 | 7 | 8 | 8 | 8 | 6 | 5 | 8 | 7 | **68** |49| G18 FR/EN localization atlas | 7 | 8 | 6 | 8 | 8 | 6 | 8 | 6 | 3 | 7 | **67** |50| G19 editing as perturbation probe | 8 | 8 | 8 | 6 | 7 | 7 | 7 | 5 | 4 | 6 | **66** |51| G04 tuned-lens drift metric | 6 | 5 | 7 | 9 | 7 | 7 | 6 | 8 | 3 | 5 | **63** |52| G15 Rosetta-mining on LLM pairs | 7 | 6 | 7 | 7 | 7 | 8 | 7 | 6 | 3 | 5 | **63** |5354## Reasoning (grouped; scores that drive the ranking)5556**G23, G20, G10, G13 — the infrastructure quartet.** These top the table not57because they are the boldest science but because every other row *presupposes*58them: G23 (map cards) is the charter's core deliverable and verified absent in59the field (NV 9, AV 10); G20 is expH — nothing else's feasibility column is60*measured* until it runs, and it reuses localvm's harness (SY 10, RK 10); G10 is61expA/expD — the denominator of every future claim (RP 10); G13 is a cheap62standard that upgrades everything else's epistemics. **Decision: these are not63"candidates" to bet on — they are the mandatory substrate, scheduled first64regardless of ranking.** Their risk is near-zero and their outputs are atlas65artifacts in their own right (cost-frontier table, noise-floor table, schema).6667**G17 (83) — the strongest research candidate.** Q1 needs a *number*, not an68adjective; the index is Tracr-falsifiable before any real model (ES 9), cheap69(ablation curves, FA 9), general across models and granularities (GN 9), and70its outputs are exactly the comparable, versionable artifacts the charter wants71(AV 9). Main deduction: hydra-effect contamination requires resample + joint72ablations (IC 7).7374**G06 (81) — highest scientific value in the table (SV 10).** If weight-only75statistics predict activation-level maps, the accuracy/cost frontier of local76interpretability moves for everyone; if they don't, a harsh pre-registered77negative (R²<0.2) is still a Level-1 publication. RK 5 is the honest price of a78real hypothesis. Absorbs G03/G08/G09 as sub-studies (same regression harness,79different targets) — G08's causal verifiability (ES 9) makes it the anchor80sub-study to run first.8182**G01 (79) + G02 (73) — the quantization cluster.** G01 is the entry point:83highest feasibility of the cluster (FA 9), clean controls, and the atlas story84("the same model, mapped at four precisions") is uniquely ours (AV 9). G0285follows once G01's noise floors exist; its FA 7 reflects the86gradients-through-quantized-weights problem. G04 (63) and G05 (71) stay in the87cluster's backlog: G04 until lens seed-variance is measured, G05 because IC 488(C-level engineering) makes it a tooling milestone rather than a first bet.8990**G11 (78) + G12 (76) — the verification science.** Both are epistemically91top-shelf (ES 9–10) and both are largely *byproducts* of running G17+G1092honestly: agreement rates and survival rates fall out of the same runs.93Scheduled as measurement layers on candidate experiments rather than94standalone prototypes — this is why their totals do not need to beat G17's.9596**G24 (74) — the localvm bridge.** SV 9 / SY 10 / NV 9, but RK 5 (working sets97may not be domain-stable) and IC 5 (needs localvm's engine as consumer). Kept98as the fourth candidate at reduced scope: run the falsifier (domain-conditioned99block-ablation on a 3B model) before committing to the full bridge.100101**G07 (74) and G08 (80) — high totals, bounded ambition.** Both are cheap,102reproducible atlas layers (RP 10) rather than research programs; they are103folded into candidate work as first map deliverables (G08 → candidate 1's104anchor; G07 → an early standalone atlas product to exercise the map-card105pipeline end-to-end).106107**Deferred (G14 73, G21 73, G16 70, G05 71, G22 68, G18 67, G19 66, G15 63,108G04 63).** Cross-model coordinates (G14/G15/G16) are blocked on G10's noise109floors by design — alignment claims without floors are exactly the field's110failure mode. G21 waits for a working SAE recipe (itself gated by G20's111throughput numbers). G22 is built incrementally as experiments demand it, not112as a big-bang library. G18 is a strong *application* of G17 once the index113exists. G19 is deferred on port cost (CUDA-first editing stacks). Nothing here114is rejected; the ranking sequences, it does not kill.115116## Selected prototype candidates117118| Slot | Program | Core gaps | First falsifier |119|---|---|---|---|120| **candidate_01** | **Weight-only pre-screen** — weight statistics vs activation-level ground truth | G06 (+G08 anchor, G03/G09 extensions) | super-weight causal catalog on 2 models; then Pythia-410M regression, kill at CV R²<0.2 |121| **candidate_02** | **Quantization deformation atlas** — the same model mapped at FP16/Q8/Q4(/Q2) | G01 (+G02, then G04/G05) | Qwen3-0.6B FP16 vs Q4 probe drift vs noise floor, 10 properties × 5 seeds × 2 datasets |122| **candidate_03** | **Localization index** — a Tracr-validated concentration measure for Q1 | G17 (+G11/G12 as measurement layers) | index must read ≈1 on Tracr, ≈0 on random-init; else broken before real models |123| **candidate_04** (reduced scope) | **Working-set bridge to localvm** | G24 (+G09) | domain-conditioned block-ablation on a 3B model; <10% working-set reduction kills per-domain prefetch |124125**Mandatory substrate scheduled before/alongside candidates:** G20 (expH),126G10 (expA/expD), G23 (map cards v0 co-evolved with expA's first output),127G13 (adopted as schema rule, zero research cost).128129**Sequencing rationale.** The three main candidates share the substrate but130fail independently (weights-regression, drift-vs-floor, index-validation are131different kernels), satisfying the charter's demand for 3–5 genuinely distinct132bets. Every candidate produces versioned map artifacts even in its negative133outcome — the property the charter's §7 explicitly prefers.134