|
1 |
+--- |
|
2 |
+project: modelmap |
|
3 |
+document: Candidate ranking (Phase 4) |
|
4 |
+author: Simon-Pierre Boucher |
|
5 |
+contact: contact@spboucher.ai |
|
6 |
+website: https://modelmap.io |
|
7 |
+created: 2026-08-12 |
|
8 |
+modified: 2026-08-12 |
|
9 |
+status: reviewed |
|
10 |
+--- |
|
11 |
+ |
|
12 |
+# Candidate ranking — 24 gaps scored on ten axes |
|
13 |
+ |
|
14 |
+Phase 4 deliverable (charter §7). Every gap from `research_gaps.md` scored 1–10 |
|
15 |
+per axis. **Convention: 10 is always favorable** — for *Implementation |
|
16 |
+complexity*, 10 = simple; for *Risk*, 10 = low risk. Total = sum (max 100). |
|
17 |
+Scores are prior judgments to be revised as evidence lands; the reasoning |
|
18 |
+column is the commitment, the numbers are its summary. |
|
19 |
+ |
|
20 |
+Axes: **SV** scientific value · **ES** epistemic soundness · **NV** novelty · |
|
21 |
+**FA** feasibility on Apple Silicon · **RP** reproducibility potential · |
|
22 |
+**GN** cross-model generality · **AV** public atlas value · **IC** implementation |
|
23 |
+complexity (10=simple) · **SY** synergy with localvm · **RK** risk (10=low). |
|
24 |
+ |
|
25 |
+## Score table (sorted by total) |
|
26 |
+ |
|
27 |
+| Gap | SV | ES | NV | FA | RP | GN | AV | IC | SY | RK | **Total** | |
|
28 |
+|---|---|---|---|---|---|---|---|---|---|---|---| |
|
29 |
+| G23 map cards schema | 7 | 8 | 9 | 10 | 10 | 10 | 10 | 8 | 6 | 9 | **87** | |
|
30 |
+| G20 capture cost frontier (expH) | 6 | 9 | 9 | 10 | 9 | 7 | 9 | 7 | 10 | 10 | **86** | |
|
31 |
+| G17 localization index | 9 | 9 | 8 | 9 | 9 | 9 | 9 | 7 | 6 | 8 | **83** | |
|
32 |
+| G10 noise-floor atlas (expA/D) | 9 | 8 | 8 | 8 | 10 | 9 | 9 | 7 | 5 | 9 | **82** | |
|
33 |
+| G06 weight→activation regression | 10 | 7 | 9 | 9 | 9 | 8 | 9 | 7 | 8 | 5 | **81** | |
|
34 |
+| G08 super-weight cartography | 6 | 9 | 6 | 10 | 10 | 8 | 8 | 9 | 5 | 9 | **80** | |
|
35 |
+| G01 probe-drift across quantization | 8 | 7 | 8 | 9 | 9 | 8 | 9 | 8 | 5 | 8 | **79** | |
|
36 |
+| G11 method-agreement maps | 9 | 9 | 8 | 8 | 9 | 8 | 9 | 6 | 5 | 7 | **78** | |
|
37 |
+| G03 weight-geometry → quant sensitivity | 9 | 6 | 9 | 9 | 8 | 8 | 8 | 7 | 6 | 6 | **76** | |
|
38 |
+| G12 correlational→causal survival rate | 9 | 10 | 8 | 7 | 8 | 8 | 9 | 6 | 4 | 7 | **76** | |
|
39 |
+| G13 ablation-scheme curves standard | 6 | 8 | 6 | 9 | 9 | 9 | 8 | 8 | 3 | 9 | **75** | |
|
40 |
+| G07 base↔instruct delta atlas | 6 | 4 | 7 | 10 | 10 | 8 | 8 | 9 | 4 | 8 | **74** | |
|
41 |
+| G24 working-set maps (localvm bridge) | 9 | 8 | 9 | 7 | 7 | 6 | 8 | 5 | 10 | 5 | **74** | |
|
42 |
+| G02 circuit survival under quantization | 8 | 8 | 9 | 7 | 8 | 8 | 8 | 6 | 4 | 7 | **73** | |
|
43 |
+| G09 cheap maps → pruning tolerance | 7 | 7 | 7 | 8 | 8 | 7 | 7 | 7 | 9 | 6 | **73** | |
|
44 |
+| G14 landmark-anchored alignment | 8 | 8 | 8 | 8 | 8 | 9 | 8 | 6 | 4 | 6 | **73** | |
|
45 |
+| G21 shuffle-regime SAE replication | 8 | 8 | 9 | 7 | 9 | 7 | 7 | 6 | 5 | 7 | **73** | |
|
46 |
+| G05 quantized-runtime capture (llama.cpp) | 7 | 7 | 9 | 7 | 8 | 9 | 7 | 4 | 7 | 6 | **71** | |
|
47 |
+| G16 cross-scale emergence maps | 8 | 7 | 7 | 8 | 8 | 8 | 8 | 6 | 4 | 6 | **70** | |
|
48 |
+| G22 MLX capture layer | 5 | 6 | 7 | 8 | 8 | 8 | 6 | 5 | 8 | 7 | **68** | |
|
49 |
+| G18 FR/EN localization atlas | 7 | 8 | 6 | 8 | 8 | 6 | 8 | 6 | 3 | 7 | **67** | |
|
50 |
+| G19 editing as perturbation probe | 8 | 8 | 8 | 6 | 7 | 7 | 7 | 5 | 4 | 6 | **66** | |
|
51 |
+| G04 tuned-lens drift metric | 6 | 5 | 7 | 9 | 7 | 7 | 6 | 8 | 3 | 5 | **63** | |
|
52 |
+| G15 Rosetta-mining on LLM pairs | 7 | 6 | 7 | 7 | 7 | 8 | 7 | 6 | 3 | 5 | **63** | |
|
53 |
+ |
|
54 |
+## Reasoning (grouped; scores that drive the ranking) |
|
55 |
+ |
|
56 |
+**G23, G20, G10, G13 — the infrastructure quartet.** These top the table not |
|
57 |
+because they are the boldest science but because every other row *presupposes* |
|
58 |
+them: G23 (map cards) is the charter's core deliverable and verified absent in |
|
59 |
+the field (NV 9, AV 10); G20 is expH — nothing else's feasibility column is |
|
60 |
+*measured* until it runs, and it reuses localvm's harness (SY 10, RK 10); G10 is |
|
61 |
+expA/expD — the denominator of every future claim (RP 10); G13 is a cheap |
|
62 |
+standard that upgrades everything else's epistemics. **Decision: these are not |
|
63 |
+"candidates" to bet on — they are the mandatory substrate, scheduled first |
|
64 |
+regardless of ranking.** Their risk is near-zero and their outputs are atlas |
|
65 |
+artifacts in their own right (cost-frontier table, noise-floor table, schema). |
|
66 |
+ |
|
67 |
+**G17 (83) — the strongest research candidate.** Q1 needs a *number*, not an |
|
68 |
+adjective; the index is Tracr-falsifiable before any real model (ES 9), cheap |
|
69 |
+(ablation curves, FA 9), general across models and granularities (GN 9), and |
|
70 |
+its outputs are exactly the comparable, versionable artifacts the charter wants |
|
71 |
+(AV 9). Main deduction: hydra-effect contamination requires resample + joint |
|
72 |
+ablations (IC 7). |
|
73 |
+ |
|
74 |
+**G06 (81) — highest scientific value in the table (SV 10).** If weight-only |
|
75 |
+statistics predict activation-level maps, the accuracy/cost frontier of local |
|
76 |
+interpretability moves for everyone; if they don't, a harsh pre-registered |
|
77 |
+negative (R²<0.2) is still a Level-1 publication. RK 5 is the honest price of a |
|
78 |
+real hypothesis. Absorbs G03/G08/G09 as sub-studies (same regression harness, |
|
79 |
+different targets) — G08's causal verifiability (ES 9) makes it the anchor |
|
80 |
+sub-study to run first. |
|
81 |
+ |
|
82 |
+**G01 (79) + G02 (73) — the quantization cluster.** G01 is the entry point: |
|
83 |
+highest feasibility of the cluster (FA 9), clean controls, and the atlas story |
|
84 |
+("the same model, mapped at four precisions") is uniquely ours (AV 9). G02 |
|
85 |
+follows once G01's noise floors exist; its FA 7 reflects the |
|
86 |
+gradients-through-quantized-weights problem. G04 (63) and G05 (71) stay in the |
|
87 |
+cluster's backlog: G04 until lens seed-variance is measured, G05 because IC 4 |
|
88 |
+(C-level engineering) makes it a tooling milestone rather than a first bet. |
|
89 |
+ |
|
90 |
+**G11 (78) + G12 (76) — the verification science.** Both are epistemically |
|
91 |
+top-shelf (ES 9–10) and both are largely *byproducts* of running G17+G10 |
|
92 |
+honestly: agreement rates and survival rates fall out of the same runs. |
|
93 |
+Scheduled as measurement layers on candidate experiments rather than |
|
94 |
+standalone prototypes — this is why their totals do not need to beat G17's. |
|
95 |
+ |
|
96 |
+**G24 (74) — the localvm bridge.** SV 9 / SY 10 / NV 9, but RK 5 (working sets |
|
97 |
+may not be domain-stable) and IC 5 (needs localvm's engine as consumer). Kept |
|
98 |
+as the fourth candidate at reduced scope: run the falsifier (domain-conditioned |
|
99 |
+block-ablation on a 3B model) before committing to the full bridge. |
|
100 |
+ |
|
101 |
+**G07 (74) and G08 (80) — high totals, bounded ambition.** Both are cheap, |
|
102 |
+reproducible atlas layers (RP 10) rather than research programs; they are |
|
103 |
+folded into candidate work as first map deliverables (G08 → candidate 1's |
|
104 |
+anchor; G07 → an early standalone atlas product to exercise the map-card |
|
105 |
+pipeline end-to-end). |
|
106 |
+ |
|
107 |
+**Deferred (G14 73, G21 73, G16 70, G05 71, G22 68, G18 67, G19 66, G15 63, |
|
108 |
+G04 63).** Cross-model coordinates (G14/G15/G16) are blocked on G10's noise |
|
109 |
+floors by design — alignment claims without floors are exactly the field's |
|
110 |
+failure mode. G21 waits for a working SAE recipe (itself gated by G20's |
|
111 |
+throughput numbers). G22 is built incrementally as experiments demand it, not |
|
112 |
+as a big-bang library. G18 is a strong *application* of G17 once the index |
|
113 |
+exists. G19 is deferred on port cost (CUDA-first editing stacks). Nothing here |
|
114 |
+is rejected; the ranking sequences, it does not kill. |
|
115 |
+ |
|
116 |
+## Selected prototype candidates |
|
117 |
+ |
|
118 |
+| Slot | Program | Core gaps | First falsifier | |
|
119 |
+|---|---|---|---| |
|
120 |
+| **candidate_01** | **Weight-only pre-screen** — weight statistics vs activation-level ground truth | G06 (+G08 anchor, G03/G09 extensions) | super-weight causal catalog on 2 models; then Pythia-410M regression, kill at CV R²<0.2 | |
|
121 |
+| **candidate_02** | **Quantization deformation atlas** — the same model mapped at FP16/Q8/Q4(/Q2) | G01 (+G02, then G04/G05) | Qwen3-0.6B FP16 vs Q4 probe drift vs noise floor, 10 properties × 5 seeds × 2 datasets | |
|
122 |
+| **candidate_03** | **Localization index** — a Tracr-validated concentration measure for Q1 | G17 (+G11/G12 as measurement layers) | index must read ≈1 on Tracr, ≈0 on random-init; else broken before real models | |
|
123 |
+| **candidate_04** (reduced scope) | **Working-set bridge to localvm** | G24 (+G09) | domain-conditioned block-ablation on a 3B model; <10% working-set reduction kills per-domain prefetch | |
|
124 |
+ |
|
125 |
+**Mandatory substrate scheduled before/alongside candidates:** G20 (expH), |
|
126 |
+G10 (expA/expD), G23 (map cards v0 co-evolved with expA's first output), |
|
127 |
+G13 (adopted as schema rule, zero research cost). |
|
128 |
+ |
|
129 |
+**Sequencing rationale.** The three main candidates share the substrate but |
|
130 |
+fail independently (weights-regression, drift-vs-floor, index-validation are |
|
131 |
+different kernels), satisfying the charter's demand for 3–5 genuinely distinct |
|
132 |
+bets. Every candidate produces versioned map artifacts even in its negative |
|
133 |
+outcome — the property the charter's §7 explicitly prefers. |
|
134 |
|