SPB Git

spb/modelmap Public License

Internal cartography of local LLMs on Apple Silicon — registered, gated, negative-first. Public atlas at modelmap.io.

🔗 https://modelmap.io
interpretabilityllmapple-siliconmlxreproducibility
Python 66.3% JavaScript 24.5% CSS 8.1% Shell 0.7%

Publications tab + TR-2026-01: first official technical report

- /publications listing + article pages: abstract box, byline, status
  badge, live-rendered figures via {{fig:*}} tokens (7 figures from the
  same versioned artifacts as the atlas), how-to-cite block
- TR-2026-01 'Registered, Gated, Negative-First' — full synthesis of
  expH/expA/expC results, gate record, survival ledger, limitations,
  reproducibility section

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 1 h ago (Aug 12, 2026) parent b7e357e

Showing 5 changed files with +314 and −2

added publications/TR-2026-01.md +210 −0
@@ -0,0 +1,210 @@
1 +---
2 +project: modelmap
3 +document: TR-2026-01 — technical report
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +website: https://modelmap.io
7 +created: 2026-08-12
8 +status: final
9 +pub_id: TR-2026-01
10 +title: "Registered, Gated, Negative-First: Standing Up a Reproducible Cartography Pipeline for a Local LLM in One Day"
11 +abstract: "We report the first results of modelmap, a research program that maps the internal structure of open-weight language models entirely on consumer Apple Silicon, under a discipline of pre-registered hypotheses, mandatory nulls, and machine-enforced publication gates. In eleven registered runs across three micro-experiments on a single day, the pipeline (i) measured the systems substrate of local interpretability — activation capture is nearly free under MLX lazy evaluation (1.004× plain prefill on a real 4-bit checkpoint) while activation-store format rankings invert between warm and cold cache, showing IO granularity rather than container choice governs throughput; (ii) demonstrated that per-layer linear probes on lexically separable classes are indistinguishable from a random-initialization architecture twin — a published negative result — and that token-balanced structural properties recover trained-model signal only as real-minus-twin differentials; and (iii) causally tested the resulting agreement map: its decodability ranking failed two intervention tests (correlational-to-causal survival 0/2), while a difference-of-means agreement direction proved robustly necessary across the early layer band (2–15), removing 73–75% of the model's grammatical preference under rank-1 erasure, replicated across six fresh estimators — published at Level 2. Two further pre-registered attempts to promote the claim to Level 3 (dose-response steering) were refused by the publication gate, including one that would have shipped a false claim had a single earlier observation been trusted. All figures below are rendered live from the same versioned artifacts as the public atlas."
12 +cite: |
13 + Boucher, S.-P. (2026). Registered, Gated, Negative-First: Standing Up a
14 + Reproducible Cartography Pipeline for a Local LLM in One Day.
15 + modelmap Technical Report TR-2026-01. https://modelmap.io/publications/TR-2026-01
16 +---
17 +
18 +## 1. Motivation and scope
19 +
20 +Most interpretability results are produced on CUDA clusters, reported without
21 +seed or dataset replication, and published without the failed attempts that
22 +surrounded them. The modelmap program inverts each of these defaults: all
23 +computation runs on consumer Apple Silicon (a MacBook Pro M5 Max, 48 GB, and
24 +one Mac Studio M3 Ultra used for a cold-cache pass), every experimental run is
25 +preceded by a **registered hypothesis with an explicit falsification
26 +criterion**, every observational map carries **mandatory nulls** (shuffled
27 +labels, random-initialization twins, random-direction controls), and atlas
28 +entries pass through a **machine-enforced publication gate** that refuses any
29 +map whose evidence does not meet its claimed confidence level.
30 +
31 +This report covers everything the pipeline produced on 2026-08-12 — the
32 +program's first full day of measurement — spanning three micro-experiments:
33 +capture-cost frontier (expH, 3 runs), probe reliability (expA, 2 runs), and
34 +causal verification (expC, 6 runs). The subject model throughout is
35 +**Qwen3-0.6B in 4-bit MLX quantization** (28 layers, d_model 1024): a genuinely
36 +*local* model, studied in the form people actually run.
37 +
38 +Confidence vocabulary (used throughout, and rendered as badges in the
39 +[atlas](/atlas)): **Level 0** anecdotal · **Level 1** correlational,
40 +controlled, replicated · **Level 2** method-robust (≥2 techniques agree) ·
41 +**Level 3** causal (intervention confirms).
42 +
43 +## 2. Systems substrate: what capture actually costs on a Mac (expH)
44 +
45 +Three registered runs established the engineering envelope for everything
46 +that follows.
47 +
48 +**Capture is nearly free under MLX.** Wrapping every decoder layer with a
49 +retaining tap costs 1.02× plain inference on a synthetic 12-layer model and
50 +**1.004×** on the real Qwen3-0.6B-4bit checkpoint (28.5 → 28.6 ms per
51 +901-token prefill); adding bf16→fp16 casting and streaming writes to an
52 +mmap store raises this to only 1.28×. The same taps on PyTorch-MPS cost
53 +1.22× (retention) to 1.47× (with copy+write) — eager hook materialization is
54 +measurably more expensive than lazy-graph retention. To our knowledge run #3
55 +is also the first Python-level activation capture from an actually-quantized
56 +local checkpoint, a capability the Phase-1 literature survey found in no
57 +existing tool.
58 +
59 +{{fig:capture}}
60 +
61 +**The storage hypothesis was falsified — instructively.** Run #1 (warm cache,
62 +M5 Max) found raw `mmap` beating chunked zarr by 3.2–10.8× on the
63 +random-batch read pattern that SAE-style training requires, and we registered
64 +the hypothesis that this ordering would survive a cold cache. Run #2 (M3
65 +Ultra, `purge` before every timed repetition) **inverted it**: cold, mmap
66 +collapses to page-fault IO (~8–16 KiB reads at queue depth 1, 0.14 GB/s)
67 +while zarr's 32 MiB chunk reads sustain 0.62 GB/s. The portable rule is that
68 +**IO granularity, not container format, governs activation-store
69 +throughput** — a conclusion that independently converges with the sister
70 +project localvm-research's cold-cache SSD measurements on different hardware.
71 +
72 +{{fig:storage}}
73 +
74 +*Design consequences adopted:* mmap-backed flat stores for in-RAM working
75 +sets; chunk-aligned batched access mandatory for larger-than-RAM stores; MLX
76 +as the preferred capture backend; cache regime reported as a first-class
77 +dimension of every cost table.
78 +
79 +## 3. Probing: the architecture null eats the map, then earns its keep (expA)
80 +
81 +**Run #1 — a published negative result.** On template-generated binary
82 +properties (French/English, code/prose, arithmetic-context), per-layer linear
83 +probes on mean-pooled representations hit ceiling (accuracy 1.000, all 28
84 +layers, all properties, both promptsets). The registered validity gate
85 +required a random-initialization *architecture twin* to sit below 0.05
86 +selectivity; instead **the twin also scored 1.00 everywhere** (mean
87 +real−twin selectivity within ±0.06). The map measured the tokenizer +
88 +architecture prior, not learned computation — and notably, the standard
89 +shuffled-label control passed everywhere and would never have caught this;
90 +only the twin null did. This became the atlas's first entry
91 +([probes/v1](/atlas/qwen3-0.6b-4bit/probes/v1), `negative_result: true`).
92 +
93 +{{fig:probes-null}}
94 +
95 +**Run #2 — differential maps on structure-borne properties.** Version-2
96 +promptsets remove lexical separability (classes share vocabulary; the
97 +property lives in structure; class token-overlap certificates ship in the
98 +corpus manifest). Outcomes, published per property in
99 +[probes/v2](/atlas/qwen3-0.6b-4bit/probes/v2) at Level 1:
100 +
101 +- **agreement** (subject–verb, is/are balanced across classes): trained-model
102 + signal on 25/28 layers (max real−twin selectivity +0.379; accuracy 0.967
103 + vs twin 0.729);
104 +- **arith_valid** (correct vs off-by-delta equations): accuracy 0.86–0.90 vs
105 + twin ~0.56–0.58, strongest in last-token representations — consistent with
106 + computation completing at the equals sign;
107 +- **word_order** (grammatical vs scrambled): **null-dominated** — the twin
108 + reaches 0.96 accuracy from surface statistics alone; flagged as invalid
109 + evidence of learned syntax at this granularity.
110 +
111 +{{fig:probes-agreement}}
112 +
113 +{{fig:probes-arith}}
114 +
115 +{{fig:probes-word-order}}
116 +
117 +Off the ceiling, the project's first real noise-floor numbers appeared:
118 +top-5-layer replication across probe seeds is ~0.54, and dataset shift
119 +exceeds seed variance on only 3–8 of 28 layers — neither noise source
120 +dominates, so both must be reported. Doctrine adopted: **probe maps are
121 +publishable only as real-minus-twin differentials**, and the twin null is a
122 +permanent fixture of every observational map.
123 +
124 +## 4. Causal verification: the survival ledger and the band map (expC)
125 +
126 +Six registered runs asked whether the agreement probe map means what it
127 +appears to mean.
128 +
129 +**The decodability ranking died twice (survival ledger 0/2).** Skipping the
130 +map's top-5 layers damaged the behavioral agreement margin *less* than
131 +skipping five random layers (run #1); normalizing by general NLL damage
132 +confirmed nothing in the skip family singles out the probe layers (run #2,
133 +P1). A surgical per-layer scan — erasing the difference-of-means agreement
134 +direction at each layer, netted against random-direction controls — produced
135 +a causal profile that **anti-correlates** with the probe ranking (Spearman
136 +ρ = −0.136). Where information is most decodable is not where computation is
137 +causally load-bearing: the Hase-class dissociation, measured end-to-end in
138 +one pipeline.
139 +
140 +**What survived instead: a band claim at Level 2.** The erasure scan's
141 +stable content is that removing the agreement direction at *any single
142 +layer* in the early band (2–15) destroys most of the behavior. A first
143 +per-layer version of this map was **refused** by the publication gate (run
144 +#3: mean pairwise ρ = 0.495 across five direction estimators, below the
145 +registered 0.7). Narrowed to band granularity with everything fresh — six
146 +new direction sources from disjoint promptset halves, a new behavioral
147 +bank — the claim passed decisively (run #4: early-band mean specific damage
148 ++3.23…+3.35 on a +4.45 baseline; spread 0.12 across six estimators). This is
149 +[interventions/v1](/atlas/qwen3-0.6b-4bit/interventions/v1), the atlas's
150 +first Level-2 entry.
151 +
152 +{{fig:causal-band}}
153 +
154 +**Level 3 was attempted twice and refused twice.** Dose-response steering
155 +(adding α·σ·u) at three early-band layers failed its conjunctive criterion
156 +(run #5) — layer 4 sits in an overdose regime where *any* ~2σ perturbation,
157 +random included, collapses the behavior — while layer 12 passed every gate
158 +with a textbook monotone dose curve. A minimal single-layer L12 claim was
159 +then re-registered with four fresh estimators and a third fresh bank (run
160 +#6): **it failed replication** — only 1/4 sources monotone, and the
161 +specificity control did not reproduce (random-direction |Δ| 3.36 vs bound
162 +1.14; run #5's passing value of 0.45 is now legible as sampling luck over
163 +three draws). Had run #5's observation been published directly, the atlas
164 +would contain a false Level-3 claim today. Final verdict: the agreement
165 +direction is **necessary but not a reliable additive handle**, and
166 +interventions/v1 stands as the arc's terminal claim.
167 +
168 +## 5. What one day establishes
169 +
170 +1. **The full charter loop runs end-to-end on consumer hardware**: register →
171 + capture → map → null-check → causally verify → gate → publish (or refuse),
172 + with wall-clock costs of seconds to minutes per run at 0.6B scale.
173 +2. **Nulls are not decorative.** The architecture twin killed a
174 + ceiling-perfect probe map that shuffled-label controls waved through; the
175 + random-direction and fresh-replication controls killed two plausible
176 + Level-3 claims.
177 +3. **Granularity is a claim.** Per-layer rankings failed replication where
178 + band-level claims passed; map artifacts must declare the resolution at
179 + which they are stable.
180 +4. **The measured correlational→causal survival rate is 0/2** for layer
181 + rankings on this model/behavior — a number the field rarely publishes and
182 + the reason this atlas labels every entry with its evidence level.
183 +5. **Gate record: refuse → pass (L2) → refuse → refuse.** The atlas never
184 + received a claim its evidence did not carry.
185 +
186 +## 6. Limitations
187 +
188 +One model (Qwen3-0.6B, 4-bit MLX), one behavior family for the causal arc,
189 +template-generated corpora (v1 corpora now serve as positive controls; v2
190 +carry token-overlap certificates but remain synthetic), single-machine
191 +timing cells (3 repeats each, two Macs total across expH), and specificity
192 +controls that used 3 random draws (a methodology rule now mandates ≥10 with
193 +percentile bounds). Cross-model generality (Qwen3-1.7B), quantization drift
194 +of the band map (FP16/Q8/Q4), and natural-corpus replication are the
195 +registered next questions.
196 +
197 +## 7. Reproducibility
198 +
199 +Every number in this report regenerates from the repository at commit
200 +`b7e357e` (or later) with the exact commands recorded in each atlas entry's
201 +`mapcard.json` (`regenerate_command`) and each experiment's
202 +`implementation/` directory. Raw result JSON — including full hardware and
203 +software manifests for every run — is browsable under [Results](/results).
204 +Hypotheses were registered in the experiments' `hypothesis.md` files before
205 +each run; the append-only [research log](/doc/research/LOG.md) records every
206 +decision in order. Promptsets ship with SHA-256 checksums
207 +([benchmarks/promptsets](/file/benchmarks/promptsets/manifest_v2.json)).
208 +
209 +*All research artifacts: © 2026 Simon-Pierre Boucher — contact@spboucher.ai —
210 +https://modelmap.io.*
modified site/lib/render.js +1 −0
@@ -67,6 +67,7 @@ const NAV = [
67 67 ["/research", "Research"],
68 68 ["/experiments", "Experiments"],
69 69 ["/atlas", "Atlas"],
70 + ["/publications", "Publications"],
70 71 ["/results", "Results"],
71 72 ["/code", "Code"],
72 73 ["/comments", "Comments"],
modified site/public/style.css +25 −0
@@ -262,6 +262,31 @@ h1, h2, h3, .page-title, .section-title { font-family: var(--font-display); font
262 262 .file-head { display: flex; justify-content: space-between; gap: 10px; flex-wrap: wrap; padding: 10px 16px; border-bottom: 1px solid var(--grid); background: var(--surface-2); }
263 263 .file-card .codeblock { border: 0; border-radius: 0; margin: 0; }
264 264
265 +/* ---------------------------------------------------------------- publications */
266 +.pub-card {
267 + display: block; background: var(--surface); border: 1px solid var(--grid);
268 + border-radius: var(--radius); padding: 24px 28px; margin: 18px 0; color: var(--ink);
269 + box-shadow: var(--shadow-sm); transition: transform 0.18s, box-shadow 0.18s, border-color 0.18s;
270 +}
271 +.pub-card:hover { border-color: var(--accent); transform: translateY(-2px); box-shadow: var(--shadow-md); text-decoration: none; }
272 +.pub-card h3 { margin: 8px 0 8px; font-size: 20px; }
273 +.pub-card p { color: var(--ink-2); font-size: 14.5px; margin: 0 0 10px; }
274 +.pub-card-meta { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; }
275 +.pub-id { font-family: var(--font-mono); font-size: 12.5px; font-weight: 600; color: var(--accent-deep);
276 + background: var(--accent-bg); border-radius: 6px; padding: 2px 8px; }
277 +.pub-date { font-family: var(--font-mono); font-size: 12px; color: var(--muted); }
278 +.pub-head { margin: 18px 0 26px; padding-bottom: 22px; border-bottom: 1px solid var(--grid); }
279 +.pub-head h1 { font-size: clamp(26px, 4vw, 34px); line-height: 1.18; margin: 12px 0 8px; }
280 +.pub-byline { color: var(--ink-2); font-size: 14px; margin: 0 0 16px; }
281 +.pub-abstract {
282 + background: var(--surface-2); border: 1px solid var(--grid); border-left: 3px solid var(--accent);
283 + border-radius: 10px; padding: 16px 20px; font-size: 14.5px; color: var(--ink-2); line-height: 1.7;
284 +}
285 +.pub .md h2 { font-family: var(--font-display); }
286 +.pub .md > p:first-of-type { font-size: 16.5px; }
287 +.pub-cite { margin-top: 36px; padding-top: 18px; border-top: 1px solid var(--grid); }
288 +.pub-cite strong { font-size: 13px; text-transform: uppercase; letter-spacing: 0.08em; color: var(--muted); }
289 +
265 290 /* ---------------------------------------------------------------- comments */
266 291 .comment-form { display: grid; gap: 14px; max-width: 640px; }
267 292 .comment-form label { font-size: 13.5px; font-weight: 600; color: var(--ink-2); display: block; margin-bottom: 5px; }
modified site/server.js +77 −1
@@ -82,7 +82,7 @@ function interventionsMapCharts(mapRel) {
82 82 }
83 83
84 84 /** expH charts: warm/cold storage panels + capture-overhead grouped bars. */
85 −function expHCharts() {
85 +function storageChart() {
86 86 const d = C.expHData();
87 87 let html = "";
88 88 if (d.warm && d.cold) {
@@ -100,6 +100,12 @@ function expHCharts() {
100 100 "blocks. The rule is IO granularity, not the container.",
101 101 });
102 102 }
103 + return html;
104 +}
105 +
106 +function captureChart() {
107 + const d = C.expHData();
108 + let html = "";
103 109 if (d.compute.length) {
104 110 const key = (c) => `${c.backend}·${c.source}`;
105 111 const groupsMap = new Map();
@@ -127,6 +133,10 @@ function expHCharts() {
127 133 return html;
128 134 }
129 135
136 +function expHCharts() {
137 + return storageChart() + captureChart();
138 +}
139 +
130 140 const app = express();
131 141 const PORT = process.env.PORT || 8140;
132 142
@@ -512,6 +522,72 @@ capture-feasibility frontier (Experiment H).</p>
512 522 page(res, { title: "About", active: "About", body });
513 523 });
514 524
525 +// ---------------------------------------------------------------- publications
526 +const FIGS = {
527 + "causal-band": () => interventionsMapCharts(),
528 + "probes-agreement": () => probeMapCharts("agreement", MAP_V2),
529 + "probes-arith": () => probeMapCharts("arith_valid", MAP_V2),
530 + "probes-word-order": () => probeMapCharts("word_order", MAP_V2),
531 + "probes-null": () => probeMapCharts("lang_id"),
532 + "storage": () => storageChart(),
533 + "capture": () => captureChart(),
534 +};
535 +
536 +function listPublications() {
537 + return (C.listDir("publications") || [])
538 + .filter((f) => f.name.endsWith(".md"))
539 + .map((f) => {
540 + const md = C.readMarkdown(f.rel);
541 + return md ? { rel: f.rel, id: f.name.replace(".md", ""), data: md.data } : null;
542 + })
543 + .filter(Boolean)
544 + .sort((a, b) => String(b.data.created).localeCompare(String(a.data.created)));
545 +}
546 +
547 +app.get("/publications", (req, res) => {
548 + const pubs = listPublications();
549 + const cards = pubs.map((p) => `<a class="pub-card" href="/publications/${p.id}">
550 + <div class="pub-card-meta"><span class="pub-id">${R.esc(p.data.pub_id || p.id)}</span>
551 + <span class="badge badge-${p.data.status === "final" ? "done" : "in-progress"}">${R.esc(p.data.status || "draft")}</span>
552 + <span class="pub-date">${R.esc(String(p.data.created))}</span></div>
553 + <h3>${R.esc(p.data.title || p.id)}</h3>
554 + <p>${R.esc(p.data.abstract ? String(p.data.abstract).slice(0, 300) + "…" : "")}</p>
555 + <span class="more">Read the report →</span></a>`).join("");
556 + const body = `<h1 class="page-title">Publications</h1>
557 +<p class="lede-small">Official technical reports of this project. Every number is regenerable from
558 +the stated commit; figures are rendered live from the same versioned results as the
559 +<a href="/atlas">Atlas</a>; negative results and publication-gate refusals are reported
560 +with the same prominence as positive findings.</p>
561 +${cards || '<p class="note-line">No publications yet.</p>'}`;
562 + page(res, { title: "Publications", active: "Publications", body });
563 +});
564 +
565 +app.get("/publications/:id", (req, res) => {
566 + const rel = `publications/${req.params.id}.md`;
567 + const md = C.readMarkdown(rel);
568 + if (!md) return notFound(res);
569 + let html = R.markdownToHtml(md.content, rel);
570 + html = html.replace(/<p>\{\{fig:([a-z0-9-]+)\}\}<\/p>/g, (_, key) =>
571 + FIGS[key] ? FIGS[key]() : "");
572 + const d = md.data;
573 + const body = `<article class="doc pub">
574 + <p class="crumb"><a href="/publications">Publications</a> / ${R.esc(d.pub_id || req.params.id)}</p>
575 + <header class="pub-head">
576 + <div class="pub-card-meta"><span class="pub-id">${R.esc(d.pub_id || req.params.id)}</span>
577 + <span class="badge badge-${d.status === "final" ? "done" : "in-progress"}">${R.esc(d.status || "draft")}</span>
578 + <span class="pub-date">${R.esc(String(d.created))}</span></div>
579 + <h1>${R.esc(d.title || req.params.id)}</h1>
580 + <p class="pub-byline">${R.esc(d.author || "Simon-Pierre Boucher")} ·
581 + <a href="mailto:${R.esc(d.contact || "contact@spboucher.ai")}">${R.esc(d.contact || "contact@spboucher.ai")}</a> ·
582 + <a href="https://modelmap.io">modelmap.io</a></p>
583 + ${d.abstract ? `<div class="pub-abstract"><strong>Abstract.</strong> ${R.esc(String(d.abstract))}</div>` : ""}
584 + </header>
585 + <div class="md">${html}</div>
586 + ${d.cite ? `<div class="pub-cite"><strong>How to cite</strong><pre class="codeblock"><code>${R.esc(String(d.cite))}</code></pre></div>` : ""}
587 +</article>`;
588 + page(res, { title: d.title || req.params.id, active: "Publications", body });
589 +});
590 +
515 591 // ---------------------------------------------------------------- comments
516 592 function commentsPage(res, { flash = "", flashKind = "ok" } = {}) {
517 593 const items = Comments.list();
modified site/sync-content.sh +1 −1
@@ -26,7 +26,7 @@ rsync -a --delete \
26 26 --include='Makefile' --include='pyproject.toml' \
27 27 --include='research/***' --include='experiments/***' --include='benchmarks/***' \
28 28 --include='src/***' --include='tools/***' --include='results/***' \
29 − --include='atlas/***' --include='docs/***' \
29 + --include='atlas/***' --include='docs/***' --include='publications/***' \
30 30 --exclude='*' \
31 31 "$REPO_ROOT/" "$CONTENT/"
32 32
33 33