publications/TR-2026-01.md at main · modelmap

# project: modelmap document: TR-2026-01 — technical report author: Simon-Pierre Boucher contact: contact@spboucher.ai website: https://modelmap.io created: 2026-08-12 status: final pub_id: TR-2026-01 title: "Registered, Gated, Negative-First: Standing Up a Reproducible Cartography Pipeline for a Local LLM in One Day" abstract: "We report the first results of modelmap, a research program that maps the internal structure of open-weight language models entirely on consumer Apple Silicon, under a discipline of pre-registered hypotheses, mandatory nulls, and machine-enforced publication gates. In eleven registered runs across three micro-experiments on a single day, the pipeline (i) measured the systems substrate of local interpretability — activation capture is nearly free under MLX lazy evaluation (1.004× plain prefill on a real 4-bit checkpoint) while activation-store format rankings invert between warm and cold cache, showing IO granularity rather than container choice governs throughput; (ii) demonstrated that per-layer linear probes on lexically separable classes are indistinguishable from a random-initialization architecture twin — a published negative result — and that token-balanced structural properties recover trained-model signal only as real-minus-twin differentials; and (iii) causally tested the resulting agreement map: its decodability ranking failed two intervention tests (correlational-to-causal survival 0/2), while a difference-of-means agreement direction proved robustly necessary across the early layer band (2–15), removing 73–75% of the model's grammatical preference under rank-1 erasure, replicated across six fresh estimators — published at Level 2. Two further pre-registered attempts to promote the claim to Level 3 (dose-response steering) were refused by the publication gate, including one that would have shipped a false claim had a single earlier observation been trusted. All figures below are rendered live from the same versioned artifacts as the public atlas." cite: | Boucher, S.-P. (2026). Registered, Gated, Negative-First: Standing Up a Reproducible Cartography Pipeline for a Local LLM in One Day. modelmap Technical Report TR-2026-01. https://modelmap.io/publications/TR-2026-01

# 1. Motivation and scope

Most interpretability results are produced on CUDA clusters, reported without seed or dataset replication, and published without the failed attempts that surrounded them. The modelmap program inverts each of these defaults: all computation runs on consumer Apple Silicon (a MacBook Pro M5 Max, 48 GB, and one Mac Studio M3 Ultra used for a cold-cache pass), every experimental run is preceded by a registered hypothesis with an explicit falsification criterion, every observational map carries mandatory nulls (shuffled labels, random-initialization twins, random-direction controls), and atlas entries pass through a machine-enforced publication gate that refuses any map whose evidence does not meet its claimed confidence level.

This report covers everything the pipeline produced on 2026-08-12 — the program's first full day of measurement — spanning three micro-experiments: capture-cost frontier (expH, 3 runs), probe reliability (expA, 2 runs), and causal verification (expC, 6 runs). The subject model throughout is Qwen3-0.6B in 4-bit MLX quantization (28 layers, d_model 1024): a genuinely local model, studied in the form people actually run.

Confidence vocabulary (used throughout, and rendered as badges in the atlas): Level 0 anecdotal · Level 1 correlational, controlled, replicated · Level 2 method-robust (≥2 techniques agree) · Level 3 causal (intervention confirms).

# 2. Systems substrate: what capture actually costs on a Mac (expH)

Three registered runs established the engineering envelope for everything that follows.

Capture is nearly free under MLX. Wrapping every decoder layer with a retaining tap costs 1.02× plain inference on a synthetic 12-layer model and 1.004× on the real Qwen3-0.6B-4bit checkpoint (28.5 → 28.6 ms per 901-token prefill); adding bf16→fp16 casting and streaming writes to an mmap store raises this to only 1.28×. The same taps on PyTorch-MPS cost 1.22× (retention) to 1.47× (with copy+write) — eager hook materialization is measurably more expensive than lazy-graph retention. To our knowledge run #3 is also the first Python-level activation capture from an actually-quantized local checkpoint, a capability the Phase-1 literature survey found in no existing tool.

The storage hypothesis was falsified — instructively. Run #1 (warm cache, M5 Max) found raw mmap beating chunked zarr by 3.2–10.8× on the random-batch read pattern that SAE-style training requires, and we registered the hypothesis that this ordering would survive a cold cache. Run #2 (M3 Ultra, purge before every timed repetition) inverted it: cold, mmap collapses to page-fault IO (~8–16 KiB reads at queue depth 1, 0.14 GB/s) while zarr's 32 MiB chunk reads sustain 0.62 GB/s. The portable rule is that IO granularity, not container format, governs activation-store throughput — a conclusion that independently converges with the sister project localvm-research's cold-cache SSD measurements on different hardware.

Design consequences adopted: mmap-backed flat stores for in-RAM working sets; chunk-aligned batched access mandatory for larger-than-RAM stores; MLX as the preferred capture backend; cache regime reported as a first-class dimension of every cost table.

# 3. Probing: the architecture null eats the map, then earns its keep (expA)

Run #1 — a published negative result. On template-generated binary properties (French/English, code/prose, arithmetic-context), per-layer linear probes on mean-pooled representations hit ceiling (accuracy 1.000, all 28 layers, all properties, both promptsets). The registered validity gate required a random-initialization architecture twin to sit below 0.05 selectivity; instead the twin also scored 1.00 everywhere (mean real−twin selectivity within ±0.06). The map measured the tokenizer + architecture prior, not learned computation — and notably, the standard shuffled-label control passed everywhere and would never have caught this; only the twin null did. This became the atlas's first entry (probes/v1, negative_result: true).

Run #2 — differential maps on structure-borne properties. Version-2 promptsets remove lexical separability (classes share vocabulary; the property lives in structure; class token-overlap certificates ship in the corpus manifest). Outcomes, published per property in probes/v2 at Level 1:

agreement (subject–verb, is/are balanced across classes): trained-model signal on 25/28 layers (max real−twin selectivity +0.379; accuracy 0.967 vs twin 0.729);
arith_valid (correct vs off-by-delta equations): accuracy 0.86–0.90 vs twin ~0.56–0.58, strongest in last-token representations — consistent with computation completing at the equals sign;
word_order (grammatical vs scrambled): null-dominated — the twin reaches 0.96 accuracy from surface statistics alone; flagged as invalid evidence of learned syntax at this granularity.

Off the ceiling, the project's first real noise-floor numbers appeared: top-5-layer replication across probe seeds is ~0.54, and dataset shift exceeds seed variance on only 3–8 of 28 layers — neither noise source dominates, so both must be reported. Doctrine adopted: probe maps are publishable only as real-minus-twin differentials, and the twin null is a permanent fixture of every observational map.

# 4. Causal verification: the survival ledger and the band map (expC)

Six registered runs asked whether the agreement probe map means what it appears to mean.

The decodability ranking died twice (survival ledger 0/2). Skipping the map's top-5 layers damaged the behavioral agreement margin less than skipping five random layers (run #1); normalizing by general NLL damage confirmed nothing in the skip family singles out the probe layers (run #2, P1). A surgical per-layer scan — erasing the difference-of-means agreement direction at each layer, netted against random-direction controls — produced a causal profile that anti-correlates with the probe ranking (Spearman ρ = −0.136). Where information is most decodable is not where computation is causally load-bearing: the Hase-class dissociation, measured end-to-end in one pipeline.

What survived instead: a band claim at Level 2. The erasure scan's stable content is that removing the agreement direction at any single layer in the early band (2–15) destroys most of the behavior. A first per-layer version of this map was refused by the publication gate (run #3: mean pairwise ρ = 0.495 across five direction estimators, below the registered 0.7). Narrowed to band granularity with everything fresh — six new direction sources from disjoint promptset halves, a new behavioral bank — the claim passed decisively (run #4: early-band mean specific damage +3.23…+3.35 on a +4.45 baseline; spread 0.12 across six estimators). This is interventions/v1, the atlas's first Level-2 entry.

Level 3 was attempted twice and refused twice. Dose-response steering (adding α·σ·u) at three early-band layers failed its conjunctive criterion (run #5) — layer 4 sits in an overdose regime where any ~2σ perturbation, random included, collapses the behavior — while layer 12 passed every gate with a textbook monotone dose curve. A minimal single-layer L12 claim was then re-registered with four fresh estimators and a third fresh bank (run #6): it failed replication — only 1/4 sources monotone, and the specificity control did not reproduce (random-direction |Δ| 3.36 vs bound 1.14; run #5's passing value of 0.45 is now legible as sampling luck over three draws). Had run #5's observation been published directly, the atlas would contain a false Level-3 claim today. Final verdict: the agreement direction is necessary but not a reliable additive handle, and interventions/v1 stands as the arc's terminal claim.

# 5. What one day establishes

The full charter loop runs end-to-end on consumer hardware: register → capture → map → null-check → causally verify → gate → publish (or refuse), with wall-clock costs of seconds to minutes per run at 0.6B scale.
Nulls are not decorative. The architecture twin killed a ceiling-perfect probe map that shuffled-label controls waved through; the random-direction and fresh-replication controls killed two plausible Level-3 claims.
Granularity is a claim. Per-layer rankings failed replication where band-level claims passed; map artifacts must declare the resolution at which they are stable.
The measured correlational→causal survival rate is 0/2 for layer rankings on this model/behavior — a number the field rarely publishes and the reason this atlas labels every entry with its evidence level.
Gate record: refuse → pass (L2) → refuse → refuse. The atlas never received a claim its evidence did not carry.

# 6. Limitations

One model (Qwen3-0.6B, 4-bit MLX), one behavior family for the causal arc, template-generated corpora (v1 corpora now serve as positive controls; v2 carry token-overlap certificates but remain synthetic), single-machine timing cells (3 repeats each, two Macs total across expH), and specificity controls that used 3 random draws (a methodology rule now mandates ≥10 with percentile bounds). Cross-model generality (Qwen3-1.7B), quantization drift of the band map (FP16/Q8/Q4), and natural-corpus replication are the registered next questions.

# 7. Reproducibility

Every number in this report regenerates from the repository at commit b7e357e (or later) with the exact commands recorded in each atlas entry's mapcard.json (regenerate_command) and each experiment's implementation/ directory. Raw result JSON — including full hardware and software manifests for every run — is browsable under Results. Hypotheses were registered in the experiments' hypothesis.md files before each run; the append-only research log records every decision in order. Promptsets ship with SHA-256 checksums (benchmarks/promptsets).