spb/modelmap Public License

1---2project: modelmap3document: TR-2026-01 — technical report4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128status: final9pub_id: TR-2026-0110title: "Registered, Gated, Negative-First: Standing Up a Reproducible Cartography Pipeline for a Local LLM in One Day"11abstract: "We report the first results of modelmap, a research program that maps the internal structure of open-weight language models entirely on consumer Apple Silicon, under a discipline of pre-registered hypotheses, mandatory nulls, and machine-enforced publication gates. In eleven registered runs across three micro-experiments on a single day, the pipeline (i) measured the systems substrate of local interpretability — activation capture is nearly free under MLX lazy evaluation (1.004× plain prefill on a real 4-bit checkpoint) while activation-store format rankings invert between warm and cold cache, showing IO granularity rather than container choice governs throughput; (ii) demonstrated that per-layer linear probes on lexically separable classes are indistinguishable from a random-initialization architecture twin — a published negative result — and that token-balanced structural properties recover trained-model signal only as real-minus-twin differentials; and (iii) causally tested the resulting agreement map: its decodability ranking failed two intervention tests (correlational-to-causal survival 0/2), while a difference-of-means agreement direction proved robustly necessary across the early layer band (2–15), removing 73–75% of the model's grammatical preference under rank-1 erasure, replicated across six fresh estimators — published at Level 2. Two further pre-registered attempts to promote the claim to Level 3 (dose-response steering) were refused by the publication gate, including one that would have shipped a false claim had a single earlier observation been trusted. All figures below are rendered live from the same versioned artifacts as the public atlas."12cite: |13  Boucher, S.-P. (2026). Registered, Gated, Negative-First: Standing Up a14  Reproducible Cartography Pipeline for a Local LLM in One Day.15  modelmap Technical Report TR-2026-01. https://modelmap.io/publications/TR-2026-0116---1718## 1. Motivation and scope1920Most interpretability results are produced on CUDA clusters, reported without21seed or dataset replication, and published without the failed attempts that22surrounded them. The modelmap program inverts each of these defaults: all23computation runs on consumer Apple Silicon (a MacBook Pro M5 Max, 48 GB, and24one Mac Studio M3 Ultra used for a cold-cache pass), every experimental run is25preceded by a **registered hypothesis with an explicit falsification26criterion**, every observational map carries **mandatory nulls** (shuffled27labels, random-initialization twins, random-direction controls), and atlas28entries pass through a **machine-enforced publication gate** that refuses any29map whose evidence does not meet its claimed confidence level.3031This report covers everything the pipeline produced on 2026-08-12 — the32program's first full day of measurement — spanning three micro-experiments:33capture-cost frontier (expH, 3 runs), probe reliability (expA, 2 runs), and34causal verification (expC, 6 runs). The subject model throughout is35**Qwen3-0.6B in 4-bit MLX quantization** (28 layers, d_model 1024): a genuinely36*local* model, studied in the form people actually run.3738Confidence vocabulary (used throughout, and rendered as badges in the39[atlas](/atlas)): **Level 0** anecdotal · **Level 1** correlational,40controlled, replicated · **Level 2** method-robust (≥2 techniques agree) ·41**Level 3** causal (intervention confirms).4243## 2. Systems substrate: what capture actually costs on a Mac (expH)4445Three registered runs established the engineering envelope for everything46that follows.4748**Capture is nearly free under MLX.** Wrapping every decoder layer with a49retaining tap costs 1.02× plain inference on a synthetic 12-layer model and50**1.004×** on the real Qwen3-0.6B-4bit checkpoint (28.5 → 28.6 ms per51901-token prefill); adding bf16→fp16 casting and streaming writes to an52mmap store raises this to only 1.28×. The same taps on PyTorch-MPS cost531.22× (retention) to 1.47× (with copy+write) — eager hook materialization is54measurably more expensive than lazy-graph retention. To our knowledge run #355is also the first Python-level activation capture from an actually-quantized56local checkpoint, a capability the Phase-1 literature survey found in no57existing tool.5859{{fig:capture}}6061**The storage hypothesis was falsified — instructively.** Run #1 (warm cache,62M5 Max) found raw `mmap` beating chunked zarr by 3.2–10.8× on the63random-batch read pattern that SAE-style training requires, and we registered64the hypothesis that this ordering would survive a cold cache. Run #2 (M365Ultra, `purge` before every timed repetition) **inverted it**: cold, mmap66collapses to page-fault IO (~8–16 KiB reads at queue depth 1, 0.14 GB/s)67while zarr's 32 MiB chunk reads sustain 0.62 GB/s. The portable rule is that68**IO granularity, not container format, governs activation-store69throughput** — a conclusion that independently converges with the sister70project localvm-research's cold-cache SSD measurements on different hardware.7172{{fig:storage}}7374*Design consequences adopted:* mmap-backed flat stores for in-RAM working75sets; chunk-aligned batched access mandatory for larger-than-RAM stores; MLX76as the preferred capture backend; cache regime reported as a first-class77dimension of every cost table.7879## 3. Probing: the architecture null eats the map, then earns its keep (expA)8081**Run #1 — a published negative result.** On template-generated binary82properties (French/English, code/prose, arithmetic-context), per-layer linear83probes on mean-pooled representations hit ceiling (accuracy 1.000, all 2884layers, all properties, both promptsets). The registered validity gate85required a random-initialization *architecture twin* to sit below 0.0586selectivity; instead **the twin also scored 1.00 everywhere** (mean87real−twin selectivity within ±0.06). The map measured the tokenizer +88architecture prior, not learned computation — and notably, the standard89shuffled-label control passed everywhere and would never have caught this;90only the twin null did. This became the atlas's first entry91([probes/v1](/atlas/qwen3-0.6b-4bit/probes/v1), `negative_result: true`).9293{{fig:probes-null}}9495**Run #2 — differential maps on structure-borne properties.** Version-296promptsets remove lexical separability (classes share vocabulary; the97property lives in structure; class token-overlap certificates ship in the98corpus manifest). Outcomes, published per property in99[probes/v2](/atlas/qwen3-0.6b-4bit/probes/v2) at Level 1:100101- **agreement** (subject–verb, is/are balanced across classes): trained-model102  signal on 25/28 layers (max real−twin selectivity +0.379; accuracy 0.967103  vs twin 0.729);104- **arith_valid** (correct vs off-by-delta equations): accuracy 0.86–0.90 vs105  twin ~0.56–0.58, strongest in last-token representations — consistent with106  computation completing at the equals sign;107- **word_order** (grammatical vs scrambled): **null-dominated** — the twin108  reaches 0.96 accuracy from surface statistics alone; flagged as invalid109  evidence of learned syntax at this granularity.110111{{fig:probes-agreement}}112113{{fig:probes-arith}}114115{{fig:probes-word-order}}116117Off the ceiling, the project's first real noise-floor numbers appeared:118top-5-layer replication across probe seeds is ~0.54, and dataset shift119exceeds seed variance on only 3–8 of 28 layers — neither noise source120dominates, so both must be reported. Doctrine adopted: **probe maps are121publishable only as real-minus-twin differentials**, and the twin null is a122permanent fixture of every observational map.123124## 4. Causal verification: the survival ledger and the band map (expC)125126Six registered runs asked whether the agreement probe map means what it127appears to mean.128129**The decodability ranking died twice (survival ledger 0/2).** Skipping the130map's top-5 layers damaged the behavioral agreement margin *less* than131skipping five random layers (run #1); normalizing by general NLL damage132confirmed nothing in the skip family singles out the probe layers (run #2,133P1). A surgical per-layer scan — erasing the difference-of-means agreement134direction at each layer, netted against random-direction controls — produced135a causal profile that **anti-correlates** with the probe ranking (Spearman136ρ = −0.136). Where information is most decodable is not where computation is137causally load-bearing: the Hase-class dissociation, measured end-to-end in138one pipeline.139140**What survived instead: a band claim at Level 2.** The erasure scan's141stable content is that removing the agreement direction at *any single142layer* in the early band (2–15) destroys most of the behavior. A first143per-layer version of this map was **refused** by the publication gate (run144#3: mean pairwise ρ = 0.495 across five direction estimators, below the145registered 0.7). Narrowed to band granularity with everything fresh — six146new direction sources from disjoint promptset halves, a new behavioral147bank — the claim passed decisively (run #4: early-band mean specific damage148+3.23…+3.35 on a +4.45 baseline; spread 0.12 across six estimators). This is149[interventions/v1](/atlas/qwen3-0.6b-4bit/interventions/v1), the atlas's150first Level-2 entry.151152{{fig:causal-band}}153154**Level 3 was attempted twice and refused twice.** Dose-response steering155(adding α·σ·u) at three early-band layers failed its conjunctive criterion156(run #5) — layer 4 sits in an overdose regime where *any* ~2σ perturbation,157random included, collapses the behavior — while layer 12 passed every gate158with a textbook monotone dose curve. A minimal single-layer L12 claim was159then re-registered with four fresh estimators and a third fresh bank (run160#6): **it failed replication** — only 1/4 sources monotone, and the161specificity control did not reproduce (random-direction |Δ| 3.36 vs bound1621.14; run #5's passing value of 0.45 is now legible as sampling luck over163three draws). Had run #5's observation been published directly, the atlas164would contain a false Level-3 claim today. Final verdict: the agreement165direction is **necessary but not a reliable additive handle**, and166interventions/v1 stands as the arc's terminal claim.167168## 5. What one day establishes1691701. **The full charter loop runs end-to-end on consumer hardware**: register →171   capture → map → null-check → causally verify → gate → publish (or refuse),172   with wall-clock costs of seconds to minutes per run at 0.6B scale.1732. **Nulls are not decorative.** The architecture twin killed a174   ceiling-perfect probe map that shuffled-label controls waved through; the175   random-direction and fresh-replication controls killed two plausible176   Level-3 claims.1773. **Granularity is a claim.** Per-layer rankings failed replication where178   band-level claims passed; map artifacts must declare the resolution at179   which they are stable.1804. **The measured correlational→causal survival rate is 0/2** for layer181   rankings on this model/behavior — a number the field rarely publishes and182   the reason this atlas labels every entry with its evidence level.1835. **Gate record: refuse → pass (L2) → refuse → refuse.** The atlas never184   received a claim its evidence did not carry.185186## 6. Limitations187188One model (Qwen3-0.6B, 4-bit MLX), one behavior family for the causal arc,189template-generated corpora (v1 corpora now serve as positive controls; v2190carry token-overlap certificates but remain synthetic), single-machine191timing cells (3 repeats each, two Macs total across expH), and specificity192controls that used 3 random draws (a methodology rule now mandates ≥10 with193percentile bounds). Cross-model generality (Qwen3-1.7B), quantization drift194of the band map (FP16/Q8/Q4), and natural-corpus replication are the195registered next questions.196197## 7. Reproducibility198199Every number in this report regenerates from the repository at commit200`b7e357e` (or later) with the exact commands recorded in each atlas entry's201`mapcard.json` (`regenerate_command`) and each experiment's202`implementation/` directory. Raw result JSON — including full hardware and203software manifests for every run — is browsable under [Results](/results).204Hypotheses were registered in the experiments' `hypothesis.md` files before205each run; the append-only [research log](/doc/research/LOG.md) records every206decision in order. Promptsets ship with SHA-256 checksums207([benchmarks/promptsets](/file/benchmarks/promptsets/manifest_v2.json)).208209*All research artifacts: © 2026 Simon-Pierre Boucher — contact@spboucher.ai —210https://modelmap.io.*211