modelmap — Internal Cartography of Local Large Language Models
modelmap discovers, measures, and maps the internal structure of pretrained open-weight LLMs entirely on consumer Apple Silicon — and publishes the results as a reproducible, confidence-labeled public atlas at modelmap.io. Every map is versioned, provenanced (commit + config + model hash + hardware manifest), regenerable by one command, and labeled with the evidence level it actually earned. Negative results and publication-gate refusals are first-class citizens.
"we think the model does X" → "here is the map, its evidence level, and the script that rebuilds it"
Headline results (2026-08-12 · Qwen3-0.6B-4bit · full report: TR-2026-01)
| Finding | Evidence |
|---|---|
| Activation capture is nearly free under MLX — 1.004× plain prefill on a real 4-bit checkpoint (first Python capture from an actually-quantized local model) | expH, 3 registered runs |
| Activation-store format rankings invert warm→cold — mmap 3–11× ahead warm, zarr 4.4× ahead cold; IO granularity governs, not the container | expH run #2 (falsified own hypothesis) |
| Probes on separable classes measure the tokenizer, not the model — a random-init architecture twin matches the trained model at ceiling; shuffled-label controls miss it | expA run #1 (published negative, atlas probes/v1) |
| Differential (real−twin) maps recover trained signal on structure-borne properties: agreement 25/28 signal layers, equation-validity 0.90 vs twin 0.58 | expA run #2 (Level 1, atlas probes/v2) |
| Decodability ≠ causal support — the probe map's layer ranking failed two intervention tests (survival ledger 0/2) | expC runs #1–#2 |
| A single agreement direction is causally necessary across layers 2–15 — rank-1 erasure removes 73–75% of the behavior, replicated across six fresh estimators | expC run #4 (Level 2, atlas interventions/v1) |
| Two Level-3 attempts refused by the publication gate — incl. one that would have shipped a false claim without fresh re-registration | expC runs #5–#6 |
How it works
- Registered hypotheses — every run is preceded by a written hypothesis
with an explicit falsification criterion (
experiments/*/hypothesis.md). - Mandatory nulls — shuffled labels, random-init architecture twins, random-direction controls, FDR correction across unit scans.
- Machine-enforced publication gates —
tools/publish.pyrefuses any atlas entry whose map card doesn't validate; per-run gates refuse claims that fail replication (make_*_mapcard.py). - Confidence taxonomy on every artifact: L0 anecdotal · L1 correlational · L2 method-robust · L3 causal.
- Everything local — MLX / PyTorch-MPS on 16–64 GB Macs; quantized checkpoints are studied in the form people actually run.
Repository layout
text
research/ charter-driven paper trail: log, state of the art, 24 gaps, ranking
src/modelmap/ capture (MLX taps, quantized models), probes, stats, atlas schema
experiments/ micro-experiments A–H + candidates (hypothesis → run → analysis)
atlas/ versioned map artifacts: map.json + provenance + confidence + map card
publications/ official technical reports (rendered with live figures on the site)
site/ modelmap.io (Express, server-rendered SVG maps, mobile-first)
benchmarks/ harness, hardware manifests, checksummed promptsets
tools/ check_headers, new_experiment, new_map, publish (the gate)Quickstart (Apple Silicon, macOS 14+)
bash
make setup # venv + deps (numpy, mlx, torch, zarr, safetensors)
make test # 5 correctness tests (probes vs planted structure, FDR, map cards)
make lint headers # ruff + mandatory author-header check
make site-run # build and preview modelmap.io locally on :8140
# regenerate any published map: see its mapcard.json "regenerate_command"Author
Simon-Pierre Boucher · contact@spboucher.ai · modelmap.io Sister project: localvm-research (out-of-core LLM execution on consumer Macs).
Citation
text
Boucher, S.-P. (2026). modelmap — Internal Cartography of Local Large
Language Models. https://modelmap.io (see also Technical Report TR-2026-01).All rights reserved (research code). © 2026 Simon-Pierre Boucher.