spb/modelmap Public License
Internal cartography of local LLMs on Apple Silicon — registered, gated, negative-first. Public atlas at modelmap.io.
Python 66.3%
JavaScript 24.5%
CSS 8.1%
Shell 0.7%
1[](https://modelmap.io)23# modelmap — Internal Cartography of Local Large Language Models45678910111213141516**modelmap** discovers, measures, and maps the internal structure of17pretrained open-weight LLMs **entirely on consumer Apple Silicon** — and18publishes the results as a reproducible, confidence-labeled public atlas at19**[modelmap.io](https://modelmap.io)**. Every map is versioned, provenanced20(commit + config + model hash + hardware manifest), regenerable by one21command, and labeled with the evidence level it actually earned. Negative22results and publication-gate refusals are first-class citizens.2324> *"we think the model does X"* → *"here is the map, its evidence level,25> and the script that rebuilds it"*2627## Headline results (2026-08-12 · Qwen3-0.6B-4bit · full report: [TR-2026-01](https://modelmap.io/publications/TR-2026-01))2829| Finding | Evidence |30|---|---|31| **Activation capture is nearly free under MLX** — 1.004× plain prefill on a real 4-bit checkpoint (first Python capture from an actually-quantized local model) | expH, 3 registered runs |32| **Activation-store format rankings invert warm→cold** — mmap 3–11× ahead warm, zarr 4.4× ahead cold; IO granularity governs, not the container | expH run #2 (falsified own hypothesis) |33| **Probes on separable classes measure the tokenizer, not the model** — a random-init architecture twin matches the trained model at ceiling; shuffled-label controls miss it | expA run #1 (published negative, atlas probes/v1) |34| **Differential (real−twin) maps recover trained signal** on structure-borne properties: agreement 25/28 signal layers, equation-validity 0.90 vs twin 0.58 | expA run #2 (Level 1, atlas probes/v2) |35| **Decodability ≠ causal support** — the probe map's layer ranking failed two intervention tests (survival ledger **0/2**) | expC runs #1–#2 |36| **A single agreement direction is causally necessary across layers 2–15** — rank-1 erasure removes 73–75% of the behavior, replicated across six fresh estimators | expC run #4 (**Level 2**, atlas interventions/v1) |37| **Two Level-3 attempts refused by the publication gate** — incl. one that would have shipped a false claim without fresh re-registration | expC runs #5–#6 |3839## How it works4041- **Registered hypotheses** — every run is preceded by a written hypothesis42 with an explicit falsification criterion (`experiments/*/hypothesis.md`).43- **Mandatory nulls** — shuffled labels, random-init architecture twins,44 random-direction controls, FDR correction across unit scans.45- **Machine-enforced publication gates** — `tools/publish.py` refuses any46 atlas entry whose map card doesn't validate; per-run gates refuse claims47 that fail replication (`make_*_mapcard.py`).48- **Confidence taxonomy** on every artifact: L0 anecdotal · L1 correlational49 · L2 method-robust · L3 causal.50- **Everything local** — MLX / PyTorch-MPS on 16–64 GB Macs; quantized51 checkpoints are studied in the form people actually run.5253## Repository layout5455```text56research/ charter-driven paper trail: log, state of the art, 24 gaps, ranking57src/modelmap/ capture (MLX taps, quantized models), probes, stats, atlas schema58experiments/ micro-experiments A–H + candidates (hypothesis → run → analysis)59atlas/ versioned map artifacts: map.json + provenance + confidence + map card60publications/ official technical reports (rendered with live figures on the site)61site/ modelmap.io (Express, server-rendered SVG maps, mobile-first)62benchmarks/ harness, hardware manifests, checksummed promptsets63tools/ check_headers, new_experiment, new_map, publish (the gate)64```6566## Quickstart (Apple Silicon, macOS 14+)6768```bash69make setup # venv + deps (numpy, mlx, torch, zarr, safetensors)70make test # 5 correctness tests (probes vs planted structure, FDR, map cards)71make lint headers # ruff + mandatory author-header check72make site-run # build and preview modelmap.io locally on :814073# regenerate any published map: see its mapcard.json "regenerate_command"74```7576## Author7778**Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai) · [modelmap.io](https://modelmap.io)79Sister project: [localvm-research](https://www.localvm.dev) (out-of-core LLM execution on consumer Macs).8081## Citation8283```text84Boucher, S.-P. (2026). modelmap — Internal Cartography of Local Large85Language Models. https://modelmap.io (see also Technical Report TR-2026-01).86```8788*All rights reserved (research code). © 2026 Simon-Pierre Boucher.*89