README: English rewrite with metric badges and author info · 46cfa7b · modelmap

5

contact: contact@spboucher.ai

6

website: https://modelmap.io

7

created: 2026-08-12

8

−status: draft

8

+modified: 2026-08-12

9

+status: reviewed

9

10

---

10

11

12

# modelmap — Internal Cartography of Local Large Language Models

12

13

−**modelmap** investigates what can actually be known, measured, localized, and mapped

14

−about the internal organization of pretrained open-weight LLMs — where knowledge lives,

15

−how computation is distributed, which structures are stable across inputs, layers,

16

−scales, and model families — **using only local, consumer-grade Apple Silicon hardware**,

17

−and publishes the resulting maps as a rigorous, reproducible public atlas at

18

−**[modelmap.io](https://modelmap.io)**.

19

−

20

−## Core research questions

21

−

22

−- **Q1 — Localization.** Where do capabilities, knowledge domains, languages, and behaviors reside? Localized or diffuse?

23

−- **Q2 — Structure.** Which internal structures are stable across inputs — and at what granularity are they real rather than probing artifacts?

24

−- **Q3 — Comparability.** Can internal maps be compared across sizes, checkpoints, quantization levels, and families?

25

−- **Q4 — Cost.** Which mapping techniques are feasible on a 16–64 GB Mac, and what is the accuracy/cost frontier of local interpretability?

26

−- **Q5 — Utility.** Do the maps predict anything useful — quantization sensitivity, pruning tolerance, working sets, failure modes?

27

−

28

−The distinction kept sharp throughout:

14

+![status](https://img.shields.io/badge/status-active_research-6d4fc4)

15

+![platform](https://img.shields.io/badge/platform-macOS_·_Apple_Silicon-0b0b0b)

16

+![compute](https://img.shields.io/badge/compute-MLX_·_PyTorch--MPS-eb6834)

17

+![atlas](https://img.shields.io/badge/atlas_entries-3-53389e)

18

+![max confidence](https://img.shields.io/badge/max_confidence-Level_2_·_method--robust-0e9f4e)

19

+![causal survival](https://img.shields.io/badge/correlational→causal_survival-0%2F2-9a6a00)

20

+![registered runs](https://img.shields.io/badge/registered_runs-11-53389e)

21

+![gate record](https://img.shields.io/badge/publication_gate-3_refusals_·_1_pass-8b8798)

22

+![tests](https://img.shields.io/badge/tests-5_passing-0e9f4e)

23

+![license](https://img.shields.io/badge/license-all_rights_reserved-555)

24

+

25

+**modelmap** discovers, measures, and maps the internal structure of

26

+pretrained open-weight LLMs **entirely on consumer Apple Silicon** — and

27

+publishes the results as a reproducible, confidence-labeled public atlas at

28

+**[modelmap.io](https://modelmap.io)**. Every map is versioned, provenanced

29

+(commit + config + model hash + hardware manifest), regenerable by one

30

+command, and labeled with the evidence level it actually earned. Negative

31

+results and publication-gate refusals are first-class citizens.

32

+

33

+> *"we think the model does X"* → *"here is the map, its evidence level,

34

+> and the script that rebuilds it"*

35

+

36

+## Headline results (2026-08-12 · Qwen3-0.6B-4bit · full report: [TR-2026-01](https://modelmap.io/publications/TR-2026-01))

37

+

38

+| Finding | Evidence |

39

+|---|---|

40

+| **Activation capture is nearly free under MLX** — 1.004× plain prefill on a real 4-bit checkpoint (first Python capture from an actually-quantized local model) | expH, 3 registered runs |

41

+| **Activation-store format rankings invert warm→cold** — mmap 3–11× ahead warm, zarr 4.4× ahead cold; IO granularity governs, not the container | expH run #2 (falsified own hypothesis) |

42

+| **Probes on separable classes measure the tokenizer, not the model** — a random-init architecture twin matches the trained model at ceiling; shuffled-label controls miss it | expA run #1 (published negative, atlas probes/v1) |

43

+| **Differential (real−twin) maps recover trained signal** on structure-borne properties: agreement 25/28 signal layers, equation-validity 0.90 vs twin 0.58 | expA run #2 (Level 1, atlas probes/v2) |

44

+| **Decodability ≠ causal support** — the probe map's layer ranking failed two intervention tests (survival ledger **0/2**) | expC runs #1–#2 |

45

+| **A single agreement direction is causally necessary across layers 2–15** — rank-1 erasure removes 73–75% of the behavior, replicated across six fresh estimators | expC run #4 (**Level 2**, atlas interventions/v1) |

46

+| **Two Level-3 attempts refused by the publication gate** — incl. one that would have shipped a false claim without fresh re-registration | expC runs #5–#6 |

47

+

48

+## How it works

49

+

50

+- **Registered hypotheses** — every run is preceded by a written hypothesis

51

+ with an explicit falsification criterion (`experiments/*/hypothesis.md`).

52

+- **Mandatory nulls** — shuffled labels, random-init architecture twins,

53

+ random-direction controls, FDR correction across unit scans.

54

+- **Machine-enforced publication gates** — `tools/publish.py` refuses any

55

+ atlas entry whose map card doesn't validate; per-run gates refuse claims

56

+ that fail replication (`make_*_mapcard.py`).

57

+- **Confidence taxonomy** on every artifact: L0 anecdotal · L1 correlational

58

+ · L2 method-robust · L3 causal.

59

+- **Everything local** — MLX / PyTorch-MPS on 16–64 GB Macs; quantized

60

+ checkpoints are studied in the form people actually run.

61

+

62

+## Repository layout

29

63

30

64

```text

31

−what a probing method reports

32

−≠ what the model actually computes

33

−≠ what is stable across methods, seeds, and datasets

34

−≠ what is causally verified by intervention

65

+research/ charter-driven paper trail: log, state of the art, 24 gaps, ranking

66

+src/modelmap/ capture (MLX taps, quantized models), probes, stats, atlas schema

67

+experiments/ micro-experiments A–H + candidates (hypothesis → run → analysis)

68

+atlas/ versioned map artifacts: map.json + provenance + confidence + map card

69

+publications/ official technical reports (rendered with live figures on the site)

70

+site/ modelmap.io (Express, server-rendered SVG maps, mobile-first)

71

+benchmarks/ harness, hardware manifests, checksummed promptsets

72

+tools/ check_headers, new_experiment, new_map, publish (the gate)

35

73

```

36

74

37

−## Confidence taxonomy

38

−

39

−Every atlas entry carries an explicit evidence level, visible on modelmap.io:

75

+## Quickstart (Apple Silicon, macOS 14+)

40

76

41

−```text

42

−Level 0 — anecdotal (single run, no controls; never published alone)

43

−Level 1 — correlational (controlled, replicated ≥3 seeds, ≥2 datasets)

44

−Level 2 — method-robust (Level 1 + agreement across ≥2 independent techniques)

45

−Level 3 — causal (Level 2 + intervention confirms the claim)

77

+```bash

78

+make setup # venv + deps (numpy, mlx, torch, zarr, safetensors)

79

+make test # 5 correctness tests (probes vs planted structure, FDR, map cards)

80

+make lint headers # ruff + mandatory author-header check

81

+make site-run # build and preview modelmap.io locally on :8140

82

+# regenerate any published map: see its mapcard.json "regenerate_command"

46

83

```

47

84

48

−## Status

49

−

50

−Project bootstrapped 2026-08-12. Phase 1 (ultra-deep literature research) is next.

51

−The full research charter is in [CLAUDE.md](CLAUDE.md); the auditable paper trail

52

−lives in [research/LOG.md](research/LOG.md).

85

+## Author

53

86

54

−Sister project: **localvm-research** (out-of-core LLM execution on consumer Macs) —

55

−findings cross-pollinate, in particular working-set prediction (Q5).

87

+**Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai) · [modelmap.io](https://modelmap.io)

88

+Sister project: [localvm-research](https://www.localvm.dev) (out-of-core LLM execution on consumer Macs).

56

89

57

−## Layout

90

+## Citation

58

91

59

92

```text

60

−research/ paper trail: log, state of the art, gaps, ranking, methodology

61

−src/modelmap/ the mapping library (MLX / PyTorch-MPS / Metal; CUDA never core)

62

−experiments/ micro-experiments A–H and prototype candidates (registered hypotheses)

63

−atlas/ versioned map artifacts with provenance + confidence levels

64

−site/ modelmap.io platform source (builds and previews locally on the Mac)

65

−benchmarks/ harness, hardware manifest, versioned prompt corpora

66

−tools/ check_headers, new_experiment, new_map, publish

93

+Boucher, S.-P. (2026). modelmap — Internal Cartography of Local Large

94

+Language Models. https://modelmap.io (see also Technical Report TR-2026-01).

67

95

```

68

96

69

−---

70

−

71

97

72

98

		`@@ -5,67 +5,93 @@ author: Simon-Pierre Boucher`
5	5	contact: contact@spboucher.ai
6	6	website: https://modelmap.io
7	7	created: 2026-08-12
8		−status: draft
	8	+modified: 2026-08-12
	9	+status: reviewed
9	10	---
10	11
11	12	# modelmap — Internal Cartography of Local Large Language Models
12	13
13		−modelmap investigates what can actually be known, measured, localized, and mapped
14		−about the internal organization of pretrained open-weight LLMs — where knowledge lives,
15		−how computation is distributed, which structures are stable across inputs, layers,
16		−scales, and model families — using only local, consumer-grade Apple Silicon hardware,
17		−and publishes the resulting maps as a rigorous, reproducible public atlas at
18		−[modelmap.io](https://modelmap.io).
19		−
20		−## Core research questions
21		−
22		−- Q1 — Localization. Where do capabilities, knowledge domains, languages, and behaviors reside? Localized or diffuse?
23		−- Q2 — Structure. Which internal structures are stable across inputs — and at what granularity are they real rather than probing artifacts?
24		−- Q3 — Comparability. Can internal maps be compared across sizes, checkpoints, quantization levels, and families?
25		−- Q4 — Cost. Which mapping techniques are feasible on a 16–64 GB Mac, and what is the accuracy/cost frontier of local interpretability?
26		−- Q5 — Utility. Do the maps predict anything useful — quantization sensitivity, pruning tolerance, working sets, failure modes?
27		−
28		−The distinction kept sharp throughout:
	14	+![status](https://img.shields.io/badge/status-active_research-6d4fc4)
	15	+![platform](https://img.shields.io/badge/platform-macOS_·_Apple_Silicon-0b0b0b)
	16	+![compute](https://img.shields.io/badge/compute-MLX_·_PyTorch--MPS-eb6834)
	17	+![atlas](https://img.shields.io/badge/atlas_entries-3-53389e)
	18	+![max confidence](https://img.shields.io/badge/max_confidence-Level_2_·_method--robust-0e9f4e)
	19	+![causal survival](https://img.shields.io/badge/correlational→causal_survival-0%2F2-9a6a00)
	20	+![registered runs](https://img.shields.io/badge/registered_runs-11-53389e)
	21	+![gate record](https://img.shields.io/badge/publication_gate-3_refusals_·_1_pass-8b8798)
	22	+![tests](https://img.shields.io/badge/tests-5_passing-0e9f4e)
	23	+![license](https://img.shields.io/badge/license-all_rights_reserved-555)
	24	+
	25	+modelmap discovers, measures, and maps the internal structure of
	26	+pretrained open-weight LLMs entirely on consumer Apple Silicon — and
	27	+publishes the results as a reproducible, confidence-labeled public atlas at
	28	+[modelmap.io](https://modelmap.io). Every map is versioned, provenanced
	29	+(commit + config + model hash + hardware manifest), regenerable by one
	30	+command, and labeled with the evidence level it actually earned. Negative
	31	+results and publication-gate refusals are first-class citizens.
	32	+
	33	+> "we think the model does X" → *"here is the map, its evidence level,
	34	+> and the script that rebuilds it"*
	35	+
	36	+## Headline results (2026-08-12 · Qwen3-0.6B-4bit · full report: [TR-2026-01](https://modelmap.io/publications/TR-2026-01))
	37	+
	38	+\| Finding \| Evidence \|
	39	+\|---\|---\|
	40	+\| Activation capture is nearly free under MLX — 1.004× plain prefill on a real 4-bit checkpoint (first Python capture from an actually-quantized local model) \| expH, 3 registered runs \|
	41	+\| Activation-store format rankings invert warm→cold — mmap 3–11× ahead warm, zarr 4.4× ahead cold; IO granularity governs, not the container \| expH run #2 (falsified own hypothesis) \|
	42	+\| Probes on separable classes measure the tokenizer, not the model — a random-init architecture twin matches the trained model at ceiling; shuffled-label controls miss it \| expA run #1 (published negative, atlas probes/v1) \|
	43	+\| Differential (real−twin) maps recover trained signal on structure-borne properties: agreement 25/28 signal layers, equation-validity 0.90 vs twin 0.58 \| expA run #2 (Level 1, atlas probes/v2) \|
	44	+\| Decodability ≠ causal support — the probe map's layer ranking failed two intervention tests (survival ledger 0/2) \| expC runs #1–#2 \|
	45	+\| A single agreement direction is causally necessary across layers 2–15 — rank-1 erasure removes 73–75% of the behavior, replicated across six fresh estimators \| expC run #4 (Level 2, atlas interventions/v1) \|
	46	+\| Two Level-3 attempts refused by the publication gate — incl. one that would have shipped a false claim without fresh re-registration \| expC runs #5–#6 \|
	47	+
	48	+## How it works
	49	+
	50	+- Registered hypotheses — every run is preceded by a written hypothesis
	51	+ with an explicit falsification criterion (`experiments/*/hypothesis.md`).
	52	+- Mandatory nulls — shuffled labels, random-init architecture twins,
	53	+ random-direction controls, FDR correction across unit scans.
	54	+- Machine-enforced publication gates — `tools/publish.py` refuses any
	55	+ atlas entry whose map card doesn't validate; per-run gates refuse claims
	56	+ that fail replication (`make_*_mapcard.py`).
	57	+- Confidence taxonomy on every artifact: L0 anecdotal · L1 correlational
	58	+ · L2 method-robust · L3 causal.
	59	+- Everything local — MLX / PyTorch-MPS on 16–64 GB Macs; quantized
	60	+ checkpoints are studied in the form people actually run.
	61	+
	62	+## Repository layout
29	63
30	64	```text
31		−what a probing method reports
32		−≠ what the model actually computes
33		−≠ what is stable across methods, seeds, and datasets
34		−≠ what is causally verified by intervention
	65	+research/ charter-driven paper trail: log, state of the art, 24 gaps, ranking
	66	+src/modelmap/ capture (MLX taps, quantized models), probes, stats, atlas schema
	67	+experiments/ micro-experiments A–H + candidates (hypothesis → run → analysis)
	68	+atlas/ versioned map artifacts: map.json + provenance + confidence + map card
	69	+publications/ official technical reports (rendered with live figures on the site)
	70	+site/ modelmap.io (Express, server-rendered SVG maps, mobile-first)
	71	+benchmarks/ harness, hardware manifests, checksummed promptsets
	72	+tools/ check_headers, new_experiment, new_map, publish (the gate)
35	73	```
36	74
37		−## Confidence taxonomy
38		−
39		−Every atlas entry carries an explicit evidence level, visible on modelmap.io:
	75	+## Quickstart (Apple Silicon, macOS 14+)
40	76
41		−```text
42		−Level 0 — anecdotal (single run, no controls; never published alone)
43		−Level 1 — correlational (controlled, replicated ≥3 seeds, ≥2 datasets)
44		−Level 2 — method-robust (Level 1 + agreement across ≥2 independent techniques)
45		−Level 3 — causal (Level 2 + intervention confirms the claim)
	77	+```bash
	78	+make setup # venv + deps (numpy, mlx, torch, zarr, safetensors)
	79	+make test # 5 correctness tests (probes vs planted structure, FDR, map cards)
	80	+make lint headers # ruff + mandatory author-header check
	81	+make site-run # build and preview modelmap.io locally on :8140
	82	+# regenerate any published map: see its mapcard.json "regenerate_command"
46	83	```
47	84
48		−## Status
49		−
50		−Project bootstrapped 2026-08-12. Phase 1 (ultra-deep literature research) is next.
51		−The full research charter is in [CLAUDE.md](CLAUDE.md); the auditable paper trail
52		−lives in [research/LOG.md](research/LOG.md).
	85	+## Author
53	86
54		−Sister project: localvm-research (out-of-core LLM execution on consumer Macs) —
55		−findings cross-pollinate, in particular working-set prediction (Q5).
	87	+Simon-Pierre Boucher · [contact@spboucher.ai](mailto:contact@spboucher.ai) · [modelmap.io](https://modelmap.io)
	88	+Sister project: [localvm-research](https://www.localvm.dev) (out-of-core LLM execution on consumer Macs).
56	89
57		−## Layout
	90	+## Citation
58	91
59	92	```text
60		−research/ paper trail: log, state of the art, gaps, ranking, methodology
61		−src/modelmap/ the mapping library (MLX / PyTorch-MPS / Metal; CUDA never core)
62		−experiments/ micro-experiments A–H and prototype candidates (registered hypotheses)
63		−atlas/ versioned map artifacts with provenance + confidence levels
64		−site/ modelmap.io platform source (builds and previews locally on the Mac)
65		−benchmarks/ harness, hardware manifest, versioned prompt corpora
66		−tools/ check_headers, new_experiment, new_map, publish
	93	+Boucher, S.-P. (2026). modelmap — Internal Cartography of Local Large
	94	+Language Models. https://modelmap.io (see also Technical Report TR-2026-01).
67	95	```
68	96
69		−---
70		−
71		−Author: Simon-Pierre Boucher — contact@spboucher.ai — https://modelmap.io — All rights reserved (research code).
	97	+All rights reserved (research code). © 2026 Simon-Pierre Boucher.
72	98