Phase 5 substrate + expH run #1: first measured result

- mapcard v0 schema + publish gate (G23), stats core (bootstrap/FDR/
  replication), probe harness with built-in shuffled-label controls
  (unit test caught a fancy-indexing shuffle bug pre-science)
- expH hypothesis registered before run; run #1 on M5 Max 48GB:
  (A) mmap 3.2-10.8x zarr on random-batch reads (confirmed >=2x bar);
  (B) MLX capture nearly free (1.02x), torch-MPS retain 1.22x (confirmed)
- expA hypothesis registered (dataset-variance > seed-variance)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Simon-Pierre Boucher committed 3 h ago (Aug 12, 2026) parent 18bd626

Showing 22 changed files with +1,292 and −21

modified Makefile +1 −1

		`@@ -16,7 +16,7 @@ setup:`
16	16	python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
17	17
18	18	lint:
19		− .venv/bin/ruff check src tools benchmarks
	19	+ .venv/bin/ruff check src tools benchmarks tests experiments
20	20
21	21	test:
22	22	.venv/bin/pytest -q

modified benchmarks/hardware_manifest.py +0 −0

modified benchmarks/harness.py +1 −1

		`@@ -41,7 +41,7 @@ def git_commit() -> str:`
41	41
42	42
43	43	def result_dir(experiment_id: str) -> Path:
44		− ts = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%SZ")
	44	+ ts = datetime.datetime.now(datetime.UTC).strftime("%Y%m%dT%H%M%SZ")
45	45	out = ROOT / "results" / experiment_id / ts
46	46	out.mkdir(parents=True)
47	47	return out

modified experiments/micro/expA_probe_reliability/hypothesis.md +34 −9

		`@@ -1,23 +1,48 @@`
1	1	---
2	2	project: modelmap
3		−document: expA_probe_reliability — hypothesis
	3	+document: expA_probe_reliability — hypothesis (registered before run)
4	4	author: Simon-Pierre Boucher
5	5	contact: contact@spboucher.ai
6	6	website: https://modelmap.io
7	7	created: 2026-08-12
8		−status: draft
	8	+modified: 2026-08-12
	9	+status: reviewed
9	10	---
10	11
11		−# Hypothesis — expA_probe_reliability
	12	+# Hypothesis — expA (probe reliability / the noise floor)
12	13
13		−> Probe reliability — noise floor of linear probes across seeds and datasets
	14	+> Probe reliability — noise floor of linear probes across seeds and datasets.
	15	+> Registered 2026-08-12, before any run. The harness
	16	+> (`src/modelmap/probes/linear.py` + `src/modelmap/stats/replication.py`) is
	17	+> implemented and unit-tested against planted structure and pure noise;
	18	+> no real-model run has happened yet.
14	19
15	20	```text
16		−Hypothesis : (to be registered before the first run)
17		−Falsification criterion : (explicit kill-number, registered in advance)
18		−Method : (including controls)
19		−Baseline / null : (shuffled labels / random directions / random init)
	21	+Hypothesis : On a 0.5B-class model, per-layer linear-probe maps
	22	+ for simple properties (language ID, code-vs-prose,
	23	+ arithmetic-context) are seed-stable (probe-accuracy
	24	+ SD across ≥5 training seeds < 2 points) but
	25	+ dataset-SENSITIVE: switching prompt corpus moves
	26	+ per-layer accuracy by more than the seed SD on at
	27	+ least a third of layers — i.e. the dataset, not the
	28	+ probe seed, is the dominant noise source
	29	+ (Bolukbasi-consistent).
	30	+Falsification criterion : Dies if seed variance ≥ dataset variance across the
	31	+ board (then seeds dominate and every later map needs
	32	+ more seeds, not more corpora); the harness itself is
	33	+ invalid if selectivity on a random-init model
	34	+ exceeds 0.05 (fake structure).
	35	+Method : Qwen3-0.6B (or Pythia-410M fallback) via mlx-lm;
	36	+ capture residual stream at every layer over ≥2
	37	+ disjoint prompt sets per property (versioned,
	38	+ checksummed promptsets); probe_with_control per
	39	+ (layer, property, seed) for 5 seeds; report
	40	+ selectivity, bootstrap CIs, BH-FDR across the layer
	41	+ scan; replication_rate on top-k layer sets.
	42	+Baseline / null : shuffled-label control (built into every probe);
	43	+ random-init model of identical architecture;
	44	+ random-direction projections.
20	45	Result : (pending)
21	46	Interpretation : (pending — with explicit confidence level)
22		−Next experiment : (pending)
	47	+Next experiment : (pending — expA output becomes the first map card)
23	48	```

added experiments/micro/expH_capture_cost_frontier/analysis.md +78 −0

		`@@ -0,0 +1,78 @@`
	1	+---
	2	+project: modelmap
	3	+document: expH_capture_cost_frontier — analysis (run #1)
	4	+author: Simon-Pierre Boucher
	5	+contact: contact@spboucher.ai
	6	+website: https://modelmap.io
	7	+created: 2026-08-12
	8	+modified: 2026-08-12
	9	+status: reviewed
	10	+---
	11	+
	12	+# Analysis — expH run #1 (storage formats + hook overhead)
	13	+
	14	+Run: `results/expH_capture_cost_frontier/20260812T052441Z/results.json`
	15	+Hardware: Apple M5 Max, 48 GB unified memory, macOS 27.0 · torch 2.13.0 (MPS) ·
	16	+MLX 0.32.0 · Python 3.14.4. Hypothesis registered before the run in
	17	+`hypothesis.md`; config, seed, commit and full manifest embedded in the JSON.
	18	+
	19	+```text
	20	+Hypothesis : (A) raw mmap ≥ 2× zarr on random-batch reads (warm);
	21	+ (B) retain-capture < 2× plain inference on both
	22	+ backends; copy+write dominates capture cost.
	23	+Falsification criterion : (A) dies at ratio < 1.25×; (B) dies at retain > 2×
	24	+ or total > 3×.
	25	+Method / Baseline : as registered (see hypothesis.md) — 3 repeats/cell,
	26	+ plain-forward baselines, sequential-scan references.
	27	+Result : (A) CONFIRMED. Random-batch reads: raw-mmap 4.09 GB/s,
	28	+ safetensors(mmap) 4.26, zarr-uncompressed 1.27,
	29	+ zarr-zstd 0.38 → mmap/zarr ratio 3.2× (uncompressed)
	30	+ and 10.8× (zstd), both past the 2× bar.
	31	+ Writes: safetensors 8.10 GB/s (single large write),
	32	+ raw-mmap chunked 2.82, zarr-uncompressed 1.49,
	33	+ zarr-zstd 0.54. Seq scans: 3.9–7.3 GB/s all formats.
	34	+ (B) CONFIRMED, asymmetrically. torch-MPS: plain
	35	+ 46.7 ms/fwd (±1.6) → retain 57.2 (1.22×) →
	36	+ retain+copy+write 68.8 (1.47×). MLX: plain 36.2
	37	+ (±2.6) → retain 36.9 (1.02×) → +copy+write 40.2
	38	+ (1.11×). Retain never exceeds 1.25×; totals never
	39	+ exceed 1.5× — far under the 2×/3× kill lines.
	40	+ The "copy+write dominates" sub-claim holds on MLX
	41	+ (+3.3 ms vs +0.7 ms retain) but on torch-MPS the
	42	+ two costs are comparable (+10.5 ms retain,
	43	+ +11.6 ms copy+write) — retain itself is expensive
	44	+ on MPS, matching its eager hook materialization.
	45	+ Bonus observation (not registered, Level 0): MLX
	46	+ baseline is 1.29× faster than torch-MPS on the
	47	+ identical architecture, and MLX capture is nearly
	48	+ free (2%), consistent with lazy-graph retention.
	49	+Interpretation : Level 0–1. Within-machine replication only
	50	+ (3 repeats/cell, tight variance ≤ ±4 ms); single
	51	+ hardware, single size class, warm cache, synthetic
	52	+ model — all declared in advance. Design decisions
	53	+ this licenses NOW: (1) activation stores are
	54	+ mmap-backed flat files (raw or safetensors container
	55	+ for self-description) — zarr is eliminated for the
	56	+ SAE-shuffle pattern unless cold-cache reverses the
	57	+ ordering; (2) MLX is the preferred capture backend,
	58	+ with torch-MPS as the compatibility path; (3)
	59	+ capture-cost is NOT the bottleneck at this scale —
	60	+ the 4 TB/500M-token storage wall (notes §4.9)
	61	+ remains the real constraint.
	62	+Next experiment : run #2 = cold-cache pass (purge/F_NOCACHE) to test
	63	+ whether the mmap>zarr ordering survives; run #3 =
	64	+ same capture modes on a real 0.5B checkpoint via
	65	+ mlx-lm vs transformers-MPS; then replicate the whole
	66	+ grid on ≥2 cluster Macs (M3U96a, M2U64) to promote
	67	+ the cost table toward Level 1 cross-hardware.
	68	+```
	69	+
	70	+## Notes
	71	+
	72	+- zarr numbers use chunk = one write-block (4096×4096); different chunk
	73	+ geometries could narrow the gap — a registered caveat, not tested here.
	74	+- The torch-MPS retain overhead (22%) is per-layer-output retention of
	75	+ ~100 MB/forward; capture pipelines that subsample layers will scale it down
	76	+ roughly linearly.
	77	+- All timing cells and raw seconds are in the results JSON with the hardware
	78	+ manifest; the figure-ready aggregation is deferred to the atlas pipeline.

modified experiments/micro/expH_capture_cost_frontier/hypothesis.md +50 −8

		`@@ -1,23 +1,65 @@`
1	1	---
2	2	project: modelmap
3		−document: expH_capture_cost_frontier — hypothesis
	3	+document: expH_capture_cost_frontier — hypothesis (run #1, registered before run)
4	4	author: Simon-Pierre Boucher
5	5	contact: contact@spboucher.ai
6	6	website: https://modelmap.io
7	7	created: 2026-08-12
8		−status: draft
	8	+modified: 2026-08-12
	9	+status: reviewed
9	10	---
10	11
11		−# Hypothesis — expH_capture_cost_frontier
	12	+# Hypothesis — expH run #1 (storage formats + hook overhead)
12	13
13		−> Capture cost frontier — what can you map on which Mac (macOS-specific)
	14	+> Capture cost frontier — what can you map on which Mac (macOS-specific).
	15	+> Run #1 scope: (A) activation-store formats on APFS, (B) capture/hook
	16	+> overhead in MLX vs PyTorch-MPS on a synthetic transformer. Registered
	17	+> 2026-08-12 before the first benchmark execution.
14	18
15	19	```text
16		−Hypothesis : (to be registered before the first run)
17		−Falsification criterion : (explicit kill-number, registered in advance)
18		−Method : (including controls)
19		−Baseline / null : (shuffled labels / random directions / random init)
	20	+Hypothesis : (A) For the SAE-shuffle access pattern (random row
	21	+ batches), raw np.memmap sustains ≥ 2× the read
	22	+ throughput of chunked zarr on APFS (warm cache),
	23	+ because dense fp16 activations gain little from
	24	+ compression while chunk decode adds latency.
	25	+ (B) Activation capture (retain per-layer outputs)
	26	+ costs < 2× plain inference on both MLX and
	27	+ PyTorch-MPS at the 12-layer/d1024 scale, and the
	28	+ device→CPU copy + disk write — not the hook/retain
	29	+ mechanism — dominates capture cost.
	30	+
	31	+Falsification criterion : (A) dies if zarr (any codec) matches or beats mmap
	32	+ on random-batch reads (ratio < 1.25×).
	33	+ (B) dies if retain-only capture overhead exceeds
	34	+ 2× plain inference on either backend; the "capture
	35	+ is cheap on consumer Macs" premise dies at > 3×
	36	+ including copy+write.
	37	+
	38	+Method : Storage: 100k rows × 4096 dims fp16 (~0.8 GB) per
	39	+ format (raw mmap, safetensors-mmap, zarr zstd, zarr
	40	+ uncompressed); sequential write; sequential scan;
	41	+ random-batch reads (4096 rows/batch, SAE-shuffle
	42	+ pattern); 3 repeats/cell. Compute: synthetic
	43	+ pre-norm transformer (12 layers, d_model 1024,
	44	+ 4 heads, seq 512, batch 8, fp16) implemented
	45	+ identically in torch-MPS and MLX; modes = plain /
	46	+ retain-on-device / retain+CPU-copy+mmap-write;
	47	+ 20 timed forwards (10 for write mode), 3 repeats,
	48	+ explicit device synchronization.
	49	+
	50	+Baseline / null : plain forward (no capture) per backend; sequential
	51	+ scan as the reference for random-read degradation.
	52	+
20	53	Result : (pending)
	54	+
21	55	Interpretation : (pending — with explicit confidence level)
	56	+
22	57	Next experiment : (pending)
23	58	```
	59	+
	60	+Known limitations, declared in advance. Warm-cache regime (the 0.8 GB
	61	+store fits in the unified buffer cache): read numbers upper-bound the cold
	62	+case; a purge/F_NOCACHE cold pass is a registered follow-up (run #2), and the
	63	+sister project's cold-cache NVMe ceiling (localvm expH: ~13.1 GB/s at
	64	+≥256 KiB, QD≥4) bounds what run #2 can find. Synthetic model ≠ real
	65	+checkpoint: run #3 will repeat mode timings on a real 0.5B model via mlx-lm.

added experiments/micro/expH_capture_cost_frontier/implementation/benchmark.py +291 −0

		`@@ -0,0 +1,291 @@`
	1	+#!/usr/bin/env python3
	2	+# =============================================================================
	3	+# Project : modelmap
	4	+# File : experiments/micro/expH_capture_cost_frontier/implementation/benchmark.py
	5	+# Purpose : Run #1 — storage-format throughput on APFS + hook overhead MLX/MPS
	6	+# Author : Simon-Pierre Boucher
	7	+# Contact : contact@spboucher.ai
	8	+# Website : https://modelmap.io
	9	+# Created : 2026-08-12
	10	+# Modified : 2026-08-12
	11	+# Platform : macOS / Apple Silicon (arm64) — MLX / Metal / MPS
	12	+# License : All rights reserved (research code)
	13	+# =============================================================================
	14	+"""expH run #1. See hypothesis.md (registered before this run).
	15	+
	16	+Part A — activation-store formats on APFS (warm cache, declared):
	17	+ raw np.memmap \| safetensors (mmap) \| zarr zstd \| zarr uncompressed
	18	+ sequential write, sequential scan, random-batch reads (SAE-shuffle).
	19	+Part B — capture overhead on a synthetic 12-layer transformer (fp16):
	20	+ torch-MPS forward hooks vs MLX retained arrays;
	21	+ modes: plain / retain-on-device / retain + CPU copy + mmap write.
	22	+"""
	23	+
	24	+from __future__ import annotations
	25	+
	26	+import json
	27	+import shutil
	28	+import subprocess
	29	+import sys
	30	+import tempfile
	31	+import time
	32	+from pathlib import Path
	33	+
	34	+import numpy as np
	35	+
	36	+ROOT = Path(__file__).resolve().parents[4]
	37	+sys.path.insert(0, str(ROOT / "benchmarks"))
	38	+from hardware_manifest import manifest
	39	+
	40	+REPEATS = 3
	41	+ROWS, DIM = 100_000, 4096 # ~0.82 GB fp16 per format
	42	+WRITE_CHUNK = 4_096 # rows per write
	43	+BATCH = 4_096 # rows per random-batch read
	44	+N_BATCHES = 24
	45	+SEED = 0
	46	+
	47	+# Part B config
	48	+L, D_MODEL, N_HEADS, SEQ, BSZ = 12, 1024, 4, 512, 8
	49	+N_FWD, N_FWD_WRITE, WARMUP = 20, 10, 3
	50	+
	51	+
	52	+def _timed(fn, repeats=REPEATS):
	53	+ out = []
	54	+ for _ in range(repeats):
	55	+ t0 = time.perf_counter()
	56	+ fn()
	57	+ out.append(time.perf_counter() - t0)
	58	+ return out
	59	+
	60	+
	61	+# ---------------------------------------------------------------- Part A
	62	+def bench_storage(workdir: Path) -> list[dict]:
	63	+ rng = np.random.default_rng(SEED)
	64	+ data = rng.standard_normal((WRITE_CHUNK, DIM)).astype(np.float16)
	65	+ batches = [rng.integers(0, ROWS, BATCH) for _ in range(N_BATCHES)]
	66	+ total_bytes = ROWS * DIM * 2
	67	+ batch_bytes = BATCH * DIM * 2 * N_BATCHES
	68	+ results = []
	69	+
	70	+ def record(fmt, op, times, nbytes):
	71	+ results.append({
	72	+ "format": fmt, "op": op, "bytes": nbytes,
	73	+ "seconds": times, "gb_per_s_mean": nbytes / 2**30 / np.mean(times),
	74	+ })
	75	+ print(f" {fmt:18s} {op:12s} {nbytes/2**30/np.mean(times):8.2f} GB/s")
	76	+
	77	+ # ---- raw np.memmap
	78	+ p = workdir / "acts.raw"
	79	+ def write_raw():
	80	+ m = np.memmap(p, dtype=np.float16, mode="w+", shape=(ROWS, DIM))
	81	+ for i in range(0, ROWS, WRITE_CHUNK):
	82	+ end = min(i + WRITE_CHUNK, ROWS)
	83	+ m[i:end] = data[: end - i]
	84	+ m.flush(); del m
	85	+ record("raw-mmap", "write", _timed(write_raw), total_bytes)
	86	+ m = np.memmap(p, dtype=np.float16, mode="r", shape=(ROWS, DIM))
	87	+ record("raw-mmap", "seq-scan", _timed(lambda m=m: float(np.asarray(m).sum(dtype=np.float32))), total_bytes)
	88	+ record("raw-mmap", "random-batch", _timed(lambda m=m: [m[b].sum(dtype=np.float32) for b in batches]), batch_bytes)
	89	+ del m
	90	+
	91	+ # ---- safetensors (mmap-backed numpy)
	92	+ from safetensors import safe_open
	93	+ from safetensors.numpy import save_file
	94	+ p = workdir / "acts.safetensors"
	95	+ full = np.memmap(workdir / "acts.raw", dtype=np.float16, mode="r", shape=(ROWS, DIM))
	96	+ def write_st(full=full, p=p):
	97	+ save_file({"acts": np.asarray(full)}, str(p))
	98	+ record("safetensors", "write", _timed(write_st), total_bytes)
	99	+ def open_st():
	100	+ return safe_open(str(p), framework="np")
	101	+ f = open_st()
	102	+ t = f.get_tensor("acts") # mmap-backed load
	103	+ record("safetensors", "seq-scan", _timed(lambda t=t: float(t.sum(dtype=np.float32))), total_bytes)
	104	+ record("safetensors", "random-batch", _timed(lambda t=t: [t[b].sum(dtype=np.float32) for b in batches]), batch_bytes)
	105	+ del t, f, full
	106	+
	107	+ # ---- zarr (zstd default) and uncompressed
	108	+ import zarr
	109	+ for codec, name in ((None, "zarr-uncompressed"), ("default", "zarr-zstd")):
	110	+ p = workdir / f"acts_{name}.zarr"
	111	+ kwargs = {} if codec == "default" else {"compressors": None}
	112	+ def write_zarr(p=p, kwargs=kwargs):
	113	+ if p.exists():
	114	+ shutil.rmtree(p)
	115	+ z = zarr.create_array(store=str(p), shape=(ROWS, DIM), chunks=(WRITE_CHUNK, DIM),
	116	+ dtype=np.float16, **kwargs)
	117	+ for i in range(0, ROWS, WRITE_CHUNK):
	118	+ end = min(i + WRITE_CHUNK, ROWS)
	119	+ z[i:end] = data[: end - i]
	120	+ record(name, "write", _timed(write_zarr), total_bytes)
	121	+ z = zarr.open_array(store=str(p), mode="r")
	122	+ record(name, "seq-scan", _timed(lambda z=z: float(z[:].sum(dtype=np.float32))), total_bytes)
	123	+ record(name, "random-batch", _timed(lambda z=z: [z[np.sort(b)].sum(dtype=np.float32) for b in batches]), batch_bytes)
	124	+
	125	+ return results
	126	+
	127	+
	128	+# ---------------------------------------------------------------- Part B
	129	+def bench_torch_mps(workdir: Path) -> list[dict]:
	130	+ import torch
	131	+ from torch import nn
	132	+ assert torch.backends.mps.is_available(), "MPS required"
	133	+ dev, dt = torch.device("mps"), torch.float16
	134	+
	135	+ class Block(nn.Module):
	136	+ def __init__(self):
	137	+ super().__init__()
	138	+ self.ln1, self.ln2 = nn.LayerNorm(D_MODEL), nn.LayerNorm(D_MODEL)
	139	+ self.attn = nn.MultiheadAttention(D_MODEL, N_HEADS, batch_first=True)
	140	+ self.mlp = nn.Sequential(nn.Linear(D_MODEL, 4 * D_MODEL), nn.GELU(),
	141	+ nn.Linear(4 * D_MODEL, D_MODEL))
	142	+ def forward(self, x):
	143	+ h = self.ln1(x)
	144	+ x = x + self.attn(h, h, h, need_weights=False)[0]
	145	+ return x + self.mlp(self.ln2(x))
	146	+
	147	+ torch.manual_seed(SEED)
	148	+ model = nn.Sequential(*[Block() for _ in range(L)]).to(dev, dt).eval()
	149	+ x = torch.randn(BSZ, SEQ, D_MODEL, device=dev, dtype=dt)
	150	+ store = np.memmap(workdir / "torch_capture.raw", dtype=np.float16, mode="w+",
	151	+ shape=(N_FWD_WRITE * L * BSZ * SEQ, D_MODEL))
	152	+
	153	+ def run(n, capture, to_disk):
	154	+ captured, row = [], 0
	155	+ hooks = []
	156	+ if capture:
	157	+ def hook(_m, _i, out):
	158	+ captured.append(out)
	159	+ hooks = [b.register_forward_hook(hook) for b in model]
	160	+ with torch.no_grad():
	161	+ for _ in range(WARMUP):
	162	+ model(x)
	163	+ torch.mps.synchronize()
	164	+ t0 = time.perf_counter()
	165	+ for _ in range(n):
	166	+ captured.clear()
	167	+ model(x)
	168	+ if to_disk:
	169	+ for c in captured:
	170	+ a = c.to("cpu").numpy().reshape(-1, D_MODEL)
	171	+ store[row:row + a.shape[0]] = a
	172	+ row += a.shape[0]
	173	+ torch.mps.synchronize()
	174	+ dt_s = time.perf_counter() - t0
	175	+ for h in hooks:
	176	+ h.remove()
	177	+ return dt_s / n
	178	+
	179	+ out = []
	180	+ for mode, cap, disk, n in (("plain", False, False, N_FWD),
	181	+ ("retain", True, False, N_FWD),
	182	+ ("retain+copy+write", True, True, N_FWD_WRITE)):
	183	+ times = [run(n, cap, disk) for _ in range(REPEATS)]
	184	+ out.append({"backend": "torch-mps", "mode": mode,
	185	+ "s_per_forward": times,
	186	+ "tokens_per_s_mean": BSZ * SEQ / np.mean(times)})
	187	+ print(f" torch-mps {mode:22s} {np.mean(times)*1000:8.1f} ms/fwd")
	188	+ return out
	189	+
	190	+
	191	+def bench_mlx(workdir: Path) -> list[dict]:
	192	+ import mlx.core as mx
	193	+ import mlx.nn as mnn
	194	+
	195	+ class Block(mnn.Module):
	196	+ def __init__(self):
	197	+ super().__init__()
	198	+ self.ln1, self.ln2 = mnn.LayerNorm(D_MODEL), mnn.LayerNorm(D_MODEL)
	199	+ self.attn = mnn.MultiHeadAttention(D_MODEL, N_HEADS)
	200	+ self.fc1, self.fc2 = mnn.Linear(D_MODEL, 4 * D_MODEL), mnn.Linear(4 * D_MODEL, D_MODEL)
	201	+ def __call__(self, x):
	202	+ h = self.ln1(x)
	203	+ x = x + self.attn(h, h, h)
	204	+ return x + self.fc2(mnn.gelu(self.fc1(self.ln2(x))))
	205	+
	206	+ mx.random.seed(SEED)
	207	+ blocks = [Block() for _ in range(L)]
	208	+ for b in blocks:
	209	+ b.set_dtype(mx.float16)
	210	+ x = mx.random.normal((BSZ, SEQ, D_MODEL)).astype(mx.float16)
	211	+ store = np.memmap(workdir / "mlx_capture.raw", dtype=np.float16, mode="w+",
	212	+ shape=(N_FWD_WRITE * L * BSZ * SEQ, D_MODEL))
	213	+
	214	+ def fwd(capture):
	215	+ captured, h = [], x
	216	+ for b in blocks:
	217	+ h = b(h)
	218	+ if capture:
	219	+ captured.append(h)
	220	+ return h, captured
	221	+
	222	+ def run(n, capture, to_disk):
	223	+ row = 0
	224	+ for _ in range(WARMUP):
	225	+ out, cap = fwd(capture)
	226	+ mx.eval(out, *cap)
	227	+ t0 = time.perf_counter()
	228	+ for _ in range(n):
	229	+ out, cap = fwd(capture)
	230	+ mx.eval(out, *cap)
	231	+ if to_disk:
	232	+ for c in cap:
	233	+ a = np.array(c, copy=False).reshape(-1, D_MODEL)
	234	+ store[row:row + a.shape[0]] = a
	235	+ row += a.shape[0]
	236	+ return (time.perf_counter() - t0) / n
	237	+
	238	+ out = []
	239	+ for mode, cap, disk, n in (("plain", False, False, N_FWD),
	240	+ ("retain", True, False, N_FWD),
	241	+ ("retain+copy+write", True, True, N_FWD_WRITE)):
	242	+ times = [run(n, cap, disk) for _ in range(REPEATS)]
	243	+ out.append({"backend": "mlx", "mode": mode,
	244	+ "s_per_forward": times,
	245	+ "tokens_per_s_mean": BSZ * SEQ / np.mean(times)})
	246	+ print(f" mlx {mode:22s} {np.mean(times)*1000:8.1f} ms/fwd")
	247	+ return out
	248	+
	249	+
	250	+# ---------------------------------------------------------------- main
	251	+def main() -> int:
	252	+ workdir = Path(tempfile.mkdtemp(prefix="modelmap_expH_"))
	253	+ print(f"expH run #1 — workdir {workdir}")
	254	+ try:
	255	+ print("Part A — storage formats (warm cache):")
	256	+ storage = bench_storage(workdir)
	257	+ print("Part B — capture overhead:")
	258	+ compute = bench_torch_mps(workdir) + bench_mlx(workdir)
	259	+ finally:
	260	+ shutil.rmtree(workdir, ignore_errors=True)
	261	+
	262	+ commit = subprocess.run(["git", "rev-parse", "HEAD"], cwd=ROOT,
	263	+ capture_output=True, text=True, check=False).stdout.strip()
	264	+ ts = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime())
	265	+ outdir = ROOT / "results" / "expH_capture_cost_frontier" / ts
	266	+ outdir.mkdir(parents=True)
	267	+ doc = {
	268	+ "experiment": "expH_capture_cost_frontier",
	269	+ "run": 1,
	270	+ "scope": "storage formats (warm cache) + capture overhead, synthetic model",
	271	+ "commit": commit,
	272	+ "config": {
	273	+ "storage": {"rows": ROWS, "dim": DIM, "write_chunk": WRITE_CHUNK,
	274	+ "batch": BATCH, "n_batches": N_BATCHES, "repeats": REPEATS,
	275	+ "cache": "warm (declared limitation; cold pass = run #2)"},
	276	+ "compute": {"layers": L, "d_model": D_MODEL, "heads": N_HEADS,
	277	+ "seq": SEQ, "batch": BSZ, "dtype": "float16",
	278	+ "n_forwards": N_FWD, "warmup": WARMUP, "repeats": REPEATS},
	279	+ "seed": SEED,
	280	+ },
	281	+ "manifest": manifest(),
	282	+ "storage": storage,
	283	+ "compute": compute,
	284	+ }
	285	+ (outdir / "results.json").write_text(json.dumps(doc, indent=2) + "\n")
	286	+ print(f"results -> {outdir / 'results.json'}")
	287	+ return 0
	288	+
	289	+
	290	+if __name__ == "__main__":
	291	+ sys.exit(main())

modified research/LOG.md +45 −0

		`@@ -195,3 +195,48 @@ patterns; (2) expA probe-reliability harness with the full controls`
195	195	doctrine; (3) map-cards v0 schema in src/modelmap/atlas enforced by
196	196	tools/publish.py. Register hypotheses in experiments/micro/expH and expA
197	197	before their first runs.
	198	+
	199	+---
	200	+
	201	+## 2026-08-12 05:30 EDT — Phase 5 substrate + expH run #1 (FIRST MEASURED RESULT)
	202	+
	203	+Question. Build the substrate; then: which activation-store format wins
	204	+on APFS, and what does capture actually cost on MLX vs PyTorch-MPS?
	205	+
	206	+Built.
	207	+- `src/modelmap/atlas/mapcard.py` — map-card schema v0 (G23): provenance +
	208	+ per-level requirements (L1: ≥3 seeds, ≥2 promptsets, replication_rate;
	209	+ L2: ≥2 methods; L3: interventions + featurizer class). `tools/publish.py`
	210	+ is the gate: refuses any atlas entry without a valid card.
	211	+- `src/modelmap/stats/replication.py` — bootstrap CIs, BH-FDR, selectivity,
	212	+ Jaccard replication_rate (the §8.3 metric).
	213	+- `src/modelmap/probes/linear.py` — probe_with_control: every probe carries
	214	+ its shuffled-label control by construction. **The unit test caught a real
	215	+ bug before any science: `rng.shuffle(y[tr])` shuffles a fancy-indexed
	216	+ COPY, so the "control" was training on true labels.** Fixed; tests now
	217	+ verify the probe separates planted structure (selectivity>0.3) from pure
	218	+ noise (\|selectivity\|<0.15). 5/5 tests pass; ruff clean; env = Python
	219	+ 3.14.4, numpy 2.5.2, torch 2.13 (MPS), mlx 0.32, zarr 3.3.
	220	+
	221	+expH run #1 (hypothesis registered BEFORE the run; M5 Max 48 GB,
	222	+macOS 27.0; results/expH_capture_cost_frontier/20260812T052441Z/):
	223	+- (A) CONFIRMED. Random-batch reads (SAE-shuffle pattern, warm cache):
	224	+ raw-mmap 4.09 GB/s, safetensors(mmap) 4.26, zarr-uncompressed 1.27,
	225	+ zarr-zstd 0.38 → mmap beats zarr 3.2×/10.8×, past the pre-registered 2×
	226	+ bar. safetensors write wins at 8.10 GB/s (single large write).
	227	+- (B) CONFIRMED, asymmetrically. Capture overhead: torch-MPS retain
	228	+ 1.22× plain, +copy+write 1.47×; **MLX retain 1.02× (capture nearly free,
	229	+ lazy-graph retention), +copy+write 1.11×.** MLX baseline also 1.29×
	230	+ faster than torch-MPS on the identical synthetic 12-layer model.
	231	+- Confidence: Level 0–1 (3 repeats, tight variance, but single machine,
	232	+ warm cache, synthetic model — all declared in advance).
	233	+- Decisions licensed: activation stores = mmap-backed flat files (zarr
	234	+ eliminated for the shuffle pattern pending the cold-cache pass); MLX is
	235	+ the preferred capture backend; capture cost is not the bottleneck — the
	236	+ storage wall is.
	237	+
	238	+Next. expH run #2 (cold cache) and #3 (real 0.5B checkpoint via
	239	+mlx-lm); then expA first run (hypothesis registered: dataset variance >
	240	+seed variance; harness validity gate = selectivity<0.05 on random-init).
	241	+expA's output becomes the first real map card. Replicate expH grid on
	242	+M3U96a/M2U64 to move the cost table cross-hardware.

added results/expH_capture_cost_frontier/20260812T052441Z/results.json +252 −0

		`@@ -0,0 +1,252 @@`
	1	+{
	2	+ "experiment": "expH_capture_cost_frontier",
	3	+ "run": 1,
	4	+ "scope": "storage formats (warm cache) + capture overhead, synthetic model",
	5	+ "commit": "18bd626391ff0edd868a0c2e3d257df055247b55",
	6	+ "config": {
	7	+ "storage": {
	8	+ "rows": 100000,
	9	+ "dim": 4096,
	10	+ "write_chunk": 4096,
	11	+ "batch": 4096,
	12	+ "n_batches": 24,
	13	+ "repeats": 3,
	14	+ "cache": "warm (declared limitation; cold pass = run #2)"
	15	+ },
	16	+ "compute": {
	17	+ "layers": 12,
	18	+ "d_model": 1024,
	19	+ "heads": 4,
	20	+ "seq": 512,
	21	+ "batch": 8,
	22	+ "dtype": "float16",
	23	+ "n_forwards": 20,
	24	+ "warmup": 3,
	25	+ "repeats": 3
	26	+ },
	27	+ "seed": 0
	28	+ },
	29	+ "manifest": {
	30	+ "author": "Simon-Pierre Boucher",
	31	+ "contact": "contact@spboucher.ai",
	32	+ "website": "https://modelmap.io",
	33	+ "chip": {
	34	+ "brand": "Apple M5 Max",
	35	+ "cores_total": 18,
	36	+ "cores_performance": 6,
	37	+ "cores_efficiency": 12
	38	+ },
	39	+ "memory": {
	40	+ "unified_gb": 48.0,
	41	+ "pagesize": 16384
	42	+ },
	43	+ "os": {
	44	+ "system": "Darwin",
	45	+ "version": "27.0",
	46	+ "arch": "arm64"
	47	+ },
	48	+ "software": {
	49	+ "python": "3.14.4",
	50	+ "numpy": "2.5.2",
	51	+ "mlx": "0.32.0",
	52	+ "torch": "2.13.0",
	53	+ "safetensors": "0.8.0"
	54	+ }
	55	+ },
	56	+ "storage": [
	57	+ {
	58	+ "format": "raw-mmap",
	59	+ "op": "write",
	60	+ "bytes": 819200000,
	61	+ "seconds": [
	62	+ 0.22015058299984958,
	63	+ 0.2848355409996657,
	64	+ 0.30594250000012835
	65	+ ],
	66	+ "gb_per_s_mean": 2.8224658639944686
	67	+ },
	68	+ {
	69	+ "format": "raw-mmap",
	70	+ "op": "seq-scan",
	71	+ "bytes": 819200000,
	72	+ "seconds": [
	73	+ 0.16795029199965938,
	74	+ 0.10112945800028683,
	75	+ 0.10645208299956721
	76	+ ],
	77	+ "gb_per_s_mean": 6.094871747871148
	78	+ },
	79	+ {
	80	+ "format": "raw-mmap",
	81	+ "op": "random-batch",
	82	+ "bytes": 805306368,
	83	+ "seconds": [
	84	+ 0.1789496669998698,
	85	+ 0.1846267920000173,
	86	+ 0.1863629999998011
	87	+ ],
	88	+ "gb_per_s_mean": 4.0913594454426585
	89	+ },
	90	+ {
	91	+ "format": "safetensors",
	92	+ "op": "write",
	93	+ "bytes": 819200000,
	94	+ "seconds": [
	95	+ 0.11196637499961071,
	96	+ 0.08140020800055936,
	97	+ 0.0892282079994402
	98	+ ],
	99	+ "gb_per_s_mean": 8.09929422718254
	100	+ },
	101	+ {
	102	+ "format": "safetensors",
	103	+ "op": "seq-scan",
	104	+ "bytes": 819200000,
	105	+ "seconds": [
	106	+ 0.10347433399965666,
	107	+ 0.10620929100059584,
	108	+ 0.1054912500003411
	109	+ ],
	110	+ "gb_per_s_mean": 7.262058434609323
	111	+ },
	112	+ {
	113	+ "format": "safetensors",
	114	+ "op": "random-batch",
	115	+ "bytes": 805306368,
	116	+ "seconds": [
	117	+ 0.1736507920004442,
	118	+ 0.176909834000071,
	119	+ 0.17781483300041145
	120	+ ],
	121	+ "gb_per_s_mean": 4.258335548464703
	122	+ },
	123	+ {
	124	+ "format": "zarr-uncompressed",
	125	+ "op": "write",
	126	+ "bytes": 819200000,
	127	+ "seconds": [
	128	+ 0.4914736250002534,
	129	+ 0.5174622080003246,
	130	+ 0.524438000000373
	131	+ ],
	132	+ "gb_per_s_mean": 1.4926682000928473
	133	+ },
	134	+ {
	135	+ "format": "zarr-uncompressed",
	136	+ "op": "seq-scan",
	137	+ "bytes": 819200000,
	138	+ "seconds": [
	139	+ 0.16390270800002327,
	140	+ 0.15708874999927502,
	141	+ 0.1571942090004086
	142	+ ],
	143	+ "gb_per_s_mean": 4.78646374688684
	144	+ },
	145	+ {
	146	+ "format": "zarr-uncompressed",
	147	+ "op": "random-batch",
	148	+ "bytes": 805306368,
	149	+ "seconds": [
	150	+ 0.5920942090006065,
	151	+ 0.6018026250003459,
	152	+ 0.5726199580003595
	153	+ ],
	154	+ "gb_per_s_mean": 1.2736929590411328
	155	+ },
	156	+ {
	157	+ "format": "zarr-zstd",
	158	+ "op": "write",
	159	+ "bytes": 819200000,
	160	+ "seconds": [
	161	+ 1.582709333000821,
	162	+ 1.3768872919999922,
	163	+ 1.252639000000272
	164	+ ],
	165	+ "gb_per_s_mean": 0.5433737718255992
	166	+ },
	167	+ {
	168	+ "format": "zarr-zstd",
	169	+ "op": "seq-scan",
	170	+ "bytes": 819200000,
	171	+ "seconds": [
	172	+ 0.24328949999926408,
	173	+ 0.17038491599942063,
	174	+ 0.16977412500000355
	175	+ ],
	176	+ "gb_per_s_mean": 3.922913845079177
	177	+ },
	178	+ {
	179	+ "format": "zarr-zstd",
	180	+ "op": "random-batch",
	181	+ "bytes": 805306368,
	182	+ "seconds": [
	183	+ 2.0657694999999876,
	184	+ 1.9662967499998558,
	185	+ 1.9579576249998354
	186	+ ],
	187	+ "gb_per_s_mean": 0.3756245462377428
	188	+ }
	189	+ ],
	190	+ "compute": [
	191	+ {
	192	+ "backend": "torch-mps",
	193	+ "mode": "plain",
	194	+ "s_per_forward": [
	195	+ 0.04537852500002373,
	196	+ 0.045744383349983764,
	197	+ 0.04896532289999413
	198	+ ],
	199	+ "tokens_per_s_mean": 87716.14781880443
	200	+ },
	201	+ {
	202	+ "backend": "torch-mps",
	203	+ "mode": "retain",
	204	+ "s_per_forward": [
	205	+ 0.053923208349988275,
	206	+ 0.05791100205001385,
	207	+ 0.05981330204999722
	208	+ ],
	209	+ "tokens_per_s_mean": 71588.57023098122
	210	+ },
	211	+ {
	212	+ "backend": "torch-mps",
	213	+ "mode": "retain+copy+write",
	214	+ "s_per_forward": [
	215	+ 0.07440289169999233,
	216	+ 0.06590515420002703,
	217	+ 0.06603583339992838
	218	+ ],
	219	+ "tokens_per_s_mean": 59551.075814261436
	220	+ },
	221	+ {
	222	+ "backend": "mlx",
	223	+ "mode": "plain",
	224	+ "s_per_forward": [
	225	+ 0.03250455415000033,
	226	+ 0.03844294169998648,
	227	+ 0.03760368125003879
	228	+ ],
	229	+ "tokens_per_s_mean": 113200.0622036286
	230	+ },
	231	+ {
	232	+ "backend": "mlx",
	233	+ "mode": "retain",
	234	+ "s_per_forward": [
	235	+ 0.0370830708000085,
	236	+ 0.036970339600020455,
	237	+ 0.03677087500000198
	238	+ ],
	239	+ "tokens_per_s_mean": 110878.22453034802
	240	+ },
	241	+ {
	242	+ "backend": "mlx",
	243	+ "mode": "retain+copy+write",
	244	+ "s_per_forward": [
	245	+ 0.044162579200019536,
	246	+ 0.03750625419997959,
	247	+ 0.03900341250000565
	248	+ ],
	249	+ "tokens_per_s_mean": 101829.54587737157
	250	+ }
	251	+ ]
	252	+}

added site/data/atlas-index.json +6 −0

		`@@ -0,0 +1,6 @@`
	1	+{
	2	+ "author": "Simon-Pierre Boucher",
	3	+ "contact": "contact@spboucher.ai",
	4	+ "website": "https://modelmap.io",
	5	+ "entries": []
	6	+}

added src/modelmap.egg-info/PKG-INFO +13 −0

		`@@ -0,0 +1,13 @@`
	1	+Metadata-Version: 2.4
	2	+Name: modelmap
	3	+Version: 0.1.0
	4	+Summary: Internal cartography of local large language models on Apple Silicon
	5	+Author-email: Simon-Pierre Boucher <contact@spboucher.ai>
	6	+Requires-Python: >=3.11
	7	+License-File: LICENSE
	8	+Requires-Dist: numpy
	9	+Requires-Dist: safetensors
	10	+Provides-Extra: dev
	11	+Requires-Dist: ruff; extra == "dev"
	12	+Requires-Dist: pytest; extra == "dev"
	13	+Dynamic: license-file

added src/modelmap.egg-info/SOURCES.txt +25 −0

		`@@ -0,0 +1,25 @@`
	1	+LICENSE
	2	+README.md
	3	+pyproject.toml
	4	+src/modelmap/__init__.py
	5	+src/modelmap.egg-info/PKG-INFO
	6	+src/modelmap.egg-info/SOURCES.txt
	7	+src/modelmap.egg-info/dependency_links.txt
	8	+src/modelmap.egg-info/requires.txt
	9	+src/modelmap.egg-info/top_level.txt
	10	+src/modelmap/atlas/__init__.py
	11	+src/modelmap/atlas/mapcard.py
	12	+src/modelmap/backends/__init__.py
	13	+src/modelmap/capture/__init__.py
	14	+src/modelmap/features/__init__.py
	15	+src/modelmap/geometry/__init__.py
	16	+src/modelmap/graphs/__init__.py
	17	+src/modelmap/instrumentation/__init__.py
	18	+src/modelmap/interventions/__init__.py
	19	+src/modelmap/models/__init__.py
	20	+src/modelmap/probes/__init__.py
	21	+src/modelmap/probes/linear.py
	22	+src/modelmap/stats/__init__.py
	23	+src/modelmap/stats/replication.py
	24	+src/modelmap/viz/__init__.py
	25	+tests/test_stats_and_probes.py
		\ No newline at end of file

added src/modelmap.egg-info/dependency_links.txt +1 −0

		`@@ -0,0 +1 @@`
	1	+

added src/modelmap.egg-info/requires.txt +6 −0

		`@@ -0,0 +1,6 @@`
	1	+numpy
	2	+safetensors
	3	+
	4	+[dev]
	5	+ruff
	6	+pytest

added src/modelmap.egg-info/top_level.txt +1 −0

		`@@ -0,0 +1 @@`
	1	+modelmap

added src/modelmap/atlas/mapcard.py +131 −0

		`@@ -0,0 +1,131 @@`
	1	+# =============================================================================
	2	+# Project : modelmap
	3	+# File : src/modelmap/atlas/mapcard.py
	4	+# Purpose : Map card schema v0 — machine-readable provenance + confidence
	5	+# Author : Simon-Pierre Boucher
	6	+# Contact : contact@spboucher.ai
	7	+# Website : https://modelmap.io
	8	+# Created : 2026-08-12
	9	+# Modified : 2026-08-12
	10	+# Platform : macOS / Apple Silicon (arm64)
	11	+# License : All rights reserved (research code)
	12	+# =============================================================================
	13	+"""Map card v0 (gap G23, ranked #1 in Phase 4).
	14	+
	15	+A map card is the datasheet of a published map. `tools/publish.py` refuses to
	16	+export any atlas entry whose card does not validate. Fields follow the Phase 2
	17	+conclusions: pinned intervention protocol, ablation-scheme coverage, featurizer
	18	+class, seed replication, per-dataset agreement — plus the charter's provenance
	19	+block (commit, config, model hash, hardware, dates).
	20	+
	21	+The schema will evolve (v1+) with the first real maps; version is embedded so
	22	+old cards remain interpretable.
	23	+"""
	24	+
	25	+from __future__ import annotations
	26	+
	27	+import json
	28	+from dataclasses import asdict, dataclass, field
	29	+from pathlib import Path
	30	+
	31	+SCHEMA_VERSION = "0.1"
	32	+
	33	+CONFIDENCE_LEVELS = {
	34	+ 0: "anecdotal",
	35	+ 1: "correlational",
	36	+ 2: "method-robust",
	37	+ 3: "causal",
	38	+}
	39	+
	40	+# Fields that must be non-empty for ANY publishable card.
	41	+REQUIRED_ALWAYS = (
	42	+ "map_id", "map_type", "model_id", "model_hash", "quantization",
	43	+ "commit", "config", "created", "hardware_manifest",
	44	+ "confidence_level", "regenerate_command",
	45	+)
	46	+
	47	+# Additional requirements per confidence level (charter §10 taxonomy).
	48	+REQUIRED_LEVEL_1 = ("seeds", "prompt_sets", "controls")
	49	+REQUIRED_LEVEL_2 = REQUIRED_LEVEL_1 + ("methods_in_agreement",)
	50	+REQUIRED_LEVEL_3 = REQUIRED_LEVEL_2 + ("interventions",)
	51	+
	52	+
	53	+@dataclass
	54	+class MapCard:
	55	+ """v0 map card. All list/dict fields default empty; validate() enforces."""
	56	+
	57	+ map_id: str = "" # atlas/<model>/<type>/<version>
	58	+ map_type: str = "" # probes \| geometry \| features \| circuits \| ...
	59	+ model_id: str = "" # e.g. Qwen/Qwen3-0.6B
	60	+ model_hash: str = "" # checkpoint content hash
	61	+ quantization: str = "" # fp16 \| q8 \| q4 \| q2 \| mixed:<desc>
	62	+ commit: str = "" # modelmap git commit that generated the map
	63	+ config: str = "" # path or inline JSON of the run config
	64	+ created: str = "" # YYYY-MM-DD
	65	+ hardware_manifest: dict = field(default_factory=dict)
	66	+ confidence_level: int = 0
	67	+ regenerate_command: str = "" # exact command that rebuilds the map
	68	+
	69	+ seeds: list = field(default_factory=list) # ≥3 for Level 1+
	70	+ prompt_sets: list = field(default_factory=list) # ≥2 for Level 1+ (name+checksum)
	71	+ controls: list = field(default_factory=list) # e.g. shuffled-labels, random-init
	72	+ methods_in_agreement: list = field(default_factory=list) # ≥2 for Level 2+
	73	+ interventions: list = field(default_factory=list) # ≥1 for Level 3
	74	+
	75	+ replication_rate: float \| None = None # map-to-map agreement under reseeding
	76	+ per_dataset_agreement: float \| None = None
	77	+ ablation_schemes: list = field(default_factory=list) # curves published, e.g. zero/mean/resample
	78	+ featurizer_class: str = "" # natural-basis \| linear \| orthogonal \| DAS-optimized
	79	+ intervention_protocol: str = "" # pinned corruption + metric (Phase 2 rule)
	80	+ negative_result: bool = False
	81	+ notes: str = ""
	82	+
	83	+ author: str = "Simon-Pierre Boucher"
	84	+ contact: str = "contact@spboucher.ai"
	85	+ website: str = "https://modelmap.io"
	86	+ schema_version: str = SCHEMA_VERSION
	87	+
	88	+ # ------------------------------------------------------------------ api
	89	+ def validate(self) -> list[str]:
	90	+ """Return a list of violations; empty list means publishable."""
	91	+ errs: list[str] = []
	92	+ for f in REQUIRED_ALWAYS:
	93	+ v = getattr(self, f)
	94	+ if v in ("", None) or v == {}:
	95	+ errs.append(f"missing required field: {f}")
	96	+ lvl = self.confidence_level
	97	+ if lvl not in CONFIDENCE_LEVELS:
	98	+ errs.append(f"confidence_level must be 0-3, got {lvl!r}")
	99	+ return errs
	100	+ extra = {1: REQUIRED_LEVEL_1, 2: REQUIRED_LEVEL_2, 3: REQUIRED_LEVEL_3}.get(lvl, ())
	101	+ for f in extra:
	102	+ if not getattr(self, f):
	103	+ errs.append(f"Level {lvl} requires non-empty: {f}")
	104	+ if lvl >= 1:
	105	+ if len(self.seeds) < 3:
	106	+ errs.append("Level 1+ requires >=3 seeds")
	107	+ if len(self.prompt_sets) < 2:
	108	+ errs.append("Level 1+ requires >=2 prompt sets")
	109	+ if self.replication_rate is None:
	110	+ errs.append("Level 1+ requires replication_rate")
	111	+ if lvl >= 2 and len(self.methods_in_agreement) < 2:
	112	+ errs.append("Level 2 requires >=2 independent methods in agreement")
	113	+ if lvl >= 3 and not self.featurizer_class:
	114	+ errs.append("Level 3 requires featurizer_class disclosure")
	115	+ return errs
	116	+
	117	+ def to_json(self) -> str:
	118	+ return json.dumps(asdict(self), indent=2, ensure_ascii=False) + "\n"
	119	+
	120	+ @classmethod
	121	+ def from_json(cls, text: str) -> MapCard:
	122	+ data = json.loads(text)
	123	+ known = {f for f in cls.__dataclass_fields__} # tolerate future fields
	124	+ return cls(**{k: v for k, v in data.items() if k in known})
	125	+
	126	+ def save(self, path: Path) -> None:
	127	+ path.write_text(self.to_json())
	128	+
	129	+ @classmethod
	130	+ def load(cls, path: Path) -> MapCard:
	131	+ return cls.from_json(path.read_text())

added src/modelmap/probes/linear.py +108 −0

		`@@ -0,0 +1,108 @@`
	1	+# =============================================================================
	2	+# Project : modelmap
	3	+# File : src/modelmap/probes/linear.py
	4	+# Purpose : Linear probes with the mandatory controls doctrine (expA core)
	5	+# Author : Simon-Pierre Boucher
	6	+# Contact : contact@spboucher.ai
	7	+# Website : https://modelmap.io
	8	+# Created : 2026-08-12
	9	+# Modified : 2026-08-12
	10	+# Platform : macOS / Apple Silicon (arm64)
	11	+# License : All rights reserved (research code)
	12	+# =============================================================================
	13	+"""Linear probing with controls baked in, never optional (notes §4.1/§4.8).
	14	+
	15	+Every probe run returns task accuracy AND its shuffled-label control on the
	16	+same split, so selectivity is always computable. Multinomial logistic
	17	+regression via plain NumPy gradient descent — no sklearn dependency, fully
	18	+deterministic given a seed, runs on any Mac.
	19	+"""
	20	+
	21	+from __future__ import annotations
	22	+
	23	+from dataclasses import dataclass
	24	+
	25	+import numpy as np
	26	+
	27	+
	28	+@dataclass
	29	+class ProbeResult:
	30	+ task_accuracy: float
	31	+ control_accuracy: float # shuffled-label control, same capacity/split
	32	+ selectivity: float
	33	+ n_train: int
	34	+ n_test: int
	35	+ seed: int
	36	+
	37	+
	38	+def _train_logreg(
	39	+ x: np.ndarray, y: np.ndarray, n_classes: int,
	40	+ l2: float, lr: float, epochs: int, rng: np.random.Generator,
	41	+) -> np.ndarray:
	42	+ """Multinomial logistic regression, full-batch GD. Returns (d+1, C)."""
	43	+ n, d = x.shape
	44	+ xb = np.hstack([x, np.ones((n, 1), dtype=x.dtype)])
	45	+ w = rng.normal(0, 0.01, size=(d + 1, n_classes)).astype(np.float64)
	46	+ onehot = np.eye(n_classes)[y]
	47	+ for _ in range(epochs):
	48	+ logits = xb @ w
	49	+ logits -= logits.max(axis=1, keepdims=True)
	50	+ p = np.exp(logits)
	51	+ p /= p.sum(axis=1, keepdims=True)
	52	+ grad = xb.T @ (p - onehot) / n + l2 * w
	53	+ w -= lr * grad
	54	+ return w
	55	+
	56	+
	57	+def _accuracy(w: np.ndarray, x: np.ndarray, y: np.ndarray) -> float:
	58	+ xb = np.hstack([x, np.ones((x.shape[0], 1), dtype=x.dtype)])
	59	+ return float((np.argmax(xb @ w, axis=1) == y).mean())
	60	+
	61	+
	62	+def probe_with_control(
	63	+ activations: np.ndarray,
	64	+ labels: np.ndarray,
	65	+ seed: int,
	66	+ train_frac: float = 0.8,
	67	+ l2: float = 1e-3,
	68	+ lr: float = 0.5,
	69	+ epochs: int = 300,
	70	+) -> ProbeResult:
	71	+ """Train task probe + shuffled-label control with identical capacity/split.
	72	+
	73	+ The control shuffles TRAINING labels only (a probe that can still score
	74	+ high is memorizing, not reading structure); control test accuracy is
	75	+ evaluated against the true test labels, per Hewitt & Liang.
	76	+ """
	77	+ x = np.asarray(activations, dtype=np.float64)
	78	+ y = np.asarray(labels)
	79	+ classes, y_idx = np.unique(y, return_inverse=True)
	80	+ n_classes = len(classes)
	81	+ if n_classes < 2:
	82	+ raise ValueError("probe needs >=2 classes")
	83	+
	84	+ rng = np.random.default_rng(seed)
	85	+ # standardize features on train split only
	86	+ perm = rng.permutation(len(y_idx))
	87	+ n_train = int(train_frac * len(y_idx))
	88	+ tr, te = perm[:n_train], perm[n_train:]
	89	+ mu, sd = x[tr].mean(0), x[tr].std(0) + 1e-8
	90	+ xs = (x - mu) / sd
	91	+
	92	+ w_task = _train_logreg(xs[tr], y_idx[tr], n_classes, l2, lr, epochs, rng)
	93	+ task_acc = _accuracy(w_task, xs[te], y_idx[te])
	94	+
	95	+ # NB: y_shuffled[tr] would be a fancy-indexed COPY — shuffle explicitly.
	96	+ shuffled_train = y_idx[tr].copy()
	97	+ rng.shuffle(shuffled_train)
	98	+ w_ctrl = _train_logreg(xs[tr], shuffled_train, n_classes, l2, lr, epochs, rng)
	99	+ ctrl_acc = _accuracy(w_ctrl, xs[te], y_idx[te])
	100	+
	101	+ return ProbeResult(
	102	+ task_accuracy=task_acc,
	103	+ control_accuracy=ctrl_acc,
	104	+ selectivity=task_acc - ctrl_acc,
	105	+ n_train=len(tr),
	106	+ n_test=len(te),
	107	+ seed=seed,
	108	+ )

added src/modelmap/stats/replication.py +82 −0

		`@@ -0,0 +1,82 @@`
	1	+# =============================================================================
	2	+# Project : modelmap
	3	+# File : src/modelmap/stats/replication.py
	4	+# Purpose : Bootstrap CIs, BH-FDR control, selectivity, replication metrics
	5	+# Author : Simon-Pierre Boucher
	6	+# Contact : contact@spboucher.ai
	7	+# Website : https://modelmap.io
	8	+# Created : 2026-08-12
	9	+# Modified : 2026-08-12
	10	+# Platform : macOS / Apple Silicon (arm64)
	11	+# License : All rights reserved (research code)
	12	+# =============================================================================
	13	+"""Statistical core mandated by charter §8.2 and the Dead-Salmons doctrine
	14	+(notes §4.8): bootstrap confidence intervals, Benjamini-Hochberg FDR control
	15	+for unit scans, probe selectivity, and map-replication metrics.
	16	+"""
	17	+
	18	+from __future__ import annotations
	19	+
	20	+import numpy as np
	21	+
	22	+
	23	+def bootstrap_ci(
	24	+ values: np.ndarray,
	25	+ stat=np.mean,
	26	+ n_boot: int = 10_000,
	27	+ alpha: float = 0.05,
	28	+ rng: np.random.Generator \| None = None,
	29	+) -> tuple[float, float, float]:
	30	+ """Percentile bootstrap CI. Returns (point, lo, hi)."""
	31	+ values = np.asarray(values, dtype=np.float64)
	32	+ if values.size == 0:
	33	+ raise ValueError("bootstrap_ci: empty sample")
	34	+ rng = rng or np.random.default_rng(0)
	35	+ idx = rng.integers(0, values.size, size=(n_boot, values.size))
	36	+ boots = stat(values[idx], axis=1)
	37	+ return (
	38	+ float(stat(values)),
	39	+ float(np.quantile(boots, alpha / 2)),
	40	+ float(np.quantile(boots, 1 - alpha / 2)),
	41	+ )
	42	+
	43	+
	44	+def bh_fdr(p_values: np.ndarray, q: float = 0.05) -> np.ndarray:
	45	+ """Benjamini-Hochberg: boolean mask of discoveries at FDR level q."""
	46	+ p = np.asarray(p_values, dtype=np.float64)
	47	+ m = p.size
	48	+ if m == 0:
	49	+ return np.zeros(0, dtype=bool)
	50	+ order = np.argsort(p)
	51	+ ranked = p[order]
	52	+ thresh = q * (np.arange(1, m + 1) / m)
	53	+ passing = np.nonzero(ranked <= thresh)[0]
	54	+ out = np.zeros(m, dtype=bool)
	55	+ if passing.size:
	56	+ out[order[: passing.max() + 1]] = True
	57	+ return out
	58	+
	59	+
	60	+def selectivity(task_accuracy: float, control_accuracy: float) -> float:
	61	+ """Hewitt & Liang selectivity: task acc minus shuffled-label control acc."""
	62	+ return float(task_accuracy - control_accuracy)
	63	+
	64	+
	65	+def jaccard(a: set, b: set) -> float:
	66	+ """Overlap of 'important unit' sets — the replication metric of expD."""
	67	+ if not a and not b:
	68	+ return 1.0
	69	+ return len(a & b) / len(a \| b)
	70	+
	71	+
	72	+def replication_rate(unit_sets: list[set]) -> tuple[float, float, float]:
	73	+ """Mean pairwise Jaccard across seeds/datasets, with bootstrap CI.
	74	+
	75	+ This is THE first-class atlas metric (charter §8.3): does the map
	76	+ reproduce under resampling?
	77	+ """
	78	+ n = len(unit_sets)
	79	+ if n < 2:
	80	+ raise ValueError("replication_rate needs >=2 replicates")
	81	+ pairs = [jaccard(unit_sets[i], unit_sets[j]) for i in range(n) for j in range(i + 1, n)]
	82	+ return bootstrap_ci(np.array(pairs))

added tests/test_stats_and_probes.py +76 −0

		`@@ -0,0 +1,76 @@`
	1	+# =============================================================================
	2	+# Project : modelmap
	3	+# File : tests/test_stats_and_probes.py
	4	+# Purpose : Correctness tests for stats + probe modules (tiny references)
	5	+# Author : Simon-Pierre Boucher
	6	+# Contact : contact@spboucher.ai
	7	+# Website : https://modelmap.io
	8	+# Created : 2026-08-12
	9	+# Modified : 2026-08-12
	10	+# Platform : macOS / Apple Silicon (arm64)
	11	+# License : All rights reserved (research code)
	12	+# =============================================================================
	13	+"""Charter §0.3: numerical probes get correctness tests against tiny
	14	+references — a probe must separate real structure from noise, FDR must
	15	+control false discoveries on known nulls, map cards must gate publication."""
	16	+
	17	+import numpy as np
	18	+
	19	+from modelmap.atlas.mapcard import MapCard
	20	+from modelmap.probes.linear import probe_with_control
	21	+from modelmap.stats.replication import bh_fdr, bootstrap_ci, jaccard, replication_rate
	22	+
	23	+
	24	+def test_bootstrap_ci_covers_mean():
	25	+ rng = np.random.default_rng(1)
	26	+ point, lo, hi = bootstrap_ci(rng.normal(5.0, 1.0, 500))
	27	+ assert lo < 5.0 < hi and abs(point - 5.0) < 0.2
	28	+
	29	+
	30	+def test_bh_fdr_null_and_signal():
	31	+ rng = np.random.default_rng(2)
	32	+ null_p = rng.uniform(0, 1, 1000)
	33	+ assert bh_fdr(null_p, q=0.05).mean() < 0.01 # near-zero false discoveries
	34	+ mixed = np.concatenate([null_p, np.full(50, 1e-8)])
	35	+ disc = bh_fdr(mixed, q=0.05)
	36	+ assert disc[-50:].all() # all planted signals found
	37	+
	38	+
	39	+def test_jaccard_and_replication():
	40	+ assert jaccard({1, 2}, {1, 2}) == 1.0
	41	+ assert jaccard({1}, {2}) == 0.0
	42	+ point, lo, hi = replication_rate([{1, 2, 3}, {2, 3, 4}, {1, 2, 3}])
	43	+ assert 0 < lo <= point <= hi <= 1
	44	+
	45	+
	46	+def test_probe_separates_structure_from_noise():
	47	+ """On separable data: high selectivity. On pure noise labels: ~0."""
	48	+ rng = np.random.default_rng(3)
	49	+ n, d = 600, 32
	50	+ y = rng.integers(0, 2, n)
	51	+ x = rng.normal(0, 1, (n, d))
	52	+ x[:, 0] += 3.0 * y # planted signal in one dimension
	53	+ r = probe_with_control(x, y, seed=0)
	54	+ assert r.task_accuracy > 0.9
	55	+ assert r.selectivity > 0.3
	56	+
	57	+ y_noise = rng.integers(0, 2, n)
	58	+ x_noise = rng.normal(0, 1, (n, d))
	59	+ r0 = probe_with_control(x_noise, y_noise, seed=0)
	60	+ assert abs(r0.selectivity) < 0.15 # noise floor: no fake structure
	61	+
	62	+
	63	+def test_mapcard_gates_publication():
	64	+ card = MapCard()
	65	+ assert card.validate() # empty card blocked
	66	+ card = MapCard(
	67	+ map_id="atlas/demo/probes/v0", map_type="probes", model_id="demo",
	68	+ model_hash="abc", quantization="fp16", commit="deadbeef",
	69	+ config="{}", created="2026-08-12", hardware_manifest={"chip": "test"},
	70	+ confidence_level=1, regenerate_command="make demo",
	71	+ seeds=[0, 1, 2], prompt_sets=["a#sha", "b#sha"],
	72	+ controls=["shuffled-labels"], replication_rate=0.8,
	73	+ )
	74	+ assert card.validate() == []
	75	+ card.seeds = [0] # Level 1 with 1 seed must fail
	76	+ assert any("3 seeds" in e for e in card.validate())

modified tools/new_experiment.py +1 −1

		`@@ -53,7 +53,7 @@ def main() -> int:`
53	53	print(__doc__)
54	54	return 2
55	55	rel, purpose = sys.argv[1], sys.argv[2]
56		− date = datetime.date.today().isoformat()
	56	+ date = datetime.datetime.now(tz=datetime.UTC).date().isoformat()
57	57	exp = ROOT / rel
58	58	name = exp.name
59	59	if exp.exists():

modified tools/new_map.py +1 −1

		`@@ -71,7 +71,7 @@ def main() -> int:`
71	71	return 2
72	72	model_id, map_type = sys.argv[1], sys.argv[2]
73	73	version = sys.argv[3] if len(sys.argv) > 3 else "v0"
74		− date = datetime.date.today().isoformat()
	74	+ date = datetime.datetime.now(tz=datetime.UTC).date().isoformat()
75	75	entry = ROOT / "atlas" / model_id / map_type / version
76	76	if entry.exists():
77	77	print(f"error: {entry.relative_to(ROOT)} already exists")

added tools/publish.py +89 −0

		`@@ -0,0 +1,89 @@`
	1	+#!/usr/bin/env python3
	2	+# =============================================================================
	3	+# Project : modelmap
	4	+# File : tools/publish.py
	5	+# Purpose : Export validated atlas entries to site/data — refuses incomplete
	6	+# Author : Simon-Pierre Boucher
	7	+# Contact : contact@spboucher.ai
	8	+# Website : https://modelmap.io
	9	+# Created : 2026-08-12
	10	+# Modified : 2026-08-12
	11	+# Platform : macOS / Apple Silicon (arm64)
	12	+# License : All rights reserved (research code)
	13	+# =============================================================================
	14	+"""Publish gate for the atlas (charter §14 hard constraints).
	15	+
	16	+Walks atlas/<model>/<type>/<version>/, validates each entry's mapcard.json
	17	+(schema v0) plus the presence of provenance.json and confidence.md, and exports
	18	+valid entries to site/data/atlas-index.json. Any violation blocks that entry
	19	+and is reported; nothing incomplete ever reaches modelmap.io.
	20	+"""
	21	+
	22	+from __future__ import annotations
	23	+
	24	+import json
	25	+import sys
	26	+from pathlib import Path
	27	+
	28	+ROOT = Path(__file__).resolve().parent.parent
	29	+sys.path.insert(0, str(ROOT / "src"))
	30	+
	31	+from modelmap.atlas.mapcard import MapCard
	32	+
	33	+
	34	+def entries():
	35	+ atlas = ROOT / "atlas"
	36	+ for model in sorted(p for p in atlas.iterdir() if p.is_dir()):
	37	+ for map_type in sorted(p for p in model.iterdir() if p.is_dir()):
	38	+ yield from sorted(p for p in map_type.iterdir() if p.is_dir())
	39	+
	40	+
	41	+def main() -> int:
	42	+ exported, blocked = [], []
	43	+ for entry in entries():
	44	+ rel = entry.relative_to(ROOT)
	45	+ problems = []
	46	+ for required in ("provenance.json", "confidence.md", "mapcard.json"):
	47	+ if not (entry / required).exists():
	48	+ problems.append(f"missing {required}")
	49	+ card = None
	50	+ if not problems:
	51	+ try:
	52	+ card = MapCard.load(entry / "mapcard.json")
	53	+ problems += card.validate()
	54	+ except (json.JSONDecodeError, TypeError) as e:
	55	+ problems.append(f"mapcard.json unreadable: {e}")
	56	+ if problems:
	57	+ blocked.append((str(rel), problems))
	58	+ else:
	59	+ exported.append({
	60	+ "path": str(rel),
	61	+ "map_id": card.map_id,
	62	+ "map_type": card.map_type,
	63	+ "model_id": card.model_id,
	64	+ "quantization": card.quantization,
	65	+ "confidence_level": card.confidence_level,
	66	+ "replication_rate": card.replication_rate,
	67	+ "negative_result": card.negative_result,
	68	+ "created": card.created,
	69	+ })
	70	+
	71	+ out = ROOT / "site" / "data"
	72	+ out.mkdir(parents=True, exist_ok=True)
	73	+ (out / "atlas-index.json").write_text(json.dumps({
	74	+ "author": "Simon-Pierre Boucher",
	75	+ "contact": "contact@spboucher.ai",
	76	+ "website": "https://modelmap.io",
	77	+ "entries": exported,
	78	+ }, indent=2) + "\n")
	79	+
	80	+ print(f"publish: {len(exported)} entry(ies) exported to site/data/atlas-index.json")
	81	+ for rel, problems in blocked:
	82	+ print(f"BLOCKED {rel}:")
	83	+ for p in problems:
	84	+ print(f" - {p}")
	85	+ return 1 if blocked else 0
	86	+
	87	+
	88	+if __name__ == "__main__":
	89	+ sys.exit(main())
	90