SPB Git

spb/modelmap Public License

Internal cartography of local LLMs on Apple Silicon — registered, gated, negative-first. Public atlas at modelmap.io.

🔗 https://modelmap.io
interpretabilityllmapple-siliconmlxreproducibility
Python 66.3% JavaScript 24.5% CSS 8.1% Shell 0.7%

expC run #3: publication gate refused the causal profile (as designed)

Replication failed: mean pairwise rho 0.495 (<0.7), band claim fails on
2/5 direction sources; make_interventions_mapcard.py exits 1 and the
atlas stays at two entries. The refusal decomposes the object: early band
(2-15) replicates tightly (+2.98..+3.34 of +4.24), late band (20-27) is
estimator noise (-0.44..+2.77) -- retroactively explaining run #2's
anti-correlation. Run #4 pre-registers the narrower BAND claim.
Charts: interventions profile builder + dynamic y-axis (renders when a
map passes the gate).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 3 h ago (Aug 12, 2026) parent ee5c471

Showing 8 changed files with +762 and −4

modified experiments/micro/expC_causal_verification/analysis.md +47 −0
@@ -130,3 +130,50 @@ Next experiment : run #3 — replicate the direction-erasure profile
130 130 probes/v2 entry point to it as the causal
131 131 counterpart. Then the same scan on arith_valid.
132 132 ```
133 +
134 +---
135 +
136 +# Analysis — expC run #3: the publication gate REFUSED the causal profile
137 +
138 +Run: `results/expC_causal_verification/20260812T065929Z/results.json`.
139 +Five direction sources (set A, set B, 3 bootstraps of A), shared
140 +random-direction controls, same held-out bank (192 pairs, baseline +4.24).
141 +Hypothesis, criterion, and publication rule registered before the run.
142 +
143 +```text
144 +Hypothesis : the per-layer specific-damage profile is
145 + estimator-stable — mean pairwise Spearman ρ ≥ 0.7
146 + and early(2–15) ≥ 3× late(20–27) in EVERY source.
147 +Result : FALSIFIED — mean ρ = 0.495 (min 0.176); the band
148 + claim fails for setB and bootA2 (ratio 1.2).
149 + make_interventions_mapcard.py refused the entry
150 + (exit 1), per the registered rule. The atlas
151 + stays at two entries — the gate did its job.
152 + THE STABLE PART: early-band (layers 2–15) mean
153 + specific damage replicates tightly across all
154 + five sources (+2.98, +3.22, +3.34, +3.27, +3.28)
155 + — erasing the estimated agreement direction
156 + anywhere in the early-mid band reliably destroys
157 + ~70–79% of the behavior, whatever the estimation
158 + set. THE UNSTABLE PART: the late band (20–27)
159 + swings from −0.44 to +2.77 with the estimator —
160 + the late-layer "profile" is direction-estimation
161 + noise, which also retroactively explains run #2's
162 + anti-correlation (the probe ranking lives exactly
163 + where the causal profile is noise).
164 +Interpretation : Level 0–1. The full profile is NOT a stable
165 + object; the coarser claim ("an early-band
166 + agreement direction is causally load-bearing") is
167 + the replicable candidate. Publishing gates that
168 + refuse are the mechanism that keeps the atlas
169 + honest — this refusal is itself a process result
170 + worth reporting on the site's methodology page.
171 +Next experiment : run #4 with the NARROWER pre-registered claim:
172 + early-band (2–15) mean specific damage ≥ 2.5 on
173 + fresh direction estimates (new bootstrap seeds +
174 + a held-out estimation split) and a fresh
175 + behavioral bank; late band explicitly excluded as
176 + unstable. If it passes, publish the interventions
177 + map as an early-band BAND claim (not a per-layer
178 + ranking), Level 2.
179 +```
modified experiments/micro/expC_causal_verification/hypothesis.md +39 −0
@@ -101,3 +101,42 @@ Result : (pending)
101 101 Interpretation : (pending)
102 102 Next experiment : (pending)
103 103 ```
104 +
105 +---
106 +
107 +# Hypothesis — expC run #3 (replication of the causal direction profile)
108 +
109 +Registered 2026-08-12 **before** the run. Run #2 found that erasing the
110 +diff-of-means agreement direction at any layer 2–15 destroys the behavior.
111 +By our own doctrine that profile is unpublishable until it replicates
112 +across direction estimates.
113 +
114 +```text
115 +Hypothesis : The per-layer specific-damage profile is an
116 + estimator-stable object: profiles from directions
117 + estimated on (i) set A, (ii) set B (disjoint
118 + nouns), (iii) 3 bootstrap resamples of set A,
119 + agree pairwise at Spearman ρ ≥ 0.7 (mean over the
120 + 10 pairs), AND the qualitative claim holds in
121 + EVERY source: mean specific damage over layers
122 + 2–15 ≥ 3× mean over layers 20–27.
123 +Falsification criterion : mean pairwise ρ < 0.7 or any source violating the
124 + 3× band claim → the "causal direction profile" is
125 + estimator noise; not publishable; expC pivots to
126 + steering-based verification.
127 +Method : shared random-direction controls (3 dirs/layer,
128 + computed once); agreement-direction scan repeated
129 + per source (5 sources × 28 layers × 192 pairs);
130 + same held-out bank and margins as run #2.
131 +Baseline / null : shared random-direction damage per layer.
132 +Publication rule : if confirmed → atlas qwen3-0.6b-4bit/
133 + interventions/v1 at Level 2 (probing evidences the
134 + direction, erasure confirms causal load,
135 + replicated across estimates — NOT Level 3, because
136 + the two agreeing methods share the diff-of-means
137 + estimator; an independent intervention family
138 + (activation addition) is the registered L3 path).
139 +Result : (pending)
140 +Interpretation : (pending)
141 +Next experiment : (pending)
142 +```
added experiments/micro/expC_causal_verification/implementation/benchmark_v3.py +192 −0
@@ -0,0 +1,192 @@
1 +#!/usr/bin/env python3
2 +# =============================================================================
3 +# Project : modelmap
4 +# File : experiments/micro/expC_causal_verification/implementation/benchmark_v3.py
5 +# Purpose : Run #3 — replication of the causal direction-erasure profile
6 +# Author : Simon-Pierre Boucher
7 +# Contact : contact@spboucher.ai
8 +# Website : https://modelmap.io
9 +# Created : 2026-08-12
10 +# Modified : 2026-08-12
11 +# Platform : macOS / Apple Silicon (arm64) — MLX / Metal
12 +# License : All rights reserved (research code)
13 +# =============================================================================
14 +"""expC run #3 (hypothesis registered before this run).
15 +
16 +Five direction sources (set A, set B, 3 bootstrap resamples of A); shared
17 +random-direction controls; pairwise Spearman replication of the per-layer
18 +specific-damage profile + the 2–15 vs 20–27 band claim per source.
19 +"""
20 +
21 +from __future__ import annotations
22 +
23 +import json
24 +import random
25 +import subprocess
26 +import sys
27 +import time
28 +from pathlib import Path
29 +
30 +import numpy as np
31 +
32 +ROOT = Path(__file__).resolve().parents[4]
33 +sys.path.insert(0, str(ROOT / "src"))
34 +sys.path.insert(0, str(ROOT / "benchmarks"))
35 +from hardware_manifest import manifest
36 +
37 +from modelmap.capture.mlx_capture import capture_pooled, install_taps
38 +
39 +MODEL = "mlx-community/Qwen3-0.6B-4bit"
40 +N_RANDOM_DIRS = 3
41 +SEED = 779
42 +BOOTS = 3
43 +
44 +NOUN_PAIRS = [("key", "keys"), ("crate", "crates"), ("report", "reports"), ("valve", "valves"),
45 + ("ticket", "tickets"), ("ladder", "ladders"), ("sample", "samples"), ("cable", "cables"),
46 + ("permit", "permits"), ("beacon", "beacons"), ("filter", "filters"), ("stamp", "stamps")]
47 +NEW_NEAR = ["across the yard", "above the workbench", "below the landing", "outside the depot",
48 + "around the corner", "beyond the fence", "beneath the awning", "atop the cabinet"]
49 +
50 +
51 +def spearman(a, b):
52 + ra = np.argsort(np.argsort(a)).astype(float)
53 + rb = np.argsort(np.argsort(b)).astype(float)
54 + ra -= ra.mean(); rb -= rb.mean()
55 + return float((ra * rb).sum() / np.sqrt((ra**2).sum() * (rb**2).sum()))
56 +
57 +
58 +def directions_from(reps, labels, idx=None):
59 + """Per-layer unit diff-of-means directions (+grand means)."""
60 + if idx is not None:
61 + reps, labels = reps[idx], labels[idx]
62 + dirs, mus = [], []
63 + for layer in range(reps.shape[1]):
64 + x = reps[:, layer, :]
65 + mu = x.mean(0)
66 + u = x[labels == "correct"].mean(0) - x[labels == "violated"].mean(0)
67 + u = u / (np.linalg.norm(u) + 1e-8)
68 + dirs.append(u); mus.append(mu)
69 + return dirs, mus
70 +
71 +
72 +def main() -> int:
73 + import mlx.core as mx
74 + from mlx_lm import load
75 +
76 + t0 = time.time()
77 + model, tokenizer = load(MODEL)
78 + taps = install_taps(model)
79 + n_layers = len(taps)
80 +
81 + rng = random.Random(SEED)
82 + combos = [(n, loc) for n in NOUN_PAIRS for loc in NEW_NEAR]
83 + rng.shuffle(combos)
84 + pairs = []
85 + for (sg, pl), loc in combos:
86 + pairs.append({"prefix": f"The {sg} {loc}", "singular": True})
87 + pairs.append({"prefix": f"The {pl} {loc}", "singular": False})
88 + prefix_ids = [tokenizer.encode(p["prefix"]) for p in pairs]
89 + id_is, id_are = tokenizer.encode(" is")[0], tokenizer.encode(" are")[0]
90 +
91 + def margin() -> float:
92 + out = []
93 + for ids, p in zip(prefix_ids, pairs):
94 + logits = model(mx.array([ids]))[0, -1, :]
95 + mx.eval(logits)
96 + m = float(logits[id_is] - logits[id_are])
97 + out.append(m if p["singular"] else -m)
98 + return float(np.mean(out))
99 +
100 + def erase_fn(u_np, mu_np):
101 + u = mx.array(u_np.astype(np.float32))
102 + mu = mx.array(mu_np.astype(np.float32))
103 + def fn(out):
104 + h = out.astype(mx.float32)
105 + coef = ((h - mu) * u).sum(axis=-1, keepdims=True)
106 + return (h - coef * u).astype(out.dtype)
107 + return fn
108 +
109 + base_m = margin()
110 + print(f"baseline margin {base_m:+.4f} | pairs {len(pairs)}")
111 +
112 + # ---- direction sources
113 + sources: dict[str, tuple] = {}
114 + reps_cache = {}
115 + for setname in ("A", "B"):
116 + items = [json.loads(l) for l in
117 + (ROOT / "benchmarks" / "promptsets" / f"agreement_{setname}.jsonl").read_text().splitlines()]
118 + toks = [tokenizer.encode(it["text"]) for it in items]
119 + labels = np.array([it["label"] for it in items])
120 + print(f"capture agreement_{setname}…", flush=True)
121 + reps = capture_pooled(model, taps, toks)["mean"]
122 + reps_cache[setname] = (reps, labels)
123 + sources[f"set{setname}"] = directions_from(reps, labels)
124 + repsA, labelsA = reps_cache["A"]
125 + for b in range(BOOTS):
126 + idx = np.random.default_rng(100 + b).integers(0, len(labelsA), len(labelsA))
127 + sources[f"bootA{b}"] = directions_from(repsA, labelsA, idx)
128 +
129 + # ---- shared random-direction control per layer
130 + rand_damage = []
131 + for layer in range(n_layers):
132 + ms = []
133 + for s in range(N_RANDOM_DIRS):
134 + ru = np.random.default_rng(1000 * layer + s).standard_normal(repsA.shape[-1])
135 + ru /= np.linalg.norm(ru)
136 + taps[layer].edit = erase_fn(ru.astype(np.float32), sources["setA"][1][layer])
137 + ms.append(margin())
138 + taps[layer].edit = None
139 + rand_damage.append(base_m - float(np.mean(ms)))
140 + print("random-direction controls done", flush=True)
141 +
142 + # ---- per-source agreement-direction scans
143 + profiles = {}
144 + for name, (dirs, mus) in sources.items():
145 + prof = []
146 + for layer in range(n_layers):
147 + taps[layer].edit = erase_fn(dirs[layer], mus[layer])
148 + m = margin()
149 + taps[layer].edit = None
150 + prof.append((base_m - m) - rand_damage[layer])
151 + profiles[name] = prof
152 + band_early = float(np.mean(prof[2:16]))
153 + band_late = float(np.mean(prof[20:28]))
154 + print(f"{name:8s} early(2-15)={band_early:+.3f} late(20-27)={band_late:+.3f} "
155 + f"ratio={band_early / max(band_late, 1e-6):.1f}", flush=True)
156 +
157 + names = list(profiles)
158 + rhos = [spearman(np.array(profiles[a]), np.array(profiles[b]))
159 + for i, a in enumerate(names) for b in names[i + 1:]]
160 + mean_rho = float(np.mean(rhos))
161 + band_ok = all(np.mean(profiles[n][2:16]) >= 3 * max(np.mean(profiles[n][20:28]), 1e-6)
162 + for n in names)
163 + survives = bool(mean_rho >= 0.7 and band_ok)
164 + print(f"replication: mean pairwise rho={mean_rho:.3f} (min {min(rhos):.3f}) "
165 + f"band-claim-all-sources={band_ok} -> publishable={survives}")
166 +
167 + commit = subprocess.run(["git", "rev-parse", "HEAD"], cwd=ROOT,
168 + capture_output=True, text=True, check=False).stdout.strip()
169 + ts = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime())
170 + outdir = ROOT / "results" / "expC_causal_verification" / ts
171 + outdir.mkdir(parents=True)
172 + (outdir / "results.json").write_text(json.dumps({
173 + "experiment": "expC_causal_verification", "run": 3,
174 + "scope": "replication of the direction-erasure causal profile (5 sources)",
175 + "commit": commit,
176 + "config": {"model": MODEL, "sources": names, "n_random_dirs": N_RANDOM_DIRS,
177 + "n_pairs": len(pairs), "seed": SEED, "boots": BOOTS},
178 + "manifest": manifest(),
179 + "baseline_margin": base_m,
180 + "random_direction_damage": rand_damage,
181 + "profiles": profiles,
182 + "replication": {"pairwise_rhos": rhos, "mean_rho": mean_rho,
183 + "min_rho": float(min(rhos)), "band_claim_all": band_ok,
184 + "publishable": survives},
185 + "wall_seconds": round(time.time() - t0, 1),
186 + }, indent=2) + "\n")
187 + print(f"results -> {outdir / 'results.json'}")
188 + return 0
189 +
190 +
191 +if __name__ == "__main__":
192 + sys.exit(main())
added experiments/micro/expC_causal_verification/implementation/make_interventions_mapcard.py +171 −0
@@ -0,0 +1,171 @@
1 +#!/usr/bin/env python3
2 +# =============================================================================
3 +# Project : modelmap
4 +# File : experiments/micro/expC_causal_verification/implementation/make_interventions_mapcard.py
5 +# Purpose : Publish the causal direction-erasure profile as an atlas entry
6 +# Author : Simon-Pierre Boucher
7 +# Contact : contact@spboucher.ai
8 +# Website : https://modelmap.io
9 +# Created : 2026-08-12
10 +# Modified : 2026-08-12
11 +# Platform : macOS / Apple Silicon (arm64)
12 +# License : All rights reserved (research code)
13 +# =============================================================================
14 +"""Builds atlas/qwen3-0.6b-4bit/interventions/v1 from expC run #3 —
15 +REFUSES to publish unless the run's own replication verdict is true
16 +(mean pairwise rho >= 0.7 and the band claim holds in every source)."""
17 +
18 +from __future__ import annotations
19 +
20 +import hashlib
21 +import json
22 +import sys
23 +import time
24 +from pathlib import Path
25 +
26 +import numpy as np
27 +
28 +ROOT = Path(__file__).resolve().parents[4]
29 +sys.path.insert(0, str(ROOT / "src"))
30 +
31 +from modelmap.atlas.mapcard import MapCard
32 +
33 +ENTRY = ROOT / "atlas" / "qwen3-0.6b-4bit" / "interventions" / "v1"
34 +MODEL_ID = "mlx-community/Qwen3-0.6B-4bit"
35 +
36 +
37 +def newest_run3() -> Path:
38 + for d in sorted((ROOT / "results" / "expC_causal_verification").iterdir(), reverse=True):
39 + doc = json.loads((d / "results.json").read_text())
40 + if doc.get("run") == 3:
41 + return d / "results.json"
42 + raise SystemExit("no run-3 results found")
43 +
44 +
45 +def model_hash() -> str:
46 + from mlx_lm.utils import hf_repo_to_path
47 + mp = Path(hf_repo_to_path(MODEL_ID))
48 + h = hashlib.sha256()
49 + for f in sorted(mp.glob("*.safetensors")):
50 + h.update(f.read_bytes())
51 + return h.hexdigest()
52 +
53 +
54 +def main() -> int:
55 + res_path = newest_run3()
56 + doc = json.loads(res_path.read_text())
57 + rep = doc["replication"]
58 + if not rep["publishable"]:
59 + print(f"REFUSED: replication verdict is false "
60 + f"(mean rho {rep['mean_rho']:.3f}, band {rep['band_claim_all']}) — "
61 + "per the registered publication rule this profile stays unpublished.")
62 + return 1
63 +
64 + ENTRY.mkdir(parents=True, exist_ok=True)
65 + profiles = doc["profiles"]
66 + boots = [k for k in profiles if k.startswith("bootA")]
67 + boot_mean = np.mean([profiles[b] for b in boots], axis=0).tolist()
68 + map_doc = {
69 + "author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai",
70 + "website": "https://modelmap.io",
71 + "map_type": "interventions", "model_id": MODEL_ID,
72 + "claim": "A single diff-of-means agreement direction, erased at any layer in the "
73 + "early-mid band, destroys most of the model's grammatical-agreement "
74 + "preference; late layers carry little. Per-layer specific damage = "
75 + "(agreement-direction erasure damage) minus (random-direction damage).",
76 + "behavior_metric": "logit margin correct-vs-incorrect verb on held-out minimal pairs",
77 + "baseline_margin": doc["baseline_margin"],
78 + "per_layer": {
79 + "setA": profiles["setA"], "setB": profiles["setB"],
80 + "boot_mean": boot_mean,
81 + "random_direction_damage": doc["random_direction_damage"],
82 + },
83 + "replication": rep,
84 + "source_results": str(res_path.relative_to(ROOT)),
85 + }
86 + (ENTRY / "map.json").write_text(json.dumps(map_doc, indent=2) + "\n")
87 +
88 + mhash = model_hash()
89 + created = time.strftime("%Y-%m-%d", time.gmtime())
90 + (ENTRY / "provenance.json").write_text(json.dumps({
91 + "author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai",
92 + "website": "https://modelmap.io",
93 + "model_id": MODEL_ID, "map_type": "interventions", "version": "v1",
94 + "commit": doc["commit"], "model_hash": mhash, "config": doc["config"],
95 + "seed": doc["config"]["seed"], "hardware_manifest": doc["manifest"],
96 + "created": created, "source_results": str(res_path.relative_to(ROOT)),
97 + }, indent=2) + "\n")
98 +
99 + card = MapCard(
100 + map_id="atlas/qwen3-0.6b-4bit/interventions/v1",
101 + map_type="interventions", model_id=MODEL_ID, model_hash=mhash,
102 + quantization="q4 (mlx)", commit=doc["commit"],
103 + config=str(res_path.relative_to(ROOT)), created=created,
104 + hardware_manifest=doc["manifest"], confidence_level=2,
105 + regenerate_command=(
106 + ".venv/bin/python experiments/micro/expC_causal_verification/implementation/benchmark_v3.py && "
107 + ".venv/bin/python experiments/micro/expC_causal_verification/implementation/make_interventions_mapcard.py"),
108 + seeds=[doc["config"]["seed"], "bootA0-2 (direction bootstrap)"],
109 + prompt_sets=["agreement_A (direction est.)", "agreement_B (direction est.)",
110 + "held-out minimal pairs (behavior, disjoint locations)"],
111 + controls=["random-direction erasure per layer (3 dirs, netted out)",
112 + "cross-promptset direction replication (A vs B)",
113 + "bootstrap direction replication (3 resamples)"],
114 + methods_in_agreement=["difference-in-means probing (direction exists, decodable)",
115 + "direction erasure (causally load-bearing)"],
116 + interventions=["rank-1 direction erasure at each layer's output, all positions"],
117 + replication_rate=round(rep["mean_rho"], 4),
118 + featurizer_class="linear (difference-in-means direction)",
119 + intervention_protocol="erase h' = h − ⟨h−μ,u⟩u at layer ℓ output; logit-margin metric; "
120 + "random-direction null netted out",
121 + notes="Level 2, NOT 3: the two agreeing methods share the diff-of-means estimator; "
122 + "an independent intervention family (activation addition/steering) is the "
123 + "registered Level-3 path. Layer profile ANTI-correlates with the probes/v2 "
124 + "decodability map (expC ledger 0/2) — decodability peaks and causal joints are "
125 + "different map types.",
126 + )
127 + (ENTRY / "mapcard.json").write_text(card.to_json())
128 +
129 + early = float(np.mean(profiles["setA"][2:16]))
130 + late = float(np.mean(profiles["setA"][20:28]))
131 + (ENTRY / "confidence.md").write_text(f"""---
132 +project: modelmap
133 +document: qwen3-0.6b-4bit/interventions/v1 — confidence
134 +author: Simon-Pierre Boucher
135 +contact: contact@spboucher.ai
136 +website: https://modelmap.io
137 +created: {created}
138 +status: reviewed
139 +---
140 +
141 +# Confidence — qwen3-0.6b-4bit / interventions / v1
142 +
143 +```text
144 +Level : 2
145 +Seeds : direction bootstrap x3 + cross-promptset (A/B)
146 +Prompt sets: 3 (two direction-estimation sets + held-out behavioral bank)
147 +Methods in agreement : 2 (diff-of-means probing; direction erasure) — shared
148 + estimator, hence Level 2 and not 3
149 +Causal verification : YES — rank-1 erasure with random-direction nulls
150 +```
151 +
152 +Replication: mean pairwise Spearman ρ = {rep['mean_rho']:.3f} (min {rep['min_rho']:.3f})
153 +across 5 direction sources; early-band (layers 2–15) mean specific damage
154 +{early:+.2f} vs late-band (20–27) {late:+.2f} on a {doc['baseline_margin']:+.2f} baseline margin.
155 +
156 +The claim is about a DIRECTION, not a place: the same linear feature is
157 +causally load-bearing wherever it is erased in the early-mid band. This map
158 +is the causal counterpart of probes/v2, whose decodability ranking it
159 +contradicts (survival ledger 0/2) — both are published, labeled by what they
160 +actually measure.
161 +""")
162 + errs = card.validate()
163 + if errs:
164 + print("CARD INVALID:", errs)
165 + return 1
166 + print(f"atlas entry written: {ENTRY.relative_to(ROOT)} (Level 2, rho {rep['mean_rho']:.2f})")
167 + return 0
168 +
169 +
170 +if __name__ == "__main__":
171 + sys.exit(main())
modified research/LOG.md +25 −0
@@ -423,3 +423,28 @@ map and probes/v2 gets a pointer to its causal counterpart.
423 423 (random-direction damage +0.9…+2.5 at layers 2–9) — relevant to
424 424 quantization sensitivity (candidate_02) and to localvm's working-set
425 425 question (which layers tolerate compression).
426 +
427 +---
428 +
429 +## 2026-08-12 08:45 EDT — expC run #3: the publication gate refused the causal profile
430 +
431 +**Result (pre-registered rule).** Replication FAILED: mean pairwise
432 +Spearman ρ = 0.495 (< 0.7 required; min 0.176); the early≥3×late band
433 +claim fails on 2 of 5 direction sources. `make_interventions_mapcard.py`
434 +refused the atlas entry (exit 1) — **the gate fired exactly as designed;
435 +the atlas stays at two entries.**
436 +
437 +**The decomposition the refusal exposed.** Early band (layers 2–15):
438 +tightly replicated across all five sources (+2.98…+3.34 specific damage
439 +on a +4.24 baseline — erasing the estimated agreement direction anywhere
440 +early reliably destroys ~70–79% of the behavior). Late band (20–27):
441 +estimator noise (−0.44…+2.77), which retroactively explains run #2's
442 +anti-correlation — the probe ranking lives exactly where the causal
443 +profile is noise.
444 +
445 +**Decision.** Run #4 will pre-register the NARROWER claim (early-band mean
446 +damage ≥ 2.5, fresh estimates + fresh behavioral bank; late band excluded
447 +as unstable). If it passes, the interventions map is published as a BAND
448 +claim, not a per-layer ranking, at Level 2. Meta-lesson for methodology.md:
449 +map artifacts must declare their stable granularity — per-layer rankings
450 +were too fine for this object; bands are the honest resolution.
added results/expC_causal_verification/20260812T065929Z/results.json +249 −0
@@ -0,0 +1,249 @@
1 +{
2 + "experiment": "expC_causal_verification",
3 + "run": 3,
4 + "scope": "replication of the direction-erasure causal profile (5 sources)",
5 + "commit": "ee5c4719c4b11e6eaac59ee6c6097565466a8ee6",
6 + "config": {
7 + "model": "mlx-community/Qwen3-0.6B-4bit",
8 + "sources": [
9 + "setA",
10 + "setB",
11 + "bootA0",
12 + "bootA1",
13 + "bootA2"
14 + ],
15 + "n_random_dirs": 3,
16 + "n_pairs": 192,
17 + "seed": 779,
18 + "boots": 3
19 + },
20 + "manifest": {
21 + "author": "Simon-Pierre Boucher",
22 + "contact": "contact@spboucher.ai",
23 + "website": "https://modelmap.io",
24 + "chip": {
25 + "brand": "Apple M5 Max",
26 + "cores_total": 18,
27 + "cores_performance": 6,
28 + "cores_efficiency": 12
29 + },
30 + "memory": {
31 + "unified_gb": 48.0,
32 + "pagesize": 16384
33 + },
34 + "os": {
35 + "system": "Darwin",
36 + "version": "27.0",
37 + "arch": "arm64"
38 + },
39 + "software": {
40 + "python": "3.14.4",
41 + "numpy": "2.5.2",
42 + "mlx": "0.32.0",
43 + "torch": "2.13.0",
44 + "safetensors": "0.8.0"
45 + }
46 + },
47 + "baseline_margin": 4.242838541666667,
48 + "random_direction_damage": [
49 + -0.008138020833333037,
50 + -0.004340277777776791,
51 + 2.5252549913194446,
52 + 1.079752604166667,
53 + 0.9580078125000004,
54 + 1.4726019965277777,
55 + 0.46744791666666696,
56 + 1.2357855902777781,
57 + 1.511393229166667,
58 + 0.9156358506944451,
59 + -0.09722222222222232,
60 + 0.3184678819444442,
61 + 0.053927951388889284,
62 + 0.9705403645833335,
63 + 0.20258246527777768,
64 + 0.027994791666666963,
65 + 0.14561631944444464,
66 + 0.028862847222221433,
67 + -0.12548828125,
68 + -0.12342664930555536,
69 + 0.008029513888888395,
70 + 0.06195746527777768,
71 + 0.14008246527777768,
72 + -0.10677083333333304,
73 + 0.018663194444444642,
74 + 0.020182291666666963,
75 + 0.012369791666666963,
76 + 0.015842013888889284
77 + ],
78 + "profiles": {
79 + "setA": [
80 + -0.03548177083333304,
81 + 0.025173611111110716,
82 + 1.6440158420138893,
83 + 3.082194010416667,
84 + 3.2574869791666665,
85 + 2.6995985243055562,
86 + 3.709554036458333,
87 + 2.969129774305556,
88 + 2.607747395833333,
89 + 3.210828993055555,
90 + 0.6222873263888893,
91 + 3.5495334201388893,
92 + 3.9730902777777777,
93 + 2.874430338541667,
94 + 3.7360568576388893,
95 + 3.753662109375,
96 + -0.2523871527777777,
97 + 2.1104600694444455,
98 + -1.85693359375,
99 + 2.8389214409722223,
100 + 0.28786892361111205,
101 + 0.6925998263888893,
102 + -1.0923936631944438,
103 + 1.4794921875,
104 + 0.9165581597222223,
105 + 0.7613932291666665,
106 + 1.6647135416666665,
107 + 0.7680121527777777
108 + ],
109 + "setB": [
110 + 0.011067708333333037,
111 + 0.027452256944443754,
112 + 1.6497938368055554,
113 + 3.079182942708333,
114 + 3.1854654947916665,
115 + 2.6484103732638893,
116 + 3.62841796875,
117 + 2.833021375868056,
118 + 2.566243489583333,
119 + 3.099338107638889,
120 + 4.062717013888889,
121 + 3.6388888888888893,
122 + 3.924424913194444,
123 + 3.014322916666667,
124 + 3.793348524305556,
125 + 3.9358723958333335,
126 + 3.796766493055556,
127 + 3.628363715277779,
128 + 3.8484700520833335,
129 + 3.6763237847222223,
130 + 3.4628363715277786,
131 + 3.250379774305556,
132 + 3.1177300347222228,
133 + 3.164713541666667,
134 + 2.7280815972222223,
135 + 2.57421875,
136 + 2.642252604166667,
137 + 1.2087673611111112
138 + ],
139 + "bootA0": [
140 + 0.03466796875,
141 + 0.024197048611110716,
142 + 1.6933322482638893,
143 + 3.117513020833333,
144 + 3.2520345052083335,
145 + 2.7464735243055562,
146 + 3.731770833333333,
147 + 2.970269097222222,
148 + 2.721842447916667,
149 + 3.270399305555555,
150 + 4.118381076388889,
151 + 3.8882378472222228,
152 + 4.114447699652778,
153 + 3.1420898437500004,
154 + 3.9171278211805562,
155 + 4.107259114583333,
156 + 4.017550998263889,
157 + 3.9983181423611125,
158 + 4.151774088541667,
159 + 0.3379448784722223,
160 + -0.7462293836805545,
161 + -2.1120876736111107,
162 + -3.4369574652777777,
163 + 0.4557291666666665,
164 + 0.33224826388888884,
165 + 0.3675130208333335,
166 + 0.9446614583333335,
167 + 0.6742621527777777
168 + ],
169 + "bootA1": [
170 + 0.0478515625,
171 + 0.035915798611110716,
172 + 1.6956108940972223,
173 + 3.126302083333333,
174 + 3.2410481770833335,
175 + 2.7658420138888893,
176 + 3.71826171875,
177 + 2.992892795138889,
178 + 2.735493977864583,
179 + 3.227349175347222,
180 + 4.129687839084202,
181 + 3.766370985243056,
182 + 3.957561916775173,
183 + 2.8322957356770835,
184 + 3.8114963107638893,
185 + 3.8261515299479165,
186 + 0.6756727430555558,
187 + -1.6633029513888875,
188 + -0.9513346354166661,
189 + 0.12521701388888928,
190 + 0.990993923611112,
191 + 0.06646050347222232,
192 + -0.01996527777777768,
193 + 1.48046875,
194 + 1.1066623263888888,
195 + 0.8971354166666665,
196 + 1.5628255208333335,
197 + 0.8054470486111112
198 + ],
199 + "bootA2": [
200 + 0.024739583333333037,
201 + 0.03656684027777679,
202 + 1.6524793836805554,
203 + 3.097900390625,
204 + 3.2051798502604165,
205 + 2.6715630425347223,
206 + 3.67431640625,
207 + 2.843641493055556,
208 + 2.600260416666667,
209 + 3.169976128472222,
210 + 4.146701388888889,
211 + 3.755262586805556,
212 + 3.9985622829861107,
213 + 3.0561523437500004,
214 + 3.9038628472222223,
215 + 4.074055989583333,
216 + 3.9359266493055554,
217 + 3.9249131944444455,
218 + 4.107096354166667,
219 + 4.108778211805555,
220 + 3.889268663194445,
221 + 3.7214084201388893,
222 + 3.4372287326388893,
223 + 3.234700520833333,
224 + 2.4922417534722223,
225 + 2.0730794270833335,
226 + 2.0849609375,
227 + 0.9704861111111112
228 + ]
229 + },
230 + "replication": {
231 + "pairwise_rhos": [
232 + 0.26053639846743293,
233 + 0.4493705528188287,
234 + 0.7712096332785988,
235 + 0.17624521072796934,
236 + 0.6469622331691297,
237 + 0.34318555008210183,
238 + 0.9753694581280788,
239 + 0.5413245758073344,
240 + 0.5681444991789819,
241 + 0.21510673234811165
242 + ],
243 + "mean_rho": 0.4947454844006568,
244 + "min_rho": 0.17624521072796934,
245 + "band_claim_all": false,
246 + "publishable": false
247 + },
248 + "wall_seconds": 125.7
249 +}
modified site/lib/charts.js +9 −2
@@ -40,9 +40,16 @@ function layerLineSvg({ title, series, yLabel, yMin = 0, yMax = 1, caption }) {
40 40 const Y = (v) => MT + ih - ((v - yMin) / (yMax - yMin)) * ih;
41 41
42 42 let grid = "";
43 − for (const v of [0, 0.25, 0.5, 0.75, 1].filter((v) => v >= yMin && v <= yMax)) {
43 + const defaultTicks = [0, 0.25, 0.5, 0.75, 1].filter((v) => v >= yMin && v <= yMax);
44 + const ticks = (yMin >= 0 && yMax <= 1 && defaultTicks.length >= 3)
45 + ? defaultTicks
46 + : [0, 1, 2, 3, 4].map((i) => yMin + (i / 4) * (yMax - yMin));
47 + for (const v of ticks) {
44 48 grid += `<line x1="${ML}" y1="${Y(v)}" x2="${ML + iw}" y2="${Y(v)}" class="grid"/>` +
45 − `<text x="${ML - 8}" y="${Y(v) + 4}" class="tick" text-anchor="end">${v}</text>`;
49 + `<text x="${ML - 8}" y="${Y(v) + 4}" class="tick" text-anchor="end">${fmt(v)}</text>`;
50 + }
51 + if (yMin < 0 && yMax > 0) {
52 + grid += `<line x1="${ML}" y1="${Y(0)}" x2="${ML + iw}" y2="${Y(0)}" class="axis"/>`;
46 53 }
47 54 let xt = "";
48 55 for (let i = 0; i < nx; i += Math.ceil(nx / 8)) {
modified site/server.js +30 −2
@@ -51,6 +51,32 @@ function probeMapCharts(onlyProp, mapRel) {
51 51 }).join("");
52 52 }
53 53 const MAP_V2 = "atlas/qwen3-0.6b-4bit/probes/v2/map.json";
54 +const MAP_INT = "atlas/qwen3-0.6b-4bit/interventions/v1/map.json";
55 +
56 +/** Causal direction-erasure profile chart (interventions map). */
57 +function interventionsMapCharts(mapRel) {
58 + const m = C.readJson(mapRel || MAP_INT);
59 + if (!m || !m.per_layer) return "";
60 + const all = [...m.per_layer.setA, ...m.per_layer.setB, ...m.per_layer.boot_mean];
61 + const yMin = Math.floor(Math.min(...all, 0) * 2) / 2 - 0.5;
62 + const yMax = Math.ceil(Math.max(...all) * 2) / 2 + 0.5;
63 + return Charts.layerLineSvg({
64 + title: "agreement — causal direction-erasure profile (specific damage by layer)",
65 + yLabel: "specific margin damage",
66 + yMin, yMax,
67 + series: [
68 + { label: "direction from set A", color: Charts.S1, values: m.per_layer.setA },
69 + { label: "direction from set B", color: Charts.S2, values: m.per_layer.setB },
70 + { label: "bootstrap mean (×3)", color: Charts.REF, dash: true, ref: true,
71 + values: m.per_layer.boot_mean },
72 + ],
73 + caption: "Erasing the diff-of-means agreement direction at any single early-mid layer " +
74 + "destroys most of the grammatical-agreement margin (baseline " +
75 + (m.baseline_margin ? m.baseline_margin.toFixed(2) : "—") + "); random-direction damage is " +
76 + "netted out. Level 2 — causally verified, replicated across direction estimates. " +
77 + "This profile ANTI-correlates with the probes/v2 decodability ranking (survival ledger 0/2).",
78 + });
79 +}
54 80
55 81 /** expH charts: warm/cold storage panels + capture-overhead grouped bars. */
56 82 function expHCharts() {
@@ -299,6 +325,7 @@ app.get("/experiments/:id", (req, res) => {
299 325 if (exp.id === "expA_probe_reliability") {
300 326 charts = probeMapCharts(null, MAP_V2) + probeMapCharts();
301 327 }
328 + if (exp.id === "expC_causal_verification") charts = interventionsMapCharts();
302 329 if (charts) charts = `<section class="card"><h2>Result maps</h2>${charts}</section>`;
303 330 const body = `<p class="crumb"><a href="/experiments">Experiments</a> / ${R.esc(exp.id)}</p>
304 331 <h1 class="page-title">${R.esc(exp.id)}</h1><p class="lede-small">${R.esc(exp.purpose)}</p>
@@ -351,8 +378,9 @@ app.get("/atlas/:model/:mapType/:version", (req, res) => {
351 378 .map((f) => `<li><a href="/results/${f.rel}">${f.name}</a> <span class="mono-small">${(f.size / 1024).toFixed(1)} KiB</span></li>`)
352 379 .join("");
353 380 const mapRel = path.posix.join(entry.rel, "map.json");
354 − const mapCharts = (mapType === "probes" && C.exists(mapRel))
355 − ? probeMapCharts(null, mapRel) : "";
381 + const mapCharts = !C.exists(mapRel) ? "" :
382 + mapType === "probes" ? probeMapCharts(null, mapRel) :
383 + mapType === "interventions" ? interventionsMapCharts(mapRel) : "";
356 384 const body = `<p class="crumb"><a href="/atlas">Atlas</a> / ${R.esc(entry.rel)}</p>
357 385 <h1 class="page-title">${R.esc(model)} — ${R.esc(mapType)} <span class="mono-small">${R.esc(version)}</span></h1>
358 386 <p>${R.levelBadge(entry.level)}</p>
359 387