expC run #2: survival ledger 0/2 — causal scan finds the real structure · ee5c471 · modelmap

expC run #2: survival ledger 0/2 — causal scan finds the real structure

P1 CONFIRMED: bottom-5 skip damage was general (specificity 0.51 < random
mean 1.44), top-5 above mean but below p95 — skip never singles out the
probe layers. P2 FALSIFIED: direction-erasure profile anti-correlates with
the probe profile (rho=-0.136, p=0.76). Discovery: erasing the agreement
diff-of-means direction at ANY layer 2-15 destroys the behavior (up to
+3.97/+4.24 at L12, random-direction controls netted); late probe-ranked
layers carry little; L18/L22 suppressive. The load-bearing object is an
early-constructed DIRECTION, not a late place. Tap gains an edit hook.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Simon-Pierre Boucher committed 2 h ago (Aug 12, 2026) parent 91e30fe

Showing 6 changed files with +884 and −0

modified experiments/micro/expC_causal_verification/analysis.md +64 −0

		`@@ -63,6 +63,70 @@ Next experiment : run #2 with (1) perplexity-normalized specificity`
63	63	\| # \| Correlational claim \| Intervention \| Survives? \|
64	64	\|---\|---\|---\|---\|
65	65	\| 1 \| agreement top-5 differential layers (probes/v2) \| layer-skip ablation \| No (0/1) \|
	66	+\| 2 \| agreement differential layer profile (probes/v2) \| direction-level erasure scan \| No (0/2) \|
66	67
67	68	The published survival rate is the running fraction of this table — the
68	69	charter §8.3 metric, now live.
	70	+
	71	+---
	72	+
	73	+# Analysis — expC run #2: ledger 0/2 — and the control scan finds the real structure
	74	+
	75	+Run: `results/expC_causal_verification/20260812T065406Z/results.json` (192
	76	+held-out pairs, baseline margin +4.24, NLL 5.73). Hypotheses registered
	77	+before the run.
	78	+
	79	+```text
	80	+Hypothesis (P1) : run #1's bottom-5 damage is general, not
	81	+ agreement-specific, once normalized by NLL damage.
	82	+Result (P1) : CONFIRMED. Specificity (margin damage / NLL
	83	+ damage): bottom-5 = 0.51 — BELOW the random-5 mean
	84	+ (1.44); top-5 = 2.51 — above the random mean but
	85	+ below the p95 (4.73). Run #1's verdict stands, now
	86	+ with the confound measured: early-layer skips
	87	+ break the model generally; nothing in the skip
	88	+ family singles out the probe map's top layers.
	89	+Hypothesis (P2) : per-layer direction-erasure specific damage
	90	+ correlates with the differential probe profile
	91	+ (Spearman ρ ≥ 0.4, perm-p < 0.05).
	92	+Result (P2) : FALSIFIED, decisively — ρ = −0.136, perm-p 0.757.
	93	+ Survival ledger: 0/2. The probe map's layer
	94	+ ranking is behaviorally void at this granularity.
	95	+ BUT the scan itself uncovered strong causal
	96	+ structure the probe map missed: erasing the
	97	+ layer-ℓ agreement direction (diff-of-means, with
	98	+ random-direction controls netted out) at ANY
	99	+ single layer in 2–15 destroys most of the margin
	100	+ (specific damage +2.6…+4.0 of a +4.24 baseline;
	101	+ L12: +3.97, L15: +3.75, L6: +3.71), while the
	102	+ late layers the probes ranked highest carry
	103	+ little (L21: +0.69, L20: +0.29) — and L18/L22
	104	+ erasure slightly HELPS (−1.86/−1.09), suggesting
	105	+ suppressive components.
	106	+Interpretation : Level 0–1 (single model, single direction
	107	+ estimate from set A, no seed replication of the
	108	+ intervention yet — by our own doctrine this
	109	+ causal profile is NOT publishable as an atlas
	110	+ entry until replicated). Two lessons:
	111	+ (1) correlational layer rankings did not survive
	112	+ two different causal tests — the survival rate
	113	+ the field never publishes is, so far, 0%;
	114	+ (2) the behaviorally load-bearing object is a
	115	+ low-dimensional DIRECTION present across
	116	+ early-mid layers, not a "place" — consistent with
	117	+ the linear-representation view and with why
	118	+ decodability peaks (late, after information is
	119	+ everywhere) diverge from causal joints (early,
	120	+ where the direction is constructed). Random-
	121	+ direction erasure also hurts at layers 2–9
	122	+ (+0.9…+2.5): early residual streams are fragile
	123	+ to ANY rank-1 deletion — netted out in the
	124	+ specific column.
	125	+Next experiment : run #3 — replicate the direction-erasure profile
	126	+ (direction re-estimated on set B and on 5
	127	+ bootstrap seeds; report profile replication rate)
	128	+ → if stable, publish as the atlas's first
	129	+ INTERVENTIONS map (Level toward 2–3) and make the
	130	+ probes/v2 entry point to it as the causal
	131	+ counterpart. Then the same scan on arith_valid.
	132	+```

modified experiments/micro/expC_causal_verification/hypothesis.md +45 −0

		`@@ -56,3 +56,48 @@ result licenses "these layers causally support agreement behavior", NOT`
56	56	"agreement is localized to these layers" (hydra/backup effects, notes §4.4,
57	57	can mask redundancy). Direction-level interventions (LEACE-style erasure in
58	58	the forward pass) are the registered follow-up.
	59	+
	60	+---
	61	+
	62	+# Hypothesis — expC run #2 (specificity control + direction-level surgery)
	63	+
	64	+Registered 2026-08-12 before the run, after run #1's non-confirmation.
	65	+Two declared confounds get instruments: general-damage normalization for
	66	+layer-skip, and a surgical direction-level intervention the probe map can
	67	+legitimately pass or fail.
	68	+
	69	+```text
	70	+Hypothesis (P1) : run #1's bottom-5 damage is GENERAL, not
	71	+ agreement-specific: normalizing margin damage by
	72	+ general damage (mean NLL increase on neutral
	73	+ prose), the bottom-5 specificity ratio falls at or
	74	+ below the random-5 mean ratio, and top-5 does not
	75	+ exceed the random p95 either (the run-1 verdict
	76	+ stands, now with the confound measured).
	77	+Hypothesis (P2) : direction-level erasure tracks the probe map:
	78	+ erasing the layer-ℓ agreement direction
	79	+ (difference-in-means, estimated on agreement_A
	80	+ mean-pooled reps) at all positions of ℓ's output,
	81	+ minus the damage from erasing a random direction
	82	+ at the same layer, yields a per-layer specific-
	83	+ damage profile that correlates with the
	84	+ differential probe profile: Spearman ρ ≥ 0.4
	85	+ (permutation p < 0.05, 10k perms).
	86	+Falsification criterion : (P2) ρ ≤ 0 → the localization claim also dies at
	87	+ direction level: survival ledger 0/2 and the
	88	+ agreement map's layer structure is declared
	89	+ behaviorally void at this granularity.
	90	+Method : held-out bank enlarged with 6 NEW locations
	91	+ (target ≥ 180 pairs, deduped as before); NLL on
	92	+ 20 neutral prose sentences per condition;
	93	+ direction erasure h' = h − ⟨h−μ, u⟩u applied to
	94	+ every position of layer ℓ's output, u = unit
	95	+ class-mean difference at ℓ, μ = grand mean;
	96	+ random-direction control: 3 seeds per layer,
	97	+ same procedure; all 28 layers scanned.
	98	+Baseline / null : per-layer random-direction erasure damage;
	99	+ run #1 skip conditions rerun on the enlarged bank.
	100	+Result : (pending)
	101	+Interpretation : (pending)
	102	+Next experiment : (pending)
	103	+```

added experiments/micro/expC_causal_verification/implementation/benchmark_v2.py +237 −0

		`@@ -0,0 +1,237 @@`
	1	+#!/usr/bin/env python3
	2	+# =============================================================================
	3	+# Project : modelmap
	4	+# File : experiments/micro/expC_causal_verification/implementation/benchmark_v2.py
	5	+# Purpose : Run #2 — specificity-normalized skip + direction-level erasure
	6	+# Author : Simon-Pierre Boucher
	7	+# Contact : contact@spboucher.ai
	8	+# Website : https://modelmap.io
	9	+# Created : 2026-08-12
	10	+# Modified : 2026-08-12
	11	+# Platform : macOS / Apple Silicon (arm64) — MLX / Metal
	12	+# License : All rights reserved (research code)
	13	+# =============================================================================
	14	+"""expC run #2 (hypothesis registered before this run).
	15	+
	16	+P1: layer-skip conditions rerun with general-damage (NLL) normalization.
	17	+P2: surgical test — erase the layer-l agreement direction (diff-of-means)
	18	+from every position of that layer's output; compare per-layer specific
	19	+damage (vs random-direction control) with the differential probe profile.
	20	+"""
	21	+
	22	+from __future__ import annotations
	23	+
	24	+import json
	25	+import random
	26	+import subprocess
	27	+import sys
	28	+import time
	29	+from pathlib import Path
	30	+
	31	+import numpy as np
	32	+
	33	+ROOT = Path(__file__).resolve().parents[4]
	34	+sys.path.insert(0, str(ROOT / "src"))
	35	+sys.path.insert(0, str(ROOT / "benchmarks"))
	36	+from hardware_manifest import manifest
	37	+
	38	+from modelmap.capture.mlx_capture import capture_pooled, install_taps
	39	+
	40	+MODEL = "mlx-community/Qwen3-0.6B-4bit"
	41	+MAP_JSON = ROOT / "atlas" / "qwen3-0.6b-4bit" / "probes" / "v2" / "map.json"
	42	+K = 5
	43	+N_RANDOM_DRAWS = 20
	44	+N_RANDOM_DIRS = 3
	45	+SEED = 778
	46	+
	47	+NOUN_PAIRS = [("key", "keys"), ("crate", "crates"), ("report", "reports"), ("valve", "valves"),
	48	+ ("ticket", "tickets"), ("ladder", "ladders"), ("sample", "samples"), ("cable", "cables"),
	49	+ ("permit", "permits"), ("beacon", "beacons"), ("filter", "filters"), ("stamp", "stamps")]
	50	+NEW_NEAR = ["across the yard", "above the workbench", "below the landing", "outside the depot",
	51	+ "around the corner", "beyond the fence", "beneath the awning", "atop the cabinet"]
	52	+NEUTRAL = ["The committee reviewed the plans before the meeting.",
	53	+ "A local historian kept detailed notes for years.",
	54	+ "The lead engineer questioned the original estimate.",
	55	+ "Her assistant preferred the older method.",
	56	+ "The night watchman described the process in a letter.",
	57	+ "An early visitor returned before the first frost.",
	58	+ "The town council approved the request after some debate.",
	59	+ "The apprentice carried the tools across the yard.",
	60	+ "According to the survey, the harbor required constant maintenance.",
	61	+ "The archive near the market attracted visitors from the region.",
	62	+ "The restored lighthouse stood at the edge of town.",
	63	+ "The workshop changed hands twice last century.",
	64	+ "The observatory remained open despite the storm.",
	65	+ "The vineyard was documented in the annual report.",
	66	+ "The glacier trail closed early in the season.",
	67	+ "The orchard supplied the market for decades.",
	68	+ "The library extended its hours during the recess.",
	69	+ "The clerk signed the manifest without a word.",
	70	+ "The surveyor traced the boundary along the quay.",
	71	+ "The curator shelved the samples behind the annex."]
	72	+
	73	+
	74	+def spearman(a, b):
	75	+ ra = np.argsort(np.argsort(a)).astype(float)
	76	+ rb = np.argsort(np.argsort(b)).astype(float)
	77	+ ra -= ra.mean(); rb -= rb.mean()
	78	+ return float((ra * rb).sum() / np.sqrt((ra*2).sum() (rb**2).sum()))
	79	+
	80	+
	81	+def main() -> int:
	82	+ import mlx.core as mx
	83	+ from mlx_lm import load
	84	+
	85	+ t0 = time.time()
	86	+ model, tokenizer = load(MODEL)
	87	+ taps = install_taps(model)
	88	+ n_layers = len(taps)
	89	+
	90	+ mdoc = json.loads(MAP_JSON.read_text())
	91	+ agree = mdoc["properties"]["agreement"]
	92	+ diff_profile = np.array([a["selectivity_mean"] - t["selectivity_mean"]
	93	+ for a, t in zip(agree["per_layer"]["A"],
	94	+ agree["twin_null_per_layer_A"])])
	95	+ ranked = list(np.argsort(-diff_profile))
	96	+ top_k, bottom_k = [int(x) for x in ranked[:K]], [int(x) for x in ranked[-K:]]
	97	+
	98	+ # ---- enlarged held-out bank (new locations -> no dedup collisions)
	99	+ rng = random.Random(SEED)
	100	+ combos = [(n, loc) for n in NOUN_PAIRS for loc in NEW_NEAR]
	101	+ rng.shuffle(combos)
	102	+ pairs = []
	103	+ for (sg, pl), loc in combos:
	104	+ pairs.append({"prefix": f"The {sg} {loc}", "singular": True})
	105	+ pairs.append({"prefix": f"The {pl} {loc}", "singular": False})
	106	+ print(f"held-out pairs: {len(pairs)}")
	107	+ prefix_ids = [tokenizer.encode(p["prefix"]) for p in pairs]
	108	+ id_is, id_are = tokenizer.encode(" is")[0], tokenizer.encode(" are")[0]
	109	+ neutral_ids = [tokenizer.encode(s) for s in NEUTRAL]
	110	+
	111	+ def margin() -> float:
	112	+ out = []
	113	+ for ids, p in zip(prefix_ids, pairs):
	114	+ logits = model(mx.array([ids]))[0, -1, :]
	115	+ mx.eval(logits)
	116	+ m = float(logits[id_is] - logits[id_are])
	117	+ out.append(m if p["singular"] else -m)
	118	+ return float(np.mean(out))
	119	+
	120	+ def nll() -> float:
	121	+ tot, cnt = 0.0, 0
	122	+ for ids in neutral_ids:
	123	+ x = mx.array([ids])
	124	+ logits = model(x)[0]
	125	+ logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True)
	126	+ tgt = mx.array(ids[1:])
	127	+ picked = mx.take_along_axis(logp[:-1], tgt[:, None], axis=-1)
	128	+ mx.eval(picked)
	129	+ tot += float(-picked.sum()); cnt += len(ids) - 1
	130	+ return tot / cnt
	131	+
	132	+ def with_skip(layers: set[int], fn):
	133	+ for i, t in enumerate(taps):
	134	+ t.skip = i in layers
	135	+ try:
	136	+ return fn()
	137	+ finally:
	138	+ for t in taps:
	139	+ t.skip = False
	140	+
	141	+ base_m, base_nll = margin(), nll()
	142	+ print(f"baseline margin={base_m:+.4f} nll={base_nll:.4f}")
	143	+
	144	+ # ---------------- P1: skip conditions with NLL normalization
	145	+ def skip_cell(layers):
	146	+ m = with_skip(set(layers), margin)
	147	+ n = with_skip(set(layers), nll)
	148	+ return {"layers": sorted(int(x) for x in layers),
	149	+ "margin_damage": base_m - m,
	150	+ "nll_damage": n - base_nll,
	151	+ "specificity": (base_m - m) / max(n - base_nll, 1e-3)}
	152	+
	153	+ p1 = {"top": skip_cell(top_k), "bottom": skip_cell(bottom_k), "random": []}
	154	+ cand = [i for i in range(n_layers) if i not in set(top_k)]
	155	+ for d in range(N_RANDOM_DRAWS):
	156	+ p1["random"].append(skip_cell(random.Random(SEED + 1 + d).sample(cand, K)))
	157	+ rspec = np.array([c["specificity"] for c in p1["random"]])
	158	+ print(f"P1 specificity: top={p1['top']['specificity']:.3f} bottom={p1['bottom']['specificity']:.3f} "
	159	+ f"random mean={rspec.mean():.3f} p95={np.percentile(rspec, 95):.3f}")
	160	+
	161	+ # ---------------- P2: direction-level erasure, all layers
	162	+ # directions from agreement_A mean-pooled reps at each layer
	163	+ items = [json.loads(l) for l in
	164	+ (ROOT / "benchmarks" / "promptsets" / "agreement_A.jsonl").read_text().splitlines()]
	165	+ toks = [tokenizer.encode(it["text"]) for it in items]
	166	+ labels = np.array([it["label"] for it in items])
	167	+ reps = capture_pooled(model, taps, toks)["mean"] # (n, L, d)
	168	+ dirs, mus = [], []
	169	+ for layer in range(n_layers):
	170	+ x = reps[:, layer, :]
	171	+ mu = x.mean(0)
	172	+ u = x[labels == "correct"].mean(0) - x[labels == "violated"].mean(0)
	173	+ u = u / (np.linalg.norm(u) + 1e-8)
	174	+ mus.append(mu); dirs.append(u)
	175	+
	176	+ def erase_fn(u_np, mu_np):
	177	+ u = mx.array(u_np.astype(np.float32))
	178	+ mu = mx.array(mu_np.astype(np.float32))
	179	+ def fn(out):
	180	+ h = out.astype(mx.float32)
	181	+ coef = ((h - mu) * u).sum(axis=-1, keepdims=True)
	182	+ return (h - coef * u).astype(out.dtype)
	183	+ return fn
	184	+
	185	+ per_layer = []
	186	+ for layer in range(n_layers):
	187	+ taps[layer].edit = erase_fn(dirs[layer], mus[layer])
	188	+ m_agree = margin()
	189	+ taps[layer].edit = None
	190	+ rms = []
	191	+ for s in range(N_RANDOM_DIRS):
	192	+ ru = np.random.default_rng(1000 * layer + s).standard_normal(dirs[layer].shape)
	193	+ ru /= np.linalg.norm(ru)
	194	+ taps[layer].edit = erase_fn(ru.astype(np.float32), mus[layer])
	195	+ rms.append(margin())
	196	+ taps[layer].edit = None
	197	+ specific = (base_m - m_agree) - (base_m - float(np.mean(rms)))
	198	+ per_layer.append({"layer": layer, "agree_dir_damage": base_m - m_agree,
	199	+ "random_dir_damage_mean": base_m - float(np.mean(rms)),
	200	+ "specific_damage": specific})
	201	+ print(f" L{layer:02d} agreeDir={base_m - m_agree:+.3f} randDir={base_m - float(np.mean(rms)):+.3f} "
	202	+ f"specific={specific:+.3f}", flush=True)
	203	+
	204	+ spec_profile = np.array([r["specific_damage"] for r in per_layer])
	205	+ rho = spearman(spec_profile, diff_profile)
	206	+ perm_rng = np.random.default_rng(0)
	207	+ perms = np.array([spearman(perm_rng.permutation(spec_profile), diff_profile)
	208	+ for _ in range(10_000)])
	209	+ p_perm = float((perms >= rho).mean())
	210	+ survives = bool(rho >= 0.4 and p_perm < 0.05)
	211	+ print(f"P2: Spearman rho={rho:+.3f} perm-p={p_perm:.4f} -> survives={survives}")
	212	+
	213	+ commit = subprocess.run(["git", "rev-parse", "HEAD"], cwd=ROOT,
	214	+ capture_output=True, text=True, check=False).stdout.strip()
	215	+ ts = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime())
	216	+ outdir = ROOT / "results" / "expC_causal_verification" / ts
	217	+ outdir.mkdir(parents=True)
	218	+ (outdir / "results.json").write_text(json.dumps({
	219	+ "experiment": "expC_causal_verification", "run": 2,
	220	+ "scope": "NLL-normalized skip specificity + direction-level erasure scan",
	221	+ "commit": commit,
	222	+ "config": {"model": MODEL, "k": K, "n_random_draws": N_RANDOM_DRAWS,
	223	+ "n_random_dirs": N_RANDOM_DIRS, "n_pairs": len(pairs), "seed": SEED,
	224	+ "top_layers": sorted(top_k), "bottom_layers": sorted(bottom_k)},
	225	+ "manifest": manifest(),
	226	+ "baseline": {"margin": base_m, "nll": base_nll},
	227	+ "p1_skip_specificity": p1,
	228	+ "p2_direction_erasure": {"per_layer": per_layer, "spearman_rho": rho,
	229	+ "perm_p": p_perm, "survives": survives},
	230	+ "wall_seconds": round(time.time() - t0, 1),
	231	+ }, indent=2) + "\n")
	232	+ print(f"results -> {outdir / 'results.json'}")
	233	+ return 0
	234	+
	235	+
	236	+if __name__ == "__main__":
	237	+ sys.exit(main())

modified research/LOG.md +32 −0

		`@@ -391,3 +391,35 @@ agreement direction in the forward pass at layer ℓ (a surgical test the`
391	391	probe map can legitimately pass); (3) larger held-out bank; then the same
392	392	protocol on arith_valid. The pipeline now demonstrably runs the full
393	393	charter loop: register → measure → verify causally → publish either way.
	394	+
	395	+---
	396	+
	397	+## 2026-08-12 08:15 EDT — expC run #2: ledger 0/2 — and the causal scan finds what the probes missed
	398	+
	399	+Results (192 pairs, baseline margin +4.24; both hypotheses registered).
	400	+- P1 CONFIRMED: normalized by general (NLL) damage, run #1's bottom-5
	401	+ spike was unspecific (specificity 0.51 < random mean 1.44); top-5 (2.51)
	402	+ sits above the random mean but below p95 — the skip family never singles
	403	+ out the probe map's layers.
	404	+- P2 FALSIFIED (ρ = −0.136, p = 0.76): survival ledger 0/2. The probe
	405	+ map's layer ranking anti-correlates with the causal profile.
	406	+- The discovery: erasing the diff-of-means agreement direction at ANY
	407	+ single layer 2–15 destroys most of the behavior (specific damage up to
	408	+ +3.97/+4.24 at L12), with random-direction controls netted out; the
	409	+ late layers the probes ranked highest carry little, and L18/L22 erasure
	410	+ slightly HELPS (suppressive components). The load-bearing object is a
	411	+ low-dimensional DIRECTION constructed early — not a late "place" where
	412	+ information is merely readable.
	413	+
	414	+Doctrine consequence. Decodability-peak maps and causal-joint maps are
	415	+different map types and the atlas must never conflate them (this is charter
	416	+§2's distinction, now measured in-house at survival 0/2). The causal
	417	+profile is NOT yet publishable by our own rules (single direction estimate,
	418	+no seed replication) — expC run #3 will replicate it (direction from set B
	419	++ bootstrap seeds); if stable it becomes the atlas's first INTERVENTIONS
	420	+map and probes/v2 gets a pointer to its causal counterpart.
	421	+
	422	+Also noted. Early residual streams are fragile to ANY rank-1 deletion
	423	+(random-direction damage +0.9…+2.5 at layers 2–9) — relevant to
	424	+quantization sensitivity (candidate_02) and to localvm's working-set
	425	+question (which layers tolerate compression).

added results/expC_causal_verification/20260812T065406Z/results.json +503 −0

		`@@ -0,0 +1,503 @@`
	1	+{
	2	+ "experiment": "expC_causal_verification",
	3	+ "run": 2,
	4	+ "scope": "NLL-normalized skip specificity + direction-level erasure scan",
	5	+ "commit": "91e30fe5584b9c3727fa0ebeac6b83ef35155844",
	6	+ "config": {
	7	+ "model": "mlx-community/Qwen3-0.6B-4bit",
	8	+ "k": 5,
	9	+ "n_random_draws": 20,
	10	+ "n_random_dirs": 3,
	11	+ "n_pairs": 192,
	12	+ "seed": 778,
	13	+ "top_layers": [
	14	+ 17,
	15	+ 18,
	16	+ 19,
	17	+ 21,
	18	+ 22
	19	+ ],
	20	+ "bottom_layers": [
	21	+ 0,
	22	+ 1,
	23	+ 2,
	24	+ 3,
	25	+ 4
	26	+ ]
	27	+ },
	28	+ "manifest": {
	29	+ "author": "Simon-Pierre Boucher",
	30	+ "contact": "contact@spboucher.ai",
	31	+ "website": "https://modelmap.io",
	32	+ "chip": {
	33	+ "brand": "Apple M5 Max",
	34	+ "cores_total": 18,
	35	+ "cores_performance": 6,
	36	+ "cores_efficiency": 12
	37	+ },
	38	+ "memory": {
	39	+ "unified_gb": 48.0,
	40	+ "pagesize": 16384
	41	+ },
	42	+ "os": {
	43	+ "system": "Darwin",
	44	+ "version": "27.0",
	45	+ "arch": "arm64"
	46	+ },
	47	+ "software": {
	48	+ "python": "3.14.4",
	49	+ "numpy": "2.5.2",
	50	+ "mlx": "0.32.0",
	51	+ "torch": "2.13.0",
	52	+ "safetensors": "0.8.0"
	53	+ }
	54	+ },
	55	+ "baseline": {
	56	+ "margin": 4.242838541666667,
	57	+ "nll": 5.730769230769231
	58	+ },
	59	+ "p1_skip_specificity": {
	60	+ "top": {
	61	+ "layers": [
	62	+ 17,
	63	+ 18,
	64	+ 19,
	65	+ 21,
	66	+ 22
	67	+ ],
	68	+ "margin_damage": 2.2685546875,
	69	+ "nll_damage": 0.9023668639053257,
	70	+ "specificity": 2.514004866803278
	71	+ },
	72	+ "bottom": {
	73	+ "layers": [
	74	+ 0,
	75	+ 1,
	76	+ 2,
	77	+ 3,
	78	+ 4
	79	+ ],
	80	+ "margin_damage": 4.667521158854167,
	81	+ "nll_damage": 9.171597633136095,
	82	+ "specificity": 0.5089103715137769
	83	+ },
	84	+ "random": [
	85	+ {
	86	+ "layers": [
	87	+ 5,
	88	+ 12,
	89	+ 16,
	90	+ 20,
	91	+ 23
	92	+ ],
	93	+ "margin_damage": 0.9099934895833335,
	94	+ "nll_damage": 0.859467455621302,
	95	+ "specificity": 1.0587876057802639
	96	+ },
	97	+ {
	98	+ "layers": [
	99	+ 5,
	100	+ 6,
	101	+ 10,
	102	+ 14,
	103	+ 27
	104	+ ],
	105	+ "margin_damage": 0.9892578125000004,
	106	+ "nll_damage": 0.9053254437869818,
	107	+ "specificity": 1.092709609885622
	108	+ },
	109	+ {
	110	+ "layers": [
	111	+ 4,
	112	+ 6,
	113	+ 7,
	114	+ 10,
	115	+ 12
	116	+ ],
	117	+ "margin_damage": 1.257975260416667,
	118	+ "nll_damage": 0.7588757396449699,
	119	+ "specificity": 1.657682799301496
	120	+ },
	121	+ {
	122	+ "layers": [
	123	+ 3,
	124	+ 8,
	125	+ 10,
	126	+ 16,
	127	+ 26
	128	+ ],
	129	+ "margin_damage": 1.9109700520833335,
	130	+ "nll_damage": 0.7677514792899407,
	131	+ "specificity": 2.4890476978965967
	132	+ },
	133	+ {
	134	+ "layers": [
	135	+ 6,
	136	+ 12,
	137	+ 14,
	138	+ 20,
	139	+ 24
	140	+ ],
	141	+ "margin_damage": 2.323567708333334,
	142	+ "nll_damage": 0.46745562130177465,
	143	+ "specificity": 4.9706701608649855
	144	+ },
	145	+ {
	146	+ "layers": [
	147	+ 1,
	148	+ 7,
	149	+ 9,
	150	+ 13,
	151	+ 24
	152	+ ],
	153	+ "margin_damage": 3.490702311197917,
	154	+ "nll_damage": 4.8076923076923075,
	155	+ "specificity": 0.7260660807291668
	156	+ },
	157	+ {
	158	+ "layers": [
	159	+ 3,
	160	+ 5,
	161	+ 12,
	162	+ 16,
	163	+ 27
	164	+ ],
	165	+ "margin_damage": 1.2176106770833335,
	166	+ "nll_damage": 1.300295857988166,
	167	+ "specificity": 0.9364104865851345
	168	+ },
	169	+ {
	170	+ "layers": [
	171	+ 1,
	172	+ 8,
	173	+ 11,
	174	+ 25,
	175	+ 27
	176	+ ],
	177	+ "margin_damage": 3.7530212402343754,
	178	+ "nll_damage": 3.637573964497041,
	179	+ "specificity": 1.0317374373316137
	180	+ },
	181	+ {
	182	+ "layers": [
	183	+ 4,
	184	+ 9,
	185	+ 11,
	186	+ 15,
	187	+ 25
	188	+ ],
	189	+ "margin_damage": 2.6806640625,
	190	+ "nll_damage": 1.199704142011834,
	191	+ "specificity": 2.234437615598028
	192	+ },
	193	+ {
	194	+ "layers": [
	195	+ 0,
	196	+ 5,
	197	+ 11,
	198	+ 15,
	199	+ 23
	200	+ ],
	201	+ "margin_damage": 4.0419158935546875,
	202	+ "nll_damage": 9.372781065088759,
	203	+ "specificity": 0.4312397638956705
	204	+ },
	205	+ {
	206	+ "layers": [
	207	+ 1,
	208	+ 4,
	209	+ 15,
	210	+ 25,
	211	+ 27
	212	+ ],
	213	+ "margin_damage": 3.847269694010417,
	214	+ "nll_damage": 3.924556213017751,
	215	+ "specificity": 0.9803069405017121
	216	+ },
	217	+ {
	218	+ "layers": [
	219	+ 2,
	220	+ 9,
	221	+ 14,
	222	+ 15,
	223	+ 25
	224	+ ],
	225	+ "margin_damage": 3.8997701009114585,
	226	+ "nll_damage": 6.523668639053255,
	227	+ "specificity": 0.5977878884843868
	228	+ },
	229	+ {
	230	+ "layers": [
	231	+ 0,
	232	+ 5,
	233	+ 6,
	234	+ 24,
	235	+ 26
	236	+ ],
	237	+ "margin_damage": 4.367909749348959,
	238	+ "nll_damage": 9.251479289940828,
	239	+ "specificity": 0.4721309546785891
	240	+ },
	241	+ {
	242	+ "layers": [
	243	+ 3,
	244	+ 13,
	245	+ 16,
	246	+ 23,
	247	+ 27
	248	+ ],
	249	+ "margin_damage": -0.15462239583333304,
	250	+ "nll_damage": 0.8979289940828403,
	251	+ "specificity": -0.17219891199890136
	252	+ },
	253	+ {
	254	+ "layers": [
	255	+ 0,
	256	+ 3,
	257	+ 8,
	258	+ 20,
	259	+ 24
	260	+ ],
	261	+ "margin_damage": 4.544301350911459,
	262	+ "nll_damage": 10.328402366863905,
	263	+ "specificity": 0.4399810531675947
	264	+ },
	265	+ {
	266	+ "layers": [
	267	+ 2,
	268	+ 5,
	269	+ 9,
	270	+ 13,
	271	+ 16
	272	+ ],
	273	+ "margin_damage": 4.100779215494792,
	274	+ "nll_damage": 7.884615384615384,
	275	+ "specificity": 0.5200988273310468
	276	+ },
	277	+ {
	278	+ "layers": [
	279	+ 5,
	280	+ 9,
	281	+ 12,
	282	+ 14,
	283	+ 15
	284	+ ],
	285	+ "margin_damage": 2.029947916666667,
	286	+ "nll_damage": 0.6227810650887573,
	287	+ "specificity": 3.2594888163103732
	288	+ },
	289	+ {
	290	+ "layers": [
	291	+ 12,
	292	+ 14,
	293	+ 15,
	294	+ 20,
	295	+ 24
	296	+ ],
	297	+ "margin_damage": 2.667643229166667,
	298	+ "nll_damage": 0.5650887573964498,
	299	+ "specificity": 4.720750845331588
	300	+ },
	301	+ {
	302	+ "layers": [
	303	+ 2,
	304	+ 4,
	305	+ 12,
	306	+ 15,
	307	+ 24
	308	+ ],
	309	+ "margin_damage": 4.2433929443359375,
	310	+ "nll_damage": 7.396449704142012,
	311	+ "specificity": 0.5737067260742188
	312	+ },
	313	+ {
	314	+ "layers": [
	315	+ 1,
	316	+ 7,
	317	+ 8,
	318	+ 13,
	319	+ 14
	320	+ ],
	321	+ "margin_damage": 3.3100585937500004,
	322	+ "nll_damage": 4.214497041420118,
	323	+ "specificity": 0.7853982482888735
	324	+ }
	325	+ ]
	326	+ },
	327	+ "p2_direction_erasure": {
	328	+ "per_layer": [
	329	+ {
	330	+ "layer": 0,
	331	+ "agree_dir_damage": -0.043619791666666075,
	332	+ "random_dir_damage_mean": -0.008138020833333037,
	333	+ "specific_damage": -0.03548177083333304
	334	+ },
	335	+ {
	336	+ "layer": 1,
	337	+ "agree_dir_damage": 0.020833333333333925,
	338	+ "random_dir_damage_mean": -0.004340277777776791,
	339	+ "specific_damage": 0.025173611111110716
	340	+ },
	341	+ {
	342	+ "layer": 2,
	343	+ "agree_dir_damage": 4.169270833333334,
	344	+ "random_dir_damage_mean": 2.5252549913194446,
	345	+ "specific_damage": 1.6440158420138893
	346	+ },
	347	+ {
	348	+ "layer": 3,
	349	+ "agree_dir_damage": 4.161946614583334,
	350	+ "random_dir_damage_mean": 1.079752604166667,
	351	+ "specific_damage": 3.082194010416667
	352	+ },
	353	+ {
	354	+ "layer": 4,
	355	+ "agree_dir_damage": 4.215494791666667,
	356	+ "random_dir_damage_mean": 0.9580078125000004,
	357	+ "specific_damage": 3.2574869791666665
	358	+ },
	359	+ {
	360	+ "layer": 5,
	361	+ "agree_dir_damage": 4.172200520833334,
	362	+ "random_dir_damage_mean": 1.4726019965277777,
	363	+ "specific_damage": 2.6995985243055562
	364	+ },
	365	+ {
	366	+ "layer": 6,
	367	+ "agree_dir_damage": 4.177001953125,
	368	+ "random_dir_damage_mean": 0.46744791666666696,
	369	+ "specific_damage": 3.709554036458333
	370	+ },
	371	+ {
	372	+ "layer": 7,
	373	+ "agree_dir_damage": 4.204915364583334,
	374	+ "random_dir_damage_mean": 1.2357855902777781,
	375	+ "specific_damage": 2.969129774305556
	376	+ },
	377	+ {
	378	+ "layer": 8,
	379	+ "agree_dir_damage": 4.119140625,
	380	+ "random_dir_damage_mean": 1.511393229166667,
	381	+ "specific_damage": 2.607747395833333
	382	+ },
	383	+ {
	384	+ "layer": 9,
	385	+ "agree_dir_damage": 4.12646484375,
	386	+ "random_dir_damage_mean": 0.9156358506944451,
	387	+ "specific_damage": 3.210828993055555
	388	+ },
	389	+ {
	390	+ "layer": 10,
	391	+ "agree_dir_damage": 0.525065104166667,
	392	+ "random_dir_damage_mean": -0.09722222222222232,
	393	+ "specific_damage": 0.6222873263888893
	394	+ },
	395	+ {
	396	+ "layer": 11,
	397	+ "agree_dir_damage": 3.8680013020833335,
	398	+ "random_dir_damage_mean": 0.3184678819444442,
	399	+ "specific_damage": 3.5495334201388893
	400	+ },
	401	+ {
	402	+ "layer": 12,
	403	+ "agree_dir_damage": 4.027018229166667,
	404	+ "random_dir_damage_mean": 0.053927951388889284,
	405	+ "specific_damage": 3.9730902777777777
	406	+ },
	407	+ {
	408	+ "layer": 13,
	409	+ "agree_dir_damage": 3.8449707031250004,
	410	+ "random_dir_damage_mean": 0.9705403645833335,
	411	+ "specific_damage": 2.874430338541667
	412	+ },
	413	+ {
	414	+ "layer": 14,
	415	+ "agree_dir_damage": 3.938639322916667,
	416	+ "random_dir_damage_mean": 0.20258246527777768,
	417	+ "specific_damage": 3.7360568576388893
	418	+ },
	419	+ {
	420	+ "layer": 15,
	421	+ "agree_dir_damage": 3.781656901041667,
	422	+ "random_dir_damage_mean": 0.027994791666666963,
	423	+ "specific_damage": 3.753662109375
	424	+ },
	425	+ {
	426	+ "layer": 16,
	427	+ "agree_dir_damage": -0.10677083333333304,
	428	+ "random_dir_damage_mean": 0.14561631944444464,
	429	+ "specific_damage": -0.2523871527777777
	430	+ },
	431	+ {
	432	+ "layer": 17,
	433	+ "agree_dir_damage": 2.139322916666667,
	434	+ "random_dir_damage_mean": 0.028862847222221433,
	435	+ "specific_damage": 2.1104600694444455
	436	+ },
	437	+ {
	438	+ "layer": 18,
	439	+ "agree_dir_damage": -1.982421875,
	440	+ "random_dir_damage_mean": -0.12548828125,
	441	+ "specific_damage": -1.85693359375
	442	+ },
	443	+ {
	444	+ "layer": 19,
	445	+ "agree_dir_damage": 2.715494791666667,
	446	+ "random_dir_damage_mean": -0.12342664930555536,
	447	+ "specific_damage": 2.8389214409722223
	448	+ },
	449	+ {
	450	+ "layer": 20,
	451	+ "agree_dir_damage": 0.29589843750000044,
	452	+ "random_dir_damage_mean": 0.008029513888888395,
	453	+ "specific_damage": 0.28786892361111205
	454	+ },
	455	+ {
	456	+ "layer": 21,
	457	+ "agree_dir_damage": 0.754557291666667,
	458	+ "random_dir_damage_mean": 0.06195746527777768,
	459	+ "specific_damage": 0.6925998263888893
	460	+ },
	461	+ {
	462	+ "layer": 22,
	463	+ "agree_dir_damage": -0.9523111979166661,
	464	+ "random_dir_damage_mean": 0.14008246527777768,
	465	+ "specific_damage": -1.0923936631944438
	466	+ },
	467	+ {
	468	+ "layer": 23,
	469	+ "agree_dir_damage": 1.372721354166667,
	470	+ "random_dir_damage_mean": -0.10677083333333304,
	471	+ "specific_damage": 1.4794921875
	472	+ },
	473	+ {
	474	+ "layer": 24,
	475	+ "agree_dir_damage": 0.935221354166667,
	476	+ "random_dir_damage_mean": 0.018663194444444642,
	477	+ "specific_damage": 0.9165581597222223
	478	+ },
	479	+ {
	480	+ "layer": 25,
	481	+ "agree_dir_damage": 0.7815755208333335,
	482	+ "random_dir_damage_mean": 0.020182291666666963,
	483	+ "specific_damage": 0.7613932291666665
	484	+ },
	485	+ {
	486	+ "layer": 26,
	487	+ "agree_dir_damage": 1.6770833333333335,
	488	+ "random_dir_damage_mean": 0.012369791666666963,
	489	+ "specific_damage": 1.6647135416666665
	490	+ },
	491	+ {
	492	+ "layer": 27,
	493	+ "agree_dir_damage": 0.783854166666667,
	494	+ "random_dir_damage_mean": 0.015842013888889284,
	495	+ "specific_damage": 0.7680121527777777
	496	+ }
	497	+ ],
	498	+ "spearman_rho": -0.13628899835796388,
	499	+ "perm_p": 0.7568,
	500	+ "survives": false
	501	+ },
	502	+ "wall_seconds": 75.0
	503	+}

modified src/modelmap/capture/mlx_capture.py +3 −0

		`@@ -38,11 +38,14 @@ class Tap:`
38	38	self.retained = None
39	39	self.enabled = False
40	40	self.skip = False
	41	+ self.edit = None # optional callable applied to the block output
41	42
42	43	def __call__(self, x, args, *kwargs):
43	44	if self.skip:
44	45	return x
45	46	out = self.layer(x, args, *kwargs)
	47	+ if self.edit is not None:
	48	+ out = self.edit(out)
46	49	if self.enabled:
47	50	self.retained = out
48	51	return out
49	52