expC run #6: L12 handle fails replication — Level 3 abandoned (as registered)
Only 1/4 fresh sources monotone (positive dose arm unstable); specificity failed to replicate (random |delta| 3.36 vs bound 1.14 — run #5's 0.45 was 3-draw sampling luck). Halving held 4/4: necessary but not a reliable additive handle. interventions/v1 (Level 2) stands as the final causal claim. Gate record: refuse -> pass(L2) -> refuse -> refuse. Methodology rule adopted: specificity controls need >=10 random draws + percentile bound. Charts: x-axis customization for dose-response figures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Showing 7 changed files with +431 and −27
modified
experiments/micro/expC_causal_verification/analysis.md
+44 −0
@@ -273,3 +273,47 @@ Three refusals/passes to date, all pre-registered: run #3 per-layer profile | ||
| 273 | 273 | REFUSED → run #4 band claim PASSED (Level 2 published) → run #5 L3 |
| 274 | 274 | conjunction REFUSED (v1 unchanged). The atlas never received a claim its |
| 275 | 275 | evidence didn't carry. |
| 276 | + | |
| 277 | +--- | |
| 278 | + | |
| 279 | +# Analysis — expC run #6: the L12 handle claim fails replication — Level 3 abandoned | |
| 280 | + | |
| 281 | +Run: `results/expC_causal_verification/20260812T072913Z/results.json`. | |
| 282 | +Four fresh direction sources at L12, third fresh bank (192 pairs, baseline | |
| 283 | ++4.57), fresh random-direction seeds. Registered rule: any source failing | |
| 284 | +monotonicity/halving, or specificity failing, kills the L3 claim and closes | |
| 285 | +the steering program for this object. | |
| 286 | + | |
| 287 | +```text | |
| 288 | +Hypothesis : L12 is an estimator-stable dose-controlled handle. | |
| 289 | +Result : FALSIFIED. Monotone: 1/4 sources only (Ahalf2); | |
| 290 | + the POSITIVE dose arm is unstable (Ahalf1 +2σ dips | |
| 291 | + 5.59<6.29; Bhalf1 +1σ dips below baseline; Bhalf2 | |
| 292 | + +2σ < +1σ). Halving at −2σ: 4/4 — the negative | |
| 293 | + (erasure-like) arm is robust, again. SPECIFICITY | |
| 294 | + FAILED TO REPLICATE: random-direction mean |Δ| = | |
| 295 | + 3.36 vs bound 1.14 on the fresh bank and fresh | |
| 296 | + random seeds (run #5's L12 value was 0.45 — with | |
| 297 | + only 3 random draws, that pass now reads as | |
| 298 | + sampling luck). | |
| 299 | +Interpretation : Level 1 for the negative. The direction is | |
| 300 | + NECESSARY (v1, Level 2, band-replicated) but NOT | |
| 301 | + a reliable additive handle: pushing along it does | |
| 302 | + not control the behavior in a dose-stable, | |
| 303 | + direction-specific way. Per the pre-registered | |
| 304 | + rule, Level 3 is abandoned for this object and | |
| 305 | + the steering program is closed. Had run #5's L12 | |
| 306 | + observation been published without fresh | |
| 307 | + re-registration, the atlas would now contain a | |
| 308 | + false Level-3 claim — the gate earned its keep a | |
| 309 | + third time. | |
| 310 | +Methodology lesson : 3 random-direction draws are too few for a | |
| 311 | + specificity bound; methodology.md will require | |
| 312 | + ≥10 draws with a percentile bound for any | |
| 313 | + specificity control from now on. | |
| 314 | +Next experiment : program closed here. Proceeding tracks: arith_valid | |
| 315 | + band protocol; candidate_02 (quantization drift of | |
| 316 | + the Level-2 band map); expG (band replication on | |
| 317 | + Qwen3-1.7B). The final gate record for this arc: | |
| 318 | + refuse → pass(L2) → refuse → refuse. | |
| 319 | +``` | |
modified
experiments/micro/expC_causal_verification/hypothesis.md
+39 −0
@@ -216,3 +216,42 @@ Result : (pending) | ||
| 216 | 216 | Interpretation : (pending) |
| 217 | 217 | Next experiment : (pending) |
| 218 | 218 | ``` |
| 219 | + | |
| 220 | +--- | |
| 221 | + | |
| 222 | +# Hypothesis — expC run #6 (minimal L3 claim: layer 12 as a single-layer handle) | |
| 223 | + | |
| 224 | +Registered 2026-08-12 **before** the run, after run #5's conjunction failed | |
| 225 | +while L12 passed every gate. The claim is narrowed to the object that | |
| 226 | +showed textbook behavior — scope discipline again, now applied to layers. | |
| 227 | + | |
| 228 | +```text | |
| 229 | +Hypothesis : At LAYER 12 ONLY, the diff-of-means agreement | |
| 230 | + direction is a dose-controlled causal handle, and | |
| 231 | + this is estimator-stable: for EVERY one of four | |
| 232 | + fresh direction sources (disjoint halves of | |
| 233 | + agreement_A and agreement_B, new permutation | |
| 234 | + seed), on a THIRD fresh behavioral bank (8 | |
| 235 | + never-used locations): | |
| 236 | + (a) margins strictly monotone in | |
| 237 | + α ∈ {−2,−1,0,+1,+2}·σ; | |
| 238 | + (b) margin(−2σ) ≤ 0.5 × baseline; | |
| 239 | + (c) shared specificity control at L12: 3 random | |
| 240 | + directions at ±2σ move the margin by < 25% | |
| 241 | + of baseline on average. | |
| 242 | +Falsification criterion : any source failing (a) or (b), or (c) failing — | |
| 243 | + the single-layer L3 claim dies and Level 3 is | |
| 244 | + abandoned for this object (steering program | |
| 245 | + closed; candidate_02/expG proceed from the | |
| 246 | + Level-2 band entry). | |
| 247 | +Method : same margin metric; direction + σ re-estimated | |
| 248 | + per source at L12 from half-set mean-pooled reps; | |
| 249 | + doses applied at L12 output, all positions. | |
| 250 | +Publication rule : pass → interventions/v2 at LEVEL 3 with scope | |
| 251 | + declared as {band necessity (v1) + single-layer | |
| 252 | + L12 handle}; fail → documented refusal, v1 | |
| 253 | + unchanged. | |
| 254 | +Result : (pending) | |
| 255 | +Interpretation : (pending) | |
| 256 | +Next experiment : (pending) | |
| 257 | +``` | |
added
experiments/micro/expC_causal_verification/implementation/benchmark_v6.py
+169 −0
@@ -0,0 +1,169 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# ============================================================================= | |
| 3 | +# Project : modelmap | |
| 4 | +# File : experiments/micro/expC_causal_verification/implementation/benchmark_v6.py | |
| 5 | +# Purpose : Run #6 — minimal L3 claim: layer-12 handle, four fresh sources | |
| 6 | +# Author : Simon-Pierre Boucher | |
| 7 | +# Contact : contact@spboucher.ai | |
| 8 | +# Website : https://modelmap.io | |
| 9 | +# Created : 2026-08-12 | |
| 10 | +# Modified : 2026-08-12 | |
| 11 | +# Platform : macOS / Apple Silicon (arm64) — MLX / Metal | |
| 12 | +# License : All rights reserved (research code) | |
| 13 | +# ============================================================================= | |
| 14 | +"""expC run #6 (hypothesis registered before this run). | |
| 15 | + | |
| 16 | +Single-layer (L12) dose-response handle claim, four fresh direction sources | |
| 17 | +(new-permutation disjoint halves of A and B), third fresh behavioral bank, | |
| 18 | +shared random-direction specificity control. | |
| 19 | +""" | |
| 20 | + | |
| 21 | +from __future__ import annotations | |
| 22 | + | |
| 23 | +import json | |
| 24 | +import random | |
| 25 | +import subprocess | |
| 26 | +import sys | |
| 27 | +import time | |
| 28 | +from pathlib import Path | |
| 29 | + | |
| 30 | +import numpy as np | |
| 31 | + | |
| 32 | +ROOT = Path(__file__).resolve().parents[4] | |
| 33 | +sys.path.insert(0, str(ROOT / "src")) | |
| 34 | +sys.path.insert(0, str(ROOT / "benchmarks")) | |
| 35 | +from hardware_manifest import manifest | |
| 36 | + | |
| 37 | +from modelmap.capture.mlx_capture import capture_pooled, install_taps | |
| 38 | + | |
| 39 | +MODEL = "mlx-community/Qwen3-0.6B-4bit" | |
| 40 | +LAYER = 12 | |
| 41 | +DOSES = (-2.0, -1.0, 0.0, 1.0, 2.0) | |
| 42 | +N_RANDOM_DIRS = 3 | |
| 43 | +SEED = 782 | |
| 44 | + | |
| 45 | +NOUN_PAIRS = [("key", "keys"), ("crate", "crates"), ("report", "reports"), ("valve", "valves"), | |
| 46 | + ("ticket", "tickets"), ("ladder", "ladders"), ("sample", "samples"), ("cable", "cables"), | |
| 47 | + ("permit", "permits"), ("beacon", "beacons"), ("filter", "filters"), ("stamp", "stamps")] | |
| 48 | +THIRD_NEAR = ["beside the turbine hall", "near the cold frame", "behind the switchboard", | |
| 49 | + "under the gantry", "next to the signal box", "opposite the pump house", | |
| 50 | + "inside the drying shed", "along the breakwater"] | |
| 51 | + | |
| 52 | + | |
| 53 | +def main() -> int: | |
| 54 | + import mlx.core as mx | |
| 55 | + from mlx_lm import load | |
| 56 | + | |
| 57 | + t0 = time.time() | |
| 58 | + model, tokenizer = load(MODEL) | |
| 59 | + taps = install_taps(model) | |
| 60 | + | |
| 61 | + rng = random.Random(SEED) | |
| 62 | + combos = [(n, loc) for n in NOUN_PAIRS for loc in THIRD_NEAR] | |
| 63 | + rng.shuffle(combos) | |
| 64 | + pairs = [] | |
| 65 | + for (sg, pl), loc in combos: | |
| 66 | + pairs.append({"prefix": f"The {sg} {loc}", "singular": True}) | |
| 67 | + pairs.append({"prefix": f"The {pl} {loc}", "singular": False}) | |
| 68 | + prefix_ids = [tokenizer.encode(p["prefix"]) for p in pairs] | |
| 69 | + id_is, id_are = tokenizer.encode(" is")[0], tokenizer.encode(" are")[0] | |
| 70 | + | |
| 71 | + def margin() -> float: | |
| 72 | + out = [] | |
| 73 | + for ids, p in zip(prefix_ids, pairs): | |
| 74 | + logits = model(mx.array([ids]))[0, -1, :] | |
| 75 | + mx.eval(logits) | |
| 76 | + m = float(logits[id_is] - logits[id_are]) | |
| 77 | + out.append(m if p["singular"] else -m) | |
| 78 | + return float(np.mean(out)) | |
| 79 | + | |
| 80 | + def add_fn(u_np, delta): | |
| 81 | + d = mx.array((u_np * delta).astype(np.float32)) | |
| 82 | + def fn(out): | |
| 83 | + return (out.astype(mx.float32) + d).astype(out.dtype) | |
| 84 | + return fn | |
| 85 | + | |
| 86 | + # four fresh direction sources at L12 (new permutation seed) | |
| 87 | + sources = {} | |
| 88 | + d_model = None | |
| 89 | + for s in ("A", "B"): | |
| 90 | + items = [json.loads(l) for l in | |
| 91 | + (ROOT / "benchmarks" / "promptsets" / f"agreement_{s}.jsonl").read_text().splitlines()] | |
| 92 | + toks = [tokenizer.encode(it["text"]) for it in items] | |
| 93 | + labels = np.array([it["label"] for it in items]) | |
| 94 | + print(f"capture agreement_{s}…", flush=True) | |
| 95 | + reps = capture_pooled(model, taps, toks)["mean"][:, LAYER, :] | |
| 96 | + d_model = reps.shape[-1] | |
| 97 | + n = len(labels) | |
| 98 | + perm = np.random.default_rng(SEED + ord(s)).permutation(n) | |
| 99 | + for hi, idx in enumerate((perm[: n // 2], perm[n // 2:])): | |
| 100 | + x, l = reps[idx], labels[idx] | |
| 101 | + u = x[l == "correct"].mean(0) - x[l == "violated"].mean(0) | |
| 102 | + u = u / (np.linalg.norm(u) + 1e-8) | |
| 103 | + sources[f"{s}half{hi + 1}"] = {"u": u, "sigma": float((x @ u).std())} | |
| 104 | + | |
| 105 | + base_m = margin() | |
| 106 | + print(f"baseline margin {base_m:+.4f} | pairs {len(pairs)}", flush=True) | |
| 107 | + | |
| 108 | + per_source = {} | |
| 109 | + all_ab = True | |
| 110 | + for name, sv in sources.items(): | |
| 111 | + dm = {} | |
| 112 | + for a in DOSES: | |
| 113 | + if a == 0.0: | |
| 114 | + dm[a] = base_m | |
| 115 | + continue | |
| 116 | + taps[LAYER].edit = add_fn(sv["u"], a * sv["sigma"]) | |
| 117 | + dm[a] = margin() | |
| 118 | + taps[LAYER].edit = None | |
| 119 | + seq = [dm[a] for a in DOSES] | |
| 120 | + mono = all(seq[i] < seq[i + 1] for i in range(len(seq) - 1)) | |
| 121 | + halve = dm[-2.0] <= 0.5 * base_m | |
| 122 | + all_ab &= (mono and halve) | |
| 123 | + per_source[name] = {"sigma": sv["sigma"], | |
| 124 | + "dose_margins": {str(a): dm[a] for a in DOSES}, | |
| 125 | + "monotone": mono, "halved": halve} | |
| 126 | + print(f"{name:8s} " + " ".join(f"{a:+.0f}σ:{dm[a]:+.2f}" for a in DOSES) + | |
| 127 | + f" | mono={mono} halve={halve}", flush=True) | |
| 128 | + | |
| 129 | + sigma_ref = float(np.mean([sv["sigma"] for sv in sources.values()])) | |
| 130 | + rand_changes = [] | |
| 131 | + for s in range(N_RANDOM_DIRS): | |
| 132 | + ru = np.random.default_rng(4000 + s).standard_normal(d_model) | |
| 133 | + ru /= np.linalg.norm(ru) | |
| 134 | + for a in (-2.0, 2.0): | |
| 135 | + taps[LAYER].edit = add_fn(ru.astype(np.float32), a * sigma_ref) | |
| 136 | + rand_changes.append(abs(margin() - base_m)) | |
| 137 | + taps[LAYER].edit = None | |
| 138 | + spec = float(np.mean(rand_changes)) < 0.25 * base_m | |
| 139 | + passes = bool(all_ab and spec) | |
| 140 | + print(f"L12 HANDLE CLAIM: all-sources mono+halve={all_ab} " | |
| 141 | + f"specific={spec} (randΔ {np.mean(rand_changes):.2f} vs bound {0.25 * base_m:.2f}) " | |
| 142 | + f"-> passes={passes}") | |
| 143 | + | |
| 144 | + commit = subprocess.run(["git", "rev-parse", "HEAD"], cwd=ROOT, | |
| 145 | + capture_output=True, text=True, check=False).stdout.strip() | |
| 146 | + ts = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime()) | |
| 147 | + outdir = ROOT / "results" / "expC_causal_verification" / ts | |
| 148 | + outdir.mkdir(parents=True) | |
| 149 | + (outdir / "results.json").write_text(json.dumps({ | |
| 150 | + "experiment": "expC_causal_verification", "run": 6, | |
| 151 | + "scope": "minimal L3 claim: layer-12 handle, four fresh sources, third bank", | |
| 152 | + "commit": commit, | |
| 153 | + "config": {"model": MODEL, "layer": LAYER, "doses": list(DOSES), | |
| 154 | + "sources": list(sources), "n_random_dirs": N_RANDOM_DIRS, | |
| 155 | + "n_pairs": len(pairs), "seed": SEED}, | |
| 156 | + "manifest": manifest(), | |
| 157 | + "baseline_margin": base_m, | |
| 158 | + "per_source": per_source, | |
| 159 | + "specificity": {"random_dir_mean_abs_change": float(np.mean(rand_changes)), | |
| 160 | + "bound": 0.25 * base_m, "specific": spec}, | |
| 161 | + "criterion": {"all_sources_mono_halve": all_ab, "specific": spec, "passes": passes}, | |
| 162 | + "wall_seconds": round(time.time() - t0, 1), | |
| 163 | + }, indent=2) + "\n") | |
| 164 | + print(f"results -> {outdir / 'results.json'}") | |
| 165 | + return 0 | |
| 166 | + | |
| 167 | + | |
| 168 | +if __name__ == "__main__": | |
| 169 | + sys.exit(main()) | |
modified
experiments/micro/expC_causal_verification/implementation/make_l3_mapcard.py
+31 −23
@@ -54,10 +54,10 @@ def main() -> int: | ||
| 54 | 54 | if not (V1 / "mapcard.json").exists(): |
| 55 | 55 | print("REFUSED: the Level-2 band entry (v1) must exist first.") |
| 56 | 56 | return 1 |
| 57 | − res5 = newest_run(5) | |
| 57 | + res5 = newest_run(6) | |
| 58 | 58 | doc5 = json.loads(res5.read_text()) |
| 59 | 59 | if not doc5["criterion"]["passes"]: |
| 60 | − print(f"REFUSED: run #5 criterion failed ({doc5['criterion']}) — v1 stays Level 2.") | |
| 60 | + print(f"REFUSED: run #6 criterion failed ({doc5['criterion']}) — v1 stays Level 2.") | |
| 61 | 61 | return 1 |
| 62 | 62 | res4 = newest_run(4) |
| 63 | 63 | doc4 = json.loads(res4.read_text()) |
@@ -68,16 +68,21 @@ def main() -> int: | ||
| 68 | 68 | "author": "Simon-Pierre Boucher", "contact": "contact@spboucher.ai", |
| 69 | 69 | "website": "https://modelmap.io", |
| 70 | 70 | "map_type": "interventions", "model_id": MODEL_ID, |
| 71 | − "claim": "LEVEL-3 claim: the diff-of-means agreement direction in the early band " | |
| 72 | − "(layers 2-15) is a causal HANDLE on grammatical-agreement behavior — " | |
| 73 | − "necessary (erasure removes ~73-75% of the margin, six fresh estimators, " | |
| 74 | − "run #4) AND controllable (activation-addition steering moves the margin " | |
| 75 | − "strictly monotonically with dose at layers 4/8/12, halves it at -2sigma, " | |
| 76 | − "while random directions move it by less than the registered 25% bound, " | |
| 77 | − "run #5).", | |
| 71 | + "claim": "LEVEL-3 claim, scope-limited: (i) band necessity — erasing the " | |
| 72 | + "diff-of-means agreement direction at any early-band layer (2-15) removes " | |
| 73 | + "~73-75% of the grammatical margin (six fresh estimators, run #4, Level-2 " | |
| 74 | + "record in v1); (ii) SINGLE-LAYER HANDLE at layer 12 — activation-addition " | |
| 75 | + "steering moves the margin strictly monotonically with dose and halves it " | |
| 76 | + "at -2 sigma for EVERY of four fresh direction sources on a third fresh " | |
| 77 | + "behavioral bank, while random directions at the same doses move it by " | |
| 78 | + "less than the registered 25% bound (run #6). Runs #3 and #5 record the " | |
| 79 | + "gate refusals that scoped this claim.", | |
| 78 | 80 | "band": v1_map["band"], |
| 79 | 81 | "per_layer": v1_map["per_layer"], |
| 80 | − "steering": doc5["results"], | |
| 82 | + "steering": {"layer": doc5["config"]["layer"], | |
| 83 | + "per_source": doc5["per_source"], | |
| 84 | + "specificity": doc5["specificity"], | |
| 85 | + "baseline_margin": doc5["baseline_margin"]}, | |
| 81 | 86 | "source_results": {"erasure_band": str(res4.relative_to(ROOT)), |
| 82 | 87 | "steering": str(res5.relative_to(ROOT))}, |
| 83 | 88 | } |
@@ -113,13 +118,13 @@ def main() -> int: | ||
| 113 | 118 | controls=["random-direction erasure nulls (netted)", |
| 114 | 119 | "random-direction steering specificity (< 25% bound)", |
| 115 | 120 | "six-source direction replication (run #4)", |
| 116 | − "dose-response monotonicity across 5 doses x 3 layers", | |
| 117 | − "late band excluded as estimator-unstable (run #3 refusal)"], | |
| 121 | + "four fresh sources x 5 doses at L12, third fresh bank (run #6)", | |
| 122 | + "late band excluded (run #3 refusal); L3 scope narrowed to L12 (run #5 refusal)"], | |
| 118 | 123 | methods_in_agreement=["difference-in-means probing (direction exists)", |
| 119 | 124 | "rank-1 erasure (necessity, band-replicated)", |
| 120 | 125 | "activation-addition steering (dose-controlled sufficiency)"], |
| 121 | 126 | interventions=["rank-1 direction erasure per layer (run #4)", |
| 122 | − "activation-addition steering, 5 doses x 3 layers (run #5)"], | |
| 127 | + "activation-addition steering at L12, 5 doses x 4 sources (run #6)"], | |
| 123 | 128 | replication_rate=round(json.loads((V1 / "mapcard.json").read_text())["replication_rate"], 4), |
| 124 | 129 | featurizer_class="linear (difference-in-means direction)", |
| 125 | 130 | intervention_protocol="erasure h-⟨h−μ,u⟩u; steering h+ασℓu, α∈{−2..+2}; " |
@@ -133,10 +138,12 @@ def main() -> int: | ||
| 133 | 138 | (ENTRY / "mapcard.json").write_text(card.to_json()) |
| 134 | 139 | |
| 135 | 140 | steer_lines = "\n".join( |
| 136 | − f"- L{l}: doses " + " ".join(f"{a}σ→{doc5['results']['per_layer'][l]['dose_margins'][a]:+.2f}" | |
| 137 | − for a in ("-2.0", "-1.0", "0.0", "1.0", "2.0")) + | |
| 138 | − f" (random-dir mean |Δ| {doc5['results']['per_layer'][l]['random_dir_mean_abs_change']:.2f})" | |
| 139 | − for l in map(str, doc5["config"]["test_layers"])) | |
| 141 | + f"- {name}: " + " ".join(f"{a}σ→{sv['dose_margins'][a]:+.2f}" | |
| 142 | + for a in ("-2.0", "-1.0", "0.0", "1.0", "2.0")) | |
| 143 | + for name, sv in doc5["per_source"].items()) + ( | |
| 144 | + f"\n- specificity: random-direction mean |Δ| " | |
| 145 | + f"{doc5['specificity']['random_dir_mean_abs_change']:.2f} vs bound " | |
| 146 | + f"{doc5['specificity']['bound']:.2f}") | |
| 140 | 147 | (ENTRY / "confidence.md").write_text(f"""--- |
| 141 | 148 | project: modelmap |
| 142 | 149 | document: qwen3-0.6b-4bit/interventions/v2 — confidence |
@@ -159,14 +166,15 @@ Causal verification : YES, both directions — necessity (erasure) and | ||
| 159 | 166 | dose-controlled sufficiency (steering) |
| 160 | 167 | ``` |
| 161 | 168 | |
| 162 | −Steering dose-response (baseline margin {doc5['results']['baseline_margin']:+.2f}): | |
| 169 | +Steering dose-response at layer {doc5['config']['layer']} (baseline margin {doc5['baseline_margin']:+.2f}): | |
| 163 | 170 | {steer_lines} |
| 164 | 171 | |
| 165 | −All three pre-registered gates passed: strict monotonicity at every test | |
| 166 | −layer, halving at −2σ, random-direction specificity under the 25% bound. | |
| 167 | −Scope honesty: one model, one behavior, band granularity; generality across | |
| 168 | −sizes (expG) and quantization levels (candidate_02) is untested and is the | |
| 169 | −registered next question. | |
| 172 | +All pre-registered gates passed at L12 for every fresh source: strict | |
| 173 | +monotonicity, halving at −2σ, random-direction specificity under the 25% | |
| 174 | +bound. Scope honesty: the HANDLE claim is single-layer (L12) and the | |
| 175 | +NECESSITY claim is band-level (v1); one model, one behavior. Runs #3 and #5 | |
| 176 | +are the refusals that scoped this entry. Generality across sizes (expG) and | |
| 177 | +quantization (candidate_02) is untested and registered next. | |
| 170 | 178 | """) |
| 171 | 179 | errs = card.validate() |
| 172 | 180 | if errs: |
modified
research/LOG.md
+27 −0
@@ -507,3 +507,30 @@ dose calibration + handle claim on the perturbation-tolerant sub-band; | ||
| 507 | 507 | or the minimal L12 single-layer L3 claim with fresh direction estimates. |
| 508 | 508 | Parallel tracks unchanged: arith_valid band, candidate_02 quantization |
| 509 | 509 | drift, expG cross-model. |
| 510 | + | |
| 511 | +--- | |
| 512 | + | |
| 513 | +## 2026-08-12 10:15 EDT — expC run #6: L12 handle fails replication — Level 3 abandoned, program closed | |
| 514 | + | |
| 515 | +**Result (pre-registered kill rule).** FALSIFIED on both prongs: only 1/4 | |
| 516 | +fresh sources monotone (the POSITIVE dose arm is unstable); specificity | |
| 517 | +did not replicate (random-direction |Δ| 3.36 vs bound 1.14 on fresh bank + | |
| 518 | +fresh seeds — run #5's 0.45 at L12 was sampling luck of 3 draws). Halving | |
| 519 | +at −2σ held 4/4: the negative (erasure-like) arm is robust, consistent | |
| 520 | +with the Level-2 entry. **Verdict: the agreement direction is NECESSARY | |
| 521 | +but NOT a reliable additive handle. Level 3 abandoned for this object; | |
| 522 | +steering program closed; interventions/v1 (Level 2) stands as the final | |
| 523 | +causal claim for agreement at 0.6B.** | |
| 524 | + | |
| 525 | +**The gate earned its keep a third time:** had run #5's L12 observation | |
| 526 | +been published without fresh re-registration, the atlas would now hold a | |
| 527 | +false Level-3 claim. Final arc record: refuse (r3) → pass L2 (r4) → | |
| 528 | +refuse (r5) → refuse (r6). | |
| 529 | + | |
| 530 | +**Methodology rule adopted:** specificity controls need ≥10 random- | |
| 531 | +direction draws with a percentile bound (3 was too few) — goes into | |
| 532 | +methodology.md and the map-card doctrine. | |
| 533 | + | |
| 534 | +**Proceeding tracks (from the Level-2 base):** arith_valid band protocol; | |
| 535 | +candidate_02 quantization drift of the band map; expG cross-model | |
| 536 | +replication on Qwen3-1.7B. | |
added
results/expC_causal_verification/20260812T072913Z/results.json
+115 −0
@@ -0,0 +1,115 @@ | ||
| 1 | +{ | |
| 2 | + "experiment": "expC_causal_verification", | |
| 3 | + "run": 6, | |
| 4 | + "scope": "minimal L3 claim: layer-12 handle, four fresh sources, third bank", | |
| 5 | + "commit": "f2f47f94aa921f3bc7637a0f83a22739c36c5620", | |
| 6 | + "config": { | |
| 7 | + "model": "mlx-community/Qwen3-0.6B-4bit", | |
| 8 | + "layer": 12, | |
| 9 | + "doses": [ | |
| 10 | + -2.0, | |
| 11 | + -1.0, | |
| 12 | + 0.0, | |
| 13 | + 1.0, | |
| 14 | + 2.0 | |
| 15 | + ], | |
| 16 | + "sources": [ | |
| 17 | + "Ahalf1", | |
| 18 | + "Ahalf2", | |
| 19 | + "Bhalf1", | |
| 20 | + "Bhalf2" | |
| 21 | + ], | |
| 22 | + "n_random_dirs": 3, | |
| 23 | + "n_pairs": 192, | |
| 24 | + "seed": 782 | |
| 25 | + }, | |
| 26 | + "manifest": { | |
| 27 | + "author": "Simon-Pierre Boucher", | |
| 28 | + "contact": "contact@spboucher.ai", | |
| 29 | + "website": "https://modelmap.io", | |
| 30 | + "chip": { | |
| 31 | + "brand": "Apple M5 Max", | |
| 32 | + "cores_total": 18, | |
| 33 | + "cores_performance": 6, | |
| 34 | + "cores_efficiency": 12 | |
| 35 | + }, | |
| 36 | + "memory": { | |
| 37 | + "unified_gb": 48.0, | |
| 38 | + "pagesize": 16384 | |
| 39 | + }, | |
| 40 | + "os": { | |
| 41 | + "system": "Darwin", | |
| 42 | + "version": "27.0", | |
| 43 | + "arch": "arm64" | |
| 44 | + }, | |
| 45 | + "software": { | |
| 46 | + "python": "3.14.4", | |
| 47 | + "numpy": "2.5.2", | |
| 48 | + "mlx": "0.32.0", | |
| 49 | + "torch": "2.13.0", | |
| 50 | + "safetensors": "0.8.0" | |
| 51 | + } | |
| 52 | + }, | |
| 53 | + "baseline_margin": 4.567708333333333, | |
| 54 | + "per_source": { | |
| 55 | + "Ahalf1": { | |
| 56 | + "sigma": 42.242393493652344, | |
| 57 | + "dose_margins": { | |
| 58 | + "-2.0": 1.73828125, | |
| 59 | + "-1.0": 2.4296875, | |
| 60 | + "0.0": 4.567708333333333, | |
| 61 | + "1.0": 6.28857421875, | |
| 62 | + "2.0": 5.590657552083333 | |
| 63 | + }, | |
| 64 | + "monotone": false, | |
| 65 | + "halved": true | |
| 66 | + }, | |
| 67 | + "Ahalf2": { | |
| 68 | + "sigma": 49.152095794677734, | |
| 69 | + "dose_margins": { | |
| 70 | + "-2.0": 1.2859700520833333, | |
| 71 | + "-1.0": 2.7734375, | |
| 72 | + "0.0": 4.567708333333333, | |
| 73 | + "1.0": 4.690755208333333, | |
| 74 | + "2.0": 5.4912109375 | |
| 75 | + }, | |
| 76 | + "monotone": true, | |
| 77 | + "halved": true | |
| 78 | + }, | |
| 79 | + "Bhalf1": { | |
| 80 | + "sigma": 45.88282012939453, | |
| 81 | + "dose_margins": { | |
| 82 | + "-2.0": 1.9191080729166667, | |
| 83 | + "-1.0": 3.6044921875, | |
| 84 | + "0.0": 4.567708333333333, | |
| 85 | + "1.0": 3.935546875, | |
| 86 | + "2.0": 4.4736328125 | |
| 87 | + }, | |
| 88 | + "monotone": false, | |
| 89 | + "halved": true | |
| 90 | + }, | |
| 91 | + "Bhalf2": { | |
| 92 | + "sigma": 38.40946578979492, | |
| 93 | + "dose_margins": { | |
| 94 | + "-2.0": 0.7444661458333334, | |
| 95 | + "-1.0": 2.0065104166666665, | |
| 96 | + "0.0": 4.567708333333333, | |
| 97 | + "1.0": 5.039388020833333, | |
| 98 | + "2.0": 4.46142578125 | |
| 99 | + }, | |
| 100 | + "monotone": false, | |
| 101 | + "halved": true | |
| 102 | + } | |
| 103 | + }, | |
| 104 | + "specificity": { | |
| 105 | + "random_dir_mean_abs_change": 3.3635207282172312, | |
| 106 | + "bound": 1.1419270833333333, | |
| 107 | + "specific": false | |
| 108 | + }, | |
| 109 | + "criterion": { | |
| 110 | + "all_sources_mono_halve": false, | |
| 111 | + "specific": false, | |
| 112 | + "passes": false | |
| 113 | + }, | |
| 114 | + "wall_seconds": 15.3 | |
| 115 | +} | |
modified
site/lib/charts.js
+6 −4
@@ -32,7 +32,7 @@ const fmt = (v) => (Math.abs(v) >= 100 ? v.toFixed(0) : Math.abs(v) >= 10 ? v.to | ||
| 32 | 32 | * series: [{label, color, dash?, ref?, values:[y per layer]}] |
| 33 | 33 | * Direct labels at line ends + legend; per-point hover targets. |
| 34 | 34 | */ |
| 35 | −function layerLineSvg({ title, series, yLabel, yMin = 0, yMax = 1, caption }) { | |
| 35 | +function layerLineSvg({ title, series, yLabel, yMin = 0, yMax = 1, caption, xTitle = "layer", xTickLabels = null }) { | |
| 36 | 36 | const W = 720, H = 300, ML = 52, MR = 118, MT = 16, MB = 40; |
| 37 | 37 | const iw = W - ML - MR, ih = H - MT - MB; |
| 38 | 38 | const nx = Math.max(...series.map((s) => s.values.length)); |
@@ -52,10 +52,12 @@ function layerLineSvg({ title, series, yLabel, yMin = 0, yMax = 1, caption }) { | ||
| 52 | 52 | grid += `<line x1="${ML}" y1="${Y(0)}" x2="${ML + iw}" y2="${Y(0)}" class="axis"/>`; |
| 53 | 53 | } |
| 54 | 54 | let xt = ""; |
| 55 | + const xlab = (i) => (xTickLabels ? (xTickLabels[i] ?? "") : String(i)); | |
| 55 | 56 | for (let i = 0; i < nx; i += Math.ceil(nx / 8)) { |
| 56 | − xt += `<text x="${X(i)}" y="${MT + ih + 18}" class="tick" text-anchor="middle">${i}</text>`; | |
| 57 | + xt += `<text x="${X(i)}" y="${MT + ih + 18}" class="tick" text-anchor="middle">${xlab(i)}</text>`; | |
| 57 | 58 | } |
| 58 | − xt += `<text x="${X(nx - 1)}" y="${MT + ih + 18}" class="tick" text-anchor="middle">${nx - 1}</text>`; | |
| 59 | + if (!xTickLabels) xt += `<text x="${X(nx - 1)}" y="${MT + ih + 18}" class="tick" text-anchor="middle">${xlab(nx - 1)}</text>`; | |
| 60 | + else for (let i = 1; i < nx; i += 1) if (i % Math.ceil(nx / 8) !== 0) xt += `<text x="${X(i)}" y="${MT + ih + 18}" class="tick" text-anchor="middle">${xlab(i)}</text>`; | |
| 59 | 61 | |
| 60 | 62 | let lines = "", dots = "", labels = ""; |
| 61 | 63 | const usedY = []; |
@@ -81,7 +83,7 @@ ${title ? `<figcaption class="chart-title">${esc(title)}</figcaption>` : ""} | ||
| 81 | 83 | <svg viewBox="0 0 ${W} ${H}" role="img" aria-label="${esc(title || yLabel)}"> |
| 82 | 84 | <line x1="${ML}" y1="${MT + ih}" x2="${ML + iw}" y2="${MT + ih}" class="axis"/> |
| 83 | 85 | ${grid}${xt}${lines}${dots}${labels} |
| 84 | −<text x="${ML + iw / 2}" y="${H - 4}" class="axis-title" text-anchor="middle">layer</text> | |
| 86 | +<text x="${ML + iw / 2}" y="${H - 4}" class="axis-title" text-anchor="middle">${esc(xTitle)}</text> | |
| 85 | 87 | <text transform="rotate(-90 14 ${MT + ih / 2})" x="14" y="${MT + ih / 2}" class="axis-title" text-anchor="middle">${esc(yLabel)}</text> |
| 86 | 88 | </svg> |
| 87 | 89 | <div class="chart-legend">${legend}</div> |
| 88 | 90 | |