spb/modelmap Public License

1---2project: modelmap3document: expC_causal_verification — analysis (run #1)4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128modified: 2026-08-129status: reviewed10---1112# Analysis — expC run #1: the agreement probe map FAILS causal verification1314Run: `results/expC_causal_verification/20260812T064534Z/results.json` (5.7 s).15Model: Qwen3-0.6B-4bit. Input map: agreement differential (probes/v2).16Intervention: layer-skip ablation via the Tap layer. Hypothesis and the17binary verdict criterion registered before the run.1819```text20Hypothesis              : the top-5 differential layers (by real−twin probe21                          selectivity) causally support agreement behavior —22                          skip damage ≥ p95 of random-5 draws AND ≥ 2× their23                          mean.24Falsification criterion : top-5 damage inside the random distribution.25Result                  : FALSIFIED — survival 0/1. Baseline grammatical26                          margin +4.63 (sanity holds: the model robustly27                          prefers correct agreement). Skip damage:28                          top-5 (layers 17,18,19,21,22) = +2.17 —29                          BELOW the random-5 mean (+3.14, p95 +4.85, 2030                          draws); bottom-5 differential (layers 0–4) = +4.99,31                          the LARGEST of all conditions.32Interpretation          : Level 1 for the negative claim. Where agreement33                          information is most linearly decodable above the34                          architecture null (late-mid layers) is NOT where35                          the computation is causally load-bearing for the36                          behavior. This is the Hase-class dissociation37                          (localization ≠ causal support — notes §4.6)38                          measured end-to-end in our own pipeline, on a39                          pre-registered binary verdict. The40                          correlational→causal survival ledger opens at 0/1.41                          Declared caveats bite exactly as registered:42                          (a) layer-skip is coarse — early-layer skips43                          plausibly cause GENERAL degradation, not44                          agreement-specific damage (bottom-5 +4.99 reads as45                          "the model breaks", not "agreement lives at layers46                          0–4"); a specificity control (margin damage47                          normalized by general perplexity damage) is the48                          registered follow-up; (b) held-out pairs = 8949                          after dedup against every v2 probe text (below the50                          planned 200 — combo pool exhausted; enlarge banks51                          next run).52Next experiment         : run #2 with (1) perplexity-normalized specificity53                          scores per skip condition, (2) direction-level54                          intervention (LEACE erasure of the agreement55                          direction in the forward pass at layer ℓ) — a56                          surgical test the probe map CAN legitimately pass57                          or fail, (3) larger held-out bank. Then the same58                          protocol on arith_valid.59```6061## Ledger6263| # | Correlational claim | Intervention | Survives? |64|---|---|---|---|65| 1 | agreement top-5 differential layers (probes/v2) | layer-skip ablation | **No** (0/1) |66| 2 | agreement differential layer *profile* (probes/v2) | direction-level erasure scan | **No** (0/2) |6768The published survival rate is the running fraction of this table — the69charter §8.3 metric, now live.7071---7273# Analysis — expC run #2: ledger 0/2 — and the control scan finds the real structure7475Run: `results/expC_causal_verification/20260812T065406Z/results.json` (19276held-out pairs, baseline margin +4.24, NLL 5.73). Hypotheses registered77before the run.7879```text80Hypothesis (P1)         : run #1's bottom-5 damage is general, not81                          agreement-specific, once normalized by NLL damage.82Result (P1)             : CONFIRMED. Specificity (margin damage / NLL83                          damage): bottom-5 = 0.51 — BELOW the random-5 mean84                          (1.44); top-5 = 2.51 — above the random mean but85                          below the p95 (4.73). Run #1's verdict stands, now86                          with the confound measured: early-layer skips87                          break the model generally; nothing in the skip88                          family singles out the probe map's top layers.89Hypothesis (P2)         : per-layer direction-erasure specific damage90                          correlates with the differential probe profile91                          (Spearman ρ ≥ 0.4, perm-p < 0.05).92Result (P2)             : FALSIFIED, decisively — ρ = −0.136, perm-p 0.757.93                          Survival ledger: 0/2. The probe map's layer94                          ranking is behaviorally void at this granularity.95                          BUT the scan itself uncovered strong causal96                          structure the probe map missed: erasing the97                          layer-ℓ agreement direction (diff-of-means, with98                          random-direction controls netted out) at ANY99                          single layer in 2–15 destroys most of the margin100                          (specific damage +2.6…+4.0 of a +4.24 baseline;101                          L12: +3.97, L15: +3.75, L6: +3.71), while the102                          late layers the probes ranked highest carry103                          little (L21: +0.69, L20: +0.29) — and L18/L22104                          erasure slightly HELPS (−1.86/−1.09), suggesting105                          suppressive components.106Interpretation          : Level 0–1 (single model, single direction107                          estimate from set A, no seed replication of the108                          intervention yet — by our own doctrine this109                          causal profile is NOT publishable as an atlas110                          entry until replicated). Two lessons:111                          (1) correlational layer rankings did not survive112                          two different causal tests — the survival rate113                          the field never publishes is, so far, 0%;114                          (2) the behaviorally load-bearing object is a115                          low-dimensional DIRECTION present across116                          early-mid layers, not a "place" — consistent with117                          the linear-representation view and with why118                          decodability peaks (late, after information is119                          everywhere) diverge from causal joints (early,120                          where the direction is constructed). Random-121                          direction erasure also hurts at layers 2–9122                          (+0.9…+2.5): early residual streams are fragile123                          to ANY rank-1 deletion — netted out in the124                          specific column.125Next experiment         : run #3 — replicate the direction-erasure profile126                          (direction re-estimated on set B and on 5127                          bootstrap seeds; report profile replication rate)128                          → if stable, publish as the atlas's first129                          INTERVENTIONS map (Level toward 2–3) and make the130                          probes/v2 entry point to it as the causal131                          counterpart. Then the same scan on arith_valid.132```133134---135136# Analysis — expC run #3: the publication gate REFUSED the causal profile137138Run: `results/expC_causal_verification/20260812T065929Z/results.json`.139Five direction sources (set A, set B, 3 bootstraps of A), shared140random-direction controls, same held-out bank (192 pairs, baseline +4.24).141Hypothesis, criterion, and publication rule registered before the run.142143```text144Hypothesis              : the per-layer specific-damage profile is145                          estimator-stable — mean pairwise Spearman ρ ≥ 0.7146                          and early(2–15) ≥ 3× late(20–27) in EVERY source.147Result                  : FALSIFIED — mean ρ = 0.495 (min 0.176); the band148                          claim fails for setB and bootA2 (ratio 1.2).149                          make_interventions_mapcard.py refused the entry150                          (exit 1), per the registered rule. The atlas151                          stays at two entries — the gate did its job.152                          THE STABLE PART: early-band (layers 2–15) mean153                          specific damage replicates tightly across all154                          five sources (+2.98, +3.22, +3.34, +3.27, +3.28)155                          — erasing the estimated agreement direction156                          anywhere in the early-mid band reliably destroys157                          ~70–79% of the behavior, whatever the estimation158                          set. THE UNSTABLE PART: the late band (20–27)159                          swings from −0.44 to +2.77 with the estimator —160                          the late-layer "profile" is direction-estimation161                          noise, which also retroactively explains run #2's162                          anti-correlation (the probe ranking lives exactly163                          where the causal profile is noise).164Interpretation          : Level 0–1. The full profile is NOT a stable165                          object; the coarser claim ("an early-band166                          agreement direction is causally load-bearing") is167                          the replicable candidate. Publishing gates that168                          refuse are the mechanism that keeps the atlas169                          honest — this refusal is itself a process result170                          worth reporting on the site's methodology page.171Next experiment         : run #4 with the NARROWER pre-registered claim:172                          early-band (2–15) mean specific damage ≥ 2.5 on173                          fresh direction estimates (new bootstrap seeds +174                          a held-out estimation split) and a fresh175                          behavioral bank; late band explicitly excluded as176                          unstable. If it passes, publish the interventions177                          map as an early-band BAND claim (not a per-layer178                          ranking), Level 2.179```180181---182183# Analysis — expC run #4: the BAND claim passes — first Level-2 atlas entry184185Run: `results/expC_causal_verification/20260812T071954Z/results.json`.186Everything fresh: six direction sources (disjoint halves of A and B + two187new bootstraps), new behavioral bank (8 unseen locations, 192 pairs,188baseline margin +4.45). Claim, bar, and publication rule registered before189the run.190191```text192Hypothesis              : early-band (2–15) mean specific damage ≥ 2.5 in193                          EVERY fresh source; late band reported, no claim.194Result                  : CONFIRMED — early-band means: Ahalf1 +3.35,195                          Ahalf2 +3.23, bootA +3.30, Bhalf1 +3.26,196                          Bhalf2 +3.33, bootB +3.27 (min +3.228 vs bar197                          2.5). Spread across six estimators: 0.12 — the198                          band is a tight, estimator-stable causal object.199                          Late band again unstable (+0.27…+2.83), as200                          declared; it carries no claim.201Interpretation          : Level 2, published: atlas/qwen3-0.6b-4bit/202                          interventions/v1 — the project's first Level-2203                          entry and first causal map. The claim: a single204                          linear direction (diff-of-means over correct vs205                          violated agreement), erased at ANY one layer in206                          2–15, removes ~73–75% of the model's grammatical207                          preference on held-out pairs, replicated across208                          six independent direction estimates and netted209                          against random-direction damage. NOT Level 3:210                          both agreeing methods share the diff-of-means211                          estimator — activation-addition steering is the212                          registered L3 path. Granularity discipline paid213                          off: the per-layer version of this map was214                          refused (run #3); the band version replicates.215Next experiment         : (1) L3 path: steering (add the direction) should216                          INCREASE the margin on violated-preference pairs;217                          (2) same band protocol on arith_valid; (3)218                          quantization drift of the band map (candidate_02:219                          does the early band move under Q8/FP16?);220                          (4) cross-model: does the band replicate on221                          Qwen3-1.7B (expG entry)?222```223224---225226# Analysis — expC run #5: Level-3 gate refused — and layer 12 is a textbook handle227228Run: `results/expC_causal_verification/20260812T072455Z/results.json`.229Steering doses α·σℓ·u at layers {4, 8, 12}, α ∈ {−2…+2}; conjunctive230criterion (monotone AND halved AND specific at EVERY test layer) registered231before the run. make_l3_mapcard.py refused (exit 1); v1 stays Level 2.232233```text234Hypothesis              : strict dose-response monotonicity + halving +235                          random-direction specificity at all of L4/L8/L12.236Result                  : FALSIFIED as a conjunction.237                          L12 — PASSES EVERYTHING, textbook: margins238                          +1.95 / +3.00 / +4.45 / +5.82 / +6.79 across239                          −2σ…+2σ (strictly monotone), halved at −2σ,240                          random-direction |Δ| = 0.45 vs bound 1.11.241                          L08 — near-monotone (+2σ dips: 5.70 < 6.26),242                          random |Δ| 2.97: not specific at this dose.243                          L04 — OVERDOSE REGIME: ±2σ both collapse the244                          margin (+0.08 / +0.04) and random directions are245                          equally destructive (|Δ| 4.40): at early layers,246                          ANY perturbation of magnitude ~2σ (σ estimated247                          from mean-pooled reps) breaks the computation —248                          the rank-1 fragility of run #2, now dose-resolved.249Interpretation          : Level 0–1. The direction is a clean, dose-250                          controlled causal handle at mid-band (L12) but251                          the conjunctive claim over the whole test set252                          fails, so the gate held v2 back — correctly.253                          Reading: "necessity everywhere in the band"254                          (erasure, L2 entry) coexists with "controllable255                          handle only where the layer tolerates256                          perturbation". Dose scale is a confound at early257                          layers: σ from mean-pooled statistics likely258                          overdoses positions at layers with different259                          norm profiles.260Next experiment         : run #6 (registered idea): layer-local dose261                          calibration (σ from position-level projections at262                          the target layer; or a dose-sweep to find each263                          layer's non-destructive range), then re-register264                          the handle claim on the sub-band that tolerates265                          calibrated doses (candidate: 10–14). Also register266                          the L12 single-layer handle claim with fresh267                          direction estimates as a minimal L3 candidate.268```269270## Gate record (for methodology.md)271272Three refusals/passes to date, all pre-registered: run #3 per-layer profile273REFUSED → run #4 band claim PASSED (Level 2 published) → run #5 L3274conjunction REFUSED (v1 unchanged). The atlas never received a claim its275evidence didn't carry.276277---278279# Analysis — expC run #6: the L12 handle claim fails replication — Level 3 abandoned280281Run: `results/expC_causal_verification/20260812T072913Z/results.json`.282Four fresh direction sources at L12, third fresh bank (192 pairs, baseline283+4.57), fresh random-direction seeds. Registered rule: any source failing284monotonicity/halving, or specificity failing, kills the L3 claim and closes285the steering program for this object.286287```text288Hypothesis              : L12 is an estimator-stable dose-controlled handle.289Result                  : FALSIFIED. Monotone: 1/4 sources only (Ahalf2);290                          the POSITIVE dose arm is unstable (Ahalf1 +2σ dips291                          5.59<6.29; Bhalf1 +1σ dips below baseline; Bhalf2292                          +2σ < +1σ). Halving at −2σ: 4/4 — the negative293                          (erasure-like) arm is robust, again. SPECIFICITY294                          FAILED TO REPLICATE: random-direction mean |Δ| =295                          3.36 vs bound 1.14 on the fresh bank and fresh296                          random seeds (run #5's L12 value was 0.45 — with297                          only 3 random draws, that pass now reads as298                          sampling luck).299Interpretation          : Level 1 for the negative. The direction is300                          NECESSARY (v1, Level 2, band-replicated) but NOT301                          a reliable additive handle: pushing along it does302                          not control the behavior in a dose-stable,303                          direction-specific way. Per the pre-registered304                          rule, Level 3 is abandoned for this object and305                          the steering program is closed. Had run #5's L12306                          observation been published without fresh307                          re-registration, the atlas would now contain a308                          false Level-3 claim — the gate earned its keep a309                          third time.310Methodology lesson      : 3 random-direction draws are too few for a311                          specificity bound; methodology.md will require312                          ≥10 draws with a percentile bound for any313                          specificity control from now on.314Next experiment         : program closed here. Proceeding tracks: arith_valid315                          band protocol; candidate_02 (quantization drift of316                          the Level-2 band map); expG (band replication on317                          Qwen3-1.7B). The final gate record for this arc:318                          refuse → pass(L2) → refuse → refuse.319```320