spb/modelmap Public License

1---2project: modelmap3document: expC_causal_verification — hypothesis (run #1, registered before run)4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128modified: 2026-08-129status: reviewed10---1112# Hypothesis — expC run #1 (does the agreement probe map survive ablation?)1314> Causal verification pipeline — correlational→causal survival rate.15> Registered 2026-08-12 **before** the run. Input: the differential16> (real−twin) agreement probe map from expA run #217> (atlas/qwen3-0.6b-4bit/probes/v2). Intervention: layer-skip ablation18> (block contribution removed, residual passes through) via the Tap layer.1920```text21Hypothesis              : The top-5 layers of the agreement DIFFERENTIAL map22                          causally support agreement behavior: skipping them23                          damages the model's grammatical preference24                          (mean logit margin of the correct verb form over25                          the incorrect one, on held-out minimal pairs) more26                          than skipping 5 random non-top layers — top-527                          damage ≥ the 95th percentile of the random-528                          damage distribution (20 draws), and ≥ 2× its mean.29Falsification criterion : Top-5 damage inside the random-5 distribution30                          (< 95th percentile) — the correlational map fails31                          causal verification; this survival datum (0 or 1)32                          is the first entry of expC's published33                          correlational→causal survival rate, either way.34Method                  : Behavioral metric: margin = mean over minimal35                          pairs of [logit(correct is/are) − logit(incorrect)]36                          at the verb position, teacher-forced prefix37                          "The <noun(s)> <location>". Held-out pairs: fresh38                          seed, deduped against every v2 probe text.39                          Conditions: baseline (no skip); skip top-540                          differential layers; skip 5 random layers41                          (excluding top-5), 20 draws; skip bottom-542                          differential layers (second control). Same model43                          (Qwen3-0.6B-4bit), deterministic forwards.44Baseline / null         : baseline margin (sanity: must be > 0, i.e. the45                          model actually prefers grammatical agreement —46                          else the whole question is moot at this size);47                          random-5 and bottom-5 skip distributions.48Result                  : (pending)49Interpretation          : (pending — with explicit confidence level)50Next experiment         : (pending)51```5253**Declared caveats.** Layer-skip is a coarse intervention (removes ALL of a54block's computation, not the agreement direction specifically) — a confirmed55result licenses "these layers causally support agreement behavior", NOT56"agreement is localized to these layers" (hydra/backup effects, notes §4.4,57can mask redundancy). Direction-level interventions (LEACE-style erasure in58the forward pass) are the registered follow-up.5960---6162# Hypothesis — expC run #2 (specificity control + direction-level surgery)6364Registered 2026-08-12 **before** the run, after run #1's non-confirmation.65Two declared confounds get instruments: general-damage normalization for66layer-skip, and a surgical direction-level intervention the probe map can67legitimately pass or fail.6869```text70Hypothesis (P1)         : run #1's bottom-5 damage is GENERAL, not71                          agreement-specific: normalizing margin damage by72                          general damage (mean NLL increase on neutral73                          prose), the bottom-5 specificity ratio falls at or74                          below the random-5 mean ratio, and top-5 does not75                          exceed the random p95 either (the run-1 verdict76                          stands, now with the confound measured).77Hypothesis (P2)         : direction-level erasure tracks the probe map:78                          erasing the layer-ℓ agreement direction79                          (difference-in-means, estimated on agreement_A80                          mean-pooled reps) at all positions of ℓ's output,81                          minus the damage from erasing a random direction82                          at the same layer, yields a per-layer specific-83                          damage profile that correlates with the84                          differential probe profile: Spearman ρ ≥ 0.485                          (permutation p < 0.05, 10k perms).86Falsification criterion : (P2) ρ ≤ 0 → the localization claim also dies at87                          direction level: survival ledger 0/2 and the88                          agreement map's layer structure is declared89                          behaviorally void at this granularity.90Method                  : held-out bank enlarged with 6 NEW locations91                          (target ≥ 180 pairs, deduped as before); NLL on92                          20 neutral prose sentences per condition;93                          direction erasure h' = h − ⟨h−μ, u⟩u applied to94                          every position of layer ℓ's output, u = unit95                          class-mean difference at ℓ, μ = grand mean;96                          random-direction control: 3 seeds per layer,97                          same procedure; all 28 layers scanned.98Baseline / null         : per-layer random-direction erasure damage;99                          run #1 skip conditions rerun on the enlarged bank.100Result                  : (pending)101Interpretation          : (pending)102Next experiment         : (pending)103```104105---106107# Hypothesis — expC run #3 (replication of the causal direction profile)108109Registered 2026-08-12 **before** the run. Run #2 found that erasing the110diff-of-means agreement direction at any layer 2–15 destroys the behavior.111By our own doctrine that profile is unpublishable until it replicates112across direction estimates.113114```text115Hypothesis              : The per-layer specific-damage profile is an116                          estimator-stable object: profiles from directions117                          estimated on (i) set A, (ii) set B (disjoint118                          nouns), (iii) 3 bootstrap resamples of set A,119                          agree pairwise at Spearman ρ ≥ 0.7 (mean over the120                          10 pairs), AND the qualitative claim holds in121                          EVERY source: mean specific damage over layers122                          2–15 ≥ 3× mean over layers 20–27.123Falsification criterion : mean pairwise ρ < 0.7 or any source violating the124                          3× band claim → the "causal direction profile" is125                          estimator noise; not publishable; expC pivots to126                          steering-based verification.127Method                  : shared random-direction controls (3 dirs/layer,128                          computed once); agreement-direction scan repeated129                          per source (5 sources × 28 layers × 192 pairs);130                          same held-out bank and margins as run #2.131Baseline / null         : shared random-direction damage per layer.132Publication rule        : if confirmed → atlas qwen3-0.6b-4bit/133                          interventions/v1 at Level 2 (probing evidences the134                          direction, erasure confirms causal load,135                          replicated across estimates — NOT Level 3, because136                          the two agreeing methods share the diff-of-means137                          estimator; an independent intervention family138                          (activation addition) is the registered L3 path).139Result                  : (pending)140Interpretation          : (pending)141Next experiment         : (pending)142```143144---145146# Hypothesis — expC run #4 (the narrowed BAND claim)147148Registered 2026-08-12 **before** the run, after run #3's gate refusal149showed the early band replicates (+2.98…+3.34) while the late band is150estimator noise. Everything fresh: direction estimates from DISJOINT HALVES151of agreement_A and agreement_B (never used whole-set estimates), two new152bootstrap seeds, and a NEW held-out behavioral bank (8 locations unseen by153any prior run or promptset).154155```text156Hypothesis              : EARLY-BAND claim only — mean specific damage over157                          layers 2–15 is ≥ 2.5 (on the fresh bank's158                          baseline-margin scale) in EVERY one of six fresh159                          direction sources (A-half-1, A-half-2, B-half-1,160                          B-half-2, bootA-fresh, bootB-fresh). The late161                          band (20–27) is REPORTED but carries no claim —162                          declared unstable by run #3.163Falsification criterion : any source with early-band mean < 2.5 → the band164                          claim fails replication too; the direction-erasure165                          program is closed for agreement at 0.6B and expC166                          pivots to steering-based verification.167Method                  : full 28-layer scan per source (map format168                          unchanged); shared random-direction controls169                          (3/layer) on the fresh bank; margins as before.170Publication rule        : pass → atlas interventions/v1 published as a BAND171                          map at Level 2: map.json carries the six profiles,172                          their mean/min, the random-direction damage, and173                          the band statistics; the chart plots mean, worst174                          source, and the raw random-direction reference.175                          Fail → refusal documented, program closed.176Result                  : (pending)177Interpretation          : (pending)178Next experiment         : (pending)179```180181---182183# Hypothesis — expC run #5 (Level-3 path: dose-response steering)184185Registered 2026-08-12 **before** the run. Erasure (runs #2–#4) established186necessity of the agreement direction in the early band. Steering (activation187addition) is the INDEPENDENT intervention family required for Level 3: if188the direction is the causal handle, pushing along it must move the behavior189in the predicted direction, dose-dependently, and random directions must not.190191```text192Hypothesis              : At each of three early-band test layers (4, 8, 12),193                          adding δ = α·σℓ·u to every position of the layer194                          output produces a STRICTLY MONOTONE margin in195                          α ∈ {−2, −1, 0, +1, +2} (σℓ = std of activation196                          projections on u at layer ℓ), with effect size:197                          margin(α=−2) ≤ 0.5 × baseline at every test layer.198                          Specificity: the same ±2σ doses along 3 random199                          directions move the margin by < 25% of baseline200                          (mean absolute change), at every test layer.201Falsification criterion : monotonicity broken at any test layer, OR the −2σ202                          dose fails to halve the margin anywhere, OR random203                          directions move the margin ≥ 25% — steering fails,204                          the entry stays Level 2, and the diff-of-means205                          direction is declared necessary-but-not-a-handle.206Method                  : direction u and σℓ estimated from agreement_A207                          (full set — the published v1 object); behavioral208                          bank = run #4's fresh bank (never used for209                          estimation); doses applied at one layer at a time.210Publication rule        : pass → atlas interventions/v2 at LEVEL 3 (erasure211                          necessity + steering dose-response = two212                          independent intervention families + probing;213                          v1 stays as the Level-2 record). Fail → documented,214                          v1 unchanged.215Result                  : (pending)216Interpretation          : (pending)217Next experiment         : (pending)218```219220---221222# Hypothesis — expC run #6 (minimal L3 claim: layer 12 as a single-layer handle)223224Registered 2026-08-12 **before** the run, after run #5's conjunction failed225while L12 passed every gate. The claim is narrowed to the object that226showed textbook behavior — scope discipline again, now applied to layers.227228```text229Hypothesis              : At LAYER 12 ONLY, the diff-of-means agreement230                          direction is a dose-controlled causal handle, and231                          this is estimator-stable: for EVERY one of four232                          fresh direction sources (disjoint halves of233                          agreement_A and agreement_B, new permutation234                          seed), on a THIRD fresh behavioral bank (8235                          never-used locations):236                          (a) margins strictly monotone in237                              α ∈ {−2,−1,0,+1,+2}·σ;238                          (b) margin(−2σ) ≤ 0.5 × baseline;239                          (c) shared specificity control at L12: 3 random240                              directions at ±2σ move the margin by < 25%241                              of baseline on average.242Falsification criterion : any source failing (a) or (b), or (c) failing —243                          the single-layer L3 claim dies and Level 3 is244                          abandoned for this object (steering program245                          closed; candidate_02/expG proceed from the246                          Level-2 band entry).247Method                  : same margin metric; direction + σ re-estimated248                          per source at L12 from half-set mean-pooled reps;249                          doses applied at L12 output, all positions.250Publication rule        : pass → interventions/v2 at LEVEL 3 with scope251                          declared as {band necessity (v1) + single-layer252                          L12 handle}; fail → documented refusal, v1253                          unchanged.254Result                  : (pending)255Interpretation          : (pending)256Next experiment         : (pending)257```258