spb/wpai0001_uqo

1# Preregistration — WP-AI-0001 (v0.1, 2026-11-09)23**Status:** draft registered *before* any transplant experiment. The Tier-0 pilot (`scripts/run_pilot.py`)4is explicitly *exploratory*: it serves to measure gap, verify hooks and localize the target subsystem.5Confirmatory analyses will be frozen in v1.0 of this document (hash recorded in `DECISION_LOG.md`) before6confirmatory data are generated. Changes after freeze are deviations and are logged.78## 1. Estimands910Notation as in CLAUDE.md §2. For a model pair (D, R), task family T with target stratum and control strata:1112- **Transfer gain** Δ_T = Q_T(R′) − Q_T(R), exact-match accuracy on held-out target items.13- **Control-adjusted transfer** Δ_T^adj = Δ_T − [Q_T(R_sham) − Q_T(R)] where R_sham is the *matched sham*14  (same positions, layers, dimensionality and norm; donor content replaced by the control defined per method).15- **Retention** ρ_c = Q_c(R′) / Q_c(R) for each control stratum c with Q_c(R) ≥ 0.5; otherwise report Δ_c.16- **Donor necessity** N_D = Q_T(D) − Q_T(D_ablated) for the localized component.17- **Rescue** = Q_T(D_ablated + graft) − Q_T(D_ablated).1819Unit of inference: *items nested in programs (clusters)*; model seeds are a second level. CIs: cluster20bootstrap over programs within a seed; seed-level summaries report mean, sd and min over ≥ 3 seeds.2122## 2. Hypotheses (Tier 0, micro-world A, target operator Ω = `qua`)2324- **H1 (localization).** In donors trained on all operators, interchange interventions identify a set of25  (layer, position) locations whose patching restores ≥ 0.8 of the clean answer rate at the Ω step, with26  the sham (corrupt-into-corrupt) rate ≤ chance + 0.1. Null: no location exceeds sham + 0.2.27- **H2 (sufficiency / activation graft).** Mapped donor activations grafted at the identified location28  raise the recipient's Ω-step accuracy by Δ_T^adj ≥ 0.25 (smallest effect of interest, SESOI) on29  held-out target items. Null: Δ_T^adj ≤ 0.05.30- **H3 (necessity).** Ablating the identified donor component reduces donor Ω-step accuracy by31  N_D ≥ 0.5 while control-stratum accuracy drops by ≤ 0.05.32- **H4 (selectivity).** Retention ρ_c ≥ 0.95 on `control_iid` for the graft that achieves H2.33- **H5 (rule vs cache).** The graft's effect on programs whose (Ω-neighbour, v0) combinations were absent34  from the alignment data is at least 0.8 × its effect on seen combinations. Null: ≤ 0.5 ×.35- **H6 (portability, Tier 1).** Effects replicate (Δ_T^adj ≥ SESOI) for at least one cross-family pair;36  we predict *failure* for mismatched-tokenizer pairs without a tokenizer bridge and record it as such.3738Directional predictions are stated for honesty; a null on any hypothesis is reported in full.3940## 3. Primary and secondary outcomes4142Primary (confirmatory): Δ_T^adj on `target_confirmatory` (generated after freeze from a new seed),43and ρ_c on `control_confirmatory`. Secondary: trace accuracy, Ω-step accuracy, N_D, rescue, graft44dimensionality, parameter count, latency, peak memory, seed stability.4546## 4. Multiplicity4748Localization sweeps (layers × positions × families × heads) are exploratory and selection is made on the49*development* localization set only. Confirmatory comparisons are limited to the pre-declared list in50`EXPERIMENT_PLAN.md` §5 (one primary contrast per method family; Holm correction across method families).5152## 5. Power5354Pilot variability will be recorded in `EXPERIMENT_LOG.md` after the pilot; the confirmatory sample size55(items and clusters) will be chosen by simulation from the pilot cluster bootstrap so that an effect of56SESOI = 0.25 has ≥ 0.9 power at α = 0.05 (two-sided) on the primary contrast. No effect sizes are invented.5758## 6. Exclusion rules5960Runs are excluded only for crashes, NaNs, or manifest/data-hash mismatches; all exclusions are listed in61the run manifest and in `EXPERIMENT_LOG.md`. No run is excluded for its outcome.6263## 7. Contamination safeguards6465Split hashes are stored in manifests; evaluation splits are generated with seeds disjoint from training66and alignment seeds; `assert_disjoint` is enforced at build time; alignment data for mappings never67include evaluation items (checked by item hash).6869---7071# Preregistration v1.0 — CONFIRMATORY FREEZE (Tier 0), 2026-11-097273Frozen after exploratory runs `graft_tier0_f50a28ca1bf7` (protocol v1) and `graft_tier0_30e3644aa801`74(protocol v2) and **before** any confirmatory split was generated. Confirmatory evaluation and alignment75items come from seed base 90 000 (never used before); the trained models (`artifacts/models/matrix_v1`) are76unchanged; the analysis code (`scripts/run_transplant.py` at the commit recorded in the confirmatory manifest)77is unchanged.7879## Primary method (fixed)80Activation graft, donor in the loop, at the recipient's `resid_post.0` at every read position (untargeted),81α = 1, with the alignment map = **orthogonal Procrustes with isotropic scale fitted on task-free alignment rows**82(3 000 items without Ω, disjoint from evaluation items). Model pairs: the nine donor_s{i} → recipient_s{j}83pairs (3 same-initialization, 6 different-initialization).8485## Primary outcome and contrast86Δ_T^adj on `target_iid` = [Q_T(graft) − Q_T(recipient)] − [Q_T(sham map) − Q_T(recipient)], paired on items,87cluster bootstrap over programs (2 000 draws), reported per pair and summarized over pairs (mean, sd, min).88**Decision rule:** H2 (sufficiency) is supported if the lower 95 % bound of Δ_T^adj exceeds SESOI = 0.25 in at89least 8 of the 9 pairs and the mean over pairs exceeds 0.5; retention H4 is supported if `control_iid` accuracy90with the graft is ≥ 0.95 in all nine pairs.9192## Secondary pre-declared contrasts (Holm correction over these four families, α = 0.05)931. Subspace graft k = 8 (Ω-contrast PCA through the primary map) vs random subspace k = 8, matched seeds.942. Adapter graft (donor-free ridge interface to the primary map's targets) vs unmodified recipient; and the95   same adapter evaluated in a fresh process on new splits (persistence; `scripts/eval_persistent.py`).963. Weight graft L0 block (attention + MLP + LN) vs shuffled-layer weight control, matched seeds; retention97   reported as a cost.984. Rule-vs-cache (H5): primary-map graft fitted on Ω inputs {0,…,5} (Procrustes on seen rows) evaluated on99   unseen-input items vs seen-input items; supported if unseen ≥ 0.8 × seen. The donor-free adapter is100   predicted to FAIL this test (cache), and the failure is reported as such.101102## Mechanism tests (fixed, matched seed 0)103Donor ablation (attn_out.0 zeroed at read positions) → rescue from intact donor_s1 through the task-free104Procrustes map vs sham rescue; reverse graft recipient → donor on control items (no harm expected).105106## Portability (fixed)107Width mismatch (d 128 → 96) and depth mismatch (4 → 6 layers) with the task-free Procrustes map at L1 → L1108(the layer pair selected in exploration) and L0 → L0; stride-2 pair at L2 → L2 and L3 → L3.109110## Exploratory (not confirmatory)111Everything else produced by the script (layer grid, dose, other maps, steering, other k) is reported as112exploratory and labelled so in the paper.113114## Tier 1 — selection rule frozen BEFORE the development results are read (2026-11-09)115116Development run: `tier1_taskvector_e105e3d91386` (dev split; the selection data). Confirmatory run: same script with117`--split confirmatory` (disjoint query halves), executed after this rule is recorded and without any change to the118script's conditions.119120- **Eligible tasks** per (pair, task): the recipient's *own* task vector reaches dev zero-shot accuracy ≥ 0.5 at its121  best layer (within-model sufficiency); otherwise the task is reported as "not vectorizable in the recipient" and122  excluded from the confirmatory contrast (reported, not tested).123- **Selected condition** per eligible (pair, task): the transfer condition (map family × recipient layer × α, add124  variants, rescaled variants, or the donor-in-the-loop replace graft) with the highest dev accuracy; ties broken125  toward Procrustes, lower layer, α = 1, add before replace.126- **Confirmatory contrast**: on the confirmatory queries, accuracy of the selected condition minus accuracy of the127  *best control* on the same queries (max over norm-matched random vector, unrelated-task vector, sham-map vector,128  mapped donor zero-shot state, shuffled-query donor state), paired on items, bootstrap over items (2 000 draws).129- **Decision rule (H6, per pair)**: supported if the lower 95 % bound exceeds 0.10 for at least two eligible tasks;130  "partial" if for one; "not supported" otherwise. Also reported: gain vs recipient zero-shot, recipient few-shot131  (prompting ceiling), recipient own vector (within-model ceiling), held-out NLL change and cross-task accuracy.132- Tier 2 (Qwen2.5-7B pairs, run on M3U96b) follows the same rule.133134### Amendment A1 to the Tier-1 rule (2026-11-09, recorded BEFORE the Tier-1 confirmatory run; D12)135The first development pair showed that the "mapped donor few-shot state of a *shuffled* query" condition is not a136null control: the recipient keeps its own query in context, and the donor state carries the task identity, so it137transfers the task exactly like the matched-query state (antonym, qwen3b→qwen1.5b: 0.72 vs 0.72 on dev). It is138reclassified as a **mechanism diagnostic** (task identity vs query content). The null-control set for the H6139contrast is therefore: norm-matched random vector, unrelated-task vector, sham-map vector (random orthogonal map),140and mapped donor **zero-shot** state (same map, no task information). The analysis reports both the pre-amendment141(shuffled query included) and the amended contrast.142143### Amendment A2 (2026-11-09, procedural; D13)144The first confirmatory Tier-1 run (`tier1_taskvector_confirmatory_82a78041c0ce`) re-localized the recipient's145best layer on the confirmatory queries, so some development-selected conditions did not exist in it. As the rule146intended, localization (layers, vectors) is frozen from the development split; the run is marked *invalidated*147in the registry and re-executed with `--split confirmatory` loading the development localization caches. No148numbers from the invalidated run are used.149