SPB Git

spb/wpai0001_uqo

Public
10.0 KB

# Preregistration — WP-AI-0001 (v0.1, 2026-11-09)

Status: draft registered before any transplant experiment. The Tier-0 pilot (scripts/run_pilot.py) is explicitly exploratory: it serves to measure gap, verify hooks and localize the target subsystem. Confirmatory analyses will be frozen in v1.0 of this document (hash recorded in DECISION_LOG.md) before confirmatory data are generated. Changes after freeze are deviations and are logged.

# 1. Estimands

Notation as in CLAUDE.md §2. For a model pair (D, R), task family T with target stratum and control strata:

  • Transfer gain Δ_T = Q_T(R′) − Q_T(R), exact-match accuracy on held-out target items.
  • Control-adjusted transfer Δ_T^adj = Δ_T − [Q_T(R_sham) − Q_T(R)] where R_sham is the matched sham (same positions, layers, dimensionality and norm; donor content replaced by the control defined per method).
  • Retention ρ_c = Q_c(R′) / Q_c(R) for each control stratum c with Q_c(R) ≥ 0.5; otherwise report Δ_c.
  • Donor necessity N_D = Q_T(D) − Q_T(D_ablated) for the localized component.
  • Rescue = Q_T(D_ablated + graft) − Q_T(D_ablated).

Unit of inference: items nested in programs (clusters); model seeds are a second level. CIs: cluster bootstrap over programs within a seed; seed-level summaries report mean, sd and min over ≥ 3 seeds.

# 2. Hypotheses (Tier 0, micro-world A, target operator Ω = qua)

  • H1 (localization). In donors trained on all operators, interchange interventions identify a set of (layer, position) locations whose patching restores ≥ 0.8 of the clean answer rate at the Ω step, with the sham (corrupt-into-corrupt) rate ≤ chance + 0.1. Null: no location exceeds sham + 0.2.
  • H2 (sufficiency / activation graft). Mapped donor activations grafted at the identified location raise the recipient's Ω-step accuracy by Δ_T^adj ≥ 0.25 (smallest effect of interest, SESOI) on held-out target items. Null: Δ_T^adj ≤ 0.05.
  • H3 (necessity). Ablating the identified donor component reduces donor Ω-step accuracy by N_D ≥ 0.5 while control-stratum accuracy drops by ≤ 0.05.
  • H4 (selectivity). Retention ρ_c ≥ 0.95 on control_iid for the graft that achieves H2.
  • H5 (rule vs cache). The graft's effect on programs whose (Ω-neighbour, v0) combinations were absent from the alignment data is at least 0.8 × its effect on seen combinations. Null: ≤ 0.5 ×.
  • H6 (portability, Tier 1). Effects replicate (Δ_T^adj ≥ SESOI) for at least one cross-family pair; we predict failure for mismatched-tokenizer pairs without a tokenizer bridge and record it as such.

Directional predictions are stated for honesty; a null on any hypothesis is reported in full.

# 3. Primary and secondary outcomes

Primary (confirmatory): Δ_T^adj on target_confirmatory (generated after freeze from a new seed), and ρ_c on control_confirmatory. Secondary: trace accuracy, Ω-step accuracy, N_D, rescue, graft dimensionality, parameter count, latency, peak memory, seed stability.

# 4. Multiplicity

Localization sweeps (layers × positions × families × heads) are exploratory and selection is made on the development localization set only. Confirmatory comparisons are limited to the pre-declared list in EXPERIMENT_PLAN.md §5 (one primary contrast per method family; Holm correction across method families).

# 5. Power

Pilot variability will be recorded in EXPERIMENT_LOG.md after the pilot; the confirmatory sample size (items and clusters) will be chosen by simulation from the pilot cluster bootstrap so that an effect of SESOI = 0.25 has ≥ 0.9 power at α = 0.05 (two-sided) on the primary contrast. No effect sizes are invented.

# 6. Exclusion rules

Runs are excluded only for crashes, NaNs, or manifest/data-hash mismatches; all exclusions are listed in the run manifest and in EXPERIMENT_LOG.md. No run is excluded for its outcome.

# 7. Contamination safeguards

Split hashes are stored in manifests; evaluation splits are generated with seeds disjoint from training and alignment seeds; assert_disjoint is enforced at build time; alignment data for mappings never include evaluation items (checked by item hash).


# Preregistration v1.0 — CONFIRMATORY FREEZE (Tier 0), 2026-11-09

Frozen after exploratory runs graft_tier0_f50a28ca1bf7 (protocol v1) and graft_tier0_30e3644aa801 (protocol v2) and before any confirmatory split was generated. Confirmatory evaluation and alignment items come from seed base 90 000 (never used before); the trained models (artifacts/models/matrix_v1) are unchanged; the analysis code (scripts/run_transplant.py at the commit recorded in the confirmatory manifest) is unchanged.

# Primary method (fixed)

Activation graft, donor in the loop, at the recipient's resid_post.0 at every read position (untargeted), α = 1, with the alignment map = orthogonal Procrustes with isotropic scale fitted on task-free alignment rows (3 000 items without Ω, disjoint from evaluation items). Model pairs: the nine donor_s{i} → recipient_s{j} pairs (3 same-initialization, 6 different-initialization).

# Primary outcome and contrast

Δ_T^adj on target_iid = [Q_T(graft) − Q_T(recipient)] − [Q_T(sham map) − Q_T(recipient)], paired on items, cluster bootstrap over programs (2 000 draws), reported per pair and summarized over pairs (mean, sd, min). Decision rule: H2 (sufficiency) is supported if the lower 95 % bound of Δ_T^adj exceeds SESOI = 0.25 in at least 8 of the 9 pairs and the mean over pairs exceeds 0.5; retention H4 is supported if control_iid accuracy with the graft is ≥ 0.95 in all nine pairs.

# Secondary pre-declared contrasts (Holm correction over these four families, α = 0.05)

  1. Subspace graft k = 8 (Ω-contrast PCA through the primary map) vs random subspace k = 8, matched seeds.
  2. Adapter graft (donor-free ridge interface to the primary map's targets) vs unmodified recipient; and the same adapter evaluated in a fresh process on new splits (persistence; scripts/eval_persistent.py).
  3. Weight graft L0 block (attention + MLP + LN) vs shuffled-layer weight control, matched seeds; retention reported as a cost.
  4. Rule-vs-cache (H5): primary-map graft fitted on Ω inputs {0,…,5} (Procrustes on seen rows) evaluated on unseen-input items vs seen-input items; supported if unseen ≥ 0.8 × seen. The donor-free adapter is predicted to FAIL this test (cache), and the failure is reported as such.

# Mechanism tests (fixed, matched seed 0)

Donor ablation (attn_out.0 zeroed at read positions) → rescue from intact donor_s1 through the task-free Procrustes map vs sham rescue; reverse graft recipient → donor on control items (no harm expected).

# Portability (fixed)

Width mismatch (d 128 → 96) and depth mismatch (4 → 6 layers) with the task-free Procrustes map at L1 → L1 (the layer pair selected in exploration) and L0 → L0; stride-2 pair at L2 → L2 and L3 → L3.

# Exploratory (not confirmatory)

Everything else produced by the script (layer grid, dose, other maps, steering, other k) is reported as exploratory and labelled so in the paper.

# Tier 1 — selection rule frozen BEFORE the development results are read (2026-11-09)

Development run: tier1_taskvector_e105e3d91386 (dev split; the selection data). Confirmatory run: same script with --split confirmatory (disjoint query halves), executed after this rule is recorded and without any change to the script's conditions.

  • Eligible tasks per (pair, task): the recipient's own task vector reaches dev zero-shot accuracy ≥ 0.5 at its best layer (within-model sufficiency); otherwise the task is reported as "not vectorizable in the recipient" and excluded from the confirmatory contrast (reported, not tested).
  • Selected condition per eligible (pair, task): the transfer condition (map family × recipient layer × α, add variants, rescaled variants, or the donor-in-the-loop replace graft) with the highest dev accuracy; ties broken toward Procrustes, lower layer, α = 1, add before replace.
  • Confirmatory contrast: on the confirmatory queries, accuracy of the selected condition minus accuracy of the best control on the same queries (max over norm-matched random vector, unrelated-task vector, sham-map vector, mapped donor zero-shot state, shuffled-query donor state), paired on items, bootstrap over items (2 000 draws).
  • Decision rule (H6, per pair): supported if the lower 95 % bound exceeds 0.10 for at least two eligible tasks; "partial" if for one; "not supported" otherwise. Also reported: gain vs recipient zero-shot, recipient few-shot (prompting ceiling), recipient own vector (within-model ceiling), held-out NLL change and cross-task accuracy.
  • Tier 2 (Qwen2.5-7B pairs, run on M3U96b) follows the same rule.

# Amendment A1 to the Tier-1 rule (2026-11-09, recorded BEFORE the Tier-1 confirmatory run; D12)

The first development pair showed that the "mapped donor few-shot state of a shuffled query" condition is not a null control: the recipient keeps its own query in context, and the donor state carries the task identity, so it transfers the task exactly like the matched-query state (antonym, qwen3b→qwen1.5b: 0.72 vs 0.72 on dev). It is reclassified as a mechanism diagnostic (task identity vs query content). The null-control set for the H6 contrast is therefore: norm-matched random vector, unrelated-task vector, sham-map vector (random orthogonal map), and mapped donor zero-shot state (same map, no task information). The analysis reports both the pre-amendment (shuffled query included) and the amended contrast.

# Amendment A2 (2026-11-09, procedural; D13)

The first confirmatory Tier-1 run (tier1_taskvector_confirmatory_82a78041c0ce) re-localized the recipient's best layer on the confirmatory queries, so some development-selected conditions did not exist in it. As the rule intended, localization (layers, vectors) is frozen from the development split; the run is marked invalidated in the registry and re-executed with --split confirmatory loading the development localization caches. No numbers from the invalidated run are used.

Files · main
Menu

Theme