spb/wpai0001_uqo

1\section{Experimental Design}\label{sec:design}23\begin{figure}[t]\centering4\includegraphics[width=\textwidth]{figures/fig_system_map.pdf}5\caption{System map. Top: the causal chain applied to every donor--recipient pair (localization before transfer;6alignment map fitted on task-free data; graft families and their controls; preregistered evaluation). Bottom:7the two tiers, their models, backends and hosts; the registry counts are read from the run manifests.}8\label{fig:systemmap}\end{figure}910\subsection{Tier 0: a controlled micro-world with a manufactured gap}11\paragraph{Task (OPSEQ).} A program $\langle\mathrm{bos}\rangle\, v_0\, o_1 \dots o_k\, {=}\, r_1 \dots r_k$12applies $k\in[1,5]$ operators drawn from eight invented maps on $\mathbb{Z}_{11}$ (\texttt{zib}: $x{+}1$,13\texttt{zob}: $x{+}3$, \texttt{qua}: $2x{+}1$, \texttt{lar}: $3x$, \texttt{neg}: $-x$, \texttt{flip}: $10{-}x$,14\texttt{dro}: $5x$, \texttt{mik}: $x^2$, all modulo 11) to $v_0$ and emits the execution trace $r_j$ (the value15after operator $j$). The target operator is $\Omega = \texttt{qua}$; a ninth operator \texttt{vex} ($4x{+}3$)16exists only in the vocabulary and in the training distribution of the \emph{unrelated-skill donor}. Two17adjacent operator pairs are held out from every training distribution (compositional stratum); programs of18length 6--7 form a length-OOD stratum. Every split is generated from its own seed, validated by exact19recomputation, hashed, and checked for item-level disjointness against all training, alignment and20evaluation splits involved in a run; split hashes are stored in run manifests. Design decision D2 (trace21instead of a single answer token) was taken after a single-answer variant memorized without generalizing;22its consequence---that the Tier-0 capability is a per-step map once the trace is visible---is controlled23by the \url{trace_stride} knob (stride 2 forces two operators per emitted result).2425\paragraph{Models and roles.} Decoder-only transformers (4 layers, $d{=}128$, 4 heads, $0.80$\,M parameters;26pre-LN, explicit attention with per-head outputs exposed) are trained from scratch with teacher forcing on27trace tokens (3\,000 steps, batch 256, AdamW, cosine schedule; 40\,000 training items). Roles: \emph{donor}28(all eight operators), \emph{recipient} ($\Omega$ excluded), \emph{unrelated-skill donor} ($\Omega$ excluded,29\texttt{vex} included), three seeds each; a width-mismatched recipient ($d{=}96$), a depth-mismatched30recipient (6 layers), and a stride-2 donor/recipient pair. The initialization seed equals the data seed, so31donor$_i$ and recipient$_i$ share their initial weights (``same initialization'') while off-diagonal pairs do32not (D8). All models reach dev accuracy $1.000$ (Table~\ref{tab:models}).3334\paragraph{Hook semantics.} Hook points expose the embedding, per-layer residual streams (pre, mid, post),35per-head attention outputs before the output projection and MLP outputs. With no registered intervention,36or when disabled, a hook returns the input tensor object itself; bitwise equality of logits is asserted in37the test suite and recorded in every run manifest.3839\paragraph{Localization.} Interchange interventions on 490 clean/corrupted pairs (length-3 programs with40$\Omega$ at index 1; the corrupted twin replaces $\Omega$ by \texttt{zib}; identical teacher-forced prefix up41to $r_1$) patch the clean activation into the corrupted run at every (layer, position) for the residual,42attention and MLP families and per head; zero-ablation measures necessity; the sham baseline is the43corrupted run scored on the clean answer. Sweeps are repeated and compared across donor seeds.4445\paragraph{Alignment maps.} Donor and recipient residual rows at every read position are collected on463\,000 \emph{alignment} items that contain $\Omega$ (``task rows'') and on 3\,000 items that do not47(``task-free rows''), both disjoint from evaluation items. Maps: identity (same width only), orthogonal48Procrustes with isotropic scale, ridge regression ($\lambda = 10^{-2}$ relative), CCA regression, random49orthogonal (sham), each fitted on task rows or on task-free rows. We report the number of parameters,50in-sample $R^2$ and $R^2$ on the other row type.5152\paragraph{Transplant families and controls.} Grafts act at the recipient's post-layer-0 residual stream53(\texttt{resid\_post.0}, the location identified in \S\ref{sec:results}) at every read position during greedy54decoding (``untargeted''); an oracle variant restricted to $\Omega$ steps is also reported. Families:55activation graft (donor in the loop, same token ids), subspace graft ($k \in \{1,\dots,128\}$ principal56directions of the mapped $\Omega$-contrast), steering graft (mapped mean-difference direction,57$\alpha \in \{0.5, 1, 2, 4\}$), adapter graft (donor-free affine interface fitted by ridge to predict mapped58donor rows from recipient rows), weight grafts (layer-0 attention, MLP, full block, $\Omega$ embedding,59layers 0--2 plus embedding), and a persistent graft (the adapter saved to disk and evaluated in a separate60process on new items). Controls: unmodified recipient and donor; sham map; norm-matched random rows;61shuffled-item donor rows; donor layer-3 rows (shuffled layer); unrelated-skill donor; random subspace of62equal $k$; shuffled-layer and unrelated-donor weight grafts. Conventional-learning references on the same633\,000 alignment items: a supervised affine adapter trained with labels (same parameter count as the adapter64graft) and full-model distillation from the donor's output distributions (300 steps each). Few-shot65prompting and retrieval are not applicable to the toy models and are marked unsupported.6667\paragraph{Sweeps and mechanism.} Donor-layer $\times$ recipient-layer grid; dose $\alpha \in [0, 2]$;68width and depth mismatches; stride-2; donor ablation (layer-0 attention output zeroed at read positions)69followed by rescue from an intact donor of another seed, with a sham rescue; reverse graft70(recipient $\to$ donor); rule-versus-cache test (maps and adapters fitted only on $\Omega$ inputs in71$\{0,\dots,5\}$, evaluated separately on items whose $\Omega$ inputs are all seen or all unseen).7273\subsection{Tier 1: open-weight models}74\paragraph{Capability.} Performing an input$\to$output task \emph{without demonstrations}. Four tasks with75exact-match validators: antonyms (semantic), country$\to$capital (factual), an invented suffix rule on76pseudo-words (morphology, no prior), and ``add three'' on two-digit numbers (formal). Prompts use a fixed77\texttt{Input:}/\texttt{Output:} frame; few-shot prompts carry $k{=}8$ demonstrations drawn from a78demonstration pool disjoint from the queries; development and confirmatory query sets are disjoint halves of79each item pool.8081\paragraph{Models.} Qwen2.5-1.5B-Instruct (28 layers, $d{=}1536$), Qwen2.5-3B-Instruct (36 layers,82$d{=}2048$) and Llama-3.2-3B-Instruct (28 layers, $d{=}3072$), bf16, served by MLX with decoder layers wrapped83in place (bitwise no-op verified; per-head outputs are not exposed by MLX and head-level interventions are84reported as unsupported for this backend). Pairs: same family/different size in both directions, and85different family/different tokenizer in both directions.8687\paragraph{Candidate organ and localization.} The task vector is the mean difference between the last-token88residual state under few-shot and under zero-shot prompts (demonstration items only; queries never enter a89vector). Within-donor localization adds the vector to the donor's zero-shot run at each of nine layers90(sufficiency) and removes its direction from the few-shot run at the best layer (necessity); a random vector91of equal norm is the control.9293\paragraph{Transfer and controls.} Maps are fitted between donor and recipient last-token states on 80094\emph{format-matched, task-free} prompts (same frame, random content; last 100 rows held out for $R^2$).95Conditions at the recipient's own best layer and at the proportional-depth layer: mapped vector added at96$\alpha \in \{1, 2\}$ (Procrustes, ridge, CCA), norm-rescaled variants, and a donor-in-the-loop replace graft97(the recipient's last-token state replaced by the mapped donor few-shot state for the same query). Controls:98recipient zero-shot, recipient few-shot (prompting baseline), recipient's own vector (within-model ceiling),99norm-matched random vector, unrelated-task vector, sham map, mapped donor \emph{zero-shot} state, and mapped100donor few-shot state of a \emph{shuffled} query. Collateral: next-token negative log-likelihood on held-out101task-free sentences and accuracy on the other three tasks under the transferred vector. The intervention is a102single-position modification of the prompt pass; generation continues from the modified key--value entry103(within-sequence persistence); nothing is saved.104105\subsection{Statistics and preregistration}106Cluster bootstrap (2\,000 draws) over programs (Tier 0) or items (Tier 1); paired contrasts on identical107items; control-adjusted effects against the sham map; Holm correction across the pre-declared families in the108confirmatory analysis. Exploratory runs (protocol v1 and v2) selected the primary method; the preregistration109was then frozen (v1.0) and confirmatory splits generated from new seeds (\S\ref{sec:results}).110