1\section{Learning from interventions}\label{sec:learning}23\subsection{Paired intervention-value target}4For a training state $s$ (a position $t$ with workspace $S$ after $r$ executed actions), an action $a$ and the known5next token $y=x_{t+1}$, let $D(\cdot)$ denote the readout-plus-head map of \eqref{eq:readout} and $T_a$ the6editor's transition. The intervention gain is7\begin{equation}8g(s,a;y)=\Loss\big(D(s),y\big)-\Loss\big(D(T_a(s)),y\big),9\label{eq:gain}10\end{equation}11obtained by cloning $s$, executing $a$ with the same checkpoint, and evaluating both branches on the same $y$.12A positive gain means the edit reduced the teacher-forced loss on this example. This is an intervention on the13model's own computation, not a causal effect in the world, and a token-level gain does not imply that a complete14generated answer improves (Section~\ref{sec:results-reasoning}). The label $y$ enters \eqref{eq:gain} only; the15runtime features of \eqref{eq:controller} never contain it (verified by a unit test that perturbs $x_{t+1}$ and16checks that all earlier predictions are unchanged).1718\subsection{Logging policy and propensities}19On sampled training states we evaluate four candidates: the current controller's preferred action (or a random20single if no controller exists yet), a competitive alternative from the other group (a pair if the preferred action21is a single and vice versa), and two uniform draws (one single, one pair). The logging mixture's propensity of22each candidate is recorded, enabling inverse-propensity or doubly-robust corrections of selection bias23\citep{dudik2011doublyrobust,swaminathan2015crm}. On an audit subset every one of the 24 actions is executed,24giving exact one-step oracle values. Helpful and harmful gains are both kept; sibling branches of one state never25cross a partition boundary; every label file records the SHA-256 of the editor checkpoint that produced it, and the26controller trainer refuses labels from a different checkpoint.2728\subsection{Staged optimisation}\label{sec:stages}29\begin{description}[leftmargin=1.5em]30\item[Stage A (backbone).] A common backbone checkpoint is trained (or, for the synthetic experiments, trained from31 scratch on the synthetic train split). Every comparison starts from the same weights, tokenizer and data.32\item[Stage B (editor bootstrap).] The backbone is frozen; the editor, bank projections, slot embeddings and33 readout are trained with the language loss under \emph{forced} action sequences: each position draws a budget34 $b\in\{0,1,2,4,8\}$ and $b$ uniformly random actions (singles and pairs balanced), later steps being \Stop.35 Because $U=0$ blocks all gradient into the editor at initialisation, the bootstrap requires a symmetry-breaking36 device; we study the zero initialisation, a non-zero initialisation and the scale $s$ of \eqref{eq:readout}37 explicitly (Section~\ref{sec:results-learnability}).38\item[Stage C (counterfactual collection).] With the editor fixed, states are sampled by executing $r\sim U\{0,\dots,3\}$39 random actions, and labels \eqref{eq:gain} are collected as above.40\item[Stage D (controller fit).] The controller minimises41 \begin{equation}42 \Loss_{\mathrm{ctrl}}=\alpha\,\mathrm{Huber}_{\delta}\big(\hat g,\sg(g)\big)43 +\beta\,\mathrm{BCE}\big(\hat\ell_{\mathrm{harm}},\mathbf 1[g<0]\big)44 +\gamma\Big(\tfrac{(g-\sg(\hat g))^2}{2\hat\sigma^2}+\log\hat\sigma\Big),45 \label{eq:ctrl-loss}46 \end{equation}47 on the training partition; sign accuracy, Brier score, expected calibration error and oracle regret are measured48 on the calibration partition, where $(\lambda,\kappa)$ are also selected by actual rollouts.49\item[Stage E (lock).] The final checkpoint (editor + controller + selected penalties) is frozen before any locked50 test split is read. Alternating editor/controller refinement is treated as a later ablation, not a dependency.51\end{description}52No teacher model and no distillation are used anywhere in this paper; the composite loss of the original proposal53reduces to the language loss in Stage B and \eqref{eq:ctrl-loss} in Stage D.5455\subsection{Delayed benefits}56An action may have negative immediate gain yet enable a later beneficial action. We therefore report, on audit57subsets, the exhaustive one-step oracle, the exhaustive two-action oracle ($|\Act|(|\Act|+1)$ branches per state)58and the ``greedy first, then best'' value, and decompose the gap between the learned policy and the two-step59oracle into an \emph{opportunity} term (two-step minus one-step oracle) and a \emph{prediction} term (one-step60oracle minus learned policy).61