\section{Learning from interventions}\label{sec:learning}

\subsection{Paired intervention-value target}
For a training state $s$ (a position $t$ with workspace $S$ after $r$ executed actions), an action $a$ and the known
next token $y=x_{t+1}$, let $D(\cdot)$ denote the readout-plus-head map of \eqref{eq:readout} and $T_a$ the
editor's transition. The intervention gain is
\begin{equation}
g(s,a;y)=\Loss\big(D(s),y\big)-\Loss\big(D(T_a(s)),y\big),
\label{eq:gain}
\end{equation}
obtained by cloning $s$, executing $a$ with the same checkpoint, and evaluating both branches on the same $y$.
A positive gain means the edit reduced the teacher-forced loss on this example. This is an intervention on the
model's own computation, not a causal effect in the world, and a token-level gain does not imply that a complete
generated answer improves (Section~\ref{sec:results-reasoning}). The label $y$ enters \eqref{eq:gain} only; the
runtime features of \eqref{eq:controller} never contain it (verified by a unit test that perturbs $x_{t+1}$ and
checks that all earlier predictions are unchanged).

\subsection{Logging policy and propensities}
On sampled training states we evaluate four candidates: the current controller's preferred action (or a random
single if no controller exists yet), a competitive alternative from the other group (a pair if the preferred action
is a single and vice versa), and two uniform draws (one single, one pair). The logging mixture's propensity of
each candidate is recorded, enabling inverse-propensity or doubly-robust corrections of selection bias
\citep{dudik2011doublyrobust,swaminathan2015crm}. On an audit subset every one of the 24 actions is executed,
giving exact one-step oracle values. Helpful and harmful gains are both kept; sibling branches of one state never
cross a partition boundary; every label file records the SHA-256 of the editor checkpoint that produced it, and the
controller trainer refuses labels from a different checkpoint.

\subsection{Staged optimisation}\label{sec:stages}
\begin{description}[leftmargin=1.5em]
\item[Stage A (backbone).] A common backbone checkpoint is trained (or, for the synthetic experiments, trained from
  scratch on the synthetic train split). Every comparison starts from the same weights, tokenizer and data.
\item[Stage B (editor bootstrap).] The backbone is frozen; the editor, bank projections, slot embeddings and
  readout are trained with the language loss under \emph{forced} action sequences: each position draws a budget
  $b\in\{0,1,2,4,8\}$ and $b$ uniformly random actions (singles and pairs balanced), later steps being \Stop.
  Because $U=0$ blocks all gradient into the editor at initialisation, the bootstrap requires a symmetry-breaking
  device; we study the zero initialisation, a non-zero initialisation and the scale $s$ of \eqref{eq:readout}
  explicitly (Section~\ref{sec:results-learnability}).
\item[Stage C (counterfactual collection).] With the editor fixed, states are sampled by executing $r\sim U\{0,\dots,3\}$
  random actions, and labels \eqref{eq:gain} are collected as above.
\item[Stage D (controller fit).] The controller minimises
  \begin{equation}
  \Loss_{\mathrm{ctrl}}=\alpha\,\mathrm{Huber}_{\delta}\big(\hat g,\sg(g)\big)
   +\beta\,\mathrm{BCE}\big(\hat\ell_{\mathrm{harm}},\mathbf 1[g<0]\big)
   +\gamma\Big(\tfrac{(g-\sg(\hat g))^2}{2\hat\sigma^2}+\log\hat\sigma\Big),
  \label{eq:ctrl-loss}
  \end{equation}
  on the training partition; sign accuracy, Brier score, expected calibration error and oracle regret are measured
  on the calibration partition, where $(\lambda,\kappa)$ are also selected by actual rollouts.
\item[Stage E (lock).] The final checkpoint (editor + controller + selected penalties) is frozen before any locked
  test split is read. Alternating editor/controller refinement is treated as a later ablation, not a dependency.
\end{description}
No teacher model and no distillation are used anywhere in this paper; the composite loss of the original proposal
reduces to the language loss in Stage B and \eqref{eq:ctrl-loss} in Stage D.

\subsection{Delayed benefits}
An action may have negative immediate gain yet enable a later beneficial action. We therefore report, on audit
subsets, the exhaustive one-step oracle, the exhaustive two-action oracle ($|\Act|(|\Act|+1)$ branches per state)
and the ``greedy first, then best'' value, and decompose the gap between the learned policy and the two-step
oracle into an \emph{opportunity} term (two-step minus one-step oracle) and a \emph{prediction} term (one-step
oracle minus learned policy).