SPB Git

spb/wp31_uqo

Public
2.1 KB · 33 lines latex
1\section{Adaptive intervention policy}\label{sec:policy}23The deployed router is a \emph{value controller} that scores every nonterminal action from target-free features4and never executes a candidate edit. At state $s=(z_t,S,r,\rho)$ it computes one shared state representation5$u = \mathrm{MLP}(W_z z_t + \mathrm{ctx}(r/R_{\max},\rho/R_{\max}))$ and, for each action $a$,6\begin{equation}7\bar s_a=\frac{1}{k(a)}\sum_{j\in I(a)} S_j,\qquad8\big(\hat g(s,a),\hat c(s,a),\log\hat\sigma(s,a),\hat\ell_{\mathrm{harm}}(s,a)\big)9 = W_{\mathrm{out}}\,\mathrm{SiLU}\big(u + W_s\bar s_a + \kappa_{k(a)}\big)+b,10\label{eq:controller}11\end{equation}12so that scoring all $|\Act|$ actions costs one MLP plus $|\Act|$ projections of width $w$13(Section~\ref{sec:compute}). The four heads estimate the \emph{intervention gain} $g(s,a)$ of14Section~\ref{sec:learning}, a cost proxy, a predictive scale, and the probability that the gain is negative. Actions15are chosen greedily,16\begin{equation}17Q(s,a)=\hat g(s,a)-\lambda\,c(a)-\kappa\,\hat\sigma(s,a),\qquad Q(s,\Stop)=0,\qquad18a^\star=\arg\max_{a\in\Act\cup\{\Stop\}} Q(s,a),19\label{eq:Q}20\end{equation}21where $c(a)$ is the \emph{known} analytic cost of the action (ops of one edit plus one controller evaluation,22relative to the backbone's ops per token; Section~\ref{sec:compute}); the learned $\hat c$ is kept only as a23diagnostic regressor. The state is recomputed after every executed edit, and the loop stops when \Stop{} wins or24the cap $R_{\max}$ is reached. The penalties $(\lambda,\kappa)$ are tuned on calibration rollouts only25(Section~\ref{sec:design}); sweeping $\lambda$ traces the controller's quality--cost curve.2627\paragraph{Known limitations of the rule.}28The policy is one-step greedy: it cannot represent an action whose value materialises only after a later action.29We quantify this myopia directly with a two-action lookahead oracle (Section~\ref{sec:results-oracle}) rather than30assume it away. The rule is also only as good as the calibration of $\hat g$; Proposition~\ref{prop:regret} bounds31the one-step loss of the greedy rule in terms of uniform estimation errors but says nothing about multi-step32optimality.33
Files · main
Menu

Theme