SPB Git

spb/wp31_uqo

Public
4.3 KB · 54 lines latex
1\section{Related work and novelty positioning}\label{sec:related}23We re-verified every reference at execution time from primary sources (arXiv metadata and full texts, official4repositories and licences; Appendix~\ref{app:literature} and \texttt{literature/novelty\_audit.md}). Four strands5are directly relevant.67\paragraph{Adaptive depth and latent recurrence.}8Universal Transformers \citep{dehghani2019universal} and ACT \citep{graves2016act} introduced weight-shared9recurrence over depth with learned per-position halting; PonderNet \citep{banino2021pondernet} made halting10probabilistic. Mixture-of-Depths routes tokens around blocks with a fixed capacity \citep{raposo2024mixtureofdepths},11LayerSkip trains early exits \citep{elhoushi2024layerskip}, and CALM calibrates confidence-based exits12\citep{schuster2022calm}. The recurrent-depth line scaled latent recurrence to language model pretraining:13Huginn \citep{geiping2025recurrentdepth}, Ouro \citep{zhu2025ouro}, Mixture-of-Recursions \citep{bae2025mixtureofrecursions},14looped Transformers \citep{giannou2023looped,saunshi2025reasoning}, and in 2026 RecurTrace15\citep{wang2026recurtrace}, Elastic Latent Recurrence \citep{ladhak2026elastic}, Think-at-Hard16\citep{fu2026thinkathard,you2026tah2} and several contemporaries listed in Appendix~\ref{app:literature}. All of17these refine the \emph{entire} hidden state of a position (globally), and route depth per token or per sequence.1819\paragraph{Slots, workspaces and sparse writes.}20Recurrent Independent Mechanisms \citep{goyal2019rims} and the Shared Global Workspace21\citep{goyal2022workspace} update a small number of competing modules per step; Slot Attention22\citep{locatello2020slot} and Perceiver / Perceiver AR \citep{jaegle2021perceiver,hawthorne2022perceiverar} maintain23a small bank of latents that cross-attend to a long input; Sparse Access Memory \citep{rae2016sparse} writes24sparsely into an external memory. Most recently ReFlux \citep{ye2026reflux} writes router-selected sparse25deep-to-shallow deltas into a frozen backbone, with the backbone unchanged when no route is active. \prisme{}'s26workspace differs in being \emph{ephemeral} (reset per token), separate from a read-only cached bank, and edited by27discrete 1--2-slot actions rather than by competition among modules.2829\paragraph{Loss-gain supervision of routing.}30The supervision signal we use---the realised change in next-token loss of an executed extra branch---has become31standard for \emph{binary} continue/stop decisions in 2025--2026: Ouro's second stage fits its exit gate to observed32per-token loss improvements \citep{zhu2025ouro}; TaH2 calls the same construction ``lookahead depth supervision''33\citep{you2026tah2}; RecurTrace trains a halting head on an oracle ``more depth still helps'' label34\citep{wang2026recurtrace}. Earlier, fitted value iteration and counterfactual risk minimisation established the35use of logged propensities for off-policy action evaluation \citep{ernst2005tree,dudik2011doublyrobust,swaminathan2015crm}.36\prisme{} makes the label \emph{action-conditional} over 24 spatial actions, logs propensities, and separates gain,37cost, uncertainty and harm heads.3839\paragraph{Zero-initialised identity insertion.}40Adding a zero-initialised branch so that a new module starts as the identity is a well-established device41\citep{bachlechner2020rezero,alayrac2022flamingo,zhang2023controlnet}; exact bypass of unrouted tokens exists in42Mixture-of-Depths and in TaH/TaH2, and ReFlux leaves the frozen backbone unchanged with no active route. The43differential readout of Section~\ref{sec:readout} is a modest variant of this device and is \emph{not} claimed as a44contribution.4546\paragraph{What is and is not new.}47After this audit, none of the three ingredients of \prisme{} is new on its own. What has no direct predecessor is48their combination: a per-token slot workspace with selected local writes, trained with action-conditional49counterfactual gains over a 25-action catalogue with logged propensities, behind an exact zero-budget bypass.50Accordingly, the contribution of this paper is empirical and methodological---a controlled test of whether spatial51selection supervised by counterfactual gains buys anything over uniform, dense or global alternatives at matched52cost---rather than architectural priority. Table~\ref{tab:novelty} in Appendix~\ref{app:literature} gives the53claim-by-claim comparison.54
Files · main
Menu

Theme