SPB Git

spb/wp31_uqo

Public
3.5 KB · 44 lines latex
1\section{Experimental design}\label{sec:design}23\subsection{Data}4\paragraph{Synthetic compositional suite.} Six generator families with programmatic ground truth5(Table~\ref{tab:families}): multi-step arithmetic (\textsf{arith}), variable binding with dependent substitution6and a distractor assignment (\textsf{bind}), register updates (\textsf{fsm}), conjunctive constraints with7distractor items (\textsf{excl}), rule-based selection under a budget and an exclusion (\textsf{sched}), and a8held-out composition of a binding chain feeding an arithmetic expression (\textsf{compose}). Each family has five9phrasing templates; training uses templates 0--2 at depths 1--4 and a dedicated seed stream. Evaluation regimes:10\emph{iid} (training templates and depths, new seed), \emph{template} (held-out paraphrases 3--4),11\emph{depth} (depths 5--6), and \emph{compose} (held-out family). Splits are fixed \emph{before} sampling by12(family, template, depth, seed); items whose full text duplicates an item of an earlier split are dropped and13counted (995 train, 256 dev-iid, 284 test-iid duplicates; \texttt{data/splits/synthetic\_v1/manifest.json}).14A separate \emph{calibration} stream (seed 3) is used only for controller calibration and penalty selection.15Test streams (seed 9) are locked: the evaluation driver refuses to read them until16\texttt{protocol/locked\_test\_manifest.json} records the unsealing.17\input{tables/families.tex}1819\paragraph{General language corpus.} For the 103M experiments we stream the first documents of20\textsc{FineWeb-Edu} (sample-10BT, ODC-By~1.0; \citealp{penedo2024fineweb}) in canonical shard order, deduplicate21exactly, assign documents to train/val/test by content hash (98/1/1\,\%), and train a byte-level BPE tokenizer22(32,768 symbols, digits split individually) on the train split only. The corpus holds 290.0M train, 2.91M23validation and 2.88M locked test tokens (\texttt{data/processed/fineweb\_edu\_300m/manifest.json}).2425\subsection{Systems and controls}26All systems in a comparison share the backbone checkpoint, tokenizer, data stream, step count and batch size.27Table~\ref{tab:controls} lists every system with its unique parameter count, training exposure and inference28regime. The parameter-matched plain decoder (feed-forward width 2144 instead of 2048, $+2.65$M parameters) is29trained from scratch with the backbone recipe. The dense-slot editor writes all $M$ slots per action with the same30weights. The global-recurrence adapter (Section~\ref{sec:related}) refines the whole top representation with one31shared block at width $w$, parameter-matched to the \prisme{} workspace, trained with the identical forced-budget32recipe and the identical symmetry-breaking device. Reference routers (random, uniform round-robin, fixed pair,33entropy threshold, norm-change threshold) reuse the trained \prisme{} editor and are therefore matched in34everything except the choice of action. The exhaustive one-step and two-step oracles read the target and are35reported as ceilings only.36\input{tables/controls.tex}3738\subsection{Pre-registration, seeds and uncertainty}39The primary outcome, metrics, decision thresholds and seed plan were written to40\texttt{protocol/preregistration.md} before any test split was read (Appendix~\ref{app:prereg}). Differences41between systems on the same items use a paired bootstrap over items (10,000 resamples); differences across42training seeds use the per-seed statistic with a $t$-interval. Tokens of one training run are never treated as43independent replications of training. Latency is measured on the stated device with warm-up and synchronisation.44
Files · main
Menu

Theme