SPB Git

spb/wp31_uqo

Public
4.9 KB · 60 lines latex
1\subsection*{\texorpdfstring{Pre-registration --- WP31 PRISME pilot (DRAFT v0.1, 2026-10-29; to be FROZEN before any \texttt{test\_*} split is read)}{Pre-registration --- WP31 PRISME pilot (DRAFT v0.1, 2026-10-29; to be FROZEN before any test\_* split is read)}}\label{pre-registration-wp31-prisme-pilot-draft-v0.1-2026-10-29-to-be-frozen-before-any-test_-split-is-read}23Status: DRAFT. No file named \texttt{test*.jsonl} / \texttt{test.bin} has been opened by any evaluation script so far (\texttt{protocol/locked\_test\_manifest.json} records the split hashes and the unsealing timestamp once frozen).45\subsubsection*{1. Primary benchmark suite}\label{primary-benchmark-suite}67\begin{itemize}8\tightlist9\item10  \textbf{Synthetic compositional suite v1} (\texttt{data/splits/synthetic\_v1}, generator \texttt{data/generators/synthetic.py}, manifest sha256 in \texttt{manifest.json}): families arith, bind, fsm, excl, sched (train/dev) + compose (held out). Task unit = one item (prompt + answer). Exclusion: items whose full text duplicates an earlier split (dropped at generation time; counts in manifest). Regimes: \texttt{*\_iid} (train templates \& depths, new seed), \texttt{*\_template} (held-out paraphrase templates 3--4), \texttt{*\_depth} (depths 5--6 \textgreater{} train max 4), \texttt{*\_compose} (held-out family).11\item12  \textbf{General language}: FineWeb-Edu held-out \texttt{test.bin} (2.88M tokens, locked; \texttt{val.bin} is the development set).13\item14  Independent benchmarks (GSM8K, EvalPlus) are OUT of the primary scope of the 103M pilot (a \textasciitilde200M-token backbone cannot solve them above chance); they are reported only if a run is executed, as secondary.15\end{itemize}1617\subsubsection*{2. Primary metrics}\label{primary-metrics}1819\begin{itemize}20\tightlist21\item22  Quality (reasoning): exact-match accuracy of free greedy generation on the synthetic test regimes (pooled \texttt{test\_iid+test\_template+test\_depth}, each regime also reported). Quality (language): held-out cross-entropy on \texttt{test.bin} at each budget.23\item24  Cost: (primary) measured median end-to-end decode latency per token at batch 1 on the reference device, with the backbone-only latency as the unit; (secondary) analytic ops/token (\texttt{prisme/flops.py}).25\item26  Joint superiority: a configuration dominates if quality is higher at $\leq$ the comparator's latency, or latency is lower at non-inferior quality (paired CI excludes zero).27\end{itemize}2829\subsubsection*{3. Baselines and tuning budgets}\label{baselines-and-tuning-budgets}3031Same backbone checkpoint, same data exposure, same mixed-budget training recipe and step count: (a) frozen backbone (budget 0); (b) parameter-matched plain decoder trained from scratch with the backbone recipe; (c) dense 16-slot editor; (d) PRISME editor with random / uniform / fixed-pair routing at budgets 1, 2, 4, 8; (e) entropy and norm-change routers (thresholds tuned on calibration data); (f) global-recurrence adapter (Huginn/ELR-inspired, parameter-matched); (g) learned value controller ($\lambda$, $\kappa$ tuned on calibration rollouts only); (h) exhaustive one-step oracle and two-step oracle (diagnostic only, never plotted as attainable). Hyperparameter budget per adapter: one learning rate (1e-3 for adapters, chosen in Phase 4) and the shared schedule; no per-baseline tuning beyond the router thresholds.3233\subsubsection*{4. Minimum meaningful effect}\label{minimum-meaningful-effect}3435\begin{itemize}36\tightlist37\item38  Reasoning: +3 percentage points pooled exact match at comparable latency ($\leq$ +10 \% of the comparator's), or --20 \% median latency at non-inferior accuracy (lower CI bound \textgreater{} --1 pp).39\item40  Language: zero-budget cross-entropy within 1 \% (relative) of the frozen backbone (H5 preservation).41\item42  Routing (H2/H3): learned controller loss at matched mean extra-ops within the CI of the best fixed/random schedule is a NULL for H3.43\end{itemize}4445\subsubsection*{5. Seeds}\label{seeds}4647Editor/controller stage: 5 independent seeds (1--5) at the Phase 4 scale; at 103M as many as the cluster permits within the session (target 3--5, actual count reported). The backbone is a single seed (declared limitation).4849\subsubsection*{6. Uncertainty}\label{uncertainty}5051Paired bootstrap over task items (10,000 resamples) for accuracy differences within a seed; across seeds, mean $\pm$ SD and a t-interval over seed-level differences. Tokens of a single run are never treated as independent replications of training.5253\subsubsection*{7. Stopping / resource rules}\label{stopping-resource-rules}5455Pilot compute cap: $\leq$ 2 nodes $\times$ 8 h for Stage A, $\leq$ 6 nodes $\times$ 6 h for stages B--D; if a stage cannot complete, the phase is reported as incomplete and downstream claims are withheld.5657\subsubsection*{8. Deviations}\label{deviations}5859Any change after freezing is recorded in \texttt{protocol/deviations.md} with justification and timestamp.60
Files · d6746e03e0d6828cde427e384b9ef1d2b92c62d6
Menu

Theme