1# Pre-registration — WP31 PRISME pilot (v1.0, FROZEN 2026-10-30T04:04-0400 before any `test_*` split was read)23Status: FROZEN. No file named `test*.jsonl` / `test.bin` had been opened by any evaluation script at freeze time4(`protocol/locked_test_manifest.json` records the split hashes and the unsealing timestamp).56## 0. Primary comparisons as executed (fixed at freeze, before unsealing)7* P1 (H1/H2, synthetic): exact match on the pooled test regimes at forced budget 2 for the sparse editor vs the frozen8 backbone, vs the global-recurrence adapter and vs the dense editor, 5 adapter seeds each (v2 backbone), paired by item.9* P2 (H3, synthetic): dev-iid loss of the locked controller vs the best fixed schedule and vs budget 0 (already reported10 on dev; the test regimes are evaluated at budget 0 and 2 with forced random actions only, since the controller was11 worse than budget 0 on dev for every seed — the controller is therefore evaluated on test for completeness, not as the12 primary system).13* P3 (H1/H4, language): FineWeb-Edu locked test cross-entropy at budgets 0/1/2/4/8 for the 103M backbone, the sparse14 editor, the global adapter and the parameter-matched plain decoder.15* Decision thresholds as in Section 4 below; no other analysis is pre-registered.1617## 1. Primary benchmark suite18* **Synthetic compositional suite v1** (`data/splits/synthetic_v1`, generator `data/generators/synthetic.py`,19 manifest sha256 in `manifest.json`): families arith, bind, fsm, excl, sched (train/dev) + compose (held out).20 Task unit = one item (prompt + answer). Exclusion: items whose full text duplicates an earlier split (dropped at21 generation time; counts in manifest). Regimes: `*_iid` (train templates & depths, new seed), `*_template`22 (held-out paraphrase templates 3–4), `*_depth` (depths 5–6 > train max 4), `*_compose` (held-out family).23* **General language**: FineWeb-Edu held-out `test.bin` (2.88M tokens, locked; `val.bin` is the development set).24* Independent benchmarks (GSM8K, EvalPlus) are OUT of the primary scope of the 103M pilot (a ~200M-token25 backbone cannot solve them above chance); they are reported only if a run is executed, as secondary.2627## 2. Primary metrics28* Quality (reasoning): exact-match accuracy of free greedy generation on the synthetic test regimes29 (pooled `test_iid+test_template+test_depth`, each regime also reported). Quality (language): held-out30 cross-entropy on `test.bin` at each budget.31* Cost: (primary) measured median end-to-end decode latency per token at batch 1 on the reference device, with32 the backbone-only latency as the unit; (secondary) analytic ops/token (`prisme/flops.py`).33* Joint superiority: a configuration dominates if quality is higher at ≤ the comparator's latency, or latency is34 lower at non-inferior quality (paired CI excludes zero).3536## 3. Baselines and tuning budgets37Same backbone checkpoint, same data exposure, same mixed-budget training recipe and step count:38(a) frozen backbone (budget 0); (b) parameter-matched plain decoder trained from scratch with the backbone recipe;39(c) dense 16-slot editor; (d) PRISME editor with random / uniform / fixed-pair routing at budgets 1, 2, 4, 8;40(e) entropy and norm-change routers (thresholds tuned on calibration data); (f) global-recurrence adapter41(Huginn/ELR-inspired, parameter-matched); (g) learned value controller (λ, κ tuned on calibration rollouts only);42(h) exhaustive one-step oracle and two-step oracle (diagnostic only, never plotted as attainable).43Hyperparameter budget per adapter: one learning rate (1e-3 for adapters, chosen in Phase 4) and the shared44schedule; no per-baseline tuning beyond the router thresholds.4546## 4. Minimum meaningful effect47* Reasoning: +3 percentage points pooled exact match at comparable latency (≤ +10 % of the comparator's), or48 −20 % median latency at non-inferior accuracy (lower CI bound > −1 pp).49* Language: zero-budget cross-entropy within 1 % (relative) of the frozen backbone (H5 preservation).50* Routing (H2/H3): learned controller loss at matched mean extra-ops within the CI of the best fixed/random51 schedule is a NULL for H3.5253## 5. Seeds54Editor/controller stage: 5 independent seeds (1–5) at the Phase 4 scale; at 103M as many as the cluster55permits within the session (target 3–5, actual count reported). The backbone is a single seed (declared limitation).5657## 6. Uncertainty58Paired bootstrap over task items (10,000 resamples) for accuracy differences within a seed; across seeds,59mean ± SD and a t-interval over seed-level differences. Tokens of a single run are never treated as60independent replications of training.6162## 7. Stopping / resource rules63Pilot compute cap: ≤ 2 nodes × 8 h for Stage A, ≤ 6 nodes × 6 h for stages B–D; if a stage cannot complete,64the phase is reported as incomplete and downstream claims are withheld.6566## 8. Deviations67Any change after freezing is recorded in `protocol/deviations.md` with justification and timestamp.68