# Pre-registration — WP31 PRISME pilot (DRAFT v0.1, 2026-10-29; to be FROZEN before any test_* split is read)

Status: DRAFT. No file named test*.jsonl / test.bin has been opened by any evaluation script so far (protocol/locked_test_manifest.json records the split hashes and the unsealing timestamp once frozen).

# 1. Primary benchmark suite

• Synthetic compositional suite v1 (data/splits/synthetic_v1, generator data/generators/synthetic.py, manifest sha256 in manifest.json): families arith, bind, fsm, excl, sched (train/dev) + compose (held out). Task unit = one item (prompt + answer). Exclusion: items whose full text duplicates an earlier split (dropped at generation time; counts in manifest). Regimes: *_iid (train templates & depths, new seed), *_template (held-out paraphrase templates 3–4), *_depth (depths 5–6 > train max 4), *_compose (held-out family).
• General language: FineWeb-Edu held-out test.bin (2.88M tokens, locked; val.bin is the development set).
• Independent benchmarks (GSM8K, EvalPlus) are OUT of the primary scope of the 103M pilot (a ~200M-token backbone cannot solve them above chance); they are reported only if a run is executed, as secondary.

# 2. Primary metrics

• Quality (reasoning): exact-match accuracy of free greedy generation on the synthetic test regimes (pooled test_iid+test_template+test_depth, each regime also reported). Quality (language): held-out cross-entropy on test.bin at each budget.
• Cost: (primary) measured median end-to-end decode latency per token at batch 1 on the reference device, with the backbone-only latency as the unit; (secondary) analytic ops/token (prisme/flops.py).
• Joint superiority: a configuration dominates if quality is higher at ≤ the comparator's latency, or latency is lower at non-inferior quality (paired CI excludes zero).

# 3. Baselines and tuning budgets

Same backbone checkpoint, same data exposure, same mixed-budget training recipe and step count: (a) frozen backbone (budget 0); (b) parameter-matched plain decoder trained from scratch with the backbone recipe; (c) dense 16-slot editor; (d) PRISME editor with random / uniform / fixed-pair routing at budgets 1, 2, 4, 8; (e) entropy and norm-change routers (thresholds tuned on calibration data); (f) global-recurrence adapter (Huginn/ELR-inspired, parameter-matched); (g) learned value controller (λ, κ tuned on calibration rollouts only); (h) exhaustive one-step oracle and two-step oracle (diagnostic only, never plotted as attainable). Hyperparameter budget per adapter: one learning rate (1e-3 for adapters, chosen in Phase 4) and the shared schedule; no per-baseline tuning beyond the router thresholds.

# 4. Minimum meaningful effect

• Reasoning: +3 percentage points pooled exact match at comparable latency (≤ +10 % of the comparator's), or −20 % median latency at non-inferior accuracy (lower CI bound > −1 pp).
• Language: zero-budget cross-entropy within 1 % (relative) of the frozen backbone (H5 preservation).
• Routing (H2/H3): learned controller loss at matched mean extra-ops within the CI of the best fixed/random schedule is a NULL for H3.

# 5. Seeds

Editor/controller stage: 5 independent seeds (1–5) at the Phase 4 scale; at 103M as many as the cluster permits within the session (target 3–5, actual count reported). The backbone is a single seed (declared limitation).

# 6. Uncertainty

Paired bootstrap over task items (10,000 resamples) for accuracy differences within a seed; across seeds, mean ± SD and a t-interval over seed-level differences. Tokens of a single run are never treated as independent replications of training.

# 7. Stopping / resource rules

Pilot compute cap: ≤ 2 nodes × 8 h for Stage A, ≤ 6 nodes × 6 h for stages B–D; if a stage cannot complete, the phase is reported as incomplete and downstream claims are withheld.

# 8. Deviations

Any change after freezing is recorded in protocol/deviations.md with justification and timestamp.