spb/wpai0002_uqo

1# Confirmatory scoring plan (pre-registration v1.0 + A1; executed once)23## M\* selection (frozen rule: lowest validation token cross-entropy among same-budget candidates)4Candidates = HTR-Tiny models fine-tuned from the Medium denoiser (`denoise_m1_tiny_medium_c`, 20 000 steps) on the Medium5Haiku pairs with the same step budget (20 000 max, patience 5), 32 k tokenizer:67| run | arch / objective | seed | best val CE |8|---|---|---|---|9| `medium_m1_tiny_mediumdenoiseinit` | M1 | 1 | 1.4714 |10| `medium_m1_tiny_mediumdenoiseinit_s2` | M1 | 2 | pending |11| `medium_m1_tiny_mediumdenoiseinit_s3` | M1 | 3 | **1.4703** |12| `medium_m2_tiny_mediumdenoiseinit` | M2 | 1 | 1.4861 |13| `medium_m3tokenonly_tiny_mediumdenoiseinit` (+ s2) | M3 token-only | 1, 2 | 1.4940, 1.4845 |14| `medium_m3light_tiny_mediumdenoiseinit` | M3 light objective | 1 | 1.4872 |1516M\* = the row with the lowest val CE once seed 2 is in (currently M1 seed 3). Its checkpoint sha256 is recorded in the17confirmatory report. Validation chrF (exploratory) was NOT used for the choice.1819## Sets and systems scored once20| Set | Pairs file | Split | Systems |21|---|---|---|---|22| Primary test (H1–H4) | `medium_v1_haiku` | `test` (≈ 15 600 pairs) | B0, B1, B2 (Medium), M\* (+ all other candidates as exploratory, labelled) |23| Held-out discipline (RQ3) | `medium_v1_haiku` | `heldout_domain` (physics, earth sciences) | same |24| Held-out period (RQ3) | `medium_v1_haiku` | `heldout_period` (pub_year < 1900) | same |25| Held-out rewriter: Opus 5.5 (H5) | `pilot_opus` | `test` | B0, B1, M\* |26| Held-out models: Sonnet/Haiku 5.5 (H5) | `pilot_heldout_models` | `test` | idem |27| Held-out prompts H1/H2 (H5) | `pilot_heldout_prompts` | `test` | idem |2829Command (one run per set): `scripts/evaluate.py --pairs <pairs> --outputs-dir artifacts/predictions/<set> --name conf_<set> --n-boot 10000 --confirmatory`.30Decision rules: §2 of `docs/PREREGISTRATION.md` (Holm over {H1-vs-B0, H1-vs-B1, H2-vs-B0, H2-vs-B1, H3, H4}; H5 family across the31four held-out sets; SESOIs +2.0 chrF++, +0.02 edit-sim, −0.01 number-recall margin, ≥ 10 % W1 reduction).32Expected outcome given validation: H1/H2 "statistically detectable but below SESOI" or null; H3 likely fails (−0.9 pp);33H4 likely supported (W1 0.10 vs 0.16). All will be reported as measured.34