Confirmatory scoring plan (pre-registration v1.0 + A1; executed once)
M* selection (frozen rule: lowest validation token cross-entropy among same-budget candidates)
Candidates = HTR-Tiny models fine-tuned from the Medium denoiser (denoise_m1_tiny_medium_c, 20 000 steps) on the Medium
Haiku pairs with the same step budget (20 000 max, patience 5), 32 k tokenizer:
| run | arch / objective | seed | best val CE |
|---|---|---|---|
medium_m1_tiny_mediumdenoiseinit |
M1 | 1 | 1.4714 |
medium_m1_tiny_mediumdenoiseinit_s2 |
M1 | 2 | pending |
medium_m1_tiny_mediumdenoiseinit_s3 |
M1 | 3 | 1.4703 |
medium_m2_tiny_mediumdenoiseinit |
M2 | 1 | 1.4861 |
medium_m3tokenonly_tiny_mediumdenoiseinit (+ s2) |
M3 token-only | 1, 2 | 1.4940, 1.4845 |
medium_m3light_tiny_mediumdenoiseinit |
M3 light objective | 1 | 1.4872 |
M* = the row with the lowest val CE once seed 2 is in (currently M1 seed 3). Its checkpoint sha256 is recorded in the confirmatory report. Validation chrF (exploratory) was NOT used for the choice.
Sets and systems scored once
| Set | Pairs file | Split | Systems |
|---|---|---|---|
| Primary test (H1–H4) | medium_v1_haiku |
test (≈ 15 600 pairs) |
B0, B1, B2 (Medium), M* (+ all other candidates as exploratory, labelled) |
| Held-out discipline (RQ3) | medium_v1_haiku |
heldout_domain (physics, earth sciences) |
same |
| Held-out period (RQ3) | medium_v1_haiku |
heldout_period (pub_year < 1900) |
same |
| Held-out rewriter: Opus 5.5 (H5) | pilot_opus |
test |
B0, B1, M* |
| Held-out models: Sonnet/Haiku 5.5 (H5) | pilot_heldout_models |
test |
idem |
| Held-out prompts H1/H2 (H5) | pilot_heldout_prompts |
test |
idem |
Command (one run per set): scripts/evaluate.py --pairs <pairs> --outputs-dir artifacts/predictions/<set> --name conf_<set> --n-boot 10000 --confirmatory.
Decision rules: §2 of docs/PREREGISTRATION.md (Holm over {H1-vs-B0, H1-vs-B1, H2-vs-B0, H2-vs-B1, H3, H4}; H5 family across the
four held-out sets; SESOIs +2.0 chrF++, +0.02 edit-sim, −0.01 number-recall margin, ≥ 10 % W1 reduction).
Expected outcome given validation: H1/H2 "statistically detectable but below SESOI" or null; H3 likely fails (−0.9 pp);
H4 likely supported (W1 0.10 vs 0.16). All will be reported as measured.