spb/wpai0002_uqo

1# Gate C — Leakage-safe splits, reproducible baselines, initial benchmark and statistical plan23**Date:** 2026-11-09 · **Decision: GO** · Evidence: `data/splits/pilot_pmc/splits_summary.json`, `docs/PREREGISTRATION.md`4(v1.0 frozen at commit `e74b7126`, amendment A1), `artifacts/metrics/pilot_val_exploratory/report.json`, `src/htr/evaluation/*`,5`tests/unit/test_splits_and_dedup.py`, `tests/unit/test_eval_audits.py`.67## Splits8- Unit = document; leakage groups = union of near-duplicate cluster (MinHash Jaccard ≥ 0.8) and normalised first-author9  group; deterministic hash of the group id → train 80 / val 10 / test 10; whole disciplines and periods carved out.10- Pilot (1 180 PMC docs): train 856 · val 98 · test 100 · heldout_domain 47 (chemistry, pre-fix label) · heldout_period 7911  (pub_year < 1950) · 0 group leaks (checked by `check_no_group_leak`).12- Assignments of rewritten documents are **frozen**: `make_splits.py --fixed` carries them into every later corpus build.1314## Baselines (implemented, run on the validation split — exploratory, pre-freeze scoring of val is allowed)15| System | chrF++ | edit sim | ROUGE-L | number recall | placeholders exact | W1 to originals |16|---|---|---|---|---|---|---|17| B0 identity (copy the rewrite) | 71.24 [69.69, 72.65] | 0.666 | 0.690 | 1.000 | 1.000 | 0.170 |18| B1 generic paraphrase (rule-based) | 68.42 [66.90, 69.99] | 0.561 | 0.612 | 1.000 | 1.000 | 0.281 |19| M1-tiny from scratch (27.4 M, 3 900 pairs, best val CE 3.69) | 32.86 [30.87, 34.47] | 0.287 | 0.241 | 0.571 | 0.633 | 1.699 |20| B2 T5-small fine-tune | running on M3U96b | | | | | |2122(n = 502 validation pairs; 95 % document-clustered bootstrap CIs, 2 000 resamples.) **Reading:** on this pilot the identity23baseline is strong (chrF 71) and the from-scratch tiny model is far below it with degenerate repetitions — the expected24failure of a 27 M-parameter sequence model trained from random initialisation on ≈ 1.7 M target tokens. This is a25genuine pilot finding, not a bug: the copy mechanism alone does not make identity easy to learn in 2 250 steps.2627## Benchmark and statistical plan28- Primary metrics, hypotheses H1–H6, SESOIs, Holm families, document-clustered paired bootstrap (10 000 resamples for29  confirmatory runs), M\* selection on validation only, single confirmatory scoring: `docs/PREREGISTRATION.md` v1.030  (pre-specified in-session; SESOIs await author confirmation) + A1 (common stylometric scale).31- Benchmark artefacts: pairs (placeholder form) + metadata + splits + validation records; release subset policy in32  `docs/data_governance/DATA_CARD.md`.33- Evaluation code: lexical/factual metrics, stylometric W1 and exploratory probe, Holm, identifiability and memorisation34  audits — all unit-tested.3536## Consequences for Gate D371. Add a **denoising pretraining stage** on unlabelled original passages (zero API cost; `scripts/pretrain_denoise.py`,38   `--init-from`), then fine-tune on pairs — decision D17.392. Add n-gram repetition blocking at decoding (`no_repeat_ngram=4`) — affects all systems equally.403. Treat B2 (pretrained T5) as the strongest comparator; the Medium-stage pair set (≥ 20 000 passages) is required for a41   fair from-scratch comparison and needs budget approval.42