Gate C — Leakage-safe splits, reproducible baselines, initial benchmark and statistical plan
Date: 2026-11-09 · Decision: GO · Evidence: data/splits/pilot_pmc/splits_summary.json, docs/PREREGISTRATION.md
(v1.0 frozen at commit e74b7126, amendment A1), artifacts/metrics/pilot_val_exploratory/report.json, src/htr/evaluation/*,
tests/unit/test_splits_and_dedup.py, tests/unit/test_eval_audits.py.
Splits
- Unit = document; leakage groups = union of near-duplicate cluster (MinHash Jaccard ≥ 0.8) and normalised first-author group; deterministic hash of the group id → train 80 / val 10 / test 10; whole disciplines and periods carved out.
- Pilot (1 180 PMC docs): train 856 · val 98 · test 100 · heldout_domain 47 (chemistry, pre-fix label) · heldout_period 79
(pub_year < 1950) · 0 group leaks (checked by
check_no_group_leak). - Assignments of rewritten documents are frozen:
make_splits.py --fixedcarries them into every later corpus build.
Baselines (implemented, run on the validation split — exploratory, pre-freeze scoring of val is allowed)
| System | chrF++ | edit sim | ROUGE-L | number recall | placeholders exact | W1 to originals |
|---|---|---|---|---|---|---|
| B0 identity (copy the rewrite) | 71.24 [69.69, 72.65] | 0.666 | 0.690 | 1.000 | 1.000 | 0.170 |
| B1 generic paraphrase (rule-based) | 68.42 [66.90, 69.99] | 0.561 | 0.612 | 1.000 | 1.000 | 0.281 |
| M1-tiny from scratch (27.4 M, 3 900 pairs, best val CE 3.69) | 32.86 [30.87, 34.47] | 0.287 | 0.241 | 0.571 | 0.633 | 1.699 |
| B2 T5-small fine-tune | running on M3U96b |
(n = 502 validation pairs; 95 % document-clustered bootstrap CIs, 2 000 resamples.) Reading: on this pilot the identity baseline is strong (chrF 71) and the from-scratch tiny model is far below it with degenerate repetitions — the expected failure of a 27 M-parameter sequence model trained from random initialisation on ≈ 1.7 M target tokens. This is a genuine pilot finding, not a bug: the copy mechanism alone does not make identity easy to learn in 2 250 steps.
Benchmark and statistical plan
- Primary metrics, hypotheses H1–H6, SESOIs, Holm families, document-clustered paired bootstrap (10 000 resamples for
confirmatory runs), M* selection on validation only, single confirmatory scoring:
docs/PREREGISTRATION.mdv1.0 (pre-specified in-session; SESOIs await author confirmation) + A1 (common stylometric scale). - Benchmark artefacts: pairs (placeholder form) + metadata + splits + validation records; release subset policy in
docs/data_governance/DATA_CARD.md. - Evaluation code: lexical/factual metrics, stylometric W1 and exploratory probe, Holm, identifiability and memorisation audits — all unit-tested.
Consequences for Gate D
- Add a denoising pretraining stage on unlabelled original passages (zero API cost;
scripts/pretrain_denoise.py,--init-from), then fine-tune on pairs — decision D17. - Add n-gram repetition blocking at decoding (
no_repeat_ngram=4) — affects all systems equally. - Treat B2 (pretrained T5) as the strongest comparator; the Medium-stage pair set (≥ 20 000 passages) is required for a fair from-scratch comparison and needs budget approval.