Gate E — Medium-scale training, confirmatory evaluation, negative findings
Date: 2026-11-10 · Decision: GO to Gate F (release of HTR-Tiny 0.3.0 as the documented M*; larger models in training on the
H100 are reported as they land) · Evidence: artifacts/metrics/conf_*/report.json (single confirmatory pass, 10 000 document-clustered
resamples, Holm), artifacts/metrics/conf_*/h4_w1_*.json, artifacts/metrics/medium_val_exploratory/, artifacts/metrics/pangram_*/,
docs/CONFIRMATORY_PLAN.md, docs/PREREGISTRATION.md v1.0 (+A1), run manifests under artifacts/runs/.
Setting
- Pairs: 103 934 validated (Haiku 5.5 rewrites of 20 000 passages of
medium_v1; D20); train 72 698 / val 10 520 / test 15 649 / held-out domain 2 493 / held-out period 2 574. Held-out rewriters from the pilot corpus (PMC, test split): Opus 5.5 (812), Sonnet 5.5 + Haiku 5.5 (531), held-out prompts H1/H2 with Opus (266). - M* =
medium_m1_tiny_mediumdenoiseinit_s3(HTR-Tiny M1, 33.5 M params; denoising pretraining 20 000 steps on 258 264 train passages, then 20 000 fine-tuning steps; lowest validation CE 1.470 among same-budget candidates — rule frozen before scoring). - Confirmatory scoring was run once (
scripts/run_confirmatory.sh, 2026-11-10 ~12:00 UTC) after M* was fixed.
Pre-registered hypotheses (primary test split, n = 15 649 pairs, 1 261 documents)
| H | Metric | Result (M* − baseline, 95 % CI, Holm p) | SESOI | Verdict |
|---|---|---|---|---|
| H1 vs B0 identity | chrF++ | +0.44 [+0.30, +0.58], p < 0.0001 | +2.0 | statistically detectable, practically small |
| H1 vs B1 paraphrase | chrF++ | +3.13 (B1 is −2.69 [−2.79, −2.59] vs B0) | +2.0 | supported |
| H2 vs B0 | edit similarity | +0.0024 [+0.0011, +0.0036], p < 0.0001 | +0.02 | detectable, practically small |
| H2 vs B1 | edit similarity | +0.115 (B1 −0.113) | +0.02 | supported |
| H3 non-inferiority | number recall (M* − B0) | −0.0087 [−0.0104, −0.0072] | margin −0.01 | not established (lower bound −0.0104 < −0.01) |
| H4 style distribution | W1 reduction vs rewrite | 32.5 % [29.6 %, 37.6 %] (0.154 → 0.104) | ≥ 10 % | supported |
Absolute scores (chrF++): B0 68.91 · B1 66.22 · B2 T5-small (Medium) 68.83 · M* 69.35 [68.81, 69.88] · other HTR variants 69.00–69.31. Placeholders (citations/equations/links) exact: M* 0.992 vs B2 0.733. Number recall M* 0.991.
H5 — held-out generalisation (chrF++ M* − B0; SESOI +1.0; one Holm family)
| Set | n | Δ chrF++ [95 % CI] | Holm p | W1 reduction (H4-style) | Verdict |
|---|---|---|---|---|---|
| Held-out disciplines (physics, earth sciences) | 2 493 | −0.10 [−0.36, +0.14] | 0.44 | 36.7 % [28.3, 39.5] | null on lexical gain; style gain holds |
| Held-out period (< 1900, books) | 2 574 | −0.53 [−0.75, −0.28] | < 0.001 | 32.0 % [23.9, 35.6] | worse lexically; style gain holds |
| Held-out rewriter Opus 5.5 (PMC) | 812 | +0.72 [+0.21, +1.11] | 0.016 | 33.8 % [22.1, 37.5] | detectable, below SESOI |
| Held-out rewriters Sonnet/Haiku (PMC pilot) | 531 | −0.85 [−1.33, −0.48] | < 0.001 | (0.163 → 0.108) | worse |
| Held-out prompts H1/H2 (Opus, PMC pilot) | 266 | −2.38 [−2.93, −1.92] | < 0.001 | (0.190 → 0.168) | worse |
| H5 is not supported: the lexical gain transfers to the Opus-rewritten PMC passages but not to the other held-out sets; the stylometric | |||||
| distribution gain is present in every set. |
Exploratory
- Architecture/size: on the test split all Medium HTR variants are within 0.35 chrF of each other (M2 metadata conditioning, M3 token-only, M3-light, Small 130 M); seeds 1–3 of M1 agree (69.31 / 69.30 / 69.35). The dual-latent objective brings no gain; the Small model is under-trained for its size (same step budget as Tiny).
- Detector sensitivity (RQ7, Pangram 4.1, n = 54 test + 85 Opus passages): originals 0 % flagged; Haiku rewrites 46 % "Mixed", Opus rewrites 33 %; every HTR output labelled Human (0 %); B2 7 %. Reported as detector behaviour, not authorship (dual-use risk noted).
- RQ6: no memorisation signal (max Jaccard to training originals ≈ 0, no same-passage leaks); reconstructions of two independent rewrites agree at chrF 80 (rewrites 83).
Reading
HTR learns a small, consistent, style-directed correction: it moves rewrites measurably towards the stylometric distribution of pre-2018 originals (−32 % distance, in- and out-of-distribution, and removes every detector flag in the sample) while changing the lexical surface only slightly (+0.4 chrF++, i.e. the identity baseline remains a near-ceiling for verbatim reconstruction — the non-identifiability predicted in the plan). The cost is a 0.9-point drop in number recall and 0.8 pp in placeholder exactness, flagged per output by the warnings. These are the paper's headline results; H1/H2 vs identity are reported as "detectable but below SESOI".
Next
Gate F: release HTR-Tiny 0.3.0 with these numbers in the model card, finish the paper, and add the H100 runs (Base 335 M, Large 1.07 B) and HTR-Edit (non-generative) as exploratory additions when they complete.