spb/wpai0002_uqo

1# Gate E — Medium-scale training, confirmatory evaluation, negative findings23**Date:** 2026-11-10 · **Decision: GO to Gate F** (release of HTR-Tiny 0.3.0 as the documented M\*; larger models in training on the4H100 are reported as they land) · Evidence: `artifacts/metrics/conf_*/report.json` (single confirmatory pass, 10 000 document-clustered5resamples, Holm), `artifacts/metrics/conf_*/h4_w1_*.json`, `artifacts/metrics/medium_val_exploratory/`, `artifacts/metrics/pangram_*/`,6`docs/CONFIRMATORY_PLAN.md`, `docs/PREREGISTRATION.md` v1.0 (+A1), run manifests under `artifacts/runs/`.78## Setting9- Pairs: 103 934 validated (Haiku 5.5 rewrites of 20 000 passages of `medium_v1`; D20); train 72 698 / val 10 520 / test 15 649 /10  held-out domain 2 493 / held-out period 2 574. Held-out rewriters from the pilot corpus (PMC, test split): Opus 5.5 (812), Sonnet 5.5 +11  Haiku 5.5 (531), held-out prompts H1/H2 with Opus (266).12- M\* = `medium_m1_tiny_mediumdenoiseinit_s3` (HTR-Tiny M1, 33.5 M params; denoising pretraining 20 000 steps on 258 264 train passages,13  then 20 000 fine-tuning steps; lowest validation CE 1.470 among same-budget candidates — rule frozen before scoring).14- Confirmatory scoring was run **once** (`scripts/run_confirmatory.sh`, 2026-11-10 ~12:00 UTC) after M\* was fixed.1516## Pre-registered hypotheses (primary test split, n = 15 649 pairs, 1 261 documents)17| H | Metric | Result (M\* − baseline, 95 % CI, Holm p) | SESOI | Verdict |18|---|---|---|---|---|19| H1 vs B0 identity | chrF++ | **+0.44 [+0.30, +0.58]**, p < 0.0001 | +2.0 | statistically detectable, **practically small** |20| H1 vs B1 paraphrase | chrF++ | +3.13 (B1 is −2.69 [−2.79, −2.59] vs B0) | +2.0 | **supported** |21| H2 vs B0 | edit similarity | +0.0024 [+0.0011, +0.0036], p < 0.0001 | +0.02 | detectable, practically small |22| H2 vs B1 | edit similarity | +0.115 (B1 −0.113) | +0.02 | **supported** |23| H3 non-inferiority | number recall (M\* − B0) | −0.0087 [−0.0104, −0.0072] | margin −0.01 | **not established** (lower bound −0.0104 < −0.01) |24| H4 style distribution | W1 reduction vs rewrite | **32.5 % [29.6 %, 37.6 %]** (0.154 → 0.104) | ≥ 10 % | **supported** |2526Absolute scores (chrF++): B0 68.91 · B1 66.22 · B2 T5-small (Medium) 68.83 · **M\* 69.35 [68.81, 69.88]** · other HTR variants 69.00–69.31.27Placeholders (citations/equations/links) exact: M\* 0.992 vs B2 0.733. Number recall M\* 0.991.2829## H5 — held-out generalisation (chrF++ M\* − B0; SESOI +1.0; one Holm family)30| Set | n | Δ chrF++ [95 % CI] | Holm p | W1 reduction (H4-style) | Verdict |31|---|---|---|---|---|---|32| Held-out disciplines (physics, earth sciences) | 2 493 | −0.10 [−0.36, +0.14] | 0.44 | 36.7 % [28.3, 39.5] | null on lexical gain; style gain holds |33| Held-out period (< 1900, books) | 2 574 | −0.53 [−0.75, −0.28] | < 0.001 | 32.0 % [23.9, 35.6] | **worse** lexically; style gain holds |34| Held-out rewriter Opus 5.5 (PMC) | 812 | **+0.72 [+0.21, +1.11]** | 0.016 | 33.8 % [22.1, 37.5] | detectable, below SESOI |35| Held-out rewriters Sonnet/Haiku (PMC pilot) | 531 | −0.85 [−1.33, −0.48] | < 0.001 | (0.163 → 0.108) | **worse** |36| Held-out prompts H1/H2 (Opus, PMC pilot) | 266 | −2.38 [−2.93, −1.92] | < 0.001 | (0.190 → 0.168) | **worse** |37H5 is **not supported**: the lexical gain transfers to the Opus-rewritten PMC passages but not to the other held-out sets; the stylometric38distribution gain is present in every set.3940## Exploratory41- Architecture/size: on the test split all Medium HTR variants are within 0.35 chrF of each other (M2 metadata conditioning, M342  token-only, M3-light, Small 130 M); seeds 1–3 of M1 agree (69.31 / 69.30 / 69.35). The dual-latent objective brings no gain;43  the Small model is under-trained for its size (same step budget as Tiny).44- Detector sensitivity (RQ7, Pangram 4.1, n = 54 test + 85 Opus passages): originals 0 % flagged; Haiku rewrites 46 % "Mixed", Opus45  rewrites 33 %; **every HTR output labelled Human** (0 %); B2 7 %. Reported as detector behaviour, not authorship (dual-use risk noted).46- RQ6: no memorisation signal (max Jaccard to training originals ≈ 0, no same-passage leaks); reconstructions of two independent47  rewrites agree at chrF 80 (rewrites 83).4849## Reading50HTR learns a *small, consistent, style-directed* correction: it moves rewrites measurably towards the stylometric distribution of51pre-2018 originals (−32 % distance, in- and out-of-distribution, and removes every detector flag in the sample) while changing the52lexical surface only slightly (+0.4 chrF++, i.e. the identity baseline remains a near-ceiling for verbatim reconstruction — the53non-identifiability predicted in the plan). The cost is a 0.9-point drop in number recall and 0.8 pp in placeholder exactness, flagged54per output by the warnings. These are the paper's headline results; H1/H2 vs identity are reported as "detectable but below SESOI".5556## Next57Gate F: release HTR-Tiny 0.3.0 with these numbers in the model card, finish the paper, and add the H100 runs (Base 335 M, Large 1.07 B)58and HTR-Edit (non-generative) as exploratory additions when they complete.59