spb/wpai0002_uqo

1% ============================================================================2% HTR 2.0 — exploratory tracks after Gate E (decision D22). Sources: docs/STATUS.md (HTR 2.0 table),3% docs/DECISION_LOG.md D22, docs/INCIDENTS.md I3-I4, artifacts/metrics/medium_test_htr2_exploratory/report.json4% (test, 2 000 document-clustered resamples), artifacts/metrics/htr2_split_deltas/deltas.json (val, held-out5% domain, held-out period; scripts/htr2_split_deltas.py), artifacts/metrics/decode_search_tiny_val/decode_search.json6% (T1), artifacts/runs/mrt_tiny_mstar_* (T2), artifacts/runs/disc_roberta_large (T4), docs/architecture/HTR_EDIT.md7% and artifacts/metrics/medium_val_exploratory (HTR-Edit), artifacts/metrics/pangram_test_* (detector),8% artifacts/metrics/audit_{qwen15b,flant5large}_val (memorisation). Figures/table: scripts/build_htr2_figures.py.9% ============================================================================10\section{Beyond the Vanilla Transformer: Pretrained Priors and Direct Metric Optimisation}11\label{sec:htr2}1213The confirmatory model beats identity by $+0.44$ chrF++, far below the pre-registered threshold of $+2.0$. After14Gate~E, and at the author's request for a mechanism that scores materially higher, we pre-declared four15\emph{exploratory} tracks (decision D22) and ran them on rented NVIDIA H100 instances and on the cluster:16\textbf{T1}, better decoding of the existing model (sampling, minimum Bayes risk, source-anchored reranking);17\textbf{T2}, direct optimisation of the evaluation metric by minimum-risk training; \textbf{T3}, pretrained18backbones fine-tuned on the same pairs and splits, the inverse-paraphrase recipe of \citet{krishna2020reformulating};19and \textbf{T4}, a learned original-versus-rewrite discriminator as a reranking or reward signal. We also completed20the scaling ladder of our own architecture (HTR-Small~v2, HTR-Base, HTR-Large) and trained a non-generative21editing model (HTR-Edit). Everything in this section is exploratory and post hoc: every choice was made on the22validation split, each system was applied once to the primary test split and to the two Medium held-out sets, and23the confirmatory results of Section~\ref{sec:results} are unchanged. The pre-registered SESOI of $+2.0$ is used24only as a yardstick, not as a confirmatory criterion. Table~\ref{tab:htr2-overview} and25Figure~\ref{fig:htr2-delta} summarise all systems; the complete test report, with every seed and variant, is in26Appendix~\ref{app:htr2-table}.\footnote{Test differences and intervals: \evid{artifacts/metrics/medium_test_htr2_exploratory/report.json}27(2\,000 document-clustered resamples, same pairs and baseline as the confirmatory pass). Validation, held-out-domain28and held-out-period differences: \evid{artifacts/metrics/htr2_split_deltas/deltas.json}; their ``$\pm$'' is $1.96$29times the plain standard error of the paired difference.}3031\begin{table}[tbp]32\centering33\singlespacing34\caption{HTR 2.0 systems (exploratory, post hoc). $\Delta$ is the paired chrF++ difference against the identity35baseline B0 on the validation (first 3\,000 pairs), primary test, held-out-discipline and held-out-period sets36($\pm$: $1.96 \times$ standard error; test: 95\% document-clustered bootstrap interval). Num.\ and Plac.: number37recall and placeholder exactness on test. Pangram: passages labelled \texttt{Human} by Pangram~4.1 out of the 54-pair38test sample (Section~\ref{sec:robustness}). Compute: accelerator hours of the runs behind the system (denoiser plus39fine-tuning for HTR models, a shared denoiser being counted once, at the first system that used it: the lr $2 \times 10^{-5}$40row re-uses the seed-2 denoiser, the HTR-Small and MPS-Base rows list their fine-tuning only; $\dagger$: wall-clock or41log time where no accelerator figure was recorded). Qwen42parameter counts are the nominal sizes from the model cards. The Large row ``stage-2 denoiser'' is the only Large43system whose pretraining did not diverge before fine-tuning (incident I4). The $M^\ast$ row repeats the44confirmatory numbers of Table~\ref{tab:conf-hyp}.}45\label{tab:htr2-overview}46\resizebox{\textwidth}{!}{\input{tables/tab_htr2_overview}}47\end{table}4849\begin{figure}[tbp]50\centering51\includegraphics[width=\textwidth]{figures/fig_htr2_delta_test.pdf}52\caption{Paired chrF++ difference against identity (B0) on the primary test split ($n = 15\,649$ pairs, 65253documents) for every reported system; bars are 95\% document-clustered bootstrap intervals (2\,000 resamples). The54dashed line is identity, the dotted line the pre-registered SESOI, used here only as a yardstick. Blue: this55paper's Transformers and the baselines; orange: pretrained backbones fine-tuned on the same pairs. Exploratory56except $M^\ast$.}57\label{fig:htr2-delta}58\end{figure}5960\subsection{Scaling the in-house architecture}6162\paragraph{HTR-Base.} With H100 compute, the 334.8\,M-parameter Base model was pretrained as a denoiser for 30\,00063steps (validation cross-entropy 1.225, against 1.78 for the Tiny denoiser) and fine-tuned on the pairs. It64over-fits the 72\,698 pairs quickly: at learning rate $10^{-4}$ the best validation cross-entropy (1.266) is reached65after 1\,000 steps and rises afterwards; at $2 \times 10^{-5}$ it reaches 1.259 and then rises as well. The best66checkpoint gains $+0.69$ chrF++ [$+0.56$, $+0.83$] over identity on test, against $+0.44$ for Tiny, with number67recall 0.992 and placeholder exactness 0.993; a second seed replicates it ($+0.63$ [$+0.49$, $+0.76$]) and the68low-learning-rate fine-tune gives $+0.55$. On validation the gain is $+0.83 \pm 0.13$. Unlike Tiny, Base69transfers a small gain to the held-out disciplines ($+0.29 \pm 0.13$; seed~2: $+0.32$) and is at identity on the70pre-1900 books ($-0.12 \pm 0.10$). A Base model pretrained on the cluster's Apple GPUs with small batches71(24\,000 steps, denoiser cross-entropy 1.959) is under-trained and scores \emph{below} identity ($-0.26$ on test),72which shows that the pretraining budget, not the parameter count, carries the Base gain.7374\paragraph{HTR-Large.} The 1.07\,B-parameter design point is unstable under this recipe (incident I4). Three75pretraining stages on one H100 (bf16 autocast, 8\,192-token batches) diverged at peak learning rates76$2 \times 10^{-4}$ and $8 \times 10^{-5}$, and a third stage with the copy mixture and log-softmax in fp32 reached a77best of 1.512 before drifting up again (1.68 at 14\,000 steps). The only Large system fine-tuned from a non-diverged denoiser78(stage~2, cross-entropy 1.628) scores $-0.71$ [$-0.84$, $-0.58$] on test, $-1.15$ on the held-out disciplines and79$-1.62$ on the pre-1900 books; a final fine-tune from the stage-3 checkpoint at learning rate $1.5 \times 10^{-5}$80is stable but still negative ($-0.79$ on test). We report Large as a negative scaling data point, not as evidence81about capacity: the likely causes (no QK-normalisation, bf16 attention logits at 24 heads, AdamW $\epsilon$ at82$d_{\mathrm{model}} = 1\,536$) were not resolved within the budget.8384\paragraph{HTR-Small v2.} Resuming the Small denoiser to 22\,500 steps (cross-entropy 1.849) and fine-tuning for8513\,000 steps (1.520) does not help: $-0.13 \pm 0.14$ on validation and $-0.09$ [$-0.22$, $+0.04$] on test. The86scaling picture of the in-house architecture is therefore monotone only once pretraining is adequate: Tiny87$+0.44 \to$ Base $+0.69$ on test, with Small and Large below identity for budget and stability reasons88respectively.8990\subsection{T1: decoding}9192On 1\,000 validation pairs we drew $K = 8$ nucleus samples (temperature 0.7, top-$p$ 0.9) from $M^\ast$ in addition93to its greedy output and compared decoding rules against identity.\footnote{\evid{scripts/decode_search.py},94\evid{artifacts/metrics/decode_search_tiny_val/decode_search.json}.} Greedy decoding gains $+0.54 \pm 0.22$; a single95sample loses $-1.23$; minimum-Bayes-risk selection with a chrF++ utility over the nine candidates96\citep{kumar2004mbr,eikema2020mbr} gains $+0.66$, and adding a source-anchoring term to the utility (weight 0.5)97$+0.70 \pm 0.21$, the best rule. The oracle that picks, for each passage, the candidate closest to the original98gains $+1.74 \pm 0.22$. Sampling and reranking therefore add at most 0.2 points over greedy decoding, and even99perfect selection among nine candidates stays below the SESOI; cheaper MBR variants100\citep{cheng2023mbrpruning,vamvas2024linearmbr} would not change this ceiling. We kept greedy decoding for all101reported systems.102103\subsection{T2: minimum-risk training}104105Minimum-risk training \citep{shen2016mrt} replaces the token loss with the expected risk $1 - \text{chrF++}$ (plus a106factual penalty for lost numbers and placeholders) under a renormalised distribution over a candidate set made of107the gold original, the greedy output and eight samples, anchored by a cross-entropy term.\footnote{\evid{src/htr/training/mrt.py},108\evid{scripts/train_mrt.py}; runs \evid{artifacts/runs/mrt_tiny_mstar_a005} and \evid{mrt_tiny_mstar_a05_abandoned}.}109Starting from $M^\ast$ (validation chrF++ 68.96 on the 600-pair monitoring set), two runs of 750--800 steps were110made. With the sharpness $\alpha = 0.5$ the renormalised distribution was one-hot on the greedy candidate111(probability mass on the gold 0.017), so the risk term carried no useful gradient; with $\alpha = 0.005$ the mass112spread (gold 0.06) but validation chrF++ still fell, to 68.80--68.87 in both runs. The track is negative and was113abandoned; direct optimisation of the metric cannot recover information the rewrite does not contain.114115\subsection{T4: discriminator}116117A RoBERTa-large classifier \citep{liu2019roberta} trained on the training split to tell originals from rewrites118reaches a validation AUROC of 0.994 and accuracy of 0.976.\footnote{\evid{scripts/train_discriminator.py},119\evid{artifacts/runs/disc_roberta_large}.} The LLM trace is thus almost perfectly \emph{detectable} in-distribution120even where it is barely \emph{invertible}. The discriminator was built as a reranking signal and as the reward of121a KL-regularised policy-optimisation variant in the spirit of HUMPA \citep{wang2025humpa} and AuthorMist122\citep{david2025authormist}, with a source-fidelity term; that variant is implemented but was not run after T2123showed that risk-based objectives did not improve validation, and no reported system uses the discriminator. We124note that such objectives optimise detector evasion directly, which is the dual-use risk discussed in125Section~\ref{sec:ethics}, and that we would not report evasion as a success measure.126127\subsection{HTR-Edit: non-generative editing}128129HTR-Edit is a LaserTagger-style tagger (23.6\,M parameters, encoder initialised from the Tiny denoiser) that keeps or130deletes each source token and inserts phrases from a closed vocabulary of 4\,000 frequent insertions built on the131training split; it cannot produce a word sequence that is neither in its input nor in that list, and protected132spans are forced to be kept.\footnote{\evid{docs/architecture/HTR_EDIT.md}; \evid{src/htr/models/tagger.py};\newline133\evid{artifacts/runs/medium_edit_tiny}.} Its outputs preserve every number and placeholder by construction (test1341.000 and 0.999), and it edits the 15\,649 test passages in 19 seconds on one Apple GPU. Its reconstruction quality135is, however, negative: $-2.27$ [$-2.54$, $-2.03$] chrF++ against identity on validation and $-2.20$ [$-2.36$, $-2.05$]136on test, with a stylometric distance (0.141) that barely improves on the rewrite (0.154). The scheme itself is not137the limit: realising the gold edit labels gives 81.57 chrF++ on validation against 68.43 for identity, a perfect-edit138oracle of about $+13$ points, but the tagger reaches a tag accuracy of 0.83 and an insertion recall of only 0.08,139so it deletes more than it correctly inserts. The $+13$ oracle is nevertheless informative: the headroom lies in140\emph{knowing} the edit, not in choosing among samples (the best-of-nine oracle of T1 is $+1.7$).141142\subsection{T3: pretrained backbones}143\label{sec:htr2-backbones}144145The decisive track replaces the in-house denoiser by a pretrained English prior. Following the inverse-paraphrase146formulation of STRAP \citep{krishna2020reformulating}, we fine-tuned four public backbones on exactly the same147training pairs and splits, with the metadata prefix and the placeholder protocol of Section~\ref{sec:data}148(placeholders spelled \texttt{[[C1]]} for the pretrained tokenizers): the instruction-tuned encoder--decoders149flan-T5-large (783\,M) and flan-T5-xl (2.85\,B) \citep{chung2022flan}, and the decoder-only Qwen2.5-1.5B and150Qwen2.5-3B \citep{qwen2025qwen25} by supervised fine-tuning with the loss on the original only.\footnote{\evid{src/htr/training/hf_finetune.py},151\evid{scripts/train_llm_sft.py}; runs \evid{medium_b2_flan_t5_large}, \evid{medium_b2_flan_t5_xl},152\evid{medium_llm_qwen25_15b}, \evid{medium_llm_qwen25_15b_s2}, \evid{medium_llm_qwen25_3b}. Checkpoints were chosen153by validation cross-entropy: 0.753 (flan-T5-large, 5\,000 of 11\,000 steps), 0.688 (flan-T5-xl, 5\,000 of 9\,000),1540.733 (Qwen 1.5B, both seeds), 0.701 (Qwen 3B).}155156\paragraph{Lexical gain.} All four backbones beat identity by a wide margin on every set157(Table~\ref{tab:htr2-overview}). On test, flan-T5-large gains $+1.79$ [$+1.62$, $+1.96$] and Qwen2.5-1.5B $+1.97$158[$+1.82$, $+2.10$] (seed~2: $+1.97$); the two larger backbones, flan-T5-xl ($+2.39$ [$+2.20$, $+2.57$]) and Qwen2.5-3B159($+2.27$ [$+2.13$, $+2.40$]), have interval lower bounds above the $+2.0$ SESOI. Relative to the identity ceiling that160constrained every in-house model, the pretrained prior raises the gain four- to five-fold (Tiny $+0.44$, Base161$+0.69$). Because these systems were chosen and tuned after the freeze, we do not describe H1 as supported; we162describe the result as an exploratory finding that passes the pre-registered yardstick and awaits a confirmatory163replication with a new freeze.164165\paragraph{Transfer.} The in-house models' lexical gain did not survive outside the training distribution166(Section~\ref{sec:robustness}); the backbones' gain does. On the held-out disciplines (physics, earth sciences)167flan-T5-xl gains $+1.78 \pm 0.18$ and Qwen2.5-3B $+2.05 \pm 0.13$; on the pre-1900 books flan-T5-xl gains168$+2.12 \pm 0.15$ and Qwen2.5-3B $+1.91 \pm 0.14$ (Tiny: $-0.53$; Base: $-0.12$). A general English prior that has169seen nineteenth-century prose supplies what 72\,698 pairs of recent journal text cannot teach a model trained from170scratch.171172\paragraph{The two families split the objectives.} Figure~\ref{fig:htr2-gain-style} plots the lexical gain173against the stylometric distance to the originals. The Qwen models are the closest lexically and preserve facts174almost perfectly (number recall 0.997--0.998, placeholders 0.998--0.999, better than any in-house model), but move175the feature distribution less ($\overline{W}_1$ 0.123 for Qwen 1.5B and 0.100 for Qwen 3B, against 0.154 for the176rewrite and 0.104 for $M^\ast$). The flan-T5 models move the distribution furthest of all systems (0.083 for177flan-T5-large and 0.081 for flan-T5-xl, a 46--47\% reduction, against 32\% for $M^\ast$) and178are the only backbones that leave no Pangram flag (54/54 \texttt{Human}; Qwen 1.5B 48/54, Qwen 3B 49/54,179Section~\ref{sec:robustness}), at a slightly higher factual cost (number recall 0.986--0.987). No single system is180best on all three axes, and the public platform serves both families for that reason.181182\begin{figure}[tbp]183\centering184\includegraphics[width=0.85\textwidth]{figures/fig_htr2_gain_vs_style.pdf}185\caption{Lexical gain against stylometric distance on the primary test split. $x$: paired chrF++ difference against186identity; $y$: mean standardised Wasserstein distance $\overline{W}_1$ of the outputs to the pre-2018 originals over18718 features (common scale of amendment A1); the cross marks the LLM rewrite, which identity returns unchanged.188Points to the right are closer in wording, points lower are closer in distribution. Exploratory except $M^\ast$.}189\label{fig:htr2-gain-style}190\end{figure}191192\paragraph{Caveats.} Three qualifications apply. First, the backbones' pretraining corpora are not auditable here:193the memorisation audit on validation finds no overlap with the HTR training originals for either family (maximum194Jaccard 0.000, no same-passage leak),\footnote{\evid{artifacts/metrics/audit_qwen15b_val}, \evid{audit_flant5large_val}.}195but it covers the HTR pairs only, and pre-2018 PMC and Project Gutenberg text may well be in the corpora of196flan-T5 and Qwen2.5, so part of the transfer to old books may be recall of a general prior rather than inversion197(Section~\ref{sec:threats}). Second, the T5 SentencePiece vocabulary cannot encode several characters that occur in198scientific text (\texttt{<}, \texttt{\textasciitilde}, $\pm$, $\rightarrow$, $\leq$, Greek letters); they become199\texttt{<unk>} and are dropped from the output. The released inference code now protects them with placeholders200(\texttt{protect\_unk\_chars}), but the reported flan-T5 scores were produced before that fix and slightly201under-state those models. Third, the comparison across families is not compute-matched: the backbones bring202orders of magnitude more pretraining than the in-house denoisers, which is the point of the track, not a confound203to be removed.204205\subsection{Reading}206207Across the five tracks the hierarchy is clear. Scaling our own Transformer gives modest gains when pretraining is208adequate (Tiny $+0.44 \to$ Base $+0.69$) and fails when it is not (Small, Large). Better decoding adds $\le 0.2$209points; metric-driven training adds nothing; a near-perfect discriminator does not translate into a better210inverter; and a non-generative editor is safe but negative. The lever is the pretrained English prior: fine-tuned211backbones reach $+1.8$ to $+2.4$ chrF++ over identity on test and, unlike the in-house models, carry a comparable212gain to unseen disciplines and to pre-1900 books. The non-identifiability ceiling remains: the best-of-nine oracle213over samples is $+1.7$ while the perfect-edit oracle is $+13$, so the information that a reconstructor lacks is214not retrievable by search over its own outputs. The original wording is recoverable in part, from a prior over215English, and in distribution; it is not recoverable verbatim.216