spb/wpai0002_uqo

1% ============================================================================2% Ablations and model comparisons (exploratory). Sources: docs/gates/GATE_D.md, docs/DECISION_LOG.md D17-D19,3% artifacts/metrics/pilot_val_exploratory/, artifacts/metrics/medium_val_exploratory/summary.md, docs/STATUS.md,4% artifacts/metrics/conf_test/report.json, artifacts/runs/*/train_summary.json.5% ============================================================================6\section{Ablations and Model Comparisons}7\label{sec:ablations}89Everything in this section is exploratory. Pilot-scale results use the 5\,383 validated Claude Opus~5.5 pilot pairs10(about 3\,900 for training, 502 validation pairs, 2\,000 bootstrap resamples); Medium-scale results use the Haiku11pairs of Section~\ref{sec:results}, on validation (3\,000 pairs, 2\,000 resamples) and on test. The test rows are12exploratory because only $M^\ast$ was designated before scoring.1314\subsection{Pilot scale: what made training work}1516\paragraph{From-scratch training fails.} An HTR-Tiny M1 model (27.4\,M parameters with the 16\,000-token pilot17tokenizer) trained from random initialisation on the pilot pairs reached a best validation cross-entropy of 3.6918(seeds 1--3: 3.69, 4.09, 4.31) and scored 32.9 chrF++ [30.9, 34.5] on validation, against 71.2 for identity, with19degenerate repetitions (Table~\ref{tab:pilot-val}). About 1.7\,M target tokens are not enough to learn the inverse20map from random weights. A one-batch overfit test (cross-entropy 5.28 to 0.002 in 150 steps) rules out a broken21training loop.2223\paragraph{Denoising pretraining recovers most of the gap (D17).} Pretraining the same architecture as a denoising24autoencoder on unlabelled training-split originals (12\,000 steps on about 17\,000 passages; no API cost) and then25fine-tuning on the pairs raised validation chrF++ to 63.6 [62.2, 64.9] (seed~2: 64.1 [62.5, 65.5]), with number26recall 0.931 and a stylometric distance to the originals of 0.150, below identity's 0.170. The denoiser alone,27without pair fine-tuning, scores 58.1. At pilot scale no system beats identity on lexical metrics: the28denoise-initialised model is $-7.6$ chrF++ [$-8.6$, $-6.7$] below B0, and the pretrained T5-small comparator B2 is29$-2.4$ [$-3.3$, $-1.6$] below it while having the lowest stylometric distance (0.112).3031\begin{table}[t]32\centering33\caption{Pilot-scale validation results (502 pairs, Opus~5.5 pilot rewrites; 95\% document-clustered bootstrap34intervals, 2\,000 resamples), against the pre-2018 originals. \texttt{denoise-only}: denoising pretraining without35pair fine-tuning; \texttt{m1-tiny}: from scratch. Exploratory.}36\label{tab:pilot-val}37\resizebox{\textwidth}{!}{\input{tables/tab_overall_pilotval}}38\end{table}3940\paragraph{Two loss-implementation faults (D18, D19).} Every model that included the factual-span loss41$\mathcal{L}_{\mathrm{fact}}$ as first implemented, normalised by the number of protected tokens, failed to42generalise (validation cross-entropy about 5.4, flat while training loss fell), including M1 with43$\mathcal{L}_{\mathrm{fact}}$ alone (5.41). The M3 architecture trained with the token loss only reached 3.80,44close to M1. The cause was the normalisation, not the architecture; $\mathcal{L}_{\mathrm{fact}}$ is now normalised45by all target tokens (D18). Single-term diagnostics then showed that the contrastive term alone also blocked46generalisation (5.21), whereas the semantic term alone did not (4.08). Masking same-passage items out of the47InfoNCE negatives (D19) was correct but changed nothing (full M3 5.31, no style 4.82, no adversary 5.30, no48contrastive 5.29, no fact 5.30), because only 1.3\% of off-diagonal pairs in length-bucketed batches shared a49passage.5051\paragraph{The full M3 objective does not work at pilot scale.} After both corrections, every M3 variant that52keeps the style/adversarial branch generalises worse than token-only training (about 5.3 against 3.6--3.8); the53best variant removes the style term (4.82). With the denoised backbone, an M3-light objective (all auxiliary terms54at small weights) and M3 token-only matched M1 (validation chrF++ 64.0 and 64.1 against 63.6 and 64.1 for the two M155seeds). The dual-latent objective therefore shows no benefit at pilot scale, and the auxiliary terms other than the56semantic one actively hurt when given their default weights.5758\paragraph{Seed variance.} Over three seeds, best validation cross-entropy ranged from 3.69 to 4.31 for M1 and from593.71 to 5.03 for M2 (M3 token-only: 3.58--3.80). Single-seed architecture comparisons at this scale are not60conclusive.6162\subsection{Medium scale: architecture, conditioning and size}6364Table~\ref{tab:medium-val} compares the Medium-scale candidates on validation, and the exploratory rows of65Table~\ref{tab:conf-abs} compare them on test. All HTR models were fine-tuned from the Medium denoiser (HTR-Tiny,6620\,000 steps) except where stated, with the same maximum step budget.6768\begin{table}[t]69\centering70\caption{Medium-scale validation results (3\,000 pairs; 95\% document-clustered bootstrap intervals, 2\,00071resamples), against the pre-2018 originals; paired chrF++ difference against B0. Validation chrF++ was not used for72the selection of $M^\ast$ (lowest validation cross-entropy, M1 seed~3). Exploratory.}73\label{tab:medium-val}74\small75\begin{tabular}{@{}lrrrrl@{}}76\toprule77System & chrF++ & Num.\ rec. & Placeh. & $\overline{W}_1$ & $\Delta$ chrF++ vs B0 [95\% CI] \\78\midrule79B0 identity & 68.42 & 1.000 & 1.000 & 0.164 & -- \\80B1 generic paraphrase & 65.67 & 1.000 & 1.000 & 0.226 & $-2.75$ \\81B2 T5-small (Medium pairs) & 68.38 & 0.989 & 0.713 & 0.117 & $-0.04$ [$-0.40$, $+0.30$] \\82HTR-Tiny M1, seed 1 & 68.75 & 0.991 & 0.992 & 0.100 & $+0.33$ [$+0.09$, $+0.58$] \\83HTR-Tiny M1, seed 2 & 68.81 & 0.992 & 0.994 & 0.097 & $+0.38$ [$+0.14$, $+0.63$] \\84HTR-Tiny M1, seed 3 ($M^\ast$) & 68.84 & 0.991 & 0.995 & 0.100 & $+0.42$ [$+0.20$, $+0.66$] \\85HTR-Tiny M2 (metadata) & 68.57 & 0.988 & 0.995 & 0.098 & $+0.15$ [$-0.08$, $+0.39$] \\86HTR-Tiny M3 token-only, seed 1 & 68.78 & 0.990 & 0.995 & 0.106 & $+0.35$ [$+0.13$, $+0.59$] \\87HTR-Tiny M3 token-only, seed 2 & 68.73 & 0.990 & 0.991 & 0.105 & $+0.31$ [$+0.09$, $+0.56$] \\88HTR-Tiny M3-light & 68.48 & 0.991 & 0.994 & 0.098 & $+0.06$ [$-0.18$, $+0.32$] \\89HTR-Small M1 (130.8\,M) & 68.62 & 0.990 & 0.996 & 0.095 & $+0.20$ [$-0.04$, $+0.45$] \\90HTR-Tiny M1, pilot denoiser & 67.87 & 0.985 & 0.988 & 0.126 & $-0.55$ \\91\bottomrule92\end{tabular}93\end{table}9495\paragraph{The denoising initialisation matters more than the objective.} The largest single difference in96Table~\ref{tab:medium-val} is between the two initialisations of the same M1 Tiny model: the Medium denoiser (32k97tokenizer, trained on the Medium training passages) gives 68.75 chrF++, the pilot denoiser (16k tokenizer, PMC98pilot passages) 67.87, below identity. In-domain pretraining on the broader corpus is the effective ingredient.99100\paragraph{Metadata conditioning (M2) brings no measurable gain.} On validation, M2 scores 68.57 ($+0.15$ vs B0,101interval including zero), below the three M1 seeds (68.75--68.84). On test it scores 69.17 against 69.30--69.35 for102the M1 seeds, with lower number recall (0.989 vs 0.991).103104\paragraph{The dual-latent objective brings no gain (H6 not testable as specified).} The M3 architecture trained105with the token loss alone performs like M1 (test 69.31 and 69.23 for two seeds against 69.31, 69.30 and 69.35 for106M1), which shows that the extra parameters are harmless. With the auxiliary objective switched on at small weights107(M3-light, one seed), performance drops to the bottom of the HTR range (validation $+0.06$ vs B0, test 69.00, test108number recall 0.989), and the stylometric distance is not better than M1's (test 0.100 vs 0.098--0.104). H6109requires three seeds of the full M3 objective at the M1 budget; because the full objective failed at pilot scale110and only one M3-light seed was trained at Medium scale, H6 was not run as a confirmatory test. On the evidence111available, a content/style decomposition adds nothing at this scale.112113\paragraph{HTR-Small is under-trained.} The 130.8\,M-parameter Small model was trained on the same step budget as114Tiny: its denoiser stopped at 15\,000 steps with validation cross-entropy 1.85 (Tiny denoiser: 1.78 after 20\,000115steps) and its fine-tuning at 15\,000 steps (validation cross-entropy 1.500 vs 1.470 for $M^\ast$). It has the lowest116stylometric distance of all systems (validation 0.095, test 0.093) but the lowest chrF++ of the M1 models (test11769.02, $+0.11$ [$-0.03$, $+0.25$] vs B0, not significant) and lower edit similarity than identity ($-0.0026$).118Its result is a budget statement, not a capacity curve; the Base (335\,M) and Large (1.07\,B) design points are in119training, and their results are \pending{HTR-Base and HTR-Large results}.120121\paragraph{Seeds at Medium scale.} The three M1 seeds agree closely on test (69.31, 69.30, 69.35 chrF++; paired122differences against B0 $+0.40$, $+0.39$, $+0.44$), so at Medium scale the seed variance that made pilot comparisons123inconclusive has largely disappeared; all Medium HTR variants lie within 0.35 chrF++ of each other on test.124125\paragraph{Training dynamics.} Figure~\ref{fig:training} shows training and validation token cross-entropy for126$M^\ast$, HTR-Small and M3 token-only. Validation cross-entropy flattens near 1.47--1.50 well before the step limit127for all three; early stopping ended seed~1 of M1 at 17\,000 steps and M3 token-only at 13\,000.128129\begin{figure}[t]130\centering131\includegraphics[width=0.75\textwidth]{figures/fig_training_conftest.pdf}132\caption{Token cross-entropy during pair fine-tuning at Medium scale (training: thin lines; validation: markers),133for $M^\ast$ (\texttt{medium\_m1\_tiny\_mediumdenoiseinit\_s3}), HTR-Small and M3 token-only. From the run logs in134\evid{artifacts/runs/}.}135\label{fig:training}136\end{figure}137138\paragraph{Non-generative editing.} The HTR-Edit tagger (decision D21; per-token keep/delete tags plus insertions139from a learned phrase vocabulary) was added after the freeze as an exploratory alternative that cannot hallucinate140beyond its phrase vocabulary. Its results are \pending{HTR-Edit results}.141