spb/wpai0002_uqo

1% ============================================================================2% Robustness and generalisation. Sources: artifacts/metrics/conf_heldout_*/report.json, summary.md,3% h4_w1_htr-m1-tiny-s3.json; docs/gates/GATE_E.md (H5 table); artifacts/metrics/medium_val_exploratory/audit_*.json4% (RQ6); artifacts/metrics/pangram_{test,opus}/pangram_summary.md, docs/experiments/PANGRAM_EVAL.md (RQ7);5% artifacts/runs/*/train_summary.json and data/pairs/*/ledger.json (compute and cost).6% ============================================================================7\section{Generalisation, Risks and Costs}8\label{sec:robustness}910\subsection{Held-out sets (H5)}1112H5 asks whether the chrF++ gain of $M^\ast$ over identity persists outside the training distribution (SESOI $+1.0$,13one Holm family across the held-out sets). Two held-out sets come from the Medium Haiku pairs: whole held-out14disciplines (physics and earth sciences) and a held-out period (publication before 1900, books). Three come from15the PMC pilot (test split only): pilot passages rewritten by Claude Opus~5.5, which became a held-out rewriter when16the Medium training rewriter was changed to Haiku (D20); Sonnet~5.5 and Haiku~5.5 rewrites (variants A, B); and17Opus rewrites under the held-out prompts \texttt{H1} and \texttt{H2}. Table~\ref{tab:h5} reports the results and18Figure~\ref{fig:h5} the absolute scores.1920\begin{table}[t]21\centering22\caption{H5: chrF++ difference $M^\ast - \text{B0}$ on each held-out set (95\% document-clustered bootstrap, 10\,00023resamples; $p$ Holm-adjusted across the five sets), and the H4-style stylometric reduction24$1 - \overline{W}_1(\text{recon}, \text{orig}) / \overline{W}_1(\text{rewrite}, \text{orig})$. Reduction intervals25from a 1\,000-resample bootstrap; for the last two sets only the two distances were computed (no interval). Scores26are relative to the pre-2018 originals.}27\label{tab:h5}28\small29\begin{tabularx}{\textwidth}{@{}>{\raggedright\arraybackslash}Xrrllll@{}}30\toprule31Held-out set & $n$ & Docs & $\Delta$ chrF++ [95\% CI] & $p_{\mathrm{Holm}}$ & $\overline{W}_1$ reduction & Reading \\32\midrule33Disciplines (physics, earth sc.) & 2\,493 & 154 & $-0.10$ [$-0.36$, $+0.14$] & 0.44 & 36.7\% [28.3, 39.5] & null \\34Period ($<$ 1900, books) & 2\,574 & 92 & $-0.53$ [$-0.75$, $-0.28$] & $<0.001$ & 32.0\% [23.9, 35.6] & worse \\35Rewriter Opus~5.5 (PMC pilot) & 812 & 54 & $+0.72$ [$+0.21$, $+1.11$] & 0.016 & 33.8\% [22.1, 37.5] & below SESOI \\36Sonnet/Haiku~5.5 (PMC pilot) & 531 & 55 & $-0.85$ [$-1.33$, $-0.48$] & $<0.001$ & 0.163 $\to$ 0.108 & worse \\37Prompts \texttt{H1}/\texttt{H2} (Opus, PMC) & 266 & 55 & $-2.38$ [$-2.93$, $-1.92$] & $<0.001$ & 0.190 $\to$ 0.168 & worse \\38\bottomrule39\end{tabularx}40\end{table}4142\paragraph{Outcome.} H5 is \emph{not supported}. The lexical gain transfers in one case only: on Opus rewrites of43PMC pilot passages, $M^\ast$ gains $+0.72$ chrF++ [$+0.21$, $+1.11$] (Holm $p = 0.016$), detectable but below the44SESOI. It is null on held-out disciplines ($-0.10$) and negative on the pre-1900 period ($-0.53$), on the45Sonnet/Haiku pilot rewrites ($-0.85$) and on the held-out prompts ($-2.38$). The held-out prompts produce the46lightest rewrites (B0 scores 80.32 chrF++, against 68.91 on the primary test), and, as for variants B and C in47Section~\ref{sec:results}, a model trained to undo stronger rewrites over-corrects light ones.4849\paragraph{The stylometric gain is present in every set.} The reduction in stylometric distance is 32--37\% on the50held-out disciplines, the pre-1900 period and the Opus rewrites, each with a lower bound above 20\%, and the distance51also falls for the Sonnet/Haiku rewrites (0.163 to 0.108) and, more modestly, for the held-out prompts (0.190 to520.168). The style-directed component of the correction is thus the part that generalises; its lexical53counterpart does not.5455\paragraph{Caveat: the held-out rewriter sets are pilot passages.} The Opus, Sonnet/Haiku and held-out-prompt sets56are drawn from the PMC pilot corpus (biomedical journal prose, placeholder protocol 1.0) rather than from the57Medium test split. Their differences from the primary test therefore confound the rewriter and prompt with the58passage population, and the Sonnet/Haiku set contains Haiku~5.5, the training rewriter, so it is held out in59passages and protocol but only partly in model. Sample sizes are small (266--812 pairs, 54--55 documents).6061\begin{figure}[t]62\centering63\begin{subfigure}{0.49\textwidth}64\includegraphics[width=\textwidth]{figures/fig_overall_confdomain.pdf}65\caption{Held-out disciplines ($n = 2\,493$).}66\end{subfigure}\hfill67\begin{subfigure}{0.49\textwidth}68\includegraphics[width=\textwidth]{figures/fig_overall_confperiod.pdf}69\caption{Held-out period ($n = 2\,574$).}70\end{subfigure}71\begin{subfigure}{0.49\textwidth}72\includegraphics[width=\textwidth]{figures/fig_overall_confopus.pdf}73\caption{Held-out rewriter Opus~5.5 ($n = 812$).}74\end{subfigure}\hfill75\begin{subfigure}{0.49\textwidth}76\includegraphics[width=\textwidth]{figures/fig_overall_confmodels.pdf}77\caption{Sonnet/Haiku~5.5 ($n = 531$).}78\end{subfigure}79\begin{subfigure}{0.49\textwidth}80\includegraphics[width=\textwidth]{figures/fig_overall_confprompts.pdf}81\caption{Held-out prompts \texttt{H1}/\texttt{H2} ($n = 266$).}82\end{subfigure}83\caption{Absolute chrF++, edit similarity and number recall against the pre-2018 originals on the five held-out84sets (means, 95\% document-clustered bootstrap intervals, 10\,000 resamples). $M^\ast$ is \texttt{m1-tiny-s3}; other85HTR rows are exploratory.}86\label{fig:h5}87\end{figure}8889\subsection{Memorisation, agreement and ambiguity (RQ6)}9091The audits were run on the Medium validation split (3\,000 pairs) for the seed-1 M1 Tiny model and for B0, against92the 13\,477 training originals.\footnote{\evid{artifacts/metrics/medium_val_exploratory/audit_*.json}.}9394\paragraph{Memorisation.} No reconstruction had a training original among its MinHash nearest-neighbour candidates95(LSH threshold Jaccard 0.3): the mean maximum Jaccard to the training originals is 0, no output reaches Jaccard 0.5,96the longest common word $n$-gram with a training original is 0, and there is no same-passage leak. The audit thus97finds no memorisation signal; it is an approximate audit whose sensitivity is bounded by the LSH threshold, and the98canary test of Section~\ref{sec:design} has not yet been run.99100\paragraph{Agreement between reconstructions.} For 416 validation passages (218 documents) with two independent101variant-E rewrites (intervals from 1\,000 document-clustered resamples), the two rewrites agree at 82.9 chrF++ [82.2, 83.6] and their reconstructions at 80.1 [79.3,10281.0]; each reconstruction scores 64.5 [63.4, 65.6] against the original, against 62.7 [61.6, 63.8] for the rewrite.103The model therefore does not collapse two rewrites of the same original onto a single reconstruction; it edits104each slightly, and the edits move both toward the original by a similar small amount.105106\paragraph{Ambiguity.} Among 1\,951 validation rewrites of variants A, B and E, no two rewrites of different originals107had a MinHash shingle Jaccard of 0.5 or more, so the direct test of many-to-one collisions found none at this threshold. The108non-identifiability observed here is of the weaker kind: the rewrite does not determine the original's wording,109even though distinct originals rarely produce near-identical rewrites.110111\subsection{Detector sensitivity (RQ7, exploratory)}112113At the author's request (D21), originals, rewrites and reconstructions were scored by the commercial AI-text114detector Pangram (model \texttt{pangram-4.1}) on two stratified samples: 54 test passages (50 documents) of the115Medium Haiku pairs and 85 passages (40 documents) of the Opus pilot pairs. The sample sizes were set by a budget cap116of about 20~USD per run. Pangram scores were never used for training, selection or tuning.117118\begin{table}[t]119\centering120\caption{Pangram~4.1 on originals, rewrites and reconstructions. Mean AI-likelihood is the word-weighted mean of the121window scores (0 human, 1 AI); ``flagged'' is the share labelled \texttt{AI} or \texttt{Mixed} by the API. 95\%122document-clustered bootstrap intervals (2\,000 resamples). No text in any condition received the label \texttt{AI}.123A low score is detector behaviour, not evidence of human authorship.}124\label{tab:pangram}125\small126\begin{tabular}{@{}llrll@{}}127\toprule128Sample & Condition & $n$ & Mean AI-likelihood [95\% CI] & Flagged, \% [95\% CI] \\129\midrule130Medium test (Haiku) & original (pre-2018) & 54 & 0.000 [0.000, 0.000] & 0.0 [0.0, 0.0] \\131 & rewrite (= B0) & 54 & 0.188 [0.125, 0.253] & 46.3 [32.1, 60.0] \\132 & B2 T5-small & 54 & 0.032 [0.004, 0.067] & 7.4 [1.9, 14.5] \\133 & HTR-Small & 54 & 0.000 [0.000, 0.000] & 0.0 [0.0, 0.0] \\134 & $M^\ast$ & 54 & 0.000 [0.000, 0.000] & 0.0 [0.0, 0.0] \\135\midrule136Opus pilot (PMC) & original (pre-2018) & 85 & 0.000 [0.000, 0.000] & 0.0 [0.0, 0.0] \\137 & rewrite (= B0) & 85 & 0.121 [0.078, 0.177] & 32.9 [22.5, 45.5] \\138 & $M^\ast$ & 85 & 0.000 [0.000, 0.000] & 0.0 [0.0, 0.0] \\139\bottomrule140\end{tabular}141\end{table}142143Table~\ref{tab:pangram} shows that Pangram labelled every original \texttt{Human} (139 of 139) and flagged 46\% of144the Haiku rewrites and 33\% of the Opus rewrites as \texttt{Mixed}. After reconstruction by $M^\ast$, every flagged145passage was labelled \texttt{Human} (25 of 25 and 28 of 28); B2 removed most but not all flags (22 of 25, with one146new flag). The paired reduction in AI-likelihood for $M^\ast$ is $-0.188$ [$-0.253$, $-0.125$] on the Haiku sample147and $-0.121$ [$-0.177$, $-0.078$] on the Opus sample.148149Two readings are excluded. \emph{A low detector score is not evidence of human authorship}: the reconstructions150remain close to the LLM rewrites on the surface (their chrF++ against the original is near identity's), and the151result shows only that light learned edits suffice to move this detector's output. Nor is the model an evasion tool152by design; but the measurement establishes that it can act as one, a dual-use risk in the sense of153\citet{krishna2023paraphrasing} and \citet{sadasivan2023reliably}, discussed in Section~\ref{sec:ethics}. The154samples are small, the passages short (mostly a single detector window) and the detector version is served155server-side and may change; the result is tied to Pangram~4.1.156157\subsection{Compute and cost}158159All training ran on Apple-silicon GPUs through PyTorch's MPS backend on the author's machines; no CUDA hardware was160used for the reported runs. Table~\ref{tab:compute} lists the runs behind $M^\ast$ and the comparators; accelerator161time and the $6ND$ estimate are read from each run's \evid{train_summary.json}. Energy was not measured.162163\begin{table}[t]164\centering165\caption{Compute of the main Medium-scale runs ($6ND$: six times parameters times tokens seen; accelerator seconds166on Apple-silicon MPS). The T5-small run logs no $6ND$ estimate.}167\label{tab:compute}168\resizebox{\textwidth}{!}{%169\begin{tabular}{@{}lrrrrr@{}}170\toprule171Run & Params & Steps & Tokens seen & Accel.\ s & $6ND$ FLOPs \\172\midrule173Denoiser, Tiny (\texttt{denoise\_m1\_tiny\_medium\_c}) & 33.5\,M & 20\,000 & 126.1\,M & 4\,164 & $2.54\times10^{16}$ \\174$M^\ast$ fine-tune (M1 Tiny, seed 3) & 33.5\,M & 20\,000 & 91.0\,M & 5\,678 & $1.83\times10^{16}$ \\175M1 Tiny, seed 1 / seed 2 & 33.5\,M & 17\,000 / 20\,000 & 104.1 / 90.9\,M & 3\,481 / 4\,634 & $2.09$ / $1.83\times10^{16}$ \\176M2 Tiny & 33.7\,M & 16\,000 & 72.8\,M & 3\,713 & $1.47\times10^{16}$ \\177M3 token-only Tiny, seed 1 / seed 2 & 34.5\,M & 13\,000 & 79.6\,M & 2\,688 / 2\,683 & $1.65\times10^{16}$ \\178M3-light Tiny & 34.5\,M & 16\,000 & 72.8\,M & 4\,131 & $1.50\times10^{16}$ \\179Denoiser, Small & 130.8\,M & 15\,000 & 70.2\,M & 4\,474 & $5.51\times10^{16}$ \\180HTR-Small fine-tune & 130.8\,M & 15\,000 & 68.2\,M & 4\,434 & $5.35\times10^{16}$ \\181B2 T5-small fine-tune & 60.5\,M & 3\,000 & -- & 531 & -- \\182\bottomrule183\end{tabular}}184\end{table}185186$M^\ast$ itself cost 9\,842 accelerator-seconds (2.7 hours) and $4.4\times10^{16}$ FLOPs, denoiser included. Over187all 38 run summaries (pilot and Medium, including failed diagnostics), the total is 16.7 accelerator-hours and188$3.3\times10^{17}$ FLOPs, of which 12.8 hours and $2.7\times10^{17}$ FLOPs at Medium scale. API spend, from the189job ledgers at Message Batches prices, is 26.85~USD for the Opus pilot, 1.43~USD for the Sonnet/Haiku held-out set,1901.40~USD for the held-out prompts and 13.52~USD for the 120\,000 Medium Haiku requests, a total of 43.20~USD for191rewriting. The Pangram evaluation cost about 38.93~USD (list-price estimate of 524 billed texts; Pangram does not192return invoiced amounts per call).193