spb/wpai0002_uqo

1% ============================================================================2% Experimental design3% ============================================================================4\providecommand{\ph}[1]{\texttt{[[#1]]}}5\providecommand{\evid}[1]{\texttt{\detokenize{#1}}}67\section{Experimental Design}8\label{sec:design}910\subsection{Evaluation sets and confirmatory protocol}1112Models are trained on the \texttt{train} split and developed on \texttt{val}. Five evaluation sets are13confirmatory: the document-, near-duplicate- and author-group-safe \texttt{test} split; the held-out-domain split14(whole disciplines never seen in training); the held-out-period split (the earliest publication years); test15passages rewritten with the held-out prompts \texttt{H1} and \texttt{H2}; and test passages rewritten by the16held-out models Sonnet~5.5 and Haiku~5.5. A sixth, exploratory set holds out a rewrite variant: models are trained17on variants A--D and tested on E. Until the pre-registration is frozen, only \texttt{train} and \texttt{val} may be18scored, apart from sanity checks of B0 and B1 that report no model comparison. After the freeze, confirmatory19scoring is run once; any deviation is reported as a dated amendment and the affected results are labelled20exploratory.\footnote{\evid{docs/PREREGISTRATION.md} (draft v0.1; freeze record \pending{freeze commit}).}2122\subsection{Training protocol}2324All trained systems use AdamW with $(\beta_1, \beta_2) = (0.9, 0.98)$, weight decay on weight matrices and embeddings but not on biases or normalisation parameters, gradient25clipping at norm 1, token-budget batching, and a linear warm-up of $W$ steps followed by cosine decay to 10\% of26the peak learning rate $\eta$:27\begin{equation}28  \eta(s) =29  \begin{cases}30    \eta\,(s+1)/W, & s < W,\\31    \eta\,\big[0.1 + 0.45\,\big(1 + \cos(\pi\,\rho_s)\big)\big], & s \ge W,32  \end{cases}33  \qquad \rho_s = \min\Big(1, \frac{s - W}{S_{\max} - W}\Big).34\end{equation}35Early stopping monitors the unsmoothed token cross-entropy on validation, and the best checkpoint is kept.36Mixed precision (bfloat16) is used on CUDA; Apple-silicon (MPS) runs stay in fp32. Every run writes a manifest with37the configuration, seeds, data and code hashes, environment versions, parameter count, tokens seen, accelerator38time and a training-FLOPs estimate $6ND$ ($N$ parameters, $D$ source plus target tokens), which for an39encoder--decoder is an upper bound.\footnote{\evid{src/htr/training/trainer.py}.} Each configuration is trained40with seeds 1, 2 and 3, and systems are compared at matched data and step budgets. Before any pilot training, each41architecture must pass a one-batch overfit test; on smoke data, an M1 model of the smallest test size drove the42token loss from 5.28 to 0.0022 in 150 steps.\footnote{\evid{artifacts/runs/vs_smoke-m1-nano/}.} Hyper-parameter43values are listed in Appendix~\ref{app:hyper}.4445\subsection{Metrics}4647All metrics compare a system output $\hat y$ with the original $y$, both in placeholder form. Every instance-level48metric is reported next to the identity baseline.\footnote{\evid{src/htr/evaluation/metrics.py},49\evid{src/htr/evaluation/stylometry.py}.}5051\paragraph{Verbatim reconstruction.} chrF++ \citep{popovic2015chrf} at sentence level (sacreBLEU, character52$n$-grams with word bigrams), normalised Levenshtein similarity after whitespace and case normalisation, exact53match, sentence BLEU \citep{papineni2002bleu} and ROUGE-L $F$ \citep{lin2004rouge}, and the word-length ratio.54Overlap metrics reward copying, which is why they are never reported without B0. Embedding similarity55\citep{reimers2019sentencebert,zhang2019bertscore} is a secondary semantic measure.5657\paragraph{Factual integrity.} With $\mathcal{N}(\cdot)$ the multiset of numbers extracted from a text (thousands58separators, decimals, percentages and scientific notation included) and $\mathcal{P}(\cdot)$ the multiset of59placeholders,60\begin{equation}61  \mathrm{NumRec} = \frac{|\mathcal{N}(\hat y) \cap \mathcal{N}(y)|}{|\mathcal{N}(y)|}, \qquad62  \mathrm{NumPrec} = \frac{|\mathcal{N}(\hat y) \cap \mathcal{N}(y)|}{|\mathcal{N}(\hat y)|},63\end{equation}64each set to 1 when its denominator is empty, together with placeholder exactness65$\mathbb{1}[\mathcal{P}(\hat y) = \mathcal{P}(y)]$ and recall, and the signed differences in negation cues, hedge66cues and sentence counts. These checks cover citations, equations and links through the placeholders, and numbers67directly.6869\paragraph{Stylistic distribution matching.} Eighteen stylometric features are computed per passage: mean and70standard deviation of sentence length, type--token ratio, MTLD, mean word length, long-word rate, comma, semicolon71and parenthesis rates, discourse-marker rate, passive rate, nominalisation rate, first-person rate, subordination72rate, hedge rate, negation rate, digit rate, and Flesch reading ease. For two sets of passages with feature samples73$\{u\}$ and $\{r\}$, the distance on feature $j$ is a quantile-based 1-Wasserstein distance standardised by a74spread $\varsigma_j$,75\begin{equation}76  W_{1,j} = \frac{1}{|Q|}\sum_{q \in Q} \frac{\big|\hat F^{-1}_{u,j}(q) - \hat F^{-1}_{r,j}(q)\big|}{\varsigma_j},77  \qquad Q = \{0, 0.005, \dots, 1\},78\end{equation}79where $\varsigma_j = \max\big(\mathrm{sd}_j, 0.05(|\mathrm{mean}_j| + 10^{-9}), 10^{-6}\big)$ is computed on a80reference pool so that near-constant features cannot dominate, and $\overline{W}_1$ is the mean over the 1881features. For H4, writing $Y$, $X$ and $\hat Y$ for the sets of originals, rewrites and reconstructions, the82distance $\overline{W}_1(\hat Y, Y)$ is compared with $\overline{W}_1(X, Y)$; both distances must use the same scale $\varsigma$, estimated83on the originals and rewrites of the evaluation set. These metrics are distributional: they ask whether84reconstructions as a set resemble originals as a set, which is the meaningful notion of style recovery under85non-identifiability, and they can reveal regression toward a mean style that per-item metrics hide.8687\paragraph{Exploratory style probe.} A logistic regression on standardised stylometric features is trained to88separate originals from rewrites and then applied to reconstructions; we report its held-out AUC and the mean89predicted probability of ``original'' on reconstructions. The probe must be fitted on a split disjoint from the90evaluated items. Its scores describe feature distributions and are never evidence of human authorship.9192\subsection{Statistical analysis}9394Items are not independent: one document contributes several passages, and one passage up to six rewrites. All95intervals therefore come from a document-clustered bootstrap.\footnote{\evid{src/htr/evaluation/bootstrap.py},96\evid{src/htr/evaluation/runner.py}.} In each replicate, documents are drawn with replacement and every item of a97drawn document enters the resample. For a paired difference $d_i = \mathrm{score}_i(M^\ast) -98\mathrm{score}_i(\mathrm{B0})$, the 95\% interval is the percentile interval of the replicate means $\bar d^\ast$,99and the two-sided $p$-value is $\min\big(1, 2\min(\hat P, 1 - \hat P)\big)$ with $\hat P$ the share of replicates100with $\bar d^\ast \le 0$. Confirmatory analyses use 10\,000 replicates (the development default is 1\,000). Results101are reported with means, intervals, and the numbers of items and documents, overall and by variant, discipline,102era and section role. Holm corrections and decision rules are those of Section~\ref{sec:rq}.103104\subsection{Experiments}105106Table~\ref{tab:experiments} maps the experiments to the research questions. Experiments marked ``code ready'' have107an implementation and tests but no reported run; all results are \pending{experiment results}.108109\begin{table}[t]110\centering111\caption{Experiment matrix (\texttt{docs/experiments/EXPERIMENT\_MATRIX.md}). Seeds refer to training seeds.}112\label{tab:experiments}113\small114\begin{tabularx}{\textwidth}{@{}l>{\raggedright\arraybackslash}p{0.09\textwidth}>{\raggedright\arraybackslash}X>{\raggedright\arraybackslash}p{0.2\textwidth}@{}}115\toprule116ID & RQ & Design & Status \\117\midrule118E1 & RQ1 & Paired feature shift original $\to$ rewrite per variant (18 features, Holm) & pairs available \\119E2 & -- & Pilot rewrite, validation and review (Gate B) & done (Table~\ref{tab:pilot-jobs}) \\120E3 & RQ2 & B0, B1, B2 on the test split & code ready \\121E4 & RQ2, RQ5 & M1 vs M2 vs M3 at tiny size, 3 seeds & running / \pending{E4} \\122E5 & RQ5 & Ablations: token loss only, no style, no adversary, no factual, no contrastive, no copy; M1 with text metadata prefix & code ready \\123E6, E7 & RQ4 & Held-out prompts \texttt{H1}, \texttt{H2}; held-out models Sonnet 5.5, Haiku 5.5 & pairs available \\124E8 & RQ4 & Train on variants A--D, test on E & code ready \\125E9 & RQ3 & Held-out discipline and held-out period & code ready \\126E10, E11 & RQ5 & Scaling in model size (tiny to base, large if feasible) and data ($\tfrac14, \tfrac12, 1$ of the pairs, then Medium) & needs Medium data \\127E12, E13 & RQ2, RQ3 & Gains by rewrite strength (deciles of rewrite--original similarity, variant) and input length & analysis code ready \\128E14 & RQ6 & Non-identifiability: within-original agreement, rewrite ambiguity, oracle ceiling & code ready \\129E15 & RQ6 & Memorisation: nearest-neighbour overlap, verbatim rate, canaries; contamination audit for B2 & code ready \\130E16 & RQ2, RQ6 & Blind human evaluation & planned; ethics approval pending \\131E17 & RQ6 & Error taxonomy on $\geq 200$ quarantined and worst-scoring items & planned \\132E18 & -- & Calibration of token log-probability against item chrF++ & planned \\133E19 & RQ2 & Exploratory style probe on a disjoint probe split & code ready \\134\bottomrule135\end{tabularx}136\end{table}137138\subsection{Non-identifiability}139\label{sec:design-ident}140141Three measurements quantify how much of the original a rewrite determines.\footnote{\evid{src/htr/evaluation/identifiability.py}.}142\begin{enumerate}143  \item \textbf{Within-original agreement.} Variant E yields two independent rewrites $x^{(1)}, x^{(2)}$ of the same144  original. We compare $\mathrm{chrF}(x^{(1)}, x^{(2)})$ and $\mathrm{chrF}(x^{(1)}, y)$ with the corresponding145  quantities for reconstructions, $\mathrm{chrF}(\hat y^{(1)}, \hat y^{(2)})$ and146  $\tfrac12[\mathrm{chrF}(\hat y^{(1)}, y) + \mathrm{chrF}(\hat y^{(2)}, y)]$. A model that converges on the same147  reconstruction from different rewrites while staying far from $y$ has learned a mode of $p(y \mid x)$, not $y$.148  \item \textbf{Rewrite ambiguity.} Among rewrites of variants A, B and E from \emph{distinct} passages, we find149  pairs whose MinHash Jaccard similarity over word 5-grams is at least 0.5 and measure the chrF and edit similarity150  between their originals. The distribution of these distances, which involves no model, is the empirical151  counterpart of the lower bound of Section~\ref{sec:models}.152  \item \textbf{Oracle ceiling.} The chrF between two originals that an LLM mapped to near-identical rewrites bounds153  the expected chrF of a perfect deterministic reconstructor on those items.154\end{enumerate}155156\subsection{Memorisation and contamination}157158A reconstruction model could score well by recalling training originals. For each reconstruction on an evaluation159set we find the nearest training original by MinHash over word 5-grams and record the Jaccard similarity and the160longest common word $n$-gram; we report the mean maximum Jaccard, the share of outputs with Jaccard at least 0.5 and1610.8, the share with a common $n$-gram of length at least 12, and the number of outputs whose nearest neighbour is162their own passage, which must be zero under correct splits.\footnote{\evid{src/htr/evaluation/memorization.py}.}163Synthetic canaries (invented passages with unique markers, containing no real data) are inserted into training a164fixed number of times, and their emission rate at test time is measured. For B2, the documented pretraining data165are audited for the originals where possible \citep{magar2022contamination,sainz2023contamination}.166167\subsection{Human evaluation}168169A blind evaluation will be run on 120 test passages stratified by discipline and variant, with three170domain-literate raters and four conditions per item: the original, B0, B1 and the best model. A Latin square171ensures that each rater sees each item in exactly one condition and each condition equally often; all texts are172shown in placeholder-restored form and raters are not told which is the original. Raters score, on five-point173scales, fidelity to the meaning of the source (the rewrite is shown as reference), fluency, historical-style174authenticity (``plausibly written before 2018''), and factual safety, flagging any changed number, citation,175negation or hedge (also checked automatically). Agreement is reported with Krippendorff's $\alpha$, and condition176effects with an ordinal mixed model with rater and item random effects. A 20-item pilot calibrates raters.177Recruitment and ethics approval at UQO are \pending{ethics approval}.178179\subsection{Compute and cost reporting}180181We report, per run, parameters, tokens, accelerator time, device, the $6ND$ FLOPs estimate and, where measured,182energy; energy that is not measured is stated as such. API spend is reported from the job ledgers. Training runs183on Apple-silicon machines with PyTorch's MPS backend; the hardware and accelerator-hours of each reported run are184\pending{compute table}.185