spb/wpai0002_uqo

1% ============================================================================2% Research questions and hypotheses3% ============================================================================4\section{Research Questions and Hypotheses}5\label{sec:rq}67\subsection{Research questions}89The programme is organised around six research questions.1011\begin{itemize}12  \item \textbf{RQ1 (feature shift).} What syntactic, lexical, rhetorical and discourse features change systematically13  when an LLM rewrites historical scientific prose?14  \item \textbf{RQ2 (recovery).} Which pre-rewrite features can be recovered from rewritten text beyond an identity15  baseline?16  \item \textbf{RQ3 (heterogeneity).} How does recovery vary across disciplines, publication periods, writing styles, and17  paragraph or section types?18  \item \textbf{RQ4 (generalisation).} Does a model trained on Claude rewrites generalise to rewrites from held-out prompts19  and from other models?20  \item \textbf{RQ5 (scale and conditioning).} How does performance change with data scale, model capacity and style21  conditioning?22  \item \textbf{RQ6 (losses and risks).} What factual distortions, memorisation risks and irreversible information losses23  remain?24\end{itemize}2526RQ1 concerns the rewrites alone and is answered without any trained model, from paired differences between each27rewrite and its original. RQ2--RQ5 concern the reconstruction models of Section~\ref{sec:models}. RQ6 cuts across28both: some losses are properties of the rewriting process (ambiguity, Section~\ref{sec:design}), others of the29reconstruction model (factual drift, memorisation).3031\subsection{Confirmatory hypotheses}3233Six hypotheses are confirmatory and are tested on the confirmatory splits only after the pre-registration is34frozen (Table~\ref{tab:hypotheses}). Throughout, $M^\ast$ denotes the best HTR model, selected on the validation35split only among M1, M2, M3 and their ablations; B0 is the identity baseline and B1 the generic-paraphrase36baseline (Section~\ref{sec:models}). Hypotheses are written H1--H6 in roman type; the two held-out rewrite37prompts of Section~\ref{sec:rewrite} keep their code names \texttt{H1} and \texttt{H2} in typewriter type. The unit of analysis is a validated pair (rewrite as source, original as38target, both in placeholder form), and inference is clustered by document. The smallest effect sizes of interest39(SESOI) in Table~\ref{tab:hypotheses} are the values proposed in the draft pre-registration (v0.1). They were40frozen unchanged in version 1.0 on 2026-11-09, pre-specified in-session before the author had confirmed them, at41commit \texttt{e74b71266d5c0e8c20f181a50df139f52c771d0f}, with one implementation amendment before any42confirmatory scoring (A1: a common standardisation scale for both stylometric distances in H4). The last column of43Table~\ref{tab:hypotheses} gives the outcome of the single confirmatory pass (Section~\ref{sec:results}); deviations44are listed in Section~\ref{sec:threats}.4546\begin{table}[t]47\centering48\caption{Confirmatory hypotheses, primary metrics and smallest effect sizes of interest (pre-registration v1.0,49frozen at \texttt{e74b712}, amendment A1), with the outcome of the confirmatory pass. All tests use a50document-clustered paired bootstrap.}51\label{tab:hypotheses}52\small53\begin{tabularx}{\textwidth}{@{}l>{\raggedright\arraybackslash}X>{\raggedright\arraybackslash}p{0.17\textwidth}>{\raggedright\arraybackslash}p{0.1\textwidth}l>{\raggedright\arraybackslash}p{0.2\textwidth}@{}}54\toprule55ID & Hypothesis & Primary metric & SESOI & RQ & Status (confirmatory pass) \\56\midrule57H1 & $M^\ast$ reconstructs the original better than B0 and better than B1 & chrF++ (sentence level, mean over items) & $+2.0$ points vs each & RQ2 & vs B0: $+0.44$, detectable but below SESOI; vs B1: $+3.13$, supported \\58H2 & Same comparison as H1 & normalised edit similarity & $+0.02$ & RQ2 & vs B0: $+0.0024$, detectable but below SESOI; vs B1: $+0.115$, supported \\59H3 & $M^\ast$ is non-inferior to B0 in preserving numbers & number recall vs the original & margin $-0.01$ & RQ2, RQ6 & not established (lower bound $-0.0104$) \\60H4 & Reconstructions are distributionally closer to the originals than the rewrites are & $\overline{W}_1$ over 18 standardised stylometric features & $\geq 10\%$ relative reduction & RQ2 & supported: 32.5\% [29.6, 37.6] \\61H5 & The H1 gain over B0 persists for held-out prompts (\texttt{H1}, \texttt{H2}) and held-out models & chrF++ gain vs B0 per held-out set & $+1.0$ & RQ4 & not supported (one set $+0.72$, others null or negative) \\62H6 & M3 beats token-loss M1 at the same size, data and step budget (mean of 3 seeds) & chrF++ on test & $+1.0$ & RQ5 & not tested as specified; exploratory: no gain \\63\bottomrule64\end{tabularx}65\end{table}6667\paragraph{Decision rules.} Multiplicity is controlled with Holm--Bonferroni at family-wise $\alpha = 0.05$ within68three families: the primary family \{H1 vs B0, H1 vs B1, H2 vs B0, H2 vs B1, H3, H4\}; H5 across the four held-out69sets (two prompts, two models); and H6 alone. A hypothesis is \emph{supported} only if it is significant after70Holm correction \emph{and} its point estimate reaches the SESOI. A significant effect below the SESOI is reported as71statistically detectable but practically small, and a non-significant one as a null finding with its confidence72interval. For H3 the decision is one-sided: the lower bound of the 95\% interval of $M^\ast - \text{B0}$ must73exceed the margin. For H4, documents are resampled and both distributional distances are recomputed in each74bootstrap replicate.7576\subsection{Rationale and expected direction}7778The hypotheses encode a conservative prior. Most of an original passage's content is copyable from its rewrite,79because the rewrite instructions require every proposition, number, entity, citation and equation to be preserved80(Section~\ref{sec:rewrite}). The identity baseline B0 is therefore strong on overlap metrics, and a model can beat81it only by systematically undoing the rewrite's lexical and syntactic substitutions. If LLM rewrites have82learnable regularities, as \citet{riverasoto2024paraphrase} report for paraphrases, H1 and H2 should hold; if the83substitutions are close to random given the rewrite, the best a model can do is regress toward an average style,84which H4 can detect while H1 and H2 fail. H3 protects against a model that improves overlap by rewriting factual85content. H5 separates learning the inverse of a particular instruction from learning general regularities of the86rewriting model. H6 is the test of the main architectural hypothesis; in view of87\citet{riverasoto2026unsupervised}, a positive result would replicate rather than establish the value of a88content/style decomposition, and a null result would be informative.8990\subsection{Exploratory analyses}9192The following analyses are reported, clearly labelled as exploratory: per-feature shifts for RQ1 (with Holm93correction over 18 features as a descriptive guard); breakdowns by discipline, era, section role, variant, rewrite94strength and input length (RQ3); exact match, BLEU, ROUGE-L, placeholder exactness and negation and hedge deltas;95the ablations, scaling curves, calibration and error taxonomy; an original-versus-rewrite style probe applied to96reconstructions; and the non-identifiability and memorisation measurements (RQ6). Probe and classifier scores are97evidence about feature distributions only, never about who wrote a text.98