spb/wpai0002_uqo

1% ============================================================================2% Threats to validity. Sources: docs/PREREGISTRATION.md (freeze record, A1), docs/CONFIRMATORY_PLAN.md,3% docs/DECISION_LOG.md D17-D21, artifacts/metrics/conf_*/report.json and h4_w1_*.json, docs/INCIDENTS.md.4% ============================================================================5\section{Threats to Validity}6\label{sec:threats}78\paragraph{Deviations from the frozen pre-registration.} The pre-registration (v1.0, frozen at commit9\texttt{e74b712}, amendment A1) was written while the only pairs were the PMC pilot pairs, and its freeze record10names the pilot corpus, splits and pair files. After the freeze, the training rewriter was changed from Opus~5.511to Haiku~5.5 for the Medium stage (D20, a budget decision), and the confirmatory pass was run on the Medium test12split as specified in \evid{docs/CONFIRMATORY_PLAN.md} before scoring. The hypotheses, metrics, SESOIs, selection13rule and decision rules were unchanged, and no confirmatory split was scored before $M^\ast$ was fixed, but this14change of test population was not recorded as a dated amendment. Three further departures follow. (i) The H515family was planned over four held-out sets (two prompts, two models); it was run over five, adding held-out16disciplines and period, with the two prompts pooled, Sonnet and Haiku pooled, and the Opus pilot rewrites added as17a held-out rewriter; Haiku, now the training rewriter, is in the ``held-out models'' set. (ii) The H4 interval was18computed with 2\,000 resamples on test and 1\,000 on the held-out sets instead of 10\,000; the lower bound on test19(29.6\%) is far from the 10\% SESOI, so the decision is not sensitive to this. (iii) The Holm adjustment stored in20the reports runs across systems within each metric rather than over the primary family; because no bootstrap21replicate crossed zero for any primary test with a $p$-value, the family-wise decisions are unaffected, and the H522$p$-values in Table~\ref{tab:h5} were adjusted across the five sets. H6 was not run as specified23(Section~\ref{sec:ablations}).2425\paragraph{Exploratory status of the HTR~2.0 tracks.} Everything in Section~\ref{sec:htr2} was decided after the26confirmatory pass, on evidence that included the confirmatory numbers. The tracks were pre-declared (D22) before27their systems were trained, each system was tuned on validation only and applied once to the test and held-out28sets, and the same pairs, baseline and bootstrap procedure were used, but the choice of backbones, their learning29rates, checkpoint selection by validation cross-entropy and the decision to report two seeds for Qwen2.5-1.5B and30HTR-Base were all made in the open. The $+2.0$ SESOI is therefore a yardstick for those results, not a passed31confirmatory test, and the Holm adjustment in the exploratory report runs over 26 systems within each metric.32Three further points concern the backbones. (i) \emph{Pretraining contamination.} flan-T5 and Qwen2.5 were33pretrained on web-scale corpora that plausibly include pre-2018 PMC articles and Project Gutenberg books. The34memorisation audit (maximum Jaccard 0.000 to the HTR training originals on validation) covers only the HTR pairs,35not those corpora, so part of the backbones' gain, in particular on the pre-1900 books, may come from a prior that36has seen the originals or their relatives; the test documents' presence in those corpora cannot be verified by37us. (ii) \emph{Tokenizer coverage.} The T5 SentencePiece vocabulary maps \texttt{<}, \texttt{\textasciitilde},38$\pm$, $\rightarrow$, $\leq$ and Greek letters to \texttt{<unk>}; the flan-T5 scores were produced before the39inference code protected these characters, so those models are slightly under-stated and their number recall is40affected. (iii) \emph{Compute mismatch.} The backbones bring orders of magnitude more pretraining than the41in-house denoisers; the comparison measures the value of a pretrained prior, not architecture at matched compute.4243\paragraph{Pre-registration status.} The SESOIs are the values proposed in the draft and were pre-specified44in-session by the operating assistant; the author had not confirmed them at freeze time. They were not changed45afterwards, but readers should treat them as the project's proposals rather than as externally reviewed thresholds.4647\paragraph{Construct validity.} chrF++ and edit similarity measure agreement with one particular original. Because a48rewrite is compatible with many originals, a valid reconstruction can score low; this biases H1 and H2 against any49model and is why stylistic distribution (H4) is reported separately. The stylometric distance uses 18 surface50features; a reduction shows that marginal feature distributions move toward the originals, not that the text reads51as pre-2018 prose to a domain reader (Section~\ref{sec:human}). The style probe and the detector are measurement52devices for feature distributions and detector behaviour, never for authorship.5354\paragraph{Internal validity.} $M^\ast$ was chosen by validation cross-entropy among candidates whose validation55chrF++ differed by less than 0.4 points, so the choice among them is close to arbitrary, but the selection rule was56fixed before scoring and all candidates are reported. Validation and test splits are disjoint by document,57near-duplicate cluster and author group, with zero group leaks. The memorisation audit uses MinHash candidates with a58Jaccard threshold of 0.3 and could miss short verbatim spans; the canary test has not been run. B2's pretraining data59may contain some originals; it is a secondary comparator and does not enter any confirmatory test.6061\paragraph{External validity.} The confirmatory rewrites come from a single model (Haiku~5.5) under five instructions62of one prompt version; real LLM-assisted writing involves other models, iterative prompting and human post-editing,63none of which is represented. The held-out evidence for other rewriters rests on small PMC pilot samples. The corpus64is dominated by PMC biomedical and life-science articles from 2010--2017 (80\% of test pairs), so results for older65prose and for books are based on few documents. Validation rejected 13.4\% of the Medium pairs, mostly for missing66numbers; the confirmatory population is therefore the set of rewrites that preserved content, which is the intended67population but not all LLM output.6869\paragraph{Statistical conclusion validity.} Intervals are document-clustered, but items from the same passage70(variants and samples) are nested within documents, and some strata in the heterogeneity analyses rest on fewer than7120 documents. The breakdowns are exploratory and uncorrected for multiplicity.7273\paragraph{Operational.} Two cluster nodes froze under Medium-scale pretraining (incident I1) and an invalid memory74setting aborted queued runs (I2); the affected runs were resumed from their last checkpoint or restarted with recorded75memory caps. On the H100 instances, the cuDNN attention backend built an execution plan per batch shape, which76made steps 15 times slower and produced NaN gradients under bf16 autocast; it was disabled for all CUDA runs (I3).77BART-large still produced NaN gradients in fp32 and was dropped in favour of flan-T5; resizing the T5 embedding78matrix for the placeholder tokens broke T5 v1.1, so placeholders are spelled in ASCII for the pretrained79backbones. HTR-Large diverged three times (I4). Some evaluation artefacts were produced from a working tree with80uncommitted changes (recorded as \texttt{dirty} in their manifests), and the run manifests written on the cloud81instances carry no git commit because the code was synchronised by \texttt{rsync} rather than cloned.82