SPB Git
spboucher.ai

spb/wpai0002_uqo

Public

Medium corpus build script

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 9, 2026) parent a5a274c Browse files

3 changed files +266 −6

3 changed files
Unified Split
modified paper/sections/ethics_and_licensing.tex +65 −3
@@ -1,3 +1,65 @@
1 −% ethics_and_licensing — to be written from measured results (never from assumptions)
2 −\section{ethics and licensing}
3 −\pending{section to be written}
1 +% ============================================================================
2 +% Ethics and licensing
3 +% ============================================================================
4 +\providecommand{\evid}[1]{\texttt{\detokenize{#1}}}
5 +
6 +\section{Ethics and Licensing}
7 +\label{sec:ethics}
8 +
9 +\paragraph{Rights in the source texts.} Open access is not open licensing. Every document carries its own licence
10 +code and evidence, the document's statement prevails over provider metadata, and documents whose licence is
11 +unknown, text-mining-only, no-derivatives, or the default arXiv licence are excluded from training
12 +(Table~\ref{tab:licence-tiers}). Rewrites and reconstructions are derivative works and inherit the terms of their
13 +source. Non-commercial and share-alike sources may be used for research training under the default policy but are
14 +never redistributed in the main benchmark. Public-domain status of Project Gutenberg books is verified for the
15 +United States only, and book dates inferred from contributor death years are flagged as upper bounds and can be
16 +excluded. Retraction status is checked at acquisition for PMC articles and will be re-checked before any release.
17 +Acquisition uses official bulk channels, honours \texttt{robots.txt} and rate limits, and identifies itself with a
18 +research user agent; publisher text-mining links are used only for Creative Commons content. Where a publisher's
19 +\texttt{robots.txt} names AI crawlers specifically, the policy to follow is an open decision for the author and is
20 +recorded in the project's risk register.
21 +
22 +\paragraph{Provider terms.} Rewrites are generated with Anthropic models under the provider's commercial terms. The
23 +provider's usage policy restricts the use of outputs to develop competing models. Whether training a small,
24 +task-specific reconstruction model on Claude rewrites and publishing it falls under this restriction is treated as
25 +an open question, not a conclusion: it will be checked against the terms in force and with UQO before any public
26 +weight release. Until then, rewrites are kept internal and are not redistributed. Cross-provider generalisation
27 +tests (RQ4) use only outputs whose terms permit this use; in this version all held-out models are from the same
28 +provider.
29 +
30 +\paragraph{Release.} Two weight sets are planned. \emph{Release weights} are trained only on CC0, public-domain and
31 +CC-BY data and can be published after a licence, privacy and memorisation review. \emph{Research weights} are
32 +trained on the full training set, including non-commercial and share-alike sources, and are released only if the
33 +author and a legal review decide so, given the uncertain status of licence inheritance by model weights. The
34 +benchmark redistributes only source and derivative text whose licence is in the release set, with an attribution
35 +file listing title, authors, venue, year, identifier, licence and the changes made, plus the NLM/PMC
36 +acknowledgement. Nothing is uploaded and no hosting namespace is chosen without the author's explicit approval.
37 +Takedown requests are handled within five working days by withdrawing the document from all release bundles and
38 +recording the action; if a withdrawn document was in a released model's training data, the next model card lists
39 +it, and retraining is considered if memorisation tests show verbatim recall.\footnote{\evid{docs/data_governance/LICENSE_MATRIX.md}, Sections 5--6.}
40 +
41 +\paragraph{Misuse and dual use.} A model that maps LLM-rewritten text toward the style of pre-2018 human prose can
42 +be misused in two ways: to present generated text as human-authored, and to ``humanise'' generated text so that it
43 +evades AI-text detectors, the class of attack studied by \citet{zheng2025thbench} and \citet{sadasivan2023reliably}.
44 +We take three measures. Every output carries the notice that it is a synthetic reconstruction and not evidence of
45 +human authorship (Section~\ref{sec:models}). The model card states the intended research uses and excludes
46 +authorship claims and detector evasion. And we make no detection or authenticity claims ourselves: probe scores are
47 +exploratory, and we do not report detector-evasion rates as a success measure. The residual risk is reduced, not
48 +removed, by these measures; it is weighed against the scientific value of measuring what LLM rewriting destroys.
49 +
50 +\paragraph{Privacy and memorisation.} The corpus consists of published scientific and scholarly text; we do not
51 +collect personal data beyond bibliographic metadata (author names, affiliations as printed). Because a
52 +reconstruction model might regurgitate training originals, the memorisation audit of Section~\ref{sec:design} is a
53 +release condition.
54 +
55 +\paragraph{Human participants.} No human participants have been involved so far. The qualitative audit of rewrite
56 +pairs reported in Section~\ref{sec:rewrite} was not a study with human raters. The planned human evaluation will
57 +begin only after recruitment and ethics approval at UQO, which are \pending{ethics approval}.
58 +
59 +\paragraph{Cost and environment.} API spend to date is 29.68~USD, recorded per job in the ledgers. Training compute,
60 +accelerator-hours and, where measured, energy are reported per run; energy that is not measured is stated as
61 +not measured. All large spends and long training runs require the author's explicit approval.
62 +
63 +\paragraph{Use of AI assistance.} The software, documentation and drafts of this working paper were produced with
64 +the assistance of Claude Code under the author's supervision. All numbers are traced to run manifests and ledgers,
65 +and all references were checked against bibliographic records.
modified paper/sections/experimental_design.tex +184 −3
@@ -1,3 +1,184 @@
1 −% experimental_design — to be written from measured results (never from assumptions)
2 −\section{experimental design}
3 −\pending{section to be written}
1 +% ============================================================================
2 +% Experimental design
3 +% ============================================================================
4 +\providecommand{\ph}[1]{\texttt{[[#1]]}}
5 +\providecommand{\evid}[1]{\texttt{\detokenize{#1}}}
6 +
7 +\section{Experimental Design}
8 +\label{sec:design}
9 +
10 +\subsection{Evaluation sets and confirmatory protocol}
11 +
12 +Models are trained on the \texttt{train} split and developed on \texttt{val}. Five evaluation sets are
13 +confirmatory: the document-, near-duplicate- and author-group-safe \texttt{test} split; the held-out-domain split
14 +(whole disciplines never seen in training); the held-out-period split (the earliest publication years); test
15 +passages rewritten with the held-out prompts \texttt{H1} and \texttt{H2}; and test passages rewritten by the
16 +held-out models Sonnet~5.5 and Haiku~5.5. A sixth, exploratory set holds out a rewrite variant: models are trained
17 +on variants A--D and tested on E. Until the pre-registration is frozen, only \texttt{train} and \texttt{val} may be
18 +scored, apart from sanity checks of B0 and B1 that report no model comparison. After the freeze, confirmatory
19 +scoring is run once; any deviation is reported as a dated amendment and the affected results are labelled
20 +exploratory.\footnote{\evid{docs/PREREGISTRATION.md} (draft v0.1; freeze record \pending{freeze commit}).}
21 +
22 +\subsection{Training protocol}
23 +
24 +All trained systems use AdamW with $(\beta_1, \beta_2) = (0.9, 0.98)$, weight decay on weight matrices and embeddings but not on biases or normalisation parameters, gradient
25 +clipping at norm 1, token-budget batching, and a linear warm-up of $W$ steps followed by cosine decay to 10\% of
26 +the peak learning rate $\eta$:
27 +\begin{equation}
28 + \eta(s) =
29 + \begin{cases}
30 + \eta\,(s+1)/W, & s < W,\\
31 + \eta\,\big[0.1 + 0.45\,\big(1 + \cos(\pi\,\rho_s)\big)\big], & s \ge W,
32 + \end{cases}
33 + \qquad \rho_s = \min\Big(1, \frac{s - W}{S_{\max} - W}\Big).
34 +\end{equation}
35 +Early stopping monitors the unsmoothed token cross-entropy on validation, and the best checkpoint is kept.
36 +Mixed precision (bfloat16) is used on CUDA; Apple-silicon (MPS) runs stay in fp32. Every run writes a manifest with
37 +the configuration, seeds, data and code hashes, environment versions, parameter count, tokens seen, accelerator
38 +time and a training-FLOPs estimate $6ND$ ($N$ parameters, $D$ source plus target tokens), which for an
39 +encoder--decoder is an upper bound.\footnote{\evid{src/htr/training/trainer.py}.} Each configuration is trained
40 +with seeds 1, 2 and 3, and systems are compared at matched data and step budgets. Before any pilot training, each
41 +architecture must pass a one-batch overfit test; on smoke data, an M1 model of the smallest test size drove the
42 +token loss from 5.28 to 0.0022 in 150 steps.\footnote{\evid{artifacts/runs/vs_smoke-m1-nano/}.} Hyper-parameter
43 +values are listed in Appendix~\ref{app:hyper}.
44 +
45 +\subsection{Metrics}
46 +
47 +All metrics compare a system output $\hat y$ with the original $y$, both in placeholder form. Every instance-level
48 +metric is reported next to the identity baseline.\footnote{\evid{src/htr/evaluation/metrics.py},
49 +\evid{src/htr/evaluation/stylometry.py}.}
50 +
51 +\paragraph{Verbatim reconstruction.} chrF++ \citep{popovic2015chrf} at sentence level (sacreBLEU, character
52 +$n$-grams with word bigrams), normalised Levenshtein similarity after whitespace and case normalisation, exact
53 +match, sentence BLEU \citep{papineni2002bleu} and ROUGE-L $F$ \citep{lin2004rouge}, and the word-length ratio.
54 +Overlap metrics reward copying, which is why they are never reported without B0. Embedding similarity
55 +\citep{reimers2019sentencebert,zhang2019bertscore} is a secondary semantic measure.
56 +
57 +\paragraph{Factual integrity.} With $\mathcal{N}(\cdot)$ the multiset of numbers extracted from a text (thousands
58 +separators, decimals, percentages and scientific notation included) and $\mathcal{P}(\cdot)$ the multiset of
59 +placeholders,
60 +\begin{equation}
61 + \mathrm{NumRec} = \frac{|\mathcal{N}(\hat y) \cap \mathcal{N}(y)|}{|\mathcal{N}(y)|}, \qquad
62 + \mathrm{NumPrec} = \frac{|\mathcal{N}(\hat y) \cap \mathcal{N}(y)|}{|\mathcal{N}(\hat y)|},
63 +\end{equation}
64 +each set to 1 when its denominator is empty, together with placeholder exactness
65 +$\mathbb{1}[\mathcal{P}(\hat y) = \mathcal{P}(y)]$ and recall, and the signed differences in negation cues, hedge
66 +cues and sentence counts. These checks cover citations, equations and links through the placeholders, and numbers
67 +directly.
68 +
69 +\paragraph{Stylistic distribution matching.} Eighteen stylometric features are computed per passage: mean and
70 +standard deviation of sentence length, type--token ratio, MTLD, mean word length, long-word rate, comma, semicolon
71 +and parenthesis rates, discourse-marker rate, passive rate, nominalisation rate, first-person rate, subordination
72 +rate, hedge rate, negation rate, digit rate, and Flesch reading ease. For two sets of passages with feature samples
73 +$\{u\}$ and $\{r\}$, the distance on feature $j$ is a quantile-based 1-Wasserstein distance standardised by a
74 +spread $\varsigma_j$,
75 +\begin{equation}
76 + W_{1,j} = \frac{1}{|Q|}\sum_{q \in Q} \frac{\big|\hat F^{-1}_{u,j}(q) - \hat F^{-1}_{r,j}(q)\big|}{\varsigma_j},
77 + \qquad Q = \{0, 0.005, \dots, 1\},
78 +\end{equation}
79 +where $\varsigma_j = \max\big(\mathrm{sd}_j, 0.05(|\mathrm{mean}_j| + 10^{-9}), 10^{-6}\big)$ is computed on a
80 +reference pool so that near-constant features cannot dominate, and $\overline{W}_1$ is the mean over the 18
81 +features. For H4, $\overline{W}_1(\text{reconstructions}, \text{originals})$ is compared with
82 +$\overline{W}_1(\text{rewrites}, \text{originals})$; both distances must use the same scale $\varsigma$, estimated
83 +on the originals and rewrites of the evaluation set. These metrics are distributional: they ask whether
84 +reconstructions as a set resemble originals as a set, which is the meaningful notion of style recovery under
85 +non-identifiability, and they can reveal regression toward a mean style that per-item metrics hide.
86 +
87 +\paragraph{Exploratory style probe.} A logistic regression on standardised stylometric features is trained to
88 +separate originals from rewrites and then applied to reconstructions; we report its held-out AUC and the mean
89 +predicted probability of ``original'' on reconstructions. The probe must be fitted on a split disjoint from the
90 +evaluated items. Its scores describe feature distributions and are never evidence of human authorship.
91 +
92 +\subsection{Statistical analysis}
93 +
94 +Items are not independent: one document contributes several passages, and one passage up to six rewrites. All
95 +intervals therefore come from a document-clustered bootstrap.\footnote{\evid{src/htr/evaluation/bootstrap.py},
96 +\evid{src/htr/evaluation/runner.py}.} In each replicate, documents are drawn with replacement and every item of a
97 +drawn document enters the resample. For a paired difference $d_i = \mathrm{score}_i(M^\ast) -
98 +\mathrm{score}_i(\mathrm{B0})$, the 95\% interval is the percentile interval of the replicate means $\bar d^\ast$,
99 +and the two-sided $p$-value is $\min\big(1, 2\min(\hat P, 1 - \hat P)\big)$ with $\hat P$ the share of replicates
100 +with $\bar d^\ast \le 0$. Confirmatory analyses use 10\,000 replicates (the development default is 1\,000). Results
101 +are reported with means, intervals, and the numbers of items and documents, overall and by variant, discipline,
102 +era and section role. Holm corrections and decision rules are those of Section~\ref{sec:rq}.
103 +
104 +\subsection{Experiments}
105 +
106 +Table~\ref{tab:experiments} maps the experiments to the research questions. Experiments marked ``code ready'' have
107 +an implementation and tests but no reported run; all results are \pending{experiment results}.
108 +
109 +\begin{table}[t]
110 +\centering
111 +\caption{Experiment matrix (\texttt{docs/experiments/EXPERIMENT\_MATRIX.md}). Seeds refer to training seeds.}
112 +\label{tab:experiments}
113 +\small
114 +\begin{tabularx}{\textwidth}{@{}l>{\raggedright\arraybackslash}p{0.09\textwidth}>{\raggedright\arraybackslash}X>{\raggedright\arraybackslash}p{0.2\textwidth}@{}}
115 +\toprule
116 +ID & RQ & Design & Status \\
117 +\midrule
118 +E1 & RQ1 & Paired feature shift original $\to$ rewrite per variant (18 features, Holm) & pairs available \\
119 +E2 & -- & Pilot rewrite, validation and review (Gate B) & done (Table~\ref{tab:pilot-jobs}) \\
120 +E3 & RQ2 & B0, B1, B2 on the test split & code ready \\
121 +E4 & RQ2, RQ5 & M1 vs M2 vs M3 at tiny size, 3 seeds & running / \pending{E4} \\
122 +E5 & RQ5 & Ablations: token loss only, no style, no adversary, no factual, no contrastive, no copy; M1 with text metadata prefix & code ready \\
123 +E6, E7 & RQ4 & Held-out prompts \texttt{H1}, \texttt{H2}; held-out models Sonnet 5.5, Haiku 5.5 & pairs available \\
124 +E8 & RQ4 & Train on variants A--D, test on E & code ready \\
125 +E9 & RQ3 & Held-out discipline and held-out period & code ready \\
126 +E10, E11 & RQ5 & Scaling in model size (tiny to base, large if feasible) and data ($\tfrac14, \tfrac12, 1$ of the pairs, then Medium) & needs Medium data \\
127 +E12, E13 & RQ2, RQ3 & Gains by rewrite strength (deciles of rewrite--original similarity, variant) and input length & analysis code ready \\
128 +E14 & RQ6 & Non-identifiability: within-original agreement, rewrite ambiguity, oracle ceiling & code ready \\
129 +E15 & RQ6 & Memorisation: nearest-neighbour overlap, verbatim rate, canaries; contamination audit for B2 & code ready \\
130 +E16 & RQ2, RQ6 & Blind human evaluation & planned; ethics approval pending \\
131 +E17 & RQ6 & Error taxonomy on $\geq 200$ quarantined and worst-scoring items & planned \\
132 +E18 & -- & Calibration of token log-probability against item chrF++ & planned \\
133 +E19 & RQ2 & Exploratory style probe on a disjoint probe split & code ready \\
134 +\bottomrule
135 +\end{tabularx}
136 +\end{table}
137 +
138 +\subsection{Non-identifiability}
139 +\label{sec:design-ident}
140 +
141 +Three measurements quantify how much of the original a rewrite determines.\footnote{\evid{src/htr/evaluation/identifiability.py}.}
142 +\begin{enumerate}
143 + \item \textbf{Within-original agreement.} Variant E yields two independent rewrites $x^{(1)}, x^{(2)}$ of the same
144 + original. We compare $\mathrm{chrF}(x^{(1)}, x^{(2)})$ and $\mathrm{chrF}(x^{(1)}, y)$ with the corresponding
145 + quantities for reconstructions, $\mathrm{chrF}(\hat y^{(1)}, \hat y^{(2)})$ and
146 + $\tfrac12[\mathrm{chrF}(\hat y^{(1)}, y) + \mathrm{chrF}(\hat y^{(2)}, y)]$. A model that converges on the same
147 + reconstruction from different rewrites while staying far from $y$ has learned a mode of $p(y \mid x)$, not $y$.
148 + \item \textbf{Rewrite ambiguity.} Among rewrites of variants A, B and E from \emph{distinct} passages, we find
149 + pairs whose MinHash Jaccard similarity over word 5-grams is at least 0.5 and measure the chrF and edit similarity
150 + between their originals. The distribution of these distances, which involves no model, is the empirical
151 + counterpart of the lower bound of Section~\ref{sec:models}.
152 + \item \textbf{Oracle ceiling.} The chrF between two originals that an LLM mapped to near-identical rewrites bounds
153 + the expected chrF of a perfect deterministic reconstructor on those items.
154 +\end{enumerate}
155 +
156 +\subsection{Memorisation and contamination}
157 +
158 +A reconstruction model could score well by recalling training originals. For each reconstruction on an evaluation
159 +set we find the nearest training original by MinHash over word 5-grams and record the Jaccard similarity and the
160 +longest common word $n$-gram; we report the mean maximum Jaccard, the share of outputs with Jaccard at least 0.5 and
161 +0.8, the share with a common $n$-gram of length at least 12, and the number of outputs whose nearest neighbour is
162 +their own passage, which must be zero under correct splits.\footnote{\evid{src/htr/evaluation/memorization.py}.}
163 +Synthetic canaries (invented passages with unique markers, containing no real data) are inserted into training a
164 +fixed number of times, and their emission rate at test time is measured. For B2, the documented pretraining data
165 +are audited for the originals where possible \citep{magar2022contamination,sainz2023contamination}.
166 +
167 +\subsection{Human evaluation}
168 +
169 +A blind evaluation will be run on 120 test passages stratified by discipline and variant, with three
170 +domain-literate raters and four conditions per item: the original, B0, B1 and the best model. A Latin square
171 +ensures that each rater sees each item in exactly one condition and each condition equally often; all texts are
172 +shown in placeholder-restored form and raters are not told which is the original. Raters score, on five-point
173 +scales, fidelity to the meaning of the source (the rewrite is shown as reference), fluency, historical-style
174 +authenticity (``plausibly written before 2018''), and factual safety, flagging any changed number, citation,
175 +negation or hedge (also checked automatically). Agreement is reported with Krippendorff's $\alpha$, and condition
176 +effects with an ordinal mixed model with rater and item random effects. A 20-item pilot calibrates raters.
177 +Recruitment and ethics approval at UQO are \pending{ethics approval}.
178 +
179 +\subsection{Compute and cost reporting}
180 +
181 +We report, per run, parameters, tokens, accelerator time, device, the $6ND$ FLOPs estimate and, where measured,
182 +energy; energy that is not measured is stated as such. API spend is reported from the job ledgers. Training runs
183 +on Apple-silicon machines with PyTorch's MPS backend; the hardware and accelerator-hours of each reported run are
184 +\pending{compute table}.
added scripts/build_medium_corpus.sh +17 −0
@@ -0,0 +1,17 @@
1 +#!/usr/bin/env bash
2 +# Assemble the Medium-stage corpus from every raw source directory, with splits that preserve the pilot assignments
3 +# (documents already rewritten keep their split). Usage: scripts/build_medium_corpus.sh [out_name] [heldout_disciplines] [heldout_before_year]
4 +set -euo pipefail
5 +cd "$(dirname "$0")/.."
6 +PY=.venv/bin/python
7 +NAME=${1:-medium_v1}
8 +HELD=${2:-physics,earth_sciences}
9 +BEFORE=${3:-1900}
10 +RAW=(data/raw/pmc data/raw/arxiv data/raw/crossref data/raw/books/gutenberg data/raw/books/bookshelf data/raw/books/openstax)
11 +EXISTING=()
12 +for d in "${RAW[@]}"; do [ -d "$d" ] && EXISTING+=("$d"); done
13 +echo "raw dirs: ${EXISTING[*]}"
14 +$PY scripts/build_corpus.py --raw "${EXISTING[@]}" --out "data/parsed/$NAME" --tag "$NAME"
15 +$PY scripts/relabel_disciplines.py --corpus "data/parsed/$NAME"
16 +$PY scripts/make_splits.py --corpus "data/parsed/$NAME" --heldout-disciplines "$HELD" --heldout-before-year "$BEFORE" --fixed data/splits/pilot_pmc/splits.jsonl
17 +echo "MEDIUM CORPUS BUILT: data/parsed/$NAME ; splits data/splits/$NAME/splits.jsonl"
18
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere