spb/wpai0002_uqo

1% ============================================================================2% Data and rights3% ============================================================================4\providecommand{\ph}[1]{\texttt{[[#1]]}}5\providecommand{\evid}[1]{\texttt{\detokenize{#1}}}67\section{Data and Rights}8\label{sec:data}910The originals are English scientific and scholarly passages published before 1~January 2018. The cut-off11precedes the public release of instruction-tuned LLMs by several years, so the originals can be treated as12human-authored prose without relying on a detector. Two constraints govern every step: the publication date must be13verified from the document or an official metadata record, and the document's own licence must permit the14operations we perform on it. Open access is not open licensing.1516\subsection{Sources and acquisition}1718Acquisition is metadata-first: candidate records are screened on licence and date before any full text is19downloaded, and every attempt is logged. Only official bulk channels and APIs are used, with \texttt{robots.txt}20honoured and at least two seconds between requests to the same host. Table~\ref{tab:sources} lists the sources.2122\begin{table}[t]23\centering24\caption{Corpus sources, licence and date evidence, and status at this version. Pools are counts of25candidate documents that passed metadata screening; documents actually parsed into the Medium-stage corpus are26\pending{Medium-stage counts}.}27\label{tab:sources}28\small29\begin{tabularx}{\textwidth}{@{}>{\raggedright\arraybackslash}p{0.17\textwidth}>{\raggedright\arraybackslash}X>{\raggedright\arraybackslash}X>{\raggedright\arraybackslash}p{0.2\textwidth}@{}}30\toprule31Source (channel) & Licence evidence & Date evidence & Status / measured \\32\midrule33PMC Open Access (AWS Open Data bucket \texttt{pmc-oa-}\linebreak[0]\texttt{opendata}) & per-article JSON \texttt{license\_code}; JATS \texttt{<license>} or public-domain statement overrides the provider field & JATS \texttt{<pub-date>}, print/collection before electronic; deposit dates never used & pilot built: 1\,180 documents kept \\34arXiv (OAI-PMH \texttt{arXivRaw} + e-prints) & record \texttt{<license>} URI; absent $\to$ unknown; default arXiv licence excluded & v1 submission date; latest version submitted before 2018 requested & adapter implemented; pool \pending{arXiv pool} \\35Crossref (16 verified open-access economics and finance journals) & \texttt{license[]} entry applying to the version of record & \texttt{published-print} $\to$ \texttt{published-online} $\to$ \texttt{issued} & 1\,730 eligible pre-2018 CC-licensed articles \\36Project Gutenberg (catalog feed + official mirror) & public domain in the US; eBook header checked & printed front matter; else latest contributor death year (upper bound, flagged) & 7\,799 English non-fiction texts in pool \\37NCBI Bookshelf (Literature Archive, FTP) & BITS \texttt{<permissions>} per book & file list year and XML date & 3\,941 pre-2018 books in pool \\38OpenStax/\linebreak[0]Connexions (Internet Archive) & Creative Commons statement inside the PDF & in-file date only; upload time never used & trial only \\39\bottomrule40\end{tabularx}41\end{table}4243\paragraph{PMC.} The PMC FTP packages and the Open Access Web Service were retired in August 2026; the official44bulk channel is now the PMC Article Datasets bucket on AWS Open Data, which provides per-article JSON metadata45(\texttt{is\_pmc\_openaccess}, \texttt{is\_retracted}, \texttt{license\_code}) and JATS XML with MD5 digests that we46verify on download.\footnote{Adapter \evid{src/htr/ingest/pmc_s3.py}; bucket layout retrieved 2026-11-09.}47Candidate PMCIDs are sampled in an identifier range used only as a sampling prior; the publication date is48verified per document from the JATS front matter. Author manuscripts and articles outside the open-access subset49are skipped.5051\paragraph{Economics and finance.} PMC contains few economics and finance articles: the pilot corpus has only four52documents labelled economics. These disciplines therefore come from two other adapters. The arXiv adapter53enumerates the \texttt{q-fin} and \texttt{econ} sets through OAI-PMH. Because OAI date windows select on the54record's last metadata change rather than its submission date, the v1 date is filtered locally, and the e-print of55the latest version submitted before 2018 is requested so that the text itself predates the cut-off. The Crossref56adapter covers 16 open-access economics and finance journals whose Crossref records, licences and full-text routes57were verified live; it counts 1\,730 eligible pre-2018 Creative Commons articles.\footnote{\evid{configs/data/crossref_journals.json},58\evid{data/manifests/discover_crossref_count.json}.}5960\paragraph{Reference books.} Pre-2018 English reference and scholarly books form a separate stratum. Project61Gutenberg texts are selected from the official catalog (English, text type, non-fiction Library of Congress classes62in science, medicine, technology, social sciences, geography, philosophy, education and bibliography; any fiction63class or subject excludes a record) and fetched from the official mirror, as the project's robot-access policy64requires. NCBI Bookshelf books are taken from the Literature Archive via FTP, one document per body chapter. For65Gutenberg, a date is taken from the printed front matter when present; otherwise the latest death year among the66catalogued contributors is used as an upper bound and flagged. The books stratum is the main source of pre-195067prose, because pre-1950 PMC documents are back-scanned OCR text that rarely passes the quality filters68(Section~\ref{sec:data-segments}).6970\subsection{Licence policy}7172Each document carries a canonical licence code with its evidence (the provider field or the URI found in the73document), and the operations it permits follow a single policy module.\footnote{\evid{src/htr/licenses/policy.py};74\evid{docs/data_governance/LICENSE_MATRIX.md} describes it, and the code prevails in case of disagreement.} When a75document's own licence statement disagrees with the provider metadata, the document statement is used and the76mismatch is recorded. Table~\ref{tab:licence-tiers} summarises the tiers. A document is used for training only if77its code is in the configured training set and permits training; derivative text is released only if its code is78in the release set.7980\begin{table}[t]81\centering82\caption{Licence tiers and default permissions (\texttt{htr.licenses.policy}).}83\label{tab:licence-tiers}84\small85\begin{tabularx}{\textwidth}{@{}l>{\raggedright\arraybackslash}p{0.27\textwidth}cc>{\raggedright\arraybackslash}X@{}}86\toprule87Tier & Codes & Train & Release & Conditions \\88\midrule89A & CC0, PUBLIC-DOMAIN, CC-BY & yes & yes & attribution (CC-BY); public-domain status verified per jurisdiction (US unless stated) \\90B & CC-BY-SA, CC-BY-NC, CC-BY-NC-SA & yes & no & derivatives follow ShareAlike or non-commercial terms; released, if ever, only in separate bundles \\91C & CC-BY-ND, CC-BY-NC-ND, TDM, ARXIV-NONEXCLUSIVE & no & no & ingest, parse and internal evaluation only \\92X & OTHER-OA, UNKNOWN & no & no & metadata only \\93\bottomrule94\end{tabularx}95\end{table}9697\paragraph{Observed distributions.} Licences are heterogeneous even within an ``open access'' collection. In the98pilot discovery run on PMC, 4\,187 sampled identifiers returned 1\,820 metadata objects with the licence99distribution of Table~\ref{tab:pmc-licences}.\footnote{\evid{data/manifests/discover_pmc_seed2.json}; counts are100over every metadata object found, before the open-access, retraction, manuscript and licence filters.} Under the101default policy, 66.2\% of these objects are training-eligible and 47.7\% release-eligible. Text-mining-only author102manuscripts (TDM) are the second largest group and are excluded from training.103104\begin{table}[t]105\centering106\caption{Licence codes of the 1\,820 PMC metadata objects found in the pilot discovery run (seed 2).}107\label{tab:pmc-licences}108\small109\begin{tabular}{@{}lrrc@{}}110\toprule111Licence code & Objects & Share (\%) & Tier \\112\midrule113CC-BY & 842 & 46.3 & A \\114TDM & 405 & 22.3 & C \\115CC-BY-NC & 210 & 11.5 & B \\116UNKNOWN & 137 & 7.5 & X \\117CC-BY-NC-SA & 126 & 6.9 & B \\118CC-BY-NC-ND & 73 & 4.0 & C \\119CC0 & 26 & 1.4 & A \\120CC-BY-ND & 1 & 0.1 & C \\121\midrule122Total & 1\,820 & 100.0 & \\123\bottomrule124\end{tabular}125\end{table}126127\subsection{Parsing, protection and segmentation}128\label{sec:data-segments}129130JATS (articles) and BITS (books) XML are parsed into a document--section--paragraph--sentence hierarchy with131character offsets; LaTeX e-prints, plain text and PDF have dedicated parsers. Reference lists, acknowledgements,132funding, author contributions, ethics statements and supplementary sections are excluded; captions and list items133are extracted but never eligible. Section titles are mapped to coarse roles (abstract, introduction, background,134methods, results, discussion, conclusion, theory, data, related work, limitations, other; chapter and preface for135books).136137Before any LLM call, spans that must survive rewriting verbatim are replaced by indexed placeholders: citations138(numeric and author--year) by \ph{C$n$}, inline and display mathematics by \ph{M$n$}, and URLs and DOIs by139\ph{U$n$}, numbered in order of appearance.\footnote{\evid{src/htr/parse/protect.py}. In the code the placeholders140are delimited by the mathematical white square brackets U+27E6 and U+27E7; we write them with doubled square141brackets in this paper.} Both the rewrite and the target are stored in placeholder form, and placeholders are142restored at inference. Numbers are deliberately \emph{not} masked: their formatting carries style, and masking them143would hide factual errors. They are instead extracted and checked (Section~\ref{sec:rewrite}). Because a detection144pattern changed after the pilot rewrites were requested, the protocol is versioned: pilot pairs use version 1.0 and145later jobs version 1.1, and each pair is validated with the version used at request time.146147A paragraph becomes an \emph{eligible passage}, the unit that is rewritten and reconstructed, if it belongs to a148main-text role, has 60 to 400 words and at least two sentences, is identified as English with probability at least1490.9, has a quality score of at least 0.6, and is not a list item.\footnote{\evid{src/htr/segment/passages.py}.}150Exact duplicates are removed by hash, and near-duplicates are clustered with MinHash over word 5-gram shingles (128151permutations, Jaccard threshold 0.8).\footnote{\evid{src/htr/dedup/minhash.py}.}152153\subsection{Splits}154155Splits are made \emph{before} any rewriting, so that no rewrite request is ever conditioned on a split decision156made after seeing model outputs.\footnote{\evid{src/htr/data/splits.py}; seed 20261109.} The unit of assignment is157a leakage group: a union-find over near-duplicate clusters and author groups (and optionally venues), so that no158cluster or author group straddles training and evaluation. Groups are assigned deterministically by hashing into159train, validation and test (80/10/10). Whole disciplines can be carved out as a held-out-domain split and documents160before a cut-off year as a held-out-period split. Already-published assignments are frozen so that documents whose161passages have been rewritten never change split.162163\subsection{Measured corpus to date}164165\paragraph{Gate A smoke test.} A 100-document PMC smoke test parsed every document without error; all 100166publication dates were verified from JATS and fall before 2018. The 100 documents yield 2\,964 paragraphs, of which1672\,013 are eligible passages, and the 100-document split has no group leak.\footnote{\evid{docs/gates/GATE_A.md},168\evid{data/parsed/smoke/build_summary.json}.} A manual audit of this run found and fixed two parser defects (a169doubled-bracket citation artefact and a section-title regex that mapped ``Introduction'' to ``other'').170171\paragraph{Pilot corpus.} The pilot PMC corpus was built from 4\,187 sampled identifiers, 1\,820 metadata objects172found and 1\,200 documents fetched. Of these, 1\,180 were kept: 12 exact duplicates, 5 documents dated 2018 and 3173documents excluded by their in-document licence statement were removed. The kept documents contain 34\,695174paragraphs, of which 23\,658 are eligible passages. Their licences are CC-BY (833), CC-BY-NC (198), CC-BY-NC-SA175(124), PUBLIC-DOMAIN (19) and CC0 (6), so 858 of 1\,180 documents (72.7\%) are176release-eligible.\footnote{\evid{data/parsed/pilot_pmc/build_summary.json}.} The pilot split assigns 856 documents177to train, 98 to validation, 100 to test, 47 to the held-out domain and 79 to the held-out period (publication year178before 1950), with no group leak.\footnote{\evid{data/splits/pilot_pmc/splits_summary.json}.} Discipline labels for179PMC are heuristic (journal titles and subject categories). They were corrected after the split (165 of 1\,180180documents relabelled), so the held-out-domain group corresponds to the pre-correction ``chemistry'' label and is181reported as such.182183\paragraph{Medium stage.} The target for the main experiments is the Medium stage: at least 10\,000 documents and184200\,000 eligible passages, including economics and finance articles and pre-2018 reference books, with no single185archive dominating. Ingestion at this scale is in progress; the final composition by source, discipline, era,186section role and licence is \pending{Medium-stage corpus composition table}, and the dataset flowchart is shown187in Figure~\ref{fig:flow}.188189\begin{figure}[t]190\centering191\fbox{\parbox{0.9\textwidth}{\centering\vspace{1.5em}\pending{dataset flowchart: sources $\to$ metadata screening192$\to$ licence and date verification $\to$ parsing $\to$ eligible passages $\to$ deduplication $\to$ splits $\to$193rewrites $\to$ validated pairs, with counts at each stage}\vspace{1.5em}}}194\caption{Dataset flowchart from candidate records to validated pairs. Generated from the build summaries by a195script; counts will be filled from the Medium-stage run.}196\label{fig:flow}197\end{figure}198