spb/wpai0002_uqo

1% ============================================================================2% Rewrite protocol3% ============================================================================4\providecommand{\ph}[1]{\texttt{[[#1]]}}5\providecommand{\evid}[1]{\texttt{\detokenize{#1}}}67\section{Rewrite Protocol}8\label{sec:rewrite}910The LLM side of each pair is generated under control: a pinned model, versioned prompts, logged sampling11settings, recorded costs, and automated integrity checks whose failures are quarantined rather than dropped.1213\subsection{Provider and logging}1415Rewrites are produced by Anthropic's Claude Opus~5.5 (model identifier \texttt{claude-opus-5-5}), the primary16rewriter. Claude Sonnet~5.5 (\texttt{claude-sonnet-5-5}) and Claude Haiku~5.5 (\texttt{claude-haiku-5-5}) are17reserved as held-out models for RQ4 and never contribute training pairs.\footnote{\evid{configs/rewriting/pilot.json},18\evid{src/htr/rewrite/anthropic_provider.py}.} Requests set the effort parameter to \texttt{low} and19\texttt{max\_tokens} to 1\,500; thinking is left at the provider's adaptive default and no temperature is set, so20sampling uses the provider defaults. Pilot jobs are submitted through the Message Batches interface, billed at2150\% of the standard price. Every response is logged with the model identifier returned by the API, the request22identifier, token usage, stop reason, computed cost and timestamps. Request identifiers are deterministic functions23of (passage, variant, sample index, provider, model, prompt version), which deduplicates requests and makes jobs24resumable; results are written to an append-only file. The default provider is a zero-cost mock; real calls25require an explicit flag and a budget ceiling, which was 30~USD for the pilot and is enforced on recorded spend.2627\subsection{Variants and prompts}2829Each prompt consists of a system message, a variant-specific instruction, a common preservation block, and the30passage in placeholder form wrapped in \texttt{<passage>} tags. The system message states that placeholders stand31for citations, equations and links and must be copied verbatim, exactly once per occurrence. The preservation block32requires every number, unit, statistical value, named entity, citation marker, equation, URL, negation, hedge and33expression of uncertainty to be preserved, forbids adding, removing or reordering facts, and requires a single34English paragraph with no commentary. Table~\ref{tab:variants} lists the variants; the full prompts are reproduced35verbatim in Appendix~\ref{app:prompts}. The prompt set is versioned (\texttt{rw-1.0}), and any change to a prompt36string requires a new version.3738\begin{table}[t]39\centering40\caption{Rewrite variants (prompt version \texttt{rw-1.0}). Variants A--E produce training and evaluation pairs;41\texttt{H1} and \texttt{H2} are used only on test passages for the held-out-prompt test.}42\label{tab:variants}43\small44\begin{tabularx}{\textwidth}{@{}l>{\raggedright\arraybackslash}p{0.2\textwidth}>{\raggedright\arraybackslash}Xc@{}}45\toprule46ID & Name & Instruction (summary) & Samples \\47\midrule48A & neutral paraphrase & paraphrase in neutral academic English; wording and structure may change freely & 1 \\49B & polished, standardised & polished, standardised contemporary academic prose; improve clarity, flow and concision & 1 \\50C & concise & about 70\% of the original length without dropping any fact, number, citation or qualification & 1 \\51D & structure-preserving & same number of sentences and order of ideas; rephrase each sentence & 1 \\52E & diverse sampling & fluent academic paragraph in own words; vary the wording naturally & 2 \\53\midrule54\texttt{H1} & plain-language expert & plain, direct language with short sentences for an expert reader & held out \\55\texttt{H2} & editorial polish & journal copy-editor editing for publication & held out \\56\bottomrule57\end{tabularx}58\end{table}5960The variants span a range of rewrite strength. A and E change wording freely; B pushes toward contemporary61academic register, the condition closest to the ``polishing'' studied by \citet{saha2025polished}; C compresses62and therefore must delete or merge material without dropping propositions; D constrains structure, so that63sentence-level alignment between original and rewrite is approximately preserved. Variant E is sampled twice64independently per passage, which yields two distinct rewrites of the same original for the non-identifiability65analysis (Section~\ref{sec:design}).6667\subsection{Automated pair validation}6869Each (original, rewrite) pair is checked by a versioned validator (\texttt{val-1.0}).\footnote{\evid{src/htr/validate/checks.py}.}70A pair passes only if all of the following hold, comparing the rewrite with the placeholder form of the original71that was sent to the model:72\begin{itemize}73  \item the rewrite is non-empty and identified as English;74  \item the word-length ratio is in $[0.5, 1.8]$ ($[0.4, 1.8]$ for variant C);75  \item the normalised Levenshtein similarity is in $[0.05, 0.95]$ (above 0.95 counts as a near-verbatim copy,76  below 0.05 as unrelated text);77  \item the multiset of placeholders is identical;78  \item no number of the original is missing and at most one number is added;79  \item recall of a proxy set of named entities and technical terms (capitalised non-initial tokens, acronyms,80  alphanumeric tokens such as gene names) is at least 0.6;81  \item the counts of negation and hedge cues change by at most 2 and 4, respectively;82  \item for variant D, the sentence count differs by at most one;83  \item there are no formatting or meta-text artefacts (Markdown emphasis, bullets, headings, echoed tags,84  ``Here is'' preambles);85  \item when the local sentence-embedding model is available, the cosine similarity of MiniLM embeddings is at86  least 0.70.87\end{itemize}88Failed pairs are quarantined with their reasons and counted in every acceptance rate. Bidirectional entailment,89which the mandate lists as a desirable check, is not implemented in this version. The entity and number checks are90heuristics: spelled-out numbers are counted as missing, and entity matching is a proxy.9192\subsection{Pilot rewrite job (Gate B)}9394The pilot job rewrote 974 passages selected by stratified sampling over discipline and section role within each95split (700 train, 90 validation, 150 test, 30 held-out domain, and 4 held-out period, the only eligible passages in96pre-1950 PMC documents). Each passage received one generation for variants A--D and two for E, giving 5\,84497requests. Table~\ref{tab:pilot-jobs} reports the job together with the two held-out generalisation sets, which use98test passages only.\footnote{\evid{docs/gates/GATE_B.md}; ledgers \evid{data/pairs/pilot_opus/ledger.json},99\evid{data/pairs/pilot_heldout_models/}, \evid{data/pairs/pilot_heldout_prompts/}.}100101\begin{table}[t]102\centering103\caption{Pilot rewrite jobs: requests, outcomes, automated validation and spend (Message Batches pricing, from the104job ledgers).}105\label{tab:pilot-jobs}106\small107\begin{tabularx}{\textwidth}{@{}>{\raggedright\arraybackslash}p{0.2\textwidth}>{\raggedright\arraybackslash}Xrrrr@{}}108\toprule109Job & Model / prompts & Requests & OK / refused & Validated & USD \\110\midrule111\texttt{pilot\_opus} & Opus 5.5, variants A--E & 5\,844 & 5\,801 / 43 & 5\,383 (92.1\%) & 26.85 \\112\texttt{pilot\_heldout\_}\linebreak[0]\texttt{models} & Sonnet 5.5 and Haiku 5.5, variants A, B & 600 & 600 / 0 & 531 (88.5\%) & 1.43 \\113\texttt{pilot\_heldout\_}\linebreak[0]\texttt{prompts} & Opus 5.5, prompts \texttt{H1}, \texttt{H2} & 300 & 297 / 3 & 266 (88.7\%) & 1.40 \\114\midrule115Total & & 6\,744 & & & 29.68 \\116\bottomrule117\end{tabularx}118\end{table}119120For the main job, all seven batches ended without errors or truncations; 43 requests (0.7\%) ended with a refusal121stop reason (for example on passages about hepatitis~B virus cccDNA and CRISPR), and are recorded as abstentions without resubmission. Measured usage was122691 input and 323 output tokens per request on average, above the pre-flight estimate of 492 and 259, and the job123cost 26.85~USD against the 30~USD ceiling. Validation accepted 5\,383 of 5\,844 pairs (92.1\%): 896 of 974 for A,124884 for B, 894 for C, 905 for D and 1\,804 of 1\,948 for E. The largest quarantine reason is at least one missing125number (318 pairs), followed by refusals (43), entity recall below threshold (39), extra numbers (31), formatting126artefacts (21) and negation changes (11). Total API spend to date across all jobs is 29.68~USD.127128\paragraph{Qualitative audit.} A side-by-side reading of 12 validated pairs (two per variant, three for E, across129six discipline labels) was carried out by the project's operating assistant during Gate~B; it is \emph{not} an130independent expert review. In that sample no rewrite added or removed a claim; variant~C shortened passages by13118--21\% without dropping facts and variant~D kept sentence counts; rewrites were uniformly smoother and more132nominal, and spelling was sometimes normalised. The audit of quarantined pairs led to three validator corrections133(token normalisation for entity matching, extraction of numbers glued to units, and the versioning of the134placeholder protocol described in Section~\ref{sec:data-segments}) and to the rejection of Markdown italics. Residual135quarantine is dominated by genuine number omissions, for example one of several $p$-values dropped from a long136results paragraph. The stratified expert review of at least 100 pairs required by the project mandate is137\pending{expert review of $\geq 100$ stratified pairs}.138139\paragraph{Scaling the rewrite.} Rewriting the Medium-stage corpus requires a separate budget approval. At the140measured token usage, 20\,000 passages with six generations each would cost about 550~USD with Opus~5.5 in batch141mode. The scale, model and spend of the Medium-stage job are \pending{Medium-stage rewrite job}.142