SPB Git

spb/wp45_uqo

Public
8.5 KB · 26 lines latex
1\section{Discussion: generalization, research integrity, and limitations}\label{sec:discussion}23\subsection{What the results do and do not show}45The preregistered contrasts separate two ideas that are often conflated in discussions of retrieval-augmented writing. The first is that an assistant writes better when it is told, or shown, what the target genre looks like. Conditions C and D test this idea in two forms---raw excerpts retrieved by similarity, and measured profiles with structurally annotated exemplars---and neither form improves the judge's ratings of clarity, argument logic or contribution positioning relative to a strong generic prompt (Section~\ref{sec:results}). The second idea is that an assistant writes better when it is made to verify its draft against the manuscript's own evidence and to revise it under a referee's standards. Condition E tests this idea and the ablations isolate it: the referee pass accounts for the gains in rigor and fidelity, while removing retrieval, profiles or the deterministic fact-check leaves E essentially unchanged (Section~\ref{sec:ablations}). The gains are not free on every dimension: the judge rates E's more cautious text as slightly less useful than D's and as less sharply positioned than B's, with intervals that include zero. A referee-checked draft says less, flags more, and leaves more for the author to do; whether that is a cost or a benefit depends on whether the author wants prose or a verified proposal. The deterministic audit still matters in a different way: it supplies the referee pass with concrete items (numbers not in the material, unknown citation keys, unsupported sentences) and it produces the flags that a human author must resolve. Its effect on the judge's scores is small because, in this design, the referee pass sees the audit and would in its absence have to find the same problems itself.67A capable model, then, already knows the genre well enough that profiles and exemplars add little to what a strong prompt elicits. This is consistent with the literature on LLM feedback and writing assistance, in which the limiting factor is seldom fluency and often fidelity \citep{liang2024feedback,du2024critique}. It does not mean that the profiles are without value. They are the only part of the system that a human can read: a measured, provenance-bearing description of how published articles in a field organise their abstracts, introductions, identification arguments and tables. They define what the referee pass is asked to enforce, and they remain available to authors and editors as a reference independent of any model.89\subsection{Generalization}1011Three forms of generalization were built into the design, and each is partial. Across documents: the held-out articles were never used for profile construction or prompt tuning, and the engine's gains on them are of the same kind as on the demonstration manuscripts; but there are only two of them. Across tasks: the gains concentrate on tasks where verification has purchase---referee reports, result interpretation, specification explanations, literature rewrites---and are small on tasks such as abstract improvement where the source already fixes most of the text. Across domains: on the out-of-domain manuscript (a methods paper on autonomous research with a coding agent) the finance profiles do not help and the full engine is not better than the alternatives; the judge's ratings there rest on one document and three tasks and are descriptive only. We take the out-of-domain result as a reminder that measured conventions are specific to the corpus from which they are measured, and that the engine correctly reports when they do not apply.1213\subsection{Research integrity}1415The design treats integrity as a measured outcome rather than an assumption. Every output is screened for eight-gram overlap with every corpus document; excluding the task's own source document, no output shares more than \bmOverlapOtherMax{} of its eight-grams with any article, and the engine's prompts forbid reuse of sentences and imitation of authors. Every number in a generated text is compared with the numbers in the material; the full engine produces somewhat fewer numbers that are absent from the source than the baselines and markedly fewer sentences that assert without an anchor (Section~\ref{sec:results}), and the remaining ones are flagged for the author. Every citation must use a key from the manuscript's bibliography, or in the fixture tasks an entry of the reference list; almost no condition produced an author--year citation that could not be matched, which shows that the constraint was respected but also that this metric barely discriminated between conditions. The copyrighted full texts never leave the machine except as bounded excerpts sent to the annotation API under the account's data terms, and are not committed to the repository. The paper's own tables and figures are generated from recorded runs by scripts; the preregistration was frozen before the runs; costs, model identifiers, prompt versions and request identifiers are logged for every call; and the one post-hoc refinement of a metric (separating overlap with the source document from overlap with other documents) is labelled as such.1617Two integrity caveats remain. The generator and the judge come from the same vendor, and LLM judges are known to prefer certain styles and their own outputs \citep{zheng2023judge,wang2024notfair,panickssery2024self}; we use a different model family member as judge, randomise and blind the presentation, and measure the judge's self-consistency, but we cannot exclude a shared preference for the referee pass's style of hedged, evidence-anchored prose. And the held-out articles may have been in the pretraining data of both models \citep{sainz2023contamination,golchin2024timetravel}; withholding the published introduction and screening overlap mitigates but does not remove the possibility that the generator reproduces structure it has seen.1819\subsection{Limitations}2021The limits of this study are listed in full in the repository and summarised here. The experiment is a pilot: two held-out finance documents, eight tasks, one to two repetitions per cell, sixteen to twenty-four paired units per contrast; it can detect only large effects, and null results for H1 and H2 are evidence of absence only at that scale. Its first run was discarded: two implementation defects---a manuscript loader that dropped most of the material of the Markdown fixtures, and a contradictory framing of two tasks---were found by the agent drafting the results, fixed, logged as Amendment A-004, and the affected cells were re-run and re-judged before any result was written into this paper (Section~\ref{sec:human}). The episode is itself evidence about LLM-assisted research: the defect was invisible in aggregate scores and became visible only when the judge's justifications were read. No human rater evaluated the outputs; the human-rating protocol is prepared and the blinded output sets exist, but the mandate's H1, as stated in terms of human ratings, could not be tested. Condition E uses two generation calls and therefore more tokens than A--D; its gains are not cost-matched, and we report cost and latency as outcomes. The rhetorical annotations are model-made; their agreement with deterministic rules is high (median section-role agreement \roleAgreementMedian), but model--rule agreement is not human inter-rater reliability, and the annotator saw bounded excerpts, so that some conventions (standard-error reporting in table notes, uncertainty in figures) were often recorded as unclear. The extractor is heuristic and tuned to one journal's template; its agreement with an independent extraction is high for sections, tables, figures and references and only approximate for numbered equations. The profiles describe four volumes of one journal and should not be read as norms for finance, let alone for other fields.2223\subsection{Implications for LLM-assisted scholarly writing}2425If the pattern we observe holds more broadly, the useful investment in writing assistants is not in teaching them style but in making them accountable to evidence: deterministic checks that are cheap and reproducible, a revision stage that is asked to behave like a referee, and an output contract---proposal, change log, flags, provenance, diff---that keeps the human author in the loop and the manuscript untouched until the author decides. Measured conventions have a place in such a system as the standard that the referee stage applies and as a document that authors can consult; they are not, on this evidence, a substitute for verification.26
Files · 562fd8309d732538d57e5fce2b766deebeb3f166
Menu

Theme