1% paper/sections/design.tex — WP45 / JoF-2502% Transcribes experiments/preregistration.md (frozen 2026-11-03, before any benchmark run). No results appear here.3\providecommand{\TODOmacro}[1]{\textcolor{red}{[TODO macro: \texttt{#1}]}}4\providecommand{\profval}[1]{\ifcsname #1\endcsname\csname #1\endcsname\else\TODOmacro{#1}\fi}5\providecommand{\profvalc}[1]{\ifcsname #1\endcsname\csname #1\endcsname\else\textcolor{red}{\scriptsize TODO}\fi} % compact fallback for table cells; macro name stays in the source67\section{Experimental design, hypotheses, and preregistration}8\label{sec:design}910This section transcribes the preregistered protocol \texttt{heldout\_\allowbreak{}v1} (\texttt{experiments/\allowbreak{}preregistration.\allowbreak{}md}), which was frozen on 3 November 2026, before any benchmark run, at the code version recorded in the repository history, following the logic of preregistration as a commitment device against analytic flexibility \citep{nosek2018preregistration}. Any departure from it is logged as an amendment in \texttt{docs/\allowbreak{}PROGRESS.\allowbreak{}md}. No result is reported here; Sections~\ref{sec:results} to~\ref{sec:human} report only what was actually run.1112\subsection{Research question and hypotheses}13\label{sec:design-hypotheses}1415The question is whether abstract, domain-specific writing conventions extracted from a pre-ChatGPT finance corpus improve LLM-assisted scholarly writing and review beyond standard prompting and conventional retrieval, while maintaining originality, factual accuracy, and reproducibility. The five hypotheses are directional and are tested as paired contrasts within task $\times$ document (and repetition), with the conditions defined in Table~\ref{tab:conditions}.1617\begin{description}18\item[H1] D (structured profiles and rhetorical retrieval) exceeds B (strong academic-writing prompt) on judge-rated \emph{argument clarity} and \emph{contribution positioning}. The mandate states H1 in terms of blind human ratings; because human raters were not available in this study (Section~\ref{sec:design-human}), H1 can only be tested here with the model judge and is reported as such.19\item[H2] D exceeds C (conventional retrieval over raw excerpts) on the finance writing tasks, on the same two outcomes and on \emph{usefulness}.20\item[H3] E (D plus verification and a specialist referee pass) exceeds D on \emph{methodological rigour} and \emph{fidelity}, with no loss in numeric preservation, that is, the preserved rate of E is at least that of D.21\item[H4] E has fewer \emph{unverifiable citations} (author--year strings absent from the source reference list) and fewer \emph{unsupported-claim sentences} than A, B, and C.22\item[H5] Any advantage of D or E persists on held-out documents and on the out-of-domain manuscript, and is not explained by overlap: the maximum eight-word overlap share with any corpus document is below 0.02 for every output.23\end{description}2425\subsection{Conditions and ablations}26\label{sec:design-conditions}2728The five conditions form a ladder from a minimal prompt to the full engine, so that each step isolates one ingredient (Table~\ref{tab:conditions}). Conditions B, C, and D share the same strong-prompt instruction; C and D differ only in what is retrieved and from which units (raw excerpts versus structured annotation units with a profile summary); E adds the two post-draft stages. Four ablations of E remove one ingredient at a time. All prompts are listed verbatim in Appendix~\ref{app:prompts}.2930\begin{table}[htbp]31\centering32\small33\caption{Experimental conditions and ablations (\texttt{config/\allowbreak{}evaluation.\allowbreak{}yaml}; prompts in Appendix~\ref{app:prompts}). Retrieval is restricted to the training split.}34\label{tab:conditions}35\begin{tabularx}{\linewidth}{>{\raggedright\arraybackslash}p{0.23\linewidth}X}36\toprule37Condition & Definition \\38\midrule39A \texttt{baseline} & Shared system prompt; one-line instruction (``Write the \textit{section} of this paper''). \\40B \texttt{strong\_\allowbreak{}prompt} & A plus an experienced-economist writing instruction (question early, explicit contribution, closest prior work without a laundry list, preview of findings with magnitudes, proportionate claims, plain English, one idea per paragraph, manuscript's own notation and numbers). \\41C \texttt{conventional\_\allowbreak{}rag} & B plus six raw excerpts (at most 400 characters each: section openings, table notes, captions) retrieved by hybrid similarity, presented as examples of tone and organisation only. \\42D \texttt{profiles\_\allowbreak{}retrieval} & B plus the measured profile summary (sample size, caveat, length and move distributions, contribution placement, identification, results, robustness, table and conclusion conventions) and six structured rhetorical units with provenance; a four-step procedure (evidence, move sequence stated in the change log, draft, flags); genre note when applicable. \\43E \texttt{full\_\allowbreak{}engine} & D plus a deterministic fact-check of the draft (numeric preservation, citation keys, unsupported claims, corpus overlap) and a specialist-referee revision pass that must fix every audit item while keeping every verified fact; the audit is re-run on the revised text. \\44\midrule45E \texttt{no\_\allowbreak{}retrieval} & E without the six structured units. \\46E \texttt{no\_\allowbreak{}profile} & E without the profile summary. \\47E \texttt{no\_\allowbreak{}referee} & E without the referee pass (draft plus audit only). \\48E \texttt{no\_\allowbreak{}factcheck} & E without the deterministic audit (the referee pass receives no audit). \\49\bottomrule50\end{tabularx}51\end{table}5253\subsection{Tasks and materials}54\label{sec:design-tasks}5556The experimental unit is a task $\times$ source document $\times$ condition $\times$ repetition. Source documents are drawn from the frozen held-out split (\nHeldout{} documents never used for profiles or prompt tuning): two empirical articles of different subfields, selected by the fixture script (seed 45) among held-out empirical articles with at least six tables, plus one out-of-domain manuscript, UQO Working Paper 29, a methods-oriented, computer-science-flavoured LaTeX paper by the author with verified results. The fixture manifest records, for each selected article, its identifier, DOI, source hash, and table and reference counts, together with the code version at which the fixtures were built. For each held-out article the fixture contains a Markdown manuscript built from the parsed structure in which the published introduction is withheld, a copy with the introduction for the judge's reference material where the task permits, the published introduction as a separate file, and metadata; no fabricated finance findings are used anywhere. The eight tasks follow the mandate (Table~\ref{tab:tasks}); each is implemented as a section role, an engine mode, and a research brief that states what material may be used and what must not be invented. The degraded literature passage used by the rewrite task is built deterministically from the paper's own literature paragraphs, so that the task has a known ground truth of admissible citations.5758\begin{table}[htbp]59\centering60\small61\caption{Evaluation tasks (\texttt{evaluation.\allowbreak{}py}, \texttt{TASKS}). ``Mode'' is the engine mode; the brief is paraphrased. The published introduction is withheld from every finance fixture.}62\label{tab:tasks}63\begin{tabularx}{\linewidth}{>{\raggedright\arraybackslash}p{0.22\linewidth}>{\raggedright\arraybackslash}p{0.17\linewidth}lX}64\toprule65Task & Section role & Mode & Brief (summary) \\66\midrule67\texttt{draft\_\allowbreak{}introduction} & introduction & write & Draft the introduction from the verified material (abstract, section structure, tables, figures, references); no numbers beyond the material. \\68\texttt{rewrite\_\allowbreak{}literature} & literature & upgrade & Rewrite a deterministically degraded ``laundry-list'' positioning passage into paragraphs that relate, differentiate, and build on the cited work, citing only works in the reference list. \\69\texttt{explain\_\allowbreak{}specification} & empirical design & upgrade & Improve the explanation of the research design: estimand, identifying assumptions, fixed effects, inference, as stated; add no specification. \\70\texttt{interpret\_\allowbreak{}results} & results & write & Interpret the main results from the tables' captions and notes and the stated facts; economic and statistical significance only as far as supported; never invent coefficients. \\71\texttt{improve\_\allowbreak{}table} & results & upgrade & Propose an improved caption and notes for the first main regression table (sample, unit of observation, period, definitions, standard errors, significance conventions) without inventing values. \\72\texttt{flag\_\allowbreak{}causal\_\allowbreak{}claims} & results & write & Identify every causal claim; classify as supported by the design, over-stated, or associational; name the design element that would support it; propose a bounded rewording. \\73\texttt{referee\_\allowbreak{}report} & results & write & Referee report: summary, contribution, major comments (identification, inference, measurement, robustness, presentation), minor comments, recommendation; every comment tied to a section or table. \\74\texttt{improve\_\allowbreak{}abstract} & abstract & upgrade & Improve the abstract without altering any finding, number, or claim; 100--150 words. \\75\bottomrule76\end{tabularx}77\end{table}7879\subsection{Controls}80\label{sec:design-controls}8182All conditions use the same generator model (\texttt{claude-opus-5-5}), the same reasoning effort (\texttt{medium}) and output limit, and the same manuscript material and brief. Temperature is not a configurable parameter of this model, which uses adaptive reasoning; nondeterminism is therefore handled by repetition rather than by fixing a seed. Repetition 1 covers the full grid of conditions and ablations; repetition 2 covers the main conditions A to E on four tasks (introduction, abstract, referee report, interpretation of results). The out-of-domain manuscript is run on three tasks (introduction, abstract, referee report) under the five main conditions only. Cost, tokens, latency, retries, prompt versions, and model identifiers are logged per call. Token budgets are not equalised: condition E makes two generation calls and uses more tokens than A to D; cost and latency are therefore reported as outcomes rather than controlled, and this asymmetry is stated as a limitation.8384\subsection{Outcomes and judging}85\label{sec:design-outcomes}8687The six primary outcomes are judge ratings on a 1--7 scale for clarity, argument logic, contribution positioning, methodological rigour, fidelity, and usefulness, with calibrated anchors (Appendix~\ref{app:prompts}, judge rubric version 1.0). The judge is \texttt{claude-sonnet-5-5}, a different member of the generator's model family, chosen to limit self-preference while keeping the rubric interpretable to the same vendor's models; the judge is blind to condition labels, receives all condition outputs of one (task, document, repetition) set in a random order under random letters together with the task and the source material, is told not to reward length, and returns scores, a count of invented facts, and a one-line justification per output. Judge consistency is measured by re-judging a random 25\% of the sets in a second random order and reporting the mean absolute score difference and the rank correlation between passes; judge biases that this design does not remove (verbosity, style preference, same-vendor affinity) are acknowledged \citep{zheng2023judge, wang2024notfair, panickssery2024self}. Deterministic secondary outcomes are computed for every output without a model: the preserved rate and new rate of numbers relative to the source material, the number of unverifiable author--year citations (surname and year not co-occurring in the source reference list or material), the number of unknown BibTeX keys, the number of unsupported-claim sentences, the maximum eight-word overlap share with the corpus, the number of \texttt{\textbackslash todo} placeholders, latency, cost, and tokens.8889\subsection{Analysis plan}90\label{sec:design-analysis}9192For each hypothesis contrast and outcome, the primary statistic is the mean of the paired differences within (task, document, repetition), with a 95\% cluster bootstrap interval (clusters are task $\times$ document; 2,000 resamples; seed 45) and a paired Wilcoxon signed-rank test. The Holm procedure \citep{holm1979simple} is applied across the six primary outcomes within each hypothesis. Effect sizes are the mean paired difference on the 1--7 scale and Cliff's $\delta$ \citep{cliff1993dominance}. Secondary outcomes use the same paired bootstrap, with counts also reported as rates. Each ablation is compared with full E by the same procedure. The out-of-domain manuscript is reported separately and descriptively, since it is a single document. Significance is declared only when the Holm-adjusted $p$-value is below 0.05 \emph{and} the bootstrap interval excludes zero. The sample, two finance documents and eight tasks, is small by construction: the study is a pilot of the protocol, powered to detect only large effects, and this is stated wherever results are reported. No result is written before the runs complete, and failed or refused runs are reported rather than silently dropped.9394\subsection{Contamination and leakage statement}95\label{sec:design-contamination}9697The generator and the judge are foundation models whose pretraining data may include the held-out articles, which are published and widely cited; nothing in this design can exclude that possibility, and the literature on benchmark contamination and memorisation shows that such exposure can inflate apparent performance \citep{sainz2023contamination, golchin2024timetravel, carlini2023memorization}. Three mitigations reduce its consequences without removing it: the published introduction is withheld from the material of every finance fixture, so that the drafting task cannot be solved by recall of the visible text; every output is screened for eight-word overlap with every corpus document, and H5 sets a preregistered ceiling; and author--year citations in outputs are verified against the source reference list, so that recalled but unlisted references count as unverifiable. Leakage between splits is addressed at the document level (Section~\ref{sec:corpus-split}): profiles and the retrieval index used by the engine are restricted to training documents, and the author-overlap count between splits is reported. Results are to be interpreted as conditional on these caveats.9899\subsection{Human evaluation}100\label{sec:design-human}101102Blind evaluation by qualified human reviewers was not available in this study. A rating protocol and rating sheets were prepared (\texttt{experiments/\allowbreak{}protocols/\allowbreak{}human\_\allowbreak{}rating\_\allowbreak{}sheet.\allowbreak{}md}): at least two PhD-level finance researchers, blind to condition, each rating all outputs of a (task, document) set in random order under random letters on the six primary dimensions plus a count of invented facts, with a joint calibration on two anchor sets, reliability measured by Krippendorff's $\alpha$ \citep{krippendorff2019content}, and disagreements above two points resolved by discussion while the original scores are kept for analysis. Until such ratings exist, every quality outcome of this paper is a model-judge rating and is labelled as such; the mandate's H1, which is phrased in terms of human ratings, remains untested in its original form.103