1% paper/sections/results.tex — WP45 / JoF-2502% Reports ONLY the recorded results of protocol heldout_v1 (experiments/analysis/results_heldout_v1.json).3% Every number in prose is a macro from paper/tables/numbers.tex; tables are generated by scripts/05_analyze_benchmark.py.4\providecommand{\TODOmacro}[1]{\textcolor{red}{[TODO macro: \texttt{#1}]}}56\section{Results}7\label{sec:results}89This section reports what the preregistered protocol \texttt{heldout\_\allowbreak{}v1} (Section~\ref{sec:design}) actually produced. Two cautions apply throughout. First, the study is a pilot: the finance sample is two held-out documents and eight tasks, so each preregistered contrast rests on \bmHEvsDMethodologicalRigorN{} paired units once repetitions are pooled, and the design is powered only for large effects. Second, every quality score is a rating by the model judge, not by a human reviewer (Section~\ref{sec:design-human}). A third caution emerged after the runs and is documented in Section~\ref{sec:human-defect}: for one of the two finance documents, the material builder passed only the title, the section list, and a word count to the generator, so that half of the finance units were, in effect, ``no-material'' tasks. In keeping with the preregistration, nothing was dropped or re-run; all results below include those units, and the task-level tables make the consequence visible.1011\subsection{Run accounting, cost, and latency}12\label{sec:results-accounting}1314All \bmRuns{} planned generation runs completed (\bmRunsOk{} successful, no failure or refusal), and the judge returned \bmJudgments{} valid judgement sets, including the second-pass sets used for the consistency check. Cost and latency were recorded per call and are reported as outcomes (Table~\ref{tab:secondary}). The single-call conditions are close to one another: mean cost per output rises from \bmSecACostUsd{}~USD under A to \bmSecDCostUsd{}~USD under D, and mean latency from \bmSecALatencyS{}~s to \bmSecDLatencyS{}~s. Condition E makes two generation calls (draft, then referee revision) and re-runs the deterministic audit; its mean cost is \bmSecECostUsd{}~USD and its mean latency \bmSecELatencyS{}~s, roughly twice the single-call conditions, with longer outputs (\bmSecENWords{} words on average against \bmSecANWords{} under A). The token budget was not equalised across conditions (Section~\ref{sec:design-controls}), so any advantage of E must be read as an advantage of a two-call workflow over a one-call workflow, not of a prompt over another prompt at equal cost.1516\subsection{Judge-rated primary outcomes}17\label{sec:results-primary}1819Table~\ref{tab:judge_means} gives the mean judge score by condition on the six primary dimensions for the finance documents, and Figure~\ref{fig:judge_by_condition} plots them. Three patterns stand out. Clarity is flat across the ladder (from \bmAClarity{} under A to \bmEClarity{} under E), which is consistent with the judge instruction not to reward length or polish. Methodological rigour and fidelity separate E from the other conditions: rigour rises from \bmAMethodologicalRigor{} (A) and \bmDMethodologicalRigor{} (D) to \bmEMethodologicalRigor{} (E), and fidelity from \bmDFidelity{} (D) to \bmEFidelity{} (E). Among the single-call conditions, the strong prompt and both retrieval variants lower fidelity relative to the minimal baseline (\bmAFidelity{} under A against \bmBFidelity{}, \bmCFidelity{}, and \bmDFidelity{} under B, C, and D), and the judge's count of invented facts moves in the same direction (\bmANInventedFacts{} under A, \bmBNInventedFacts{} under B, \bmCNInventedFacts{} under C, \bmDNInventedFacts{} under D, \bmENInventedFacts{} under E). The judge justifications, read in Section~\ref{sec:human}, attribute this to the stronger prompts producing more confident prose, and therefore more statements that go beyond the supplied material.2021\input{tables/tab_judge_means}2223\begin{figure}[htbp]24\centering25\includegraphics[width=\linewidth]{figures/fig_judge_by_condition.pdf}26\caption{Mean judge score by condition and dimension (finance held-out documents, first judge pass, 1--7 scale). Generated by \texttt{scripts/\allowbreak{}05\_\allowbreak{}analyze\_\allowbreak{}benchmark.\allowbreak{}py} from the recorded judgements of protocol \texttt{heldout\_\allowbreak{}v1}.}27\label{fig:judge_by_condition}28\end{figure}2930\subsection{Preregistered contrasts}31\label{sec:results-hypotheses}3233Table~\ref{tab:hypotheses} reports, for each preregistered contrast and outcome, the mean paired difference within (task, document, repetition), the cluster-bootstrap 95\% interval, the Holm-adjusted Wilcoxon $p$-value \citep{holm1979simple}, and Cliff's $\delta$ \citep{cliff1993dominance}. Under the preregistered rule, a contrast is called significant only when the Holm-adjusted $p$ is below 0.05 \emph{and} the interval excludes zero.3435\paragraph{H1 (D against B): not supported.} Structured profiles and rhetorical retrieval do not move judge-rated clarity (mean difference \bmHDvsBClarityDiff{}, 95\% CI [\bmHDvsBClarityLo, \bmHDvsBClarityHi], Holm $p = \bmHDvsBClarityP$) or contribution positioning (\bmHDvsBContributionPositioningDiff{}, [\bmHDvsBContributionPositioningLo, \bmHDvsBContributionPositioningHi], $p = \bmHDvsBContributionPositioningP$). The largest difference is on argument logic (\bmHDvsBArgumentLogicDiff{}, [\bmHDvsBArgumentLogicLo, \bmHDvsBArgumentLogicHi], $p = \bmHDvsBArgumentLogicP$), whose interval includes zero. Usefulness is likewise indistinguishable (\bmHDvsBUsefulnessDiff{}, [\bmHDvsBUsefulnessLo, \bmHDvsBUsefulnessHi]).3637\paragraph{H2 (D against C): not supported.} Measured profiles with structured units do not outperform conventional retrieval over raw excerpts on any of the three preregistered outcomes: clarity \bmHDvsCClarityDiff{} ([\bmHDvsCClarityLo, \bmHDvsCClarityHi]), contribution positioning \bmHDvsCContributionPositioningDiff{} ([\bmHDvsCContributionPositioningLo, \bmHDvsCContributionPositioningHi], Holm $p = \bmHDvsCContributionPositioningP$), and usefulness \bmHDvsCUsefulnessDiff{} ([\bmHDvsCUsefulnessLo, \bmHDvsCUsefulnessHi]). The point estimate for contribution positioning favours C, not D, although its interval includes zero.3839\paragraph{H3 (E against D): supported on both preregistered dimensions.} The full engine exceeds D on methodological rigour by \bmHEvsDMethodologicalRigorDiff{} points (95\% CI [\bmHEvsDMethodologicalRigorLo, \bmHEvsDMethodologicalRigorHi], Holm $p = \bmHEvsDMethodologicalRigorP$ to three decimals, Cliff's $\delta$ as reported in Table~\ref{tab:hypotheses}) and on fidelity by \bmHEvsDFidelityDiff{} points ([\bmHEvsDFidelityLo, \bmHEvsDFidelityHi], Holm $p = \bmHEvsDFidelityP$). Both intervals exclude zero and both adjusted $p$-values are below 0.05, so both meet the preregistered criterion. The third part of H3, no loss in numeric preservation, is met in the condition means: the preserved rate is \bmSecEPreservedRate{} under E against \bmSecDPreservedRate{} under D (Table~\ref{tab:secondary}). Usefulness, which was not part of H3 but is reported with it, is higher under E by \bmHEvsDUsefulnessDiff{} points with an interval that includes zero ([\bmHEvsDUsefulnessLo, \bmHEvsDUsefulnessHi], $p = \bmHEvsDUsefulnessP$).4041\paragraph{Exploratory contrasts.} Two contrasts that were not preregistered as hypotheses but follow from the ladder are reported with the same Holm correction across the six outcomes, and should be read as exploratory. Against the minimal baseline A, the full engine is higher on methodological rigour (\bmEvsAMethodologicalRigorDiff{}, [\bmEvsAMethodologicalRigorLo, \bmEvsAMethodologicalRigorHi], Holm $p = \bmEvsAMethodologicalRigorP$), argument logic (\bmEvsAArgumentLogicDiff{}, [\bmEvsAArgumentLogicLo, \bmEvsAArgumentLogicHi], $p = \bmEvsAArgumentLogicP$), and fidelity (\bmEvsAFidelityDiff{}, [\bmEvsAFidelityLo, \bmEvsAFidelityHi], $p = \bmEvsAFidelityP$ after rounding to three decimals; the unrounded adjusted value is below the threshold and the analysis script records the contrast as significant), the last of these only just inside the criterion; contribution positioning (\bmEvsAContributionPositioningDiff{}, [\bmEvsAContributionPositioningLo, \bmEvsAContributionPositioningHi], $p = \bmEvsAContributionPositioningP$) has an interval that excludes zero but an adjusted $p$ above the threshold, and is therefore not called significant; clarity (\bmEvsAClarityDiff{}) and usefulness (\bmEvsAUsefulnessDiff{}, [\bmEvsAUsefulnessLo, \bmEvsAUsefulnessHi]) do not differ. Against the strong prompt B, the picture is the same: rigour \bmEvsBMethodologicalRigorDiff{} ([\bmEvsBMethodologicalRigorLo, \bmEvsBMethodologicalRigorHi], $p = \bmEvsBMethodologicalRigorP$), fidelity \bmEvsBFidelityDiff{} ([\bmEvsBFidelityLo, \bmEvsBFidelityHi], $p = \bmEvsBFidelityP$), and argument logic \bmEvsBArgumentLogicDiff{} ([\bmEvsBArgumentLogicLo, \bmEvsBArgumentLogicHi], $p = \bmEvsBArgumentLogicP$) meet the criterion, while clarity (\bmEvsBClarityDiff{}), contribution positioning (\bmEvsBContributionPositioningDiff{}, [\bmEvsBContributionPositioningLo, \bmEvsBContributionPositioningHi]), and usefulness (\bmEvsBUsefulnessDiff{}, [\bmEvsBUsefulnessLo, \bmEvsBUsefulnessHi]) do not. The gain of E over B on fidelity is larger than its gain over A, because B itself lowers fidelity relative to A (Table~\ref{tab:judge_means}).4243\input{tables/tab_hypotheses}4445\subsection{Secondary outcomes}46\label{sec:results-secondary}4748Table~\ref{tab:secondary} reports the deterministic outcomes computed on every output without a model. The \emph{preserved rate} is the share of the numbers in the whole supplied material that reappear in the output. It is computed against the full material rather than against the subset a task needs, so low values are expected for drafting tasks that legitimately use only a few figures, and the measure is informative as a comparison across conditions on the same task rather than in absolute terms. On this measure E is the highest of the main conditions (\bmSecEPreservedRate{}) and A the lowest (\bmSecAPreservedRate{}). \emph{New numbers}, the count of numbers in the output that do not occur in the material, is lowest under E (\bmSecENewNumbers{} per output) and highest under D (\bmSecDNewNumbers{}); the deterministic \emph{unsupported-claim} count is likewise lowest under E (\bmSecENUnsupportedClaims{}) and highest under B (\bmSecBNUnsupportedClaims{}). The number of \verb|\todo| placeholders is similar across conditions (\bmSecATodoPlaceholders{} under A, \bmSecETodoPlaceholders{} under E, \bmSecBTodoPlaceholders{} under B at the high end), which reflects the shared instruction to flag rather than invent missing material, and the no-material document discussed in Section~\ref{sec:human-defect}. Output length grows along the ladder, from \bmSecANWords{} words under A to \bmSecENWords{} under E.4950\input{tables/tab_secondary}5152\subsection{H4: unverifiable citations, new numbers, and unsupported claims}53\label{sec:results-h4}5455H4 predicted fewer unverifiable author--year citations and fewer unsupported-claim sentences under E than under A, B, and C. The first part could not be tested as intended: the count of unverifiable citations was \bmSecANUnverifiableCitations{} in every condition (Table~\ref{tab:secondary}), so every paired difference is exactly zero (for instance \bmHFourEvsANUnverifiableCitationsDiff{} with interval [\bmHFourEvsANUnverifiableCitationsLo, \bmHFourEvsANUnverifiableCitationsHi] for E against A). The reason is a design choice rather than a property of E: every condition, including the minimal baseline, received the source reference list and was forbidden to cite anything else, and the outputs complied by emitting citation placeholders instead of author--year strings. The metric therefore did not discriminate among conditions in this protocol, and the second part of H4 carries the test. On the deterministic unsupported-claim count, E is lower than A by \bmHFourEvsANUnsupportedClaimsDiff{} sentences per output ([\bmHFourEvsANUnsupportedClaimsLo, \bmHFourEvsANUnsupportedClaimsHi], $p = \bmHFourEvsANUnsupportedClaimsP$), lower than B by \bmHFourEvsBNUnsupportedClaimsDiff{} ([\bmHFourEvsBNUnsupportedClaimsLo, \bmHFourEvsBNUnsupportedClaimsHi], $p = \bmHFourEvsBNUnsupportedClaimsP$), and lower than C by \bmHFourEvsCNUnsupportedClaimsDiff{} ([\bmHFourEvsCNUnsupportedClaimsLo, \bmHFourEvsCNUnsupportedClaimsHi], $p = \bmHFourEvsCNUnsupportedClaimsP$); the point estimates all favour E, but every interval includes zero, so H4 is not supported under the preregistered rule. New numbers not present in the source behave similarly: E against A \bmHFourEvsANewNumbersDiff{} ([\bmHFourEvsANewNumbersLo, \bmHFourEvsANewNumbersHi], $p = \bmHFourEvsANewNumbersP$), against B \bmHFourEvsBNewNumbersDiff{} ([\bmHFourEvsBNewNumbersLo, \bmHFourEvsBNewNumbersHi], $p = \bmHFourEvsBNewNumbersP$), and against C \bmHFourEvsCNewNumbersDiff{} ([\bmHFourEvsCNewNumbersLo, \bmHFourEvsCNewNumbersHi], $p = \bmHFourEvsCNewNumbersP$). The E--A and E--C intervals for new numbers exclude zero; these $p$-values were not Holm-adjusted across the three comparison conditions, and the evidence is best described as a consistent direction with modest precision. The judge's own count of invented facts, which captures unsupported statements that the regular-expression audit cannot, is reported in Section~\ref{sec:human-invented}.5657\subsection{H5: persistence on held-out documents and overlap}58\label{sec:results-h5}5960All finance results above are on held-out documents, so the first clause of H5 is the setting of the study rather than a separate test; the out-of-domain manuscript is reported descriptively in Section~\ref{sec:human-ood}. The overlap clause of H5 set a preregistered ceiling: the maximum eight-word overlap share of every output with any corpus document had to be below 0.02. \emph{As preregistered, this criterion was failed by the metric}: the largest overlap share observed was \bmOverlapMax{}, and only \bmOverlapBelow\% of outputs fell below the ceiling. Inspection of \texttt{overlap\_\allowbreak{}by\_\allowbreak{}run.\allowbreak{}csv} shows why. The corpus index contains the two held-out source documents themselves, and the overlap is almost entirely between an output and \emph{its own} source document; it is concentrated in the tasks whose brief requires reusing the paper's material, the abstract improvement (mean self-overlap \bmSelfOverlapImproveAbstract{}) and the literature rewrite (\bmSelfOverlapRewriteLiterature{}), with smaller values for the causal-claims audit (\bmSelfOverlapFlagCausalClaims{}) and the table task (\bmSelfOverlapImproveTable{}), and low values for the introduction draft (\bmSelfOverlapDraftIntroduction{}), the specification explanation (\bmSelfOverlapExplainSpecification{}), the interpretation of results (\bmSelfOverlapInterpretResults{}), and the referee report (\bmSelfOverlapRefereeReport{}). Once the output's own source document is excluded, a \emph{post-hoc refinement} that was not in the preregistration, the maximum overlap with any other corpus document is \bmOverlapOtherMax{}, the mean is \bmOverlapOtherMean{}, and \bmOverlapOtherBelow\% of outputs are below the ceiling. The refined reading is that no output reproduces text from a training document, which is the concern the ceiling was meant to address, while the preregistered metric, as written, did not anticipate that reuse of the task's own material is part of several briefs. Both readings are reported; the preregistered one is the one that was failed.6162\subsection{Task-level pattern}63\label{sec:results-tasks}6465Table~\ref{tab:rigor_by_task} shows judge-rated methodological rigour by task and condition, the dimension on which E differs most from the other conditions. E's advantage is not uniform. It is largest on the referee report (\bmTaskRefereeReportAMethodologicalRigor{} under A against \bmTaskRefereeReportEMethodologicalRigor{} under E), the literature rewrite (\bmTaskRewriteLiteratureAMethodologicalRigor{} against \bmTaskRewriteLiteratureEMethodologicalRigor{}), the interpretation of results (\bmTaskInterpretResultsAMethodologicalRigor{} against \bmTaskInterpretResultsEMethodologicalRigor{}), the table task (\bmTaskImproveTableAMethodologicalRigor{} against \bmTaskImproveTableEMethodologicalRigor{}), and the introduction draft (\bmTaskDraftIntroductionAMethodologicalRigor{} against \bmTaskDraftIntroductionEMethodologicalRigor{}); it is smallest on the causal-claims audit, where every condition already scores well (\bmTaskFlagCausalClaimsBMethodologicalRigor{} under B, \bmTaskFlagCausalClaimsEMethodologicalRigor{} under E), and on the abstract (\bmTaskImproveAbstractAMethodologicalRigor{} against \bmTaskImproveAbstractEMethodologicalRigor{}), where the brief leaves little room for methodological content. The referee report is also the one task on which D clearly exceeds A, B, and C on rigour (\bmTaskRefereeReportDMethodologicalRigor{} against \bmTaskRefereeReportCMethodologicalRigor{} under C), which Section~\ref{sec:human} traces partly to format compliance rather than to retrieval. Since each cell averages a small number of judged units, these task-level figures are descriptive.6667\begin{table}[htbp]68\centering69\small70\caption{Judge-rated methodological rigour by task and condition (finance held-out documents, 1--7 scale, means of the judged units of each task; conditions A--E as in Table~\ref{tab:conditions}). Values are macros generated from \texttt{results\_\allowbreak{}heldout\_\allowbreak{}v1.\allowbreak{}json}; each cell averages few units and is descriptive.}71\label{tab:rigor_by_task}72\begin{tabular}{lccccc}73\toprule74Task & A & B & C & D & E \\75\midrule76\texttt{draft\_introduction} & \bmTaskDraftIntroductionAMethodologicalRigor & \bmTaskDraftIntroductionBMethodologicalRigor & \bmTaskDraftIntroductionCMethodologicalRigor & \bmTaskDraftIntroductionDMethodologicalRigor & \bmTaskDraftIntroductionEMethodologicalRigor \\77\texttt{rewrite\_literature} & \bmTaskRewriteLiteratureAMethodologicalRigor & \bmTaskRewriteLiteratureBMethodologicalRigor & \bmTaskRewriteLiteratureCMethodologicalRigor & \bmTaskRewriteLiteratureDMethodologicalRigor & \bmTaskRewriteLiteratureEMethodologicalRigor \\78\texttt{explain\_specification} & \bmTaskExplainSpecificationAMethodologicalRigor & \bmTaskExplainSpecificationBMethodologicalRigor & \bmTaskExplainSpecificationCMethodologicalRigor & \bmTaskExplainSpecificationDMethodologicalRigor & \bmTaskExplainSpecificationEMethodologicalRigor \\79\texttt{interpret\_results} & \bmTaskInterpretResultsAMethodologicalRigor & \bmTaskInterpretResultsBMethodologicalRigor & \bmTaskInterpretResultsCMethodologicalRigor & \bmTaskInterpretResultsDMethodologicalRigor & \bmTaskInterpretResultsEMethodologicalRigor \\80\texttt{improve\_table} & \bmTaskImproveTableAMethodologicalRigor & \bmTaskImproveTableBMethodologicalRigor & \bmTaskImproveTableCMethodologicalRigor & \bmTaskImproveTableDMethodologicalRigor & \bmTaskImproveTableEMethodologicalRigor \\81\texttt{flag\_causal\_claims} & \bmTaskFlagCausalClaimsAMethodologicalRigor & \bmTaskFlagCausalClaimsBMethodologicalRigor & \bmTaskFlagCausalClaimsCMethodologicalRigor & \bmTaskFlagCausalClaimsDMethodologicalRigor & \bmTaskFlagCausalClaimsEMethodologicalRigor \\82\texttt{referee\_report} & \bmTaskRefereeReportAMethodologicalRigor & \bmTaskRefereeReportBMethodologicalRigor & \bmTaskRefereeReportCMethodologicalRigor & \bmTaskRefereeReportDMethodologicalRigor & \bmTaskRefereeReportEMethodologicalRigor \\83\texttt{improve\_abstract} & \bmTaskImproveAbstractAMethodologicalRigor & \bmTaskImproveAbstractBMethodologicalRigor & \bmTaskImproveAbstractCMethodologicalRigor & \bmTaskImproveAbstractDMethodologicalRigor & \bmTaskImproveAbstractEMethodologicalRigor \\84\bottomrule85\end{tabular}86\end{table}8788\subsection{Judge self-consistency}89\label{sec:results-judge}9091A random quarter of the judgement sets was re-judged in a second random order under new letters, giving \bmJudgePairs{} output pairs scored twice. The mean absolute difference between passes across all dimensions is \bmJudgeMAD{} points on the seven-point scale. By dimension, the rank correlation between passes (Spearman) and the mean absolute difference are: clarity $\rho = \bmJudgeRhoClarity$ (MAD \bmJudgeMadClarity), argument logic $\rho = \bmJudgeRhoArgumentLogic$ (\bmJudgeMadArgumentLogic), contribution positioning $\rho = \bmJudgeRhoContributionPositioning$ (\bmJudgeMadContributionPositioning), methodological rigour $\rho = \bmJudgeRhoMethodologicalRigor$ (\bmJudgeMadMethodologicalRigor), fidelity $\rho = \bmJudgeRhoFidelity$ (\bmJudgeMadFidelity), and usefulness $\rho = \bmJudgeRhoUsefulness$ (\bmJudgeMadUsefulness). The two dimensions that carry H3, rigour and fidelity, are among the more and the less stable respectively; argument logic is the least rank-stable. These figures measure the judge's agreement with itself under a change of presentation order, not its agreement with a human reader, and they do not address the biases that a single-vendor judge may share with the generator \citep{zheng2023judge, wang2024notfair, panickssery2024self}.92