spb/wp45_uqo

1% paper/sections/results.tex — WP45 / JoF-2502% Reports ONLY the recorded results of protocol heldout_v1, run 2 (experiments/analysis/results_heldout_v1.json; run 1 archived, see Amendment A-004).3% Every number in prose is a macro from paper/tables/numbers.tex; tables are generated by scripts/05_analyze_benchmark.py.4\providecommand{\TODOmacro}[1]{\textcolor{red}{[TODO macro: \texttt{#1}]}}56\section{Results}7\label{sec:results}89This section reports what the preregistered protocol \texttt{heldout\_\allowbreak{}v1} (Section~\ref{sec:design}) produced in its final execution, after the two implementation defects recorded as Amendment A-004 were corrected (Section~\ref{sec:human-defects}); the first execution is archived and is not used anywhere in this paper. Two cautions apply throughout. The study is a pilot: the finance sample is two held-out documents and eight tasks, so each preregistered contrast rests on \bmHEvsDMethodologicalRigorN{} paired units once repetitions are pooled, and the design is powered only for large effects. And every quality score is a rating by the model judge, not by a human reviewer (Section~\ref{sec:design-human}).1011\subsection{Run accounting, cost, and latency}12\label{sec:results-accounting}1314All \bmRuns{} planned generation runs completed (\bmRunsOk{} successful, no failure or refusal), and the judge returned \bmJudgments{} valid judgement sets, including the second-pass sets used for the consistency check. Cost and latency were recorded per call and are reported as outcomes (Table~\ref{tab:secondary}). The single-call conditions are close to one another: mean cost per output rises from \bmSecACostUsd{}~USD under A to \bmSecDCostUsd{}~USD under D, and mean latency from \bmSecALatencyS{}~s to \bmSecDLatencyS{}~s. Condition E makes two generation calls (draft, then referee revision) and re-runs the deterministic audit; its mean cost is \bmSecECostUsd{}~USD and its mean latency \bmSecELatencyS{}~s, about twice the single-call conditions, for outputs of similar length (\bmSecENWords{} words on average against \bmSecDNWords{} under D). The token budget was not equalised across conditions (Section~\ref{sec:design-controls}), so any advantage of E must be read as the advantage of a two-call workflow over a one-call workflow, not of one prompt over another at equal cost.1516\subsection{Judge-rated primary outcomes}17\label{sec:results-primary}1819Table~\ref{tab:judge_means} gives the mean judge score by condition on the six primary dimensions for the finance documents, and Figure~\ref{fig:judge_by_condition} plots them. Clarity is flat along the ladder (from \bmAClarity{} under A to \bmEClarity{} under E, with B highest at \bmBClarity{}), which is consistent with the judge instruction not to reward length or polish. Methodological rigour separates E from the single-call conditions: it rises from \bmAMethodologicalRigor{} (A) and \bmDMethodologicalRigor{} (D) to \bmEMethodologicalRigor{} (E), and fidelity from \bmDFidelity{} (D) to \bmEFidelity{} (E). Two dimensions move the other way. Contribution positioning is lower under E (\bmEContributionPositioning{}) than under B or D (\bmBContributionPositioning{} and \bmDContributionPositioning{}), and usefulness is lower under E (\bmEUsefulness{}) than under any single-call condition (\bmDUsefulness{} under D). The judge's count of invented facts falls from \bmDNInventedFacts{} under D to \bmENInventedFacts{} under E, and rises monotonically from A to D (\bmANInventedFacts{}, \bmBNInventedFacts{}, \bmCNInventedFacts{}, \bmDNInventedFacts{}). The qualitative reading in Section~\ref{sec:human} is that the referee pass removes unsupported statements and adds scope and identification caveats, and that the judge rates the resulting text as more rigorous and faithful but as more hedged, more placeholder-laden, and therefore slightly less useful and less sharply positioned.2021\input{tables/tab_judge_means}2223\begin{figure}[htbp]24\centering25\includegraphics[width=\linewidth]{figures/fig_judge_by_condition.pdf}26\caption{Mean judge score by condition and dimension (finance held-out documents, first judge pass, 1--7 scale). Generated by \texttt{scripts/\allowbreak{}05\_\allowbreak{}analyze\_\allowbreak{}benchmark.\allowbreak{}py} from the recorded judgements of protocol \texttt{heldout\_\allowbreak{}v1}.}27\label{fig:judge_by_condition}28\end{figure}2930\subsection{Preregistered contrasts}31\label{sec:results-hypotheses}3233Table~\ref{tab:hypotheses} reports, for each preregistered contrast and outcome, the mean paired difference within (task, document, repetition), the cluster-bootstrap 95\% interval, the Holm-adjusted Wilcoxon $p$-value \citep{holm1979simple}, and Cliff's $\delta$ \citep{cliff1993dominance}. Under the preregistered rule, a contrast is called significant only when the Holm-adjusted $p$ is below 0.05 \emph{and} the interval excludes zero.3435\paragraph{H1 (D against B): not supported.} Structured profiles and rhetorical retrieval do not move judge-rated clarity (mean difference \bmHDvsBClarityDiff{}, 95\% CI [\bmHDvsBClarityLo, \bmHDvsBClarityHi], Holm $p = \bmHDvsBClarityP$), argument logic (\bmHDvsBArgumentLogicDiff{}, [\bmHDvsBArgumentLogicLo, \bmHDvsBArgumentLogicHi], $p = \bmHDvsBArgumentLogicP$), contribution positioning (\bmHDvsBContributionPositioningDiff{}, [\bmHDvsBContributionPositioningLo, \bmHDvsBContributionPositioningHi], $p = \bmHDvsBContributionPositioningP$), or usefulness (\bmHDvsBUsefulnessDiff{}, [\bmHDvsBUsefulnessLo, \bmHDvsBUsefulnessHi]). Every interval includes zero and every point estimate is small on the seven-point scale.3637\paragraph{H2 (D against C): not supported.} Measured profiles with structured units do not outperform conventional retrieval over raw excerpts: clarity \bmHDvsCClarityDiff{} ([\bmHDvsCClarityLo, \bmHDvsCClarityHi], Holm $p = \bmHDvsCClarityP$), argument logic \bmHDvsCArgumentLogicDiff{} ([\bmHDvsCArgumentLogicLo, \bmHDvsCArgumentLogicHi], $p = \bmHDvsCArgumentLogicP$), contribution positioning \bmHDvsCContributionPositioningDiff{} ([\bmHDvsCContributionPositioningLo, \bmHDvsCContributionPositioningHi], $p = \bmHDvsCContributionPositioningP$), and usefulness \bmHDvsCUsefulnessDiff{} ([\bmHDvsCUsefulnessLo, \bmHDvsCUsefulnessHi], $p = \bmHDvsCUsefulnessP$). Within this pilot, what the engine retrieves and how it is presented makes no measurable difference to the judge once the strong prompt is in place.3839\paragraph{H3 (E against D): supported on both preregistered dimensions, with a trade-off.} The full engine exceeds D on methodological rigour by \bmHEvsDMethodologicalRigorDiff{} points (95\% CI [\bmHEvsDMethodologicalRigorLo, \bmHEvsDMethodologicalRigorHi], Holm $p = \bmHEvsDMethodologicalRigorP$, Cliff's $\delta$ in Table~\ref{tab:hypotheses}) and on fidelity by \bmHEvsDFidelityDiff{} points ([\bmHEvsDFidelityLo, \bmHEvsDFidelityHi], Holm $p = \bmHEvsDFidelityP$). Both intervals exclude zero and both adjusted $p$-values are below 0.05, so both meet the preregistered criterion; the fidelity interval is close to zero at its lower end and should be held more loosely than the rigour result. The third part of H3, no loss in numeric preservation, holds in the condition means: the preserved rate is \bmSecEPreservedRate{} under E and \bmSecDPreservedRate{} under D (Table~\ref{tab:secondary}). The cost of these gains appears on usefulness, which is \emph{lower} under E than under D by \bmHEvsDUsefulnessDiff{} points ([\bmHEvsDUsefulnessLo, \bmHEvsDUsefulnessHi], $p = \bmHEvsDUsefulnessP$); the interval includes zero, so the decline is not significant under the preregistered rule, but its sign is consistent across the exploratory contrasts below, and the judge justifications in Section~\ref{sec:human} explain it: the referee pass makes the text more cautious and more hedged, and the judge rates such text as somewhat less useful to an author.4041\paragraph{Exploratory contrasts.} Two contrasts that were not preregistered as hypotheses but follow from the ladder are reported with the same Holm correction across the six outcomes and should be read as exploratory. Against the minimal baseline A, the full engine is higher on methodological rigour (\bmEvsAMethodologicalRigorDiff{}, [\bmEvsAMethodologicalRigorLo, \bmEvsAMethodologicalRigorHi], Holm $p = \bmEvsAMethodologicalRigorP$), which meets the criterion; argument logic (\bmEvsAArgumentLogicDiff{}, [\bmEvsAArgumentLogicLo, \bmEvsAArgumentLogicHi], $p = \bmEvsAArgumentLogicP$) and fidelity (\bmEvsAFidelityDiff{}, [\bmEvsAFidelityLo, \bmEvsAFidelityHi], $p = \bmEvsAFidelityP$) favour E without meeting it; clarity (\bmEvsAClarityDiff{}), contribution positioning (\bmEvsAContributionPositioningDiff{}, [\bmEvsAContributionPositioningLo, \bmEvsAContributionPositioningHi]), and usefulness (\bmEvsAUsefulnessDiff{}, [\bmEvsAUsefulnessLo, \bmEvsAUsefulnessHi]) do not differ. Against the strong prompt B, rigour again meets the criterion (\bmEvsBMethodologicalRigorDiff{}, [\bmEvsBMethodologicalRigorLo, \bmEvsBMethodologicalRigorHi], $p = \bmEvsBMethodologicalRigorP$); fidelity (\bmEvsBFidelityDiff{}, [\bmEvsBFidelityLo, \bmEvsBFidelityHi], $p = \bmEvsBFidelityP$) and argument logic (\bmEvsBArgumentLogicDiff{}, [\bmEvsBArgumentLogicLo, \bmEvsBArgumentLogicHi], $p = \bmEvsBArgumentLogicP$) have intervals that exclude zero but adjusted $p$-values above the threshold, and are therefore not called significant. The trade-off is clearest here: relative to B, E is lower on contribution positioning by \bmEvsBContributionPositioningDiff{} points ([\bmEvsBContributionPositioningLo, \bmEvsBContributionPositioningHi], $p = \bmEvsBContributionPositioningP$), on usefulness by \bmEvsBUsefulnessDiff{} ([\bmEvsBUsefulnessLo, \bmEvsBUsefulnessHi], $p = \bmEvsBUsefulnessP$), and on clarity by \bmEvsBClarityDiff{} ([\bmEvsBClarityLo, \bmEvsBClarityHi], $p = \bmEvsBClarityP$), the last with an interval that excludes zero. None of these declines meets the significance rule, but together they describe a workflow that trades positioning and perceived usefulness for rigour and fidelity rather than one that dominates the strong prompt.4243\input{tables/tab_hypotheses}4445\subsection{Secondary outcomes}46\label{sec:results-secondary}4748Table~\ref{tab:secondary} reports the deterministic outcomes computed on every output without a model. The \emph{preserved rate} is the share of the numbers in the whole supplied material (abstract, sections, tables and notes, figures, references) that reappear in the output. It is computed against the full material rather than against the subset a task needs, so values are low by construction for every task (\bmSecAPreservedRate{} under A, \bmSecEPreservedRate{} under E) and the measure is informative only as a comparison across conditions on the same task. \emph{New numbers}, the count of numbers in the output that do not occur in the material, is lowest under E (\bmSecENewNumbers{} per output) and highest under C (\bmSecCNewNumbers{}); the deterministic \emph{unsupported-claim} count is likewise lowest under E (\bmSecENUnsupportedClaims{}) and highest under C (\bmSecCNUnsupportedClaims{}). The number of \verb|\todo| placeholders is small and similar across conditions (\bmSecATodoPlaceholders{} under A, \bmSecETodoPlaceholders{} under E), and output length is similar (\bmSecANWords{} words under A, \bmSecENWords{} under E). The \emph{unverifiable-citation} count is \bmSecANUnverifiableCitations{} per output under A through D and \bmSecENUnverifiableCitations{} under E; the non-zero values are loose author--year strings that do not match the reference list exactly, not invented references, as Section~\ref{sec:results-h4} explains.4950\input{tables/tab_secondary}5152\subsection{H4: unverifiable citations, new numbers, and unsupported claims}53\label{sec:results-h4}5455H4 predicted fewer unverifiable author--year citations and fewer unsupported-claim sentences under E than under A, B, and C. On citations the metric barely moves: every condition received the source reference list and was forbidden to cite anything else, and the outputs complied, either by citing works from the list or by emitting citation placeholders, so the counts are close to zero everywhere (Table~\ref{tab:secondary}) and the paired differences are small (E against A \bmHFourEvsANUnverifiableCitationsDiff{}, [\bmHFourEvsANUnverifiableCitationsLo, \bmHFourEvsANUnverifiableCitationsHi], $p = \bmHFourEvsANUnverifiableCitationsP$; the same for B and C). The metric therefore constrains behaviour rather than discriminating among conditions in this protocol. On the deterministic unsupported-claim count, E is lower than C by \bmHFourEvsCNUnsupportedClaimsDiff{} sentences per output ([\bmHFourEvsCNUnsupportedClaimsLo, \bmHFourEvsCNUnsupportedClaimsHi], unadjusted $p = \bmHFourEvsCNUnsupportedClaimsP$), an interval that excludes zero, and lower than A (\bmHFourEvsANUnsupportedClaimsDiff{}, [\bmHFourEvsANUnsupportedClaimsLo, \bmHFourEvsANUnsupportedClaimsHi], $p = \bmHFourEvsANUnsupportedClaimsP$) and B (\bmHFourEvsBNUnsupportedClaimsDiff{}, [\bmHFourEvsBNUnsupportedClaimsLo, \bmHFourEvsBNUnsupportedClaimsHi], $p = \bmHFourEvsBNUnsupportedClaimsP$) only directionally. New numbers absent from the source are lower under E only directionally: against A \bmHFourEvsANewNumbersDiff{} ([\bmHFourEvsANewNumbersLo, \bmHFourEvsANewNumbersHi]), against B \bmHFourEvsBNewNumbersDiff{} ([\bmHFourEvsBNewNumbersLo, \bmHFourEvsBNewNumbersHi]), against C \bmHFourEvsCNewNumbersDiff{} ([\bmHFourEvsCNewNumbersLo, \bmHFourEvsCNewNumbersHi]). These $p$-values were not Holm-adjusted across the three comparison conditions, and H4 as a whole is best described as supported in direction and, for unsupported claims against C, in magnitude, but not as a preregistered significance result. The judge's own count of invented facts, which captures unsupported statements that the regular-expression audit cannot, is reported in Section~\ref{sec:human-invented}.5657\subsection{H5: persistence on held-out documents and overlap}58\label{sec:results-h5}5960All finance results above are on held-out documents, so the first clause of H5 is the setting of the study rather than a separate test; the out-of-domain manuscript is reported descriptively in Section~\ref{sec:human-ood}. The overlap clause set a preregistered ceiling: the maximum eight-word overlap share of every output with any corpus document had to be below 0.02. \emph{As preregistered, this criterion was failed by the metric}: the largest overlap share observed was \bmOverlapMax{}, and only \bmOverlapBelow\% of outputs fell below the ceiling. Inspection of \texttt{overlap\_\allowbreak{}by\_\allowbreak{}run.\allowbreak{}csv} shows why. The corpus index contains the two held-out source documents themselves, and the overlap is almost entirely between an output and \emph{its own} source document; it is concentrated in the tasks whose brief requires reusing the paper's material, above all the abstract improvement (mean self-overlap \bmSelfOverlapImproveAbstract{}), then the literature rewrite (\bmSelfOverlapRewriteLiterature{}) and the causal-claims audit, which must quote the claims it classifies (\bmSelfOverlapFlagCausalClaims{}); the introduction draft (\bmSelfOverlapDraftIntroduction{}), the table task (\bmSelfOverlapImproveTable{}), the specification explanation (\bmSelfOverlapExplainSpecification{}), the interpretation of results (\bmSelfOverlapInterpretResults{}), and the referee report (\bmSelfOverlapRefereeReport{}) are at or near the ceiling. Once the output's own source document is excluded, a \emph{post-hoc refinement} that was not in the preregistration, the maximum overlap with any other corpus document is \bmOverlapOtherMax{}, the mean is \bmOverlapOtherMean{}, and \bmOverlapOtherBelow\% of outputs are below the ceiling. The refined reading is that no output reproduces text from a training document, which is the concern the ceiling was meant to address, while the preregistered metric, as written, did not anticipate that reuse of the task's own material is part of several briefs. Both readings are reported; the preregistered one is the one that was failed.6162\subsection{Task-level pattern}63\label{sec:results-tasks}6465Table~\ref{tab:rigor_by_task} shows judge-rated methodological rigour by task and condition, the dimension on which E differs most from the other conditions. The advantage is not uniform. It is largest on the interpretation of results (\bmTaskInterpretResultsAMethodologicalRigor{} under A against \bmTaskInterpretResultsEMethodologicalRigor{} under E), the literature rewrite (\bmTaskRewriteLiteratureAMethodologicalRigor{} against \bmTaskRewriteLiteratureEMethodologicalRigor{}), the table task (\bmTaskImproveTableAMethodologicalRigor{} against \bmTaskImproveTableEMethodologicalRigor{}), and the abstract (\bmTaskImproveAbstractAMethodologicalRigor{} against \bmTaskImproveAbstractEMethodologicalRigor{}); it is moderate on the introduction draft (\bmTaskDraftIntroductionAMethodologicalRigor{} against \bmTaskDraftIntroductionEMethodologicalRigor{}) and the specification explanation (\bmTaskExplainSpecificationAMethodologicalRigor{} against \bmTaskExplainSpecificationEMethodologicalRigor{}). On the referee report there is no advantage at all: E scores \bmTaskRefereeReportEMethodologicalRigor{}, below A (\bmTaskRefereeReportAMethodologicalRigor{}) and B (\bmTaskRefereeReportBMethodologicalRigor{}), because a referee report is already a critique and the referee pass adds caveats to a text that is itself truncated before it reaches its recommendation (Section~\ref{sec:human-modes}). The large gap on the causal-claims audit (\bmTaskFlagCausalClaimsAMethodologicalRigor{} under A against \bmTaskFlagCausalClaimsEMethodologicalRigor{} under E) is partly an artefact: the baseline returned an empty file on one of the two documents and was scored at the bottom of the scale on every dimension. The trade-off noted above is also task-specific: on the introduction draft, contribution positioning falls from \bmTaskDraftIntroductionBContributionPositioning{} under B to \bmTaskDraftIntroductionEContributionPositioning{} under E, and usefulness from \bmTaskDraftIntroductionBUsefulness{} to \bmTaskDraftIntroductionEUsefulness{}, because the revised introductions were truncated before their contribution paragraph and carried citation placeholders. Since each cell averages a small number of judged units, these task-level figures are descriptive.6667\begin{table}[htbp]68\centering69\small70\caption{Judge-rated methodological rigour by task and condition (finance held-out documents, 1--7 scale, means of the judged units of each task; conditions A--E as in Table~\ref{tab:conditions}). Values are macros generated from \texttt{results\_\allowbreak{}heldout\_\allowbreak{}v1.\allowbreak{}json}; each cell averages few units and is descriptive. The baseline's low score on \texttt{flag\_causal\_claims} includes an empty output on one document.}71\label{tab:rigor_by_task}72\begin{tabular}{lccccc}73\toprule74Task & A & B & C & D & E \\75\midrule76\texttt{draft\_introduction} & \bmTaskDraftIntroductionAMethodologicalRigor & \bmTaskDraftIntroductionBMethodologicalRigor & \bmTaskDraftIntroductionCMethodologicalRigor & \bmTaskDraftIntroductionDMethodologicalRigor & \bmTaskDraftIntroductionEMethodologicalRigor \\77\texttt{rewrite\_literature} & \bmTaskRewriteLiteratureAMethodologicalRigor & \bmTaskRewriteLiteratureBMethodologicalRigor & \bmTaskRewriteLiteratureCMethodologicalRigor & \bmTaskRewriteLiteratureDMethodologicalRigor & \bmTaskRewriteLiteratureEMethodologicalRigor \\78\texttt{explain\_specification} & \bmTaskExplainSpecificationAMethodologicalRigor & \bmTaskExplainSpecificationBMethodologicalRigor & \bmTaskExplainSpecificationCMethodologicalRigor & \bmTaskExplainSpecificationDMethodologicalRigor & \bmTaskExplainSpecificationEMethodologicalRigor \\79\texttt{interpret\_results} & \bmTaskInterpretResultsAMethodologicalRigor & \bmTaskInterpretResultsBMethodologicalRigor & \bmTaskInterpretResultsCMethodologicalRigor & \bmTaskInterpretResultsDMethodologicalRigor & \bmTaskInterpretResultsEMethodologicalRigor \\80\texttt{improve\_table} & \bmTaskImproveTableAMethodologicalRigor & \bmTaskImproveTableBMethodologicalRigor & \bmTaskImproveTableCMethodologicalRigor & \bmTaskImproveTableDMethodologicalRigor & \bmTaskImproveTableEMethodologicalRigor \\81\texttt{flag\_causal\_claims} & \bmTaskFlagCausalClaimsAMethodologicalRigor & \bmTaskFlagCausalClaimsBMethodologicalRigor & \bmTaskFlagCausalClaimsCMethodologicalRigor & \bmTaskFlagCausalClaimsDMethodologicalRigor & \bmTaskFlagCausalClaimsEMethodologicalRigor \\82\texttt{referee\_report} & \bmTaskRefereeReportAMethodologicalRigor & \bmTaskRefereeReportBMethodologicalRigor & \bmTaskRefereeReportCMethodologicalRigor & \bmTaskRefereeReportDMethodologicalRigor & \bmTaskRefereeReportEMethodologicalRigor \\83\texttt{improve\_abstract} & \bmTaskImproveAbstractAMethodologicalRigor & \bmTaskImproveAbstractBMethodologicalRigor & \bmTaskImproveAbstractCMethodologicalRigor & \bmTaskImproveAbstractDMethodologicalRigor & \bmTaskImproveAbstractEMethodologicalRigor \\84\bottomrule85\end{tabular}86\end{table}8788\subsection{Judge self-consistency}89\label{sec:results-judge}9091A random quarter of the judgement sets was re-judged in a second random order under new letters, giving \bmJudgePairs{} output pairs scored twice. The mean absolute difference between passes across all dimensions is \bmJudgeMAD{} points on the seven-point scale. By dimension, the rank correlation between passes (Spearman) and the mean absolute difference are: clarity $\rho = \bmJudgeRhoClarity$ (MAD \bmJudgeMadClarity), argument logic $\rho = \bmJudgeRhoArgumentLogic$ (\bmJudgeMadArgumentLogic), contribution positioning $\rho = \bmJudgeRhoContributionPositioning$ (\bmJudgeMadContributionPositioning), methodological rigour $\rho = \bmJudgeRhoMethodologicalRigor$ (\bmJudgeMadMethodologicalRigor), fidelity $\rho = \bmJudgeRhoFidelity$ (\bmJudgeMadFidelity), and usefulness $\rho = \bmJudgeRhoUsefulness$ (\bmJudgeMadUsefulness). Rigour and contribution positioning are the most rank-stable dimensions; clarity and usefulness are the least, which matters because the usefulness decline under E is of the same order as the judge's own instability on that dimension. These figures measure the judge's agreement with itself under a change of presentation order, not its agreement with a human reader, and they do not address the biases that a single-vendor judge may share with the generator \citep{zheng2023judge, wang2024notfair, panickssery2024self}.92