1% paper/sections/ablations.tex — WP45 / JoF-2502% Ablations of E, judge robustness, contamination and leakage checks. Recorded results of run 2 only (results_heldout_v1.json); numbers are macros.3\providecommand{\TODOmacro}[1]{\textcolor{red}{[TODO macro: \texttt{#1}]}}45\section{Ablations, robustness, contamination and leakage checks}6\label{sec:ablations}78\subsection{Ablations of the full engine}9\label{sec:ablations-e}1011Each ablation removes one ingredient from E and is compared with full E by the same paired procedure as the hypotheses (Table~\ref{tab:ablations}; mean paired difference E minus ablation, cluster-bootstrap interval, Wilcoxon $p$ without Holm adjustment). The ablations were run on the first repetition only, so each contrast rests on \bmSecENoRefereeN{} paired units; the intervals are correspondingly wide and only a large effect could be detected.1213The referee pass carries the effect. Removing it lowers methodological rigour by \bmAbENoRefereeMethodologicalRigorDiff{} points (95\% CI [\bmAbENoRefereeMethodologicalRigorLo, \bmAbENoRefereeMethodologicalRigorHi], $p = \bmAbENoRefereeMethodologicalRigorP$), fidelity by \bmAbENoRefereeFidelityDiff{} ([\bmAbENoRefereeFidelityLo, \bmAbENoRefereeFidelityHi], $p = \bmAbENoRefereeFidelityP$), and argument logic by \bmAbENoRefereeArgumentLogicDiff{} ([\bmAbENoRefereeArgumentLogicLo, \bmAbENoRefereeArgumentLogicHi], $p = \bmAbENoRefereeArgumentLogicP$); clarity, contribution positioning, and usefulness do not change (\bmAbENoRefereeClarityDiff{}, \bmAbENoRefereeContributionPositioningDiff{}, \bmAbENoRefereeUsefulnessDiff{}). The condition means confirm the reading: E without the referee pass scores at the level of D on rigour and fidelity (Table~\ref{tab:judge_means}), its judge-counted invented facts are back at the level of the single-call conditions, and its secondary outcomes (cost \bmSecENoRefereeCostUsd{}~USD, latency \bmSecENoRefereeLatencyS{}~s, unsupported claims \bmSecENoRefereeNUnsupportedClaims{} against \bmSecENUnsupportedClaims{} under E) are those of a single-call condition. The draft stage of E, which is D plus an audit the draft cannot act on, is not what separates E from D; the second call, which receives the audit and must fix every item while keeping every verified fact, is.1415The other three ablations are null within the pilot's power, and two of them point the wrong way. Removing the structured retrieval units changes rigour by \bmAbENoRetrievalMethodologicalRigorDiff{} ([\bmAbENoRetrievalMethodologicalRigorLo, \bmAbENoRetrievalMethodologicalRigorHi]), fidelity by \bmAbENoRetrievalFidelityDiff{} ([\bmAbENoRetrievalFidelityLo, \bmAbENoRetrievalFidelityHi]), and contribution positioning by \bmAbENoRetrievalContributionPositioningDiff{} ([\bmAbENoRetrievalContributionPositioningLo, \bmAbENoRetrievalContributionPositioningHi]). Removing the profile summary changes rigour by \bmAbENoProfileMethodologicalRigorDiff{} ([\bmAbENoProfileMethodologicalRigorLo, \bmAbENoProfileMethodologicalRigorHi]) and usefulness by \bmAbENoProfileUsefulnessDiff{} ([\bmAbENoProfileUsefulnessLo, \bmAbENoProfileUsefulnessHi], $p = \bmAbENoProfileUsefulnessP$): the ablation is rated slightly \emph{more} useful than full E, with an interval that excludes zero at the unadjusted level, and slightly more rigorous. Removing the deterministic audit, so that the referee pass works unaided, changes rigour by \bmAbENoFactcheckMethodologicalRigorDiff{} ([\bmAbENoFactcheckMethodologicalRigorLo, \bmAbENoFactcheckMethodologicalRigorHi]) and fidelity by \bmAbENoFactcheckFidelityDiff{} ([\bmAbENoFactcheckFidelityLo, \bmAbENoFactcheckFidelityHi]). On the deterministic secondary outcomes the audit does what it is designed to do, but modestly: without it the unverifiable-citation count is \bmSecENoFactcheckNUnverifiableCitations{} against \bmSecENUnverifiableCitations{} with it, and the unsupported-claim count \bmSecENoFactcheckNUnsupportedClaims{} against \bmSecENUnsupportedClaims{}; these are condition means on different units and were not tested. Within this pilot, then, the measured value of E is the referee call; the profile and the retrieved units add no measurable benefit on the judge's scale.1617Two cautions apply. With \bmSecENoRetrievalN{} units per ablation, an effect of half a point on the seven-point scale would often fall inside the intervals above, so ``null within the pilot's power'' does not mean ``no effect''. And the ablations were judged in the same blinded sets as the main conditions, so each first-repetition judge set contained up to nine outputs of the same (task, document), which lengthens the judge's context and may compress differences among the E-family variants that share the referee pass.1819\begin{table}[htbp]20\centering21\small22\caption{Ablations of the full engine: mean paired difference (E minus ablation) on the 1--7 judge scale, cluster-bootstrap 95\% interval, and paired Wilcoxon $p$ (unadjusted). Finance held-out documents, first repetition only (\bmSecENoRefereeN{} paired units per contrast). Values are macros generated from \texttt{results\_\allowbreak{}heldout\_\allowbreak{}v1.\allowbreak{}json}.}23\label{tab:ablations}24\begin{tabular}{llrrrr}25\toprule26Ablation & Dimension & $\Delta$ & CI low & CI high & $p$ \\27\midrule28E \texttt{no\_referee} & clarity & \bmAbENoRefereeClarityDiff & \bmAbENoRefereeClarityLo & \bmAbENoRefereeClarityHi & \bmAbENoRefereeClarityP \\29 & argument logic & \bmAbENoRefereeArgumentLogicDiff & \bmAbENoRefereeArgumentLogicLo & \bmAbENoRefereeArgumentLogicHi & \bmAbENoRefereeArgumentLogicP \\30 & contribution positioning & \bmAbENoRefereeContributionPositioningDiff & \bmAbENoRefereeContributionPositioningLo & \bmAbENoRefereeContributionPositioningHi & \bmAbENoRefereeContributionPositioningP \\31 & methodological rigour & \bmAbENoRefereeMethodologicalRigorDiff & \bmAbENoRefereeMethodologicalRigorLo & \bmAbENoRefereeMethodologicalRigorHi & \bmAbENoRefereeMethodologicalRigorP \\32 & fidelity & \bmAbENoRefereeFidelityDiff & \bmAbENoRefereeFidelityLo & \bmAbENoRefereeFidelityHi & \bmAbENoRefereeFidelityP \\33 & usefulness & \bmAbENoRefereeUsefulnessDiff & \bmAbENoRefereeUsefulnessLo & \bmAbENoRefereeUsefulnessHi & \bmAbENoRefereeUsefulnessP \\34\midrule35E \texttt{no\_retrieval} & clarity & \bmAbENoRetrievalClarityDiff & \bmAbENoRetrievalClarityLo & \bmAbENoRetrievalClarityHi & \bmAbENoRetrievalClarityP \\36 & argument logic & \bmAbENoRetrievalArgumentLogicDiff & \bmAbENoRetrievalArgumentLogicLo & \bmAbENoRetrievalArgumentLogicHi & \bmAbENoRetrievalArgumentLogicP \\37 & contribution positioning & \bmAbENoRetrievalContributionPositioningDiff & \bmAbENoRetrievalContributionPositioningLo & \bmAbENoRetrievalContributionPositioningHi & \bmAbENoRetrievalContributionPositioningP \\38 & methodological rigour & \bmAbENoRetrievalMethodologicalRigorDiff & \bmAbENoRetrievalMethodologicalRigorLo & \bmAbENoRetrievalMethodologicalRigorHi & \bmAbENoRetrievalMethodologicalRigorP \\39 & fidelity & \bmAbENoRetrievalFidelityDiff & \bmAbENoRetrievalFidelityLo & \bmAbENoRetrievalFidelityHi & \bmAbENoRetrievalFidelityP \\40 & usefulness & \bmAbENoRetrievalUsefulnessDiff & \bmAbENoRetrievalUsefulnessLo & \bmAbENoRetrievalUsefulnessHi & \bmAbENoRetrievalUsefulnessP \\41\midrule42E \texttt{no\_profile} & clarity & \bmAbENoProfileClarityDiff & \bmAbENoProfileClarityLo & \bmAbENoProfileClarityHi & \bmAbENoProfileClarityP \\43 & argument logic & \bmAbENoProfileArgumentLogicDiff & \bmAbENoProfileArgumentLogicLo & \bmAbENoProfileArgumentLogicHi & \bmAbENoProfileArgumentLogicP \\44 & contribution positioning & \bmAbENoProfileContributionPositioningDiff & \bmAbENoProfileContributionPositioningLo & \bmAbENoProfileContributionPositioningHi & \bmAbENoProfileContributionPositioningP \\45 & methodological rigour & \bmAbENoProfileMethodologicalRigorDiff & \bmAbENoProfileMethodologicalRigorLo & \bmAbENoProfileMethodologicalRigorHi & \bmAbENoProfileMethodologicalRigorP \\46 & fidelity & \bmAbENoProfileFidelityDiff & \bmAbENoProfileFidelityLo & \bmAbENoProfileFidelityHi & \bmAbENoProfileFidelityP \\47 & usefulness & \bmAbENoProfileUsefulnessDiff & \bmAbENoProfileUsefulnessLo & \bmAbENoProfileUsefulnessHi & \bmAbENoProfileUsefulnessP \\48\midrule49E \texttt{no\_factcheck} & clarity & \bmAbENoFactcheckClarityDiff & \bmAbENoFactcheckClarityLo & \bmAbENoFactcheckClarityHi & \bmAbENoFactcheckClarityP \\50 & argument logic & \bmAbENoFactcheckArgumentLogicDiff & \bmAbENoFactcheckArgumentLogicLo & \bmAbENoFactcheckArgumentLogicHi & \bmAbENoFactcheckArgumentLogicP \\51 & contribution positioning & \bmAbENoFactcheckContributionPositioningDiff & \bmAbENoFactcheckContributionPositioningLo & \bmAbENoFactcheckContributionPositioningHi & \bmAbENoFactcheckContributionPositioningP \\52 & methodological rigour & \bmAbENoFactcheckMethodologicalRigorDiff & \bmAbENoFactcheckMethodologicalRigorLo & \bmAbENoFactcheckMethodologicalRigorHi & \bmAbENoFactcheckMethodologicalRigorP \\53 & fidelity & \bmAbENoFactcheckFidelityDiff & \bmAbENoFactcheckFidelityLo & \bmAbENoFactcheckFidelityHi & \bmAbENoFactcheckFidelityP \\54 & usefulness & \bmAbENoFactcheckUsefulnessDiff & \bmAbENoFactcheckUsefulnessLo & \bmAbENoFactcheckUsefulnessHi & \bmAbENoFactcheckUsefulnessP \\55\bottomrule56\end{tabular}57\end{table}5859\subsection{Robustness to the judge pass}60\label{sec:ablations-judge}6162The scores analysed above are those of the first judge pass. The second pass on a random quarter of the sets (\bmJudgePairs{} re-judged outputs) gives a mean absolute difference of \bmJudgeMAD{} points across dimensions, with per-dimension values reported in Section~\ref{sec:results-judge}. The dimension that carries most of the H3 result, methodological rigour, is among the most rank-stable ($\rho = \bmJudgeRhoMethodologicalRigor$, MAD \bmJudgeMadMethodologicalRigor), so a change of presentation order is unlikely to reverse a difference of the size observed between E and D. Fidelity is somewhat less stable ($\rho = \bmJudgeRhoFidelity$, MAD \bmJudgeMadFidelity), and the H3 fidelity difference (\bmHEvsDFidelityDiff{} points) is of the same order as the judge's own instability on that dimension; it meets the preregistered criterion, but it should be held more loosely than the rigour result. Clarity ($\rho = \bmJudgeRhoClarity$) and usefulness ($\rho = \bmJudgeRhoUsefulness$) are the least rank-stable, which cuts both ways for the usefulness trade-off: the decline under E is small enough to be within judge noise, and the absence of a clarity difference is weak evidence. The check does not address the possibility that the judge systematically prefers, or systematically penalises, the hedged style of the referee pass, which is prompted with the same vendor's models and shares the rubric's vocabulary of ``bounded claim'' and ``identification threat''; same-vendor affinity is a known limitation of model-judged evaluations \citep{panickssery2024self, wang2024notfair} and cannot be ruled out here.6364\subsection{Contamination and leakage checks}65\label{sec:ablations-contamination}6667Four checks were run or recorded. (i)~\emph{Split discipline.} The two finance documents belong to the frozen held-out split and were never used to build profiles, the retrieval index, or the prompts; retrieval in conditions C and D was restricted to training documents, and every retrieved unit logged with the runs comes from a training document. (ii)~\emph{Withheld introductions.} The published introduction was withheld from every finance fixture, so the introduction task could not be solved by copying visible text; the same holds for the out-of-domain manuscript's introduction. (iii)~\emph{Overlap.} As reported in Section~\ref{sec:results-h5}, the preregistered overlap ceiling was exceeded by the metric (maximum \bmOverlapMax{}, \bmOverlapBelow\% of outputs below the ceiling) but only because the index contains the output's own source document and several briefs require reusing its material; against every \emph{other} corpus document the maximum is \bmOverlapOtherMax{} and all outputs are below the ceiling. No output reproduces an eight-word sequence from a training document at a share that would suggest recall of a specific paper, which is the pretraining-contamination signature this check was designed to detect \citep{sainz2023contamination, carlini2023memorization}. (iv)~\emph{Citation verification.} Every author--year string in every output was checked against the source reference list; the few unverifiable strings are inexact renderings of listed works, and no condition cited a work absent from the list. None of these checks can establish that the generator did not see the held-out articles during pretraining; they establish that the outputs do not visibly depend on such exposure. Two further points limit what they mean. The generator and the judge come from the same vendor, so a shared stylistic prior could inflate agreement between what E writes and what the judge rewards, independently of recall. And the judge's own counts show that the stronger single-call prompts did add characterisations of cited works and of the setting that the material does not contain (Section~\ref{sec:human-modes}); whether those additions are recalled from the published articles or generated from general knowledge cannot be distinguished by this design.6869\subsection{Cost and latency trade-off}70\label{sec:ablations-cost}7172The gains attributed to E are bought with a second generation call. Per output, E costs \bmSecECostUsd{}~USD and takes \bmSecELatencyS{}~s against \bmSecDCostUsd{}~USD and \bmSecDLatencyS{}~s for D, and \bmSecACostUsd{}~USD and \bmSecALatencyS{}~s for the minimal baseline; the no-referee ablation (\bmSecENoRefereeCostUsd{}~USD, \bmSecENoRefereeLatencyS{}~s) shows that the deterministic audit itself adds almost nothing and that essentially all of the extra cost is the referee call. The ablations that keep the referee pass but drop retrieval or the profile cost about as much as full E (\bmSecENoRetrievalCostUsd{} and \bmSecENoProfileCostUsd{}~USD) and score about as well or marginally better, which suggests that, within this pilot, a configuration consisting of the strong prompt plus audit plus referee pass would retain E's measured advantage on rigour and fidelity at the same two-call cost. Whether a single-call condition given the same budget would close the gap is a question for a follow-up.73