1% paper/sections/ablations.tex — WP45 / JoF-2502% Ablations of E, judge robustness, contamination and leakage checks. Recorded results only (results_heldout_v1.json); numbers are macros.3\providecommand{\TODOmacro}[1]{\textcolor{red}{[TODO macro: \texttt{#1}]}}45\section{Ablations, robustness, contamination and leakage checks}6\label{sec:ablations}78\subsection{Ablations of the full engine}9\label{sec:ablations-e}1011Each ablation removes one ingredient from E and is compared with full E by the same paired procedure as the hypotheses (Table~\ref{tab:ablations}; mean paired difference E minus ablation, cluster-bootstrap interval, Wilcoxon $p$ without Holm adjustment). The ablations were run once, on the first repetition only, so each contrast rests on \bmSecENoRefereeN{} paired units; the intervals are correspondingly wide and only a large effect could be detected.1213The referee pass is the active ingredient. Removing it lowers methodological rigour by \bmAbENoRefereeMethodologicalRigorDiff{} points (95\% CI [\bmAbENoRefereeMethodologicalRigorLo, \bmAbENoRefereeMethodologicalRigorHi], $p = \bmAbENoRefereeMethodologicalRigorP$), argument logic by \bmAbENoRefereeArgumentLogicDiff{} ([\bmAbENoRefereeArgumentLogicLo, \bmAbENoRefereeArgumentLogicHi], $p = \bmAbENoRefereeArgumentLogicP$), and fidelity by \bmAbENoRefereeFidelityDiff{} ([\bmAbENoRefereeFidelityLo, \bmAbENoRefereeFidelityHi], $p = \bmAbENoRefereeFidelityP$); usefulness falls by \bmAbENoRefereeUsefulnessDiff{} with an interval whose lower bound touches zero ([\bmAbENoRefereeUsefulnessLo, \bmAbENoRefereeUsefulnessHi]). The condition means confirm the reading: E without the referee pass scores close to D on rigour (Table~\ref{tab:judge_means}), and its secondary outcomes (cost \bmSecENoRefereeCostUsd{}~USD, latency \bmSecENoRefereeLatencyS{}~s, \bmSecENoRefereeNWords{} words) are those of a single-call condition. In other words, the draft stage of E, which is D plus an audit the draft cannot act on, is not what separates E from D; the second call, which receives the audit and must fix every item while keeping every verified fact, is.1415The other three ablations are null within the pilot's power. Removing the structured retrieval units changes rigour by \bmAbENoRetrievalMethodologicalRigorDiff{} ([\bmAbENoRetrievalMethodologicalRigorLo, \bmAbENoRetrievalMethodologicalRigorHi]), fidelity by \bmAbENoRetrievalFidelityDiff{} ([\bmAbENoRetrievalFidelityLo, \bmAbENoRetrievalFidelityHi]), and clarity by \bmAbENoRetrievalClarityDiff{} ([\bmAbENoRetrievalClarityLo, \bmAbENoRetrievalClarityHi]). Removing the profile summary changes rigour by \bmAbENoProfileMethodologicalRigorDiff{} ([\bmAbENoProfileMethodologicalRigorLo, \bmAbENoProfileMethodologicalRigorHi]) and fidelity by \bmAbENoProfileFidelityDiff{} ([\bmAbENoProfileFidelityLo, \bmAbENoProfileFidelityHi]); the point estimates on clarity (\bmAbENoProfileClarityDiff{}) and contribution positioning (\bmAbENoProfileContributionPositioningDiff{}) favour the ablation, with intervals that include zero. Removing the deterministic audit so that the referee pass works unaided changes rigour by \bmAbENoFactcheckMethodologicalRigorDiff{} ([\bmAbENoFactcheckMethodologicalRigorLo, \bmAbENoFactcheckMethodologicalRigorHi], $p = \bmAbENoFactcheckMethodologicalRigorP$) and fidelity by \bmAbENoFactcheckFidelityDiff{} ([\bmAbENoFactcheckFidelityLo, \bmAbENoFactcheckFidelityHi]). On the deterministic secondary outcomes, the audit does appear to matter for one thing it is designed to catch: the mean count of new numbers absent from the source is \bmSecENoFactcheckNewNumbers{} without the audit against \bmSecENewNumbers{} with it, while the unsupported-claim count is similar (\bmSecENoFactcheckNUnsupportedClaims{} against \bmSecENUnsupportedClaims{}). These are condition means on different units and were not tested.1617Two cautions apply. With \bmSecENoRetrievalN{} units per ablation, an effect of half a point on the seven-point scale would often fall inside the intervals above, so ``null within the pilot's power'' does not mean ``no effect''. And the ablations were judged in the same blinded sets as the main conditions, so each judge set contained up to nine outputs of the same (task, document), which lengthens the judge's context and may compress differences among the E-family variants that share the referee pass.1819\begin{table}[htbp]20\centering21\small22\caption{Ablations of the full engine: mean paired difference (E minus ablation) on the 1--7 judge scale, cluster-bootstrap 95\% interval, and paired Wilcoxon $p$ (unadjusted). Finance held-out documents, first repetition only (\bmSecENoRefereeN{} paired units per contrast). Values are macros generated from \texttt{results\_\allowbreak{}heldout\_\allowbreak{}v1.\allowbreak{}json}.}23\label{tab:ablations}24\begin{tabular}{llrrrr}25\toprule26Ablation & Dimension & $\Delta$ & CI low & CI high & $p$ \\27\midrule28E \texttt{no\_referee} & clarity & \bmAbENoRefereeClarityDiff & \bmAbENoRefereeClarityLo & \bmAbENoRefereeClarityHi & \bmAbENoRefereeClarityP \\29 & argument logic & \bmAbENoRefereeArgumentLogicDiff & \bmAbENoRefereeArgumentLogicLo & \bmAbENoRefereeArgumentLogicHi & \bmAbENoRefereeArgumentLogicP \\30 & contribution positioning & \bmAbENoRefereeContributionPositioningDiff & \bmAbENoRefereeContributionPositioningLo & \bmAbENoRefereeContributionPositioningHi & \bmAbENoRefereeContributionPositioningP \\31 & methodological rigour & \bmAbENoRefereeMethodologicalRigorDiff & \bmAbENoRefereeMethodologicalRigorLo & \bmAbENoRefereeMethodologicalRigorHi & \bmAbENoRefereeMethodologicalRigorP \\32 & fidelity & \bmAbENoRefereeFidelityDiff & \bmAbENoRefereeFidelityLo & \bmAbENoRefereeFidelityHi & \bmAbENoRefereeFidelityP \\33 & usefulness & \bmAbENoRefereeUsefulnessDiff & \bmAbENoRefereeUsefulnessLo & \bmAbENoRefereeUsefulnessHi & \bmAbENoRefereeUsefulnessP \\34\midrule35E \texttt{no\_retrieval} & clarity & \bmAbENoRetrievalClarityDiff & \bmAbENoRetrievalClarityLo & \bmAbENoRetrievalClarityHi & \bmAbENoRetrievalClarityP \\36 & argument logic & \bmAbENoRetrievalArgumentLogicDiff & \bmAbENoRetrievalArgumentLogicLo & \bmAbENoRetrievalArgumentLogicHi & \bmAbENoRetrievalArgumentLogicP \\37 & contribution positioning & \bmAbENoRetrievalContributionPositioningDiff & \bmAbENoRetrievalContributionPositioningLo & \bmAbENoRetrievalContributionPositioningHi & \bmAbENoRetrievalContributionPositioningP \\38 & methodological rigour & \bmAbENoRetrievalMethodologicalRigorDiff & \bmAbENoRetrievalMethodologicalRigorLo & \bmAbENoRetrievalMethodologicalRigorHi & \bmAbENoRetrievalMethodologicalRigorP \\39 & fidelity & \bmAbENoRetrievalFidelityDiff & \bmAbENoRetrievalFidelityLo & \bmAbENoRetrievalFidelityHi & \bmAbENoRetrievalFidelityP \\40 & usefulness & \bmAbENoRetrievalUsefulnessDiff & \bmAbENoRetrievalUsefulnessLo & \bmAbENoRetrievalUsefulnessHi & \bmAbENoRetrievalUsefulnessP \\41\midrule42E \texttt{no\_profile} & clarity & \bmAbENoProfileClarityDiff & \bmAbENoProfileClarityLo & \bmAbENoProfileClarityHi & \bmAbENoProfileClarityP \\43 & argument logic & \bmAbENoProfileArgumentLogicDiff & \bmAbENoProfileArgumentLogicLo & \bmAbENoProfileArgumentLogicHi & \bmAbENoProfileArgumentLogicP \\44 & contribution positioning & \bmAbENoProfileContributionPositioningDiff & \bmAbENoProfileContributionPositioningLo & \bmAbENoProfileContributionPositioningHi & \bmAbENoProfileContributionPositioningP \\45 & methodological rigour & \bmAbENoProfileMethodologicalRigorDiff & \bmAbENoProfileMethodologicalRigorLo & \bmAbENoProfileMethodologicalRigorHi & \bmAbENoProfileMethodologicalRigorP \\46 & fidelity & \bmAbENoProfileFidelityDiff & \bmAbENoProfileFidelityLo & \bmAbENoProfileFidelityHi & \bmAbENoProfileFidelityP \\47 & usefulness & \bmAbENoProfileUsefulnessDiff & \bmAbENoProfileUsefulnessLo & \bmAbENoProfileUsefulnessHi & \bmAbENoProfileUsefulnessP \\48\midrule49E \texttt{no\_factcheck} & clarity & \bmAbENoFactcheckClarityDiff & \bmAbENoFactcheckClarityLo & \bmAbENoFactcheckClarityHi & \bmAbENoFactcheckClarityP \\50 & argument logic & \bmAbENoFactcheckArgumentLogicDiff & \bmAbENoFactcheckArgumentLogicLo & \bmAbENoFactcheckArgumentLogicHi & \bmAbENoFactcheckArgumentLogicP \\51 & contribution positioning & \bmAbENoFactcheckContributionPositioningDiff & \bmAbENoFactcheckContributionPositioningLo & \bmAbENoFactcheckContributionPositioningHi & \bmAbENoFactcheckContributionPositioningP \\52 & methodological rigour & \bmAbENoFactcheckMethodologicalRigorDiff & \bmAbENoFactcheckMethodologicalRigorLo & \bmAbENoFactcheckMethodologicalRigorHi & \bmAbENoFactcheckMethodologicalRigorP \\53 & fidelity & \bmAbENoFactcheckFidelityDiff & \bmAbENoFactcheckFidelityLo & \bmAbENoFactcheckFidelityHi & \bmAbENoFactcheckFidelityP \\54 & usefulness & \bmAbENoFactcheckUsefulnessDiff & \bmAbENoFactcheckUsefulnessLo & \bmAbENoFactcheckUsefulnessHi & \bmAbENoFactcheckUsefulnessP \\55\bottomrule56\end{tabular}57\end{table}5859\subsection{Robustness to the judge pass}60\label{sec:ablations-judge}6162The scores analysed above are those of the first judge pass. The second pass on a random quarter of the sets (\bmJudgePairs{} re-judged outputs) gives a mean absolute difference of \bmJudgeMAD{} points across dimensions, with the per-dimension rank correlations reported in Section~\ref{sec:results-judge}. The dimension that carries most of the H3 result, methodological rigour, is among the most rank-stable ($\rho = \bmJudgeRhoMethodologicalRigor$, MAD \bmJudgeMadMethodologicalRigor), so a change of presentation order is unlikely to reverse a difference of more than a point. Fidelity is less stable (\mbox{$\rho = \bmJudgeRhoFidelity$}, MAD \bmJudgeMadFidelity), and the H3 fidelity difference (\bmHEvsDFidelityDiff{} points) is of the same order as the judge's own instability on that dimension; it meets the preregistered criterion, but it should be held more loosely than the rigour result. The check does not address the possibility that the judge systematically prefers the style of the referee pass, which is prompted with the same vendor's models and shares the same notions of ``bounded claim'' and ``identification threat'' that the rubric rewards; this same-vendor affinity is a known limitation of model-judged evaluations \citep{panickssery2024self, wang2024notfair} and cannot be ruled out here.6364\subsection{Contamination and leakage checks}65\label{sec:ablations-contamination}6667Four checks were run or recorded. (i)~\emph{Split discipline.} The two finance documents belong to the frozen held-out split and were never used to build profiles, the retrieval index, or the prompts; retrieval in conditions C and D was restricted to training documents, and the retrieved units logged with each run come from training documents only. (ii)~\emph{Withheld introductions.} The published introduction was withheld from every finance fixture, so the introduction task could not be solved by copying visible text. (iii)~\emph{Overlap.} As reported in Section~\ref{sec:results-h5}, the preregistered overlap ceiling was exceeded by the metric (maximum \bmOverlapMax{}, \bmOverlapBelow\% of outputs below the ceiling) but only because the index contains the output's own source document and several briefs require reusing its material; against every \emph{other} corpus document the maximum is \bmOverlapOtherMax{} and all outputs are below the ceiling. No output reproduces an eight-word sequence from a training document at a share that would suggest recall of a specific paper, which is the pretraining-contamination signature this check was designed to detect \citep{sainz2023contamination, carlini2023memorization}. (iv)~\emph{Citation verification.} Every author--year string in every output was checked against the source reference list, and none was unverifiable in any condition; as noted in Section~\ref{sec:results-h4}, this is because all conditions received the reference list and emitted placeholders rather than invented references, so the check constrains behaviour rather than measuring a difference. None of these checks can establish that the generator did not see the held-out articles during pretraining; they establish that the outputs do not visibly depend on such exposure. Two further points limit what the contamination checks mean. The generator and the judge come from the same vendor, so a shared stylistic prior could inflate agreement between what E writes and what the judge rewards, independently of recall. And the material defect described in Section~\ref{sec:human-defect} means that, for one finance document, the generator had no manuscript content to leak from; the outputs on that document are an unintended ``no-material'' control in which any specific fact would have had to come from recall, and the judge's counts of invented facts on those sets (Section~\ref{sec:human-invented}) show that the stronger single-call prompts did assert market-structure and design details that the material did not contain.6869\subsection{Cost and latency trade-off}70\label{sec:ablations-cost}7172The gains attributed to E are bought with a second generation call. Per output, E costs \bmSecECostUsd{}~USD and takes \bmSecELatencyS{}~s against \bmSecDCostUsd{}~USD and \bmSecDLatencyS{}~s for D, and \bmSecACostUsd{}~USD and \bmSecALatencyS{}~s for the minimal baseline; the no-referee ablation (\bmSecENoRefereeCostUsd{}~USD, \bmSecENoRefereeLatencyS{}~s) shows that the deterministic audit itself adds almost nothing, and that essentially all of the extra cost is the referee call. The ablations that keep the referee pass but drop retrieval or the profile cost about as much as full E (\bmSecENoRetrievalCostUsd{} and \bmSecENoProfileCostUsd{}~USD) and score about as well, which suggests that, within this pilot, a cheaper configuration consisting of the strong prompt plus audit plus referee pass would retain most of E's measured advantage; this is a hypothesis for a follow-up, not a result of the present design, since the two-call budget was never given to the single-call conditions.73