% paper/sections/human_failure.tex — WP45 / JoF-250
% Human evaluation status and qualitative failure analysis grounded in experiments/outputs/heldout_v1 (run 2: outputs/*.tex, judgments.jsonl, runs.jsonl).
% Quoted fragments are from GENERATED outputs only, never from the source articles. Numbers are macros.
\providecommand{\TODOmacro}[1]{\textcolor{red}{[TODO macro: \texttt{#1}]}}

\section{Human evaluation and failure analysis}
\label{sec:human}

\subsection{Human evaluation}
\label{sec:human-raters}

No human raters were available for this study. The rating protocol, calibration procedure, and reliability plan were prepared (\texttt{experiments/\allowbreak{}protocols/\allowbreak{}human\_\allowbreak{}rating\_\allowbreak{}sheet.\allowbreak{}md}; Section~\ref{sec:design-human}) and the blinded outputs exist, but no rating was collected. Every quality score in Sections~\ref{sec:results} and~\ref{sec:ablations} is therefore a rating by \texttt{claude-sonnet-5-5}, and the mandate's H1, phrased in terms of blind human ratings, remains untested in its original form. This section offers instead a qualitative reading of the outputs and of the judge's justifications: it shows \emph{how} the conditions fail, not \emph{how much}.

\subsection{Defects found and corrected before the final run}
\label{sec:human-defects}

The failure analysis was first applied to the pipeline itself, and it changed the study. Reading the outputs of the first execution revealed two implementation defects that the run accounting and the deterministic metrics had not: the Markdown loader and material builder dropped the content of one held-out document (and, as a second check showed, the abstract, table notes, figures, and reference list of both), so that the generator and the judge saw only a title and headings for half of the finance units; and the \texttt{referee\_report} and \texttt{flag\_causal\_claims} tasks were framed with the generic ``write the section'' instruction in contradiction with their briefs, so that several outputs delivered a results scaffold instead of the deliverable asked for. Both are recorded as Amendment A-004 (with A-004b) in \texttt{experiments/\allowbreak{}preregistration.\allowbreak{}md} and \texttt{docs/\allowbreak{}PROGRESS.\allowbreak{}md}; the loader and builder were fixed, the two tasks were given an engine mode in which the brief is the task, the whole finance grid was re-run and all judge sets re-judged, and the first execution is archived under \texttt{experiments/\allowbreak{}outputs/\allowbreak{}heldout\_\allowbreak{}v1\_\allowbreak{}run1/} and is not used in this paper. Hypotheses, conditions, outcomes, and analysis plan were not changed. The episode is itself a finding: a benchmark of writing assistance can run to completion with clean metrics and still measure the wrong thing, and the check that caught it was reading the outputs.

\subsection{Recurring failure modes}
\label{sec:human-modes}

Quoted fragments below are from generated outputs of the final run.

\paragraph{Statements beyond the material.} The commonest fault in A--D is a confident sentence the source does not support, often about results the material does not report or about what cited papers found. The conventional-retrieval interpretation of results announces that ``seller capital shocks are strongly related to spread changes'' and that losses ``are followed by wider spreads,'' then inserts a placeholder noting that ``the material provided does not report them''; the draft without the referee pass likewise states that shocks are ``strongly related to subsequent spread movements, whereas buyer capital shocks are not.'' The profiles condition derives a figure the source never states, writing that one variable ``explains roughly two-thirds as much spread variation as the conventional determinants of credit spreads.'' In the literature rewrites, single-call conditions characterise cited works beyond the source, for instance aggregate proxies ``such as broker--dealer leverage or the equity capital ratio of primary dealers''; the judge counted these as invented characterisations and they recur under A, B, C, and D on both documents. Even where numbers are preserved exactly, as the judge confirmed for every abstract task, the surrounding prose adds mechanism, direction, or magnitude.

\paragraph{Over-stated causal and priority language.} The strong prompt describes the regulation as having ``exogenously restricted the maturity of trade credit contracts between a large retailer and a subset of its suppliers''; the baseline adds ``plausibly unrelated'' to a shock the source does not characterise that way; priority claims (``first'' and ``to our knowledge'') appear under B and C and were penalised as unsupported.

\paragraph{Truncation.} Many long outputs end mid-sentence because the shared output limit was reached; the judge recorded truncation in nearly every referee-report, results-interpretation, introduction, and causal-audit set, under every condition. It is a cost of the fixed budget rather than of any ingredient, but a truncated referee report never reaches its recommendation and a truncated introduction never reaches its contribution paragraph, which is where E lost positioning and usefulness points (Section~\ref{sec:results-tasks}).

\paragraph{Lists, placeholders, and preamble.} Several E outputs are list-like where prose is expected (an introduction organised as bullet points; a causal audit that opens with a legend, ``Where a classification depends on estimates that are not visible here, the entry states the condition under which it holds''), and several carry \verb|\todo| citation markers although the reference list was supplied, because no BibTeX keys existed and the engine refuses to invent them. The judge rated such outputs faithful and rigorous but less usable, which is the mechanism behind the usefulness trade-off in Section~\ref{sec:results-hypotheses}.

\paragraph{Empty and wrong-format outputs.} The baseline returned an empty file for the causal-claims audit on one document and was scored at the bottom of the scale; wrappers such as \verb|\begin{abstract}| and stray Markdown appear in several single-call outputs.

\subsection{What the referee pass changes}
\label{sec:human-referee}

The change logs of the E runs show the mechanism behind H3. The pass removes or re-anchors claims the audit flagged (``The original `we find that relational contracts can mitigate' overstated what was shown''), deletes structure the material does not support (``The draft asserted that the footer reports observations, number of firms, and $R^2$. This is not supported by the material''), and bounds interpretation (``The quality-guarantee interpretation is now labelled an inference from heterogeneity''). The hedges are visible in the revised texts: ``This comparison spans two settings and two designs, so it is suggestive rather than a test,'' and ``These interactions identify heterogeneity in the effect. They do not identify the causal effect of exclusivity itself.'' The pass is not infallible, and the judge found E adding unsupported items of its own: a ``Price'' outcome and cross-references the table task's material does not establish, an intention-to-treat label and a power claim in a specification explanation, and characterisations of cited works in a literature rewrite. The audit also produces false positives the pass must argue away; several logs explain that a number flagged as new ``is a tokenization artifact'' of a LaTeX percent sign, or that numbers flagged as missing are ``bibliographic page and volume numbers, not facts to preserve.'' Two further limits appear. When the draft is already a critique, as in the referee report, the pass has little to add and the revised report is no more rigorous than the strong prompt's. And the revised text is subject to the same output cap as the draft: one of E's introductions ends mid-sentence in its design paragraph, with no results and no contribution.

\subsection{Invented facts counted by the judge}
\label{sec:human-invented}

Alongside the six scores, the judge counted invented facts per output: on average \bmANInventedFacts{} under A, \bmBNInventedFacts{} under B, \bmCNInventedFacts{} under C, \bmDNInventedFacts{} under D, and \bmENInventedFacts{} under E (Table~\ref{tab:judge_means}). The ordering matches the failure modes above: each ingredient that adds material to the prompt, the strong instruction, raw excerpts, profiles and structured units, is followed by more statements the judge could not find in the source, and the referee pass removes about half of them. These are judge counts, not verified counts (Section~\ref{sec:results-judge}).

\subsection{Out-of-domain manuscript}
\label{sec:human-ood}

The out-of-domain manuscript, the author's WP29, is a single document on three tasks, so its results are descriptive. On methodological rigour the conditions score \bmOodAMethodologicalRigor{} (A), \bmOodBMethodologicalRigor{} (B), \bmOodCMethodologicalRigor{} (C), \bmOodDMethodologicalRigor{} (D), and \bmOodEMethodologicalRigor{} (E); on fidelity \bmOodAFidelity{}, \bmOodBFidelity{}, \bmOodCFidelity{}, \bmOodDFidelity{}, and \bmOodEFidelity{}; on contribution positioning \bmOodAContributionPositioning{}, \bmOodBContributionPositioning{}, \bmOodCContributionPositioning{}, \bmOodDContributionPositioning{}, and \bmOodEContributionPositioning{}; and on usefulness \bmOodAUsefulness{}, \bmOodBUsefulness{}, \bmOodCUsefulness{}, \bmOodDUsefulness{}, and \bmOodEUsefulness{}. There is no E advantage on this manuscript: the baseline matches E on rigour and exceeds it on fidelity and usefulness. Three reasons appear in the outputs. The finance profile is the wrong template: WP29 is a methods paper reporting a null result, and conventions on contribution placement, economic significance, and robustness do not describe it; the engine's genre note is meant to say so, but the outputs reused the finance move sequence. Fidelity was lost on details that are in the manuscript's withheld introduction or are inferred rather than stated: E's abstract calls the runs ``preregistered'' where the source speaks of a frozen design, and its introduction reads the reported interval as spanning ``roughly three percentage points against D to roughly six in its favour,'' which the judge did not accept as supported. And on the referee report every condition was truncated before its recommendation, so the comparison rests on the depth of the major comments, where the strong prompt was rated highest. The out-of-domain case shows the limit of the finance profiles rather than of the referee pass as such.