1% paper/sections/human_failure.tex — WP45 / JoF-2502% Human evaluation status and qualitative failure analysis grounded in experiments/outputs/heldout_v1 (outputs/*.tex, judgments.jsonl, runs.jsonl).3% Quoted fragments are from GENERATED outputs or judge justifications only, never from the source articles. Numbers are macros.4\providecommand{\TODOmacro}[1]{\textcolor{red}{[TODO macro: \texttt{#1}]}}56\section{Human evaluation and failure analysis}7\label{sec:human}89\subsection{Human evaluation}10\label{sec:human-raters}1112No human raters were available for this study. The rating protocol, calibration procedure, and reliability plan were prepared (\texttt{experiments/\allowbreak{}protocols/\allowbreak{}human\_\allowbreak{}rating\_\allowbreak{}sheet.\allowbreak{}md}; Section~\ref{sec:design-human}) and the blinded outputs exist, but no rating was collected. Every quality score in Sections~\ref{sec:results} and~\ref{sec:ablations} is therefore a rating by \texttt{claude-sonnet-5-5}, and the mandate's H1, phrased in terms of blind human ratings, remains untested in its original form. This section offers instead a qualitative reading of the outputs and of the judge's justifications: it shows \emph{how} the conditions fail, not \emph{how much}.1314\subsection{A material defect discovered in the failure analysis}15\label{sec:human-defect}1617Reading the outputs revealed a defect that the run accounting did not. For the second finance document, every condition reported receiving only the title: the baseline introduction opens with ``The supplied material contains only the title. There was no abstract, section text, tables, figures, or BibTeX keys,'' and the full-engine referee report describes its material as ``the title and one section [\ldots] No abstract, results section, tables, figures, data description, or bibliography were supplied.'' The fixture file itself is complete. The cause is in the material builder, which collapses each Markdown fixture into one top-level section and assigns it a role: for the first document the role was ``results'' and the whole text was passed; for the second it was ``other'' and the text was dropped, leaving the title, a section list, and a word count. The judge saw the same thing (``The source contains only a title, so every output is a scaffold''). Half of the finance units were therefore no-material tasks on which the only correct behaviour was to scaffold and flag, which the rubric rewarded. In keeping with the preregistration, no run was dropped or repeated; Table~\ref{tab:rigor_by_task} and the condition means mix the two situations, and the protocol must be re-run with the builder fixed before any effect size is cited as a property of the engine.1819\subsection{Recurring failure modes}20\label{sec:human-modes}2122Quoted fragments below are from generated outputs.2324\paragraph{Statements beyond the material.} The commonest fault in A--D is a confident sentence the source does not support. The strong-prompt introduction asserts that eligibility was ``determined by a set of criteria that includes a threshold [\ldots] in annual revenues''; its specification explanation states, of the sample window, that ``we did not choose them to fit the data''; its results interpretation calls the headline estimate ``a lower bound on the importance of long maturities for trade with small suppliers'' and then asks the author to confirm. On the no-material document, B attributed findings to cited papers (``Where that work relies on the specialized nature of a market, I observe directly which institutions trade each contract''), which the judge counted as fabricated characterisations of prior work. D is not immune: the judge recorded that it ``invents the relational-contract heterogeneity finding'' and, elsewhere, asserted holder types and data sources ``none of which is supported by the source.''2526\paragraph{Over-stated causal and priority language.} The baseline introduction says the regulation ``exogenously limited the maturity of the trade credit'' and that the authors ``examine the parallel trends assumption directly''; the judge marked both as over-statements. Priority claims appear under the strong prompt (``to our knowledge first evidence''; ``first controlled test'' in the out-of-domain abstract) and were penalised as unsupported.2728\paragraph{Truncation.} Many long outputs end mid-sentence (the strong-prompt introduction stops at ``became more rest'') because the shared output limit was reached. The judge noted truncation in several referee-report, results-interpretation, and causal-audit sets, in every condition; it is a cost of the fixed budget, not of any ingredient, but a truncated referee report cannot reach its recommendation.2930\paragraph{Placeholder-heavy outputs.} Where material was missing, every condition degenerated into a template; the full-engine abstract on the no-material document is a chain of placeholders (``\textbackslash todo\{MAIN RESULT: the sign of the effect of limited capital on credit spreads, worded as in the results section.\}'')). Even on the complete document, E's introduction was criticised because ``its bullet-list format is awkward for an introduction''.3132\paragraph{Wrong deliverable.} The referee-report task was implemented with the section role ``results,'' so the generic task line asked for a results section while the brief asked for a referee report. Several outputs followed the task line: the baseline on the no-material document writes ``The research brief also asks for a referee report, while the task line asks for the results section,'' then delivers a results scaffold, and the judge placed it, with B and C, at the bottom of the set. On the out-of-domain manuscript only C produced a referee report; E produced a results rewrite. Part of E's advantage on \texttt{referee\_report} in Table~\ref{tab:rigor_by_task} is thus format repair: its referee pass on the no-material document records that it ``reframed the output as a referee report'' because the brief asked for one. This is a defect of the task definition that a re-run should fix.3334\subsection{What the referee pass changes}35\label{sec:human-referee}3637The E change logs show the mechanism behind H3. The pass removes numbers the audit flagged as new (``Removed the unsupported `10'. [\ldots] It came from an example star convention that does not appear in the material''), deletes invented structure (``The draft said `Columns differ in the fixed effects included' and gave example fixed effects. The material does not describe the columns''), and rewrites assertions as conditionals (``Recast as a conditional hypothesis [\ldots] The paper's evidence is not presupposed''). The hedges show in the revised literature rewrite (``We view it as an interpretation of the pattern of adjustments rather than a direct estimate of integration costs''). The pass is not infallible. The judge found E's table notes claiming an assumption ``is examined in Section II.A and Section II.C'' without verification, found its results interpretation presenting ``parallel pre-trends and divergence as features of the evidence before verification,'' and on one referee-report set ranked the strong prompt above E for breadth; on the no-material literature rewrite E was ``verbose and fragmented with many TODOs'' and added unsupported claims of its own. The audit also produces false positives: several change logs explain that a number flagged as missing is ``the section word count from the outline, not a manuscript fact.''3839\subsection{Invented facts counted by the judge}40\label{sec:human-invented}4142Alongside the six scores, the judge counted invented facts per output: on average \bmANInventedFacts{} under A, \bmBNInventedFacts{} under B, \bmCNInventedFacts{} under C, \bmDNInventedFacts{} under D, and \bmENInventedFacts{} under E (Table~\ref{tab:judge_means}). The ordering matches the failure modes above: the minimal prompt asserts less and so invents less, retrieval adds material the model then attributes to the paper, and the referee pass removes most of it. These are judge counts, not verified counts, and fidelity is the judge's least stable dimension between passes (Section~\ref{sec:results-judge}).4344\subsection{Out-of-domain manuscript}45\label{sec:human-ood}4647The out-of-domain manuscript, the author's WP29, is a single document on three tasks, so its results are descriptive. On methodological rigour the conditions score \bmOodAMethodologicalRigor{} (A), \bmOodBMethodologicalRigor{} (B), \bmOodCMethodologicalRigor{} (C), \bmOodDMethodologicalRigor{} (D), and \bmOodEMethodologicalRigor{} (E); on fidelity \bmOodAFidelity{}, \bmOodBFidelity{}, \bmOodCFidelity{}, \bmOodDFidelity{}, and \bmOodEFidelity{}; on contribution positioning \bmOodAContributionPositioning{}, \bmOodBContributionPositioning{}, \bmOodCContributionPositioning{}, \bmOodDContributionPositioning{}, and \bmOodEContributionPositioning{}; on usefulness \bmOodAUsefulness{}, \bmOodBUsefulness{}, \bmOodCUsefulness{}, \bmOodDUsefulness{}, and \bmOodEUsefulness{}. Conventional retrieval is the best-rated condition on this manuscript, and the full engine does not improve fidelity over the baseline. Three reasons appear in the outputs. The finance profile is the wrong template: WP29 is a methods paper with a null result, and conventions on contribution placement, economic significance, and robustness do not describe it; the engine's genre note is meant to say so, but the outputs reused the finance move sequence. The manuscript's own introduction and results were visible in the LaTeX source, and the judge penalised every condition for reproducing details ``that appear only in the withheld text or are inferred''; E added ``preregistered'' to the abstract and a reading of the interval ``which spans roughly three percentage points against D to roughly six in its favour'' that the judge did not accept. And the referee-report task failed as a deliverable: C's output was ``the only output that is actually a referee report'' and the only one to engage the manuscript's specific weaknesses (``Under the corrected rubric the benchmark has no outcome variance in any arm''). The out-of-domain case shows the limit of the finance profiles rather than of the referee pass as such.48