SPB Git

spb/wp29_uqo

Public
1.7 KB · 7 lines latex
1\section{Conclusion}2\label{sec:conclusion}34We asked whether an explicit research protocol, executable verification gates, persistent evidence records and a separate review stage improve a current coding agent's ability to complete bounded scientific tasks autonomously and validly, and what the improvement costs. For Claude Code 2.1.314 with \texttt{claude-opus-5-5}, on \nConfTasks{} tasks and \nMainRuns{} live runs under matched information, tools and budgets, the answer is that completion and integrity do not change ($\hat\Delta_{DA} = \dDA$, 95\% interval $[\dDAlo, \dDAhi]$), contract compliance does, and cost rises two- to three-fold. The ceiling under the open instruction was high; every failure that remained was, on inspection, an evaluator defect; and the harness caught its own infrastructure faults only because its checks did not trust success declarations.56The result is bounded by the tasks (bounded, with known answers, of moderate difficulty), the model and client version, the one-week execution window, and the fact that the same organisation authored tasks, oracles and treatments. It does not say that protocols are useless for longer, open-ended research, where the recovery and provenance mechanisms we tested in isolation (Section~\ref{sec:recovery}) have more to do; it does say that, for the kind of task on which a laboratory would first consider delegating, the marginal value of structure lies in auditability rather than in correctness, and that the scarce resource is a trustworthy oracle. Both conclusions argue for the same investment: specify the deliverable contract and the reference answers before the agent runs, protect them from it, and spend the budget on checking the checker.7
Files · 46a1f53512bec3cd5d9c16e8fc9d9da4129c09cc
Menu

Theme