SPB Git

spb/wp29_uqo

Public
5.6 KB · 13 lines latex
1\section{Introduction}2\label{sec:intro}34Autonomous research systems built on large language models now produce manuscripts end to end, and their authors report low costs and reviewer scores near acceptance thresholds \citep{lu2024aiscientist,yamada2025aiscientistv2,ifargan2025datatopaper}. Independent audits of the same systems report data leakage, metric misuse, post-hoc selection and fabricated novelty claims that are visible in execution traces but not in the papers \citep{luo2025pitfalls,beel2025sakana}. Between these two literatures lies a practical question that researchers who delegate work to coding agents face every day: does it matter \emph{how} the delegation is written? The instruction files that steer such agents range from a sentence to several thousand words of phases, gates, ledgers and integrity rules, and the longer ones read like preregistrations. Whether that structure changes what the agent delivers, and at what cost, has not been measured with the agent held fixed.56This paper measures it for one agent. We ran \nMainRuns{} live Claude Code sessions (client 2.1.314, model \texttt{claude-opus-5-5}) on \nConfTasks{} bounded scientific tasks in six families (data audit, estimation and inference, forecasting under information constraints, source-grounded synthesis, theory and numerical validation, integrated mini-projects), \nRepeats{} fresh sessions per task under each of four instruction regimes: an open instruction (A); a structured research protocol with phases, ledgers, stopping and recovery rules (B); the protocol plus an executable, controller-enforced gate that checks the deliverable contract and reproducibility and returns failures to the session (C); and the same plus a separate review-and-repair session (D). The executors ran in bare mode on cluster nodes that held no copy of the reference answers; every run was scored by a deterministic, task-specific evaluator on a separate host, validated beforehand on seeded correct and incorrect artefacts. The primary endpoint is intervention-free verified completion within a shared resource envelope; the primary contrast is D$-$A, with task-cluster bootstrap intervals and permutation tests at the level of the \nConfTasks{} task clusters.78The answer is a precise null. Completion was $\pA$ under the open instruction and $\pD$ under the full method; the paired difference is $\dDA$ with a 95\% interval of $[\dDAlo, \dDAhi]$, and the mechanism contrasts (B$-$A, C$-$B, D$-$C) are as flat. Critical integrity violations were rare in every arm and were dominated by a single task whose oracle, not whose executors, was wrong. What the protocol, the gate and the reviewer did buy was contract compliance (no arm C or D deliverable failed the post-hoc contract check; \nContractFailA{} arm-A and \nContractFailB{} arm-B deliverables did) and a two- to three-fold increase in inference cost per run; the quality--cost frontier is flat. A length-matched unstructured instruction, interruption-and-resumption experiments, stale-input, duplicate-job and missing-fixture faults, and three held-out end-to-end mini-projects complete the picture (Sections~\ref{sec:recovery} and~\ref{sec:transfer}).910We do not read the null as evidence that research protocols are useless. We read it as evidence about where the difficulty of delegated research actually lies for a current frontier model on bounded tasks: not in organising the work, which the model does competently without being told, but in \emph{knowing what the right answer is}. The two tasks that every arm ``failed'' were ones where twelve independent sessions behaved like careful scientists (reporting an undefined month as undefined; excluding impossible negative training hours with a documented ledger) and our oracles, written by a fresh session of the same client family and validated on seeded artefacts, had encoded a less careful answer. The cost of verification in this study was not the agents' inference budget but the construction and validation of tasks whose answers are known, oracles the executor cannot read, and checks that do not trust an exit code; the harness-level defects we caught along the way (an evaluator invoked on the wrong workspace, a ledger key collision that aborted a run after a reviewer asked for revisions) would otherwise have produced confident, wrong conclusions in either direction.1112Contributions. (i)~A delegation-contract formalisation and a small, auditable reference implementation (controller, durable state, isolated bare-mode executors, oracle-free runtime gate, separate review, deterministic hidden evaluators, authentic capture) in which every invariant that is claimed is enforced by code and every limit is stated. (ii)~A benchmark of \nConfTasks{} confirmatory and \nDevTasks{} development tasks with hidden oracles, seeded validation sets and provenance for \nConfRealData{} real-data and \nConfSourceDocs{} source-document tasks, plus three transfer mini-projects. (iii)~The first controlled comparison, to our knowledge, of instruction regimes with the executing agent, tools, inputs and budget held fixed and the oracles protected from it, preregistered and frozen with hashes before the confirmatory runs. (iv)~A fully instrumented account of how this paper itself was produced under a mandate of the same kind, kept separate from the confirmatory evidence. Section~\ref{sec:related} positions the work; Sections~\ref{sec:framework}--\ref{sec:evaluation} describe the framework, the corpus of researcher-written mandates that motivated it, the implementation and the design; Sections~\ref{sec:results}--\ref{sec:transfer} report the results; Section~\ref{sec:reflexive} is the reflexive account; Section~\ref{sec:limitations} states what the study cannot establish.13
Files · 152a55b8fa2830f50fa35e5aed99c0037f713345
Menu

Theme