SPB Git

spb/wp29_uqo

Public
8.1 KB · 49 lines latex
1\section{Conceptual framework: delegation contracts, autonomy and evidence}2\label{sec:framework}34\subsection{What is delegated when research is delegated}56When a researcher hands a scientific task to an autonomous agent, three things change hands: a description of what is wanted, a bundle of resources with which to obtain it, and the authority to decide, along the way, among the many admissible choices that any empirical analysis entails. The first two are explicit. The third is the source of most of the trouble, because the choices an agent makes on its own (which observations to drop, which standard errors to report, whether information was available when it was used, when to declare the work finished) are precisely the ones that determine whether the delivered artefact is scientifically valid. The ``garden of forking paths'' identified for human analysts \citep{gelman2014statisticalcrisis} does not disappear when the analyst is a language model; it is walked faster.78We therefore describe a delegation as a \emph{contract} rather than a prompt. A contract fixes, before execution, the questions and estimands, the inputs and their availability restrictions, the authorised actions, the ordering of phases, the verification criteria, the resource budget, the rules for recovery and amendment, and the evidence the executor must leave behind. Formally, a protocol is the tuple9\begin{equation}10\mathcal{P} = (Q, \Theta, \mathcal{D}, \mathcal{A}, \mathcal{G}, \mathcal{V}, \mathcal{B}, \mathcal{R}, \mathcal{L}),11\label{eq:contract}12\end{equation}13whose components are listed in Table~\ref{tab:contract} together with the concrete file or mechanism that serialises each of them in our implementation. We use the notation only as an index: it is not a theorem and it establishes nothing by itself. Its value is that every component has a place where it is written down, hashed and, where possible, checked by a machine rather than by the executor.1415\begin{table}[htbp]16\centering\small17\caption{Components of the delegation contract and their serialisation in the reference implementation.}18\label{tab:contract}19\begin{tabular}{p{1.1in}p{2.3in}p{2.9in}}20\toprule21Component & Meaning & Serialisation and consumer \\22\midrule23$Q$ & scientific question and scope & \texttt{TASK.md} (question section); read by executor and evaluator \\24$\Theta$ & estimands, hypotheses, admissible interpretations & \texttt{TASK.md} (estimand and availability rules); hidden \texttt{REFERENCE.md} on the evaluator host \\25$\mathcal{D}$ & inputs, versions, information availability & \texttt{data/} with \texttt{README.md}, \texttt{bundle.json} (SHA-256 of every file); verified on the node before launch \\26$\mathcal{A}$ & authorised actions and tools & launcher flags: bare mode, fixed tool inventory, no network, no sub-agents; identical across arms \\27$\mathcal{G}$ & phase dependency graph & protocol text (arms B--D) and the executor-maintained ledger \texttt{out/state/phases.json}; harness run-state machine \\28$\mathcal{V}$ & verification criteria & \texttt{gate\_spec.json} and the gate program (oracle-free, runtime); \texttt{evaluate.py} with \texttt{oracle.json} (hidden, final) \\29$\mathcal{B}$ & budgets and stopping & \texttt{configs/budgets.yaml}: USD cap, wall clock, turns, gate and review cycles; enforced by the client cap and a remote kill \\30$\mathcal{R}$ & recovery and amendment & protocol recovery rules; session resumption; attempt-numbered reruns; \texttt{protocol/amendments.csv} \\31$\mathcal{L}$ & evidence and provenance & SQLite state store (runs, transitions, sessions, events, receipts, reviews, screenshots, faults, jobs, interventions) and per-run directories \\32\bottomrule33\end{tabular}34\end{table}3536\subsection{Phase states and enforceable invariants}3738Within a contract, a phase is in one of seven states: \emph{pending}, \emph{running}, \emph{passed}, \emph{failed}, \emph{blocked}, \emph{stale} or \emph{skipped\_optional}. A phase becomes eligible only when its required predecessors are \emph{passed}; an artefact produced by a phase is valid only relative to the versions of inputs, code, configuration and criteria used to produce and check it, so that a change upstream makes downstream results \emph{stale} rather than silently wrong. At the level of the whole run, the harness keeps a second, smaller state machine (\emph{not\_started}, \emph{prepared}, \emph{running}, \emph{completed}, \emph{timeout}, \emph{budget\_exhausted}, \emph{error}, \emph{interrupted}, \emph{evaluated}, \emph{void}) whose transitions are validated in code.3940Five invariants are enforced mechanically. (1)~A run cannot be marked \emph{evaluated} without an evaluation receipt. (2)~A changed upstream input invalidates downstream results, because the gate regenerates the results from the replication script and compares them. (3)~A run cannot consume more than the envelope: each session is launched with a cap equal to the remaining budget, and the controller kills the process at the wall limit. (4)~A claim cannot be marked supported unless it references result keys that exist, and the final evaluator scores claims against protected references. (5)~A skipped optional phase cannot support a claim, because the final evaluator ignores phase ledgers and scores artefacts only. We state the limits of each checker in Section~\ref{sec:implementation}: these are workflow guarantees, not guarantees of truth. A gate that passes tells the researcher that the deliverable has the promised shape, reproduces from its own script and is internally consistent; whether it is \emph{right} is a separate question answered by an evaluator the executor never sees.4142\subsection{Observable autonomy}4344The word ``autonomous'' conflates two outcomes that must be kept apart. \emph{Workflow autonomy} means that the phases were executed without operational human guidance; \emph{scientific completion} means that the artefacts satisfy the independent task rubric. The cell of interest is \emph{intervention-free verified completion}, where both hold. Its complements are informative: \emph{autonomous execution with incorrect output} (no intervention, scientific failure) is the failure mode that audits of autonomous research systems describe \citep{luo2025pitfalls,beel2025sakana}, and \emph{correct assisted completion} (success after a substantive human intervention) is what a laboratory usually obtains in practice. The implementation records the four cells separately: the run status and terminal reason capture workflow autonomy; the evaluation receipt captures scientific completion; the intervention ledger (empty for every executor session in this study, because no human touched a running session) captures assistance. Every table and figure in the paper reports the primary endpoint as the conjunction and never relabels an assisted or budget-exceeding success as an autonomous one.4546\subsection{Why a controlled comparison}4748Published autonomous research systems typically present a single pipeline whose quality is judged by reviewer scores or by a few flagship outputs \citep{lu2024aiscientist,yamada2025aiscientistv2,gottweis2026coscientist}. Such designs cannot say which element of the pipeline (the staging, the self-review, the evidence records, the sheer length of the instructions) produced the outcome, nor whether the outcome is scientifically valid rather than persuasive. The present study holds the executor fixed (same client, same model, same tools, same inputs, same envelope) and varies only the organisation and checking of the work across four arms. The contrasts between adjacent arms isolate, as far as the platform permits, the marginal contribution of structure (B$-$A), of executable gates with controlled state transitions (C$-$B) and of a separate review-and-repair stage (D$-$C), while D$-$A measures the whole method against an information-matched open instruction. We do not claim that this identifies a ``pure effect of structure'': the protocol text is longer than the open instruction, gates return feedback and reviews consume part of the budget. A length-matched unstructured control addresses the first confound; the other two are properties of the treatments and are reported as costs rather than hidden.49
Files · main
Menu

Theme