\section{Related work}
\label{sec:related}
\label{sec:literature}

Eight bodies of work bear on whether the structure of the instructions given to an autonomous coding agent changes the scientific validity of its output, and at what cost. We read them as an argument: autonomous research systems have been evaluated, but never under a design that isolates the instruction regime (\S\ref{sec:lit-systems}); benchmarks supply executable oracles that must be protected from the executor (\S\ref{sec:lit-benchmarks}); reproducibility research shows what re-execution buys (\S\ref{sec:lit-repro}); preregistration gives the template for a protocol fixed before execution (\S\ref{sec:lit-prereg}); human--automation research rules out self-reported efficiency (\S\ref{sec:lit-delegation}); durable-execution research supplies the measurement apparatus (\S\ref{sec:lit-durable}); and claim verification motivates the citation gates (\S\ref{sec:lit-claims}). Section~\ref{sec:lit-positioning} states what this study adds.

\subsection{Autonomous research systems and what their evaluations established}
\label{sec:lit-systems}

Tool-using autonomous experimentation was first demonstrated in chemistry: a GPT-4 agent with search, code execution and laboratory-automation tools performed six tasks including palladium-catalysed cross-coupling optimisation, with a physical oracle and human supervision \citep{boiko2023coscientist}, and ChemCrow's 18 tools executed syntheses evaluated by both LLM and expert assessment \citep{bran2024chemcrow}.

End-to-end systems followed. The AI Scientist chains idea generation, code editing, experiments, plotting, paper writing and a simulated review; the preprint reports a cost below \$15 per paper and an automated reviewer with near-human agreement on scores \citep{lu2024aiscientist}, and the line of work has since appeared in \emph{Nature} \citep{lu2026aiscientist}. Its second version replaces human-written code templates with a progressive agentic tree search and a vision-language feedback loop on figures; its validity evidence is one of three autonomous manuscripts scoring above the average acceptance threshold at an ICLR workshop \citep{yamada2025aiscientistv2}. Agent Laboratory stages literature review, experimentation and report writing with optional human feedback at each stage; stage-wise feedback significantly improved research quality and expenses were 84\% lower than previous autonomous methods \citep{schmidgall2025agentlab}. data-to-paper walks agents from annotated data to a manuscript while programmatically back-tracing the flow of information so that a human can verify every number, in autopilot or copilot mode \citep{ifargan2025datatopaper}. Co-Scientist generates, critiques, ranks and evolves hypotheses in a tournament loop, with wet-laboratory validation performed downstream by collaborators \citep{gottweis2026coscientist}; the Virtual Lab has an LLM principal investigator run meetings with specialist agents and a critic while humans guide the agenda and run the experiments \citep{swanson2025virtuallab}; Kosmos runs for up to twelve hours and about 200 rollouts over a shared world model, with accuracy judged by expert reading \citep{mitchener2025kosmos}.

These evaluations established that end-to-end pipelines can produce reviewable papers cheaply, that staging plus human feedback improves outcomes and lowers cost, and that traceable evidence chains can be built in. They did not establish whether the \emph{structure} of the instruction---as distinct from the model, the scaffold or the amount of human co-piloting---changes validity: each system is a single pipeline without a control arm, any reviewer is in-system, and the outcome is a reviewer score or expert reading rather than a verified result. Audits make the gap concrete. \citet{luo2025pitfalls} define four failure modes---inappropriate benchmark selection, data leakage, metric misuse and post-hoc selection bias---and find several of them in controlled experiments on two open-source systems; \citet{beel2025sakana} re-ran the AI Scientist and found keyword-level literature search, ideas wrongly classified as novel and execution failures; \citet{ding2026verificationgap} screened 125 works and coded 26 on seven audit dimensions, finding code release common but reproducibility-grade and claim-verification artifacts rare, the ``verification gap''. Most recently XScientist proposes a git-like protocol in which hypotheses, attempts, claims, reviews and hand-offs are typed content-addressed objects with immutable checkpoints, claim--evidence closure and authority-aware gates, while stressing that a passing integrity check is not scientific truth \citep{luo2026xscientist}. It is the nearest conceptual neighbour to the protocol studied here, but it is an artifact and implementation, not an experiment. The AI Scientist-v2, Kosmos, the pitfalls audit, the survey and XScientist are preprints at the time of writing.

\subsection{Agent benchmarks for scientific programming and the oracle problem}
\label{sec:lit-benchmarks}

A parallel literature evaluates agents task by task against executable oracles. ScienceAgentBench extracts 102 tasks from 44 publications, validated by nine experts, with programs as outputs, execution-based metrics and cost tracking; the best agent solves 32.4\% of tasks and o1-preview 42.2\% at more than ten times the cost, and the authors call for assessing individual tasks before bold claims about end-to-end automation \citep{chen2025scienceagentbench}. DiscoveryBench formalises discovery as dataset, metadata and goal across 264 real and 903 synthetic tasks \citep{majumder2025discoverybench}; DSBench supplies 466 analysis and 74 modelling tasks with long contexts \citep{jing2025dsbench}; MLAgentBench evaluates a ReAct agent on 13 experimentation tasks \citep{huang2024mlagentbench}; MLE-bench curates 75 Kaggle competitions with human baselines, where the best setup medals in 16.9\%, and studies resource scaling and contamination \citep{chan2025mlebench}. CORE-Bench asks agents to reproduce published results across 270 tasks from 90 papers \citep{siegel2024corebench}; PaperBench asks them to replicate 20 ICML papers against 8,316 author-co-developed rubric items graded by an LLM judge validated on a separate judge benchmark \citep{starace2025paperbench}. BLADE grades analysis \emph{decisions} against independent expert analyses, admitting multiple valid paths \citep{gu2024blade}; SWE-bench grades 2,294 GitHub issues by tests \citep{jimenez2024swebench}.

The stream also criticises its own oracles. Task-setup and reward-design flaws---SWE-bench Verified's insufficient tests, TAU-bench (a preprint) counting empty responses as successes---mis-estimate performance by up to 100\% relative, and a checklist reduced overestimation on CVE-Bench by 33\% \citep{zhu2025abc}. Agent benchmarks focus on accuracy without cost, with inadequate holdouts and conflated developer needs \citep{kapoor2024agentsthatmatter}; the HAL harness later ran 21,730 rollouts for about \$40,000 \citep{kapoor2025hal}. Contamination must be measured per benchmark \citep{sainz2023contamination}; long-horizon reasoning and instruction following are the main agent failure causes \citep{liu2024agentbench}; and $\tau$-bench shows success and reliability over repeated trials are distinct quantities, with gpt-4o under 50\% success and pass\textasciicircum{}8 below 25\% in retail \citep{yao2024taubench}.

The evaluator itself is a threat. LLM judges show position, verbosity and self-enhancement biases even where strong judges agree with humans more than 80\% of the time \citep{zheng2023mtbench}; only the largest of thirteen judge models reach reasonable alignment, remain behind inter-human agreement and can differ from human scores by up to five points \citep{thakur2025judgingjudges}; self-recognition correlates linearly with self-preference, so a model grading its own output is a biased oracle \citep{panickssery2024selfpreference}; and a simulated research-and-review loop with human annotation shows pronounced misalignment between LLM reviewers and humans \citep{li2025llmreval}. Hence oracles here are deterministic, held on a host the executor cannot reach, and never the executor's own model grading itself. The checklist, HAL and the review simulation are preprints.

\subsection{Workflows, provenance and reproducibility}
\label{sec:lit-repro}

Workflow engines locate reproducibility in re-executable, environment-pinned definitions: Snakemake's rules scale from workstation to cluster unchanged \citep{koster2012snakemake}, and Nextflow combines dataflow with containers for cross-platform reproducibility \citep{ditommaso2017nextflow}. W3C PROV supplies the entity--activity--agent vocabulary \citep{missier2013prov}, and ReproZip shows execution provenance can be captured automatically rather than self-reported \citep{chirigati2016reprozip}. The norms are codified: track how every result was produced, avoid manual manipulation, archive program versions and seeds, publish scripts and results \citep{sandve2013tensimplerules}; reproducibility as the minimum standard when replication is impossible \citep{peng2011reproducible}; the consensus definitions of reproducibility (same data, code, conditions) and replicability (new data) \citep{nasem2019reproducibility}; and transparency, re-executability and determinism distilled from seven reproductions \citep{krafczyk2021reproduction}.

The empirical record explains why norms must run rather than be declared. Of 204 \emph{Science} papers under a data-and-code-on-request policy, artifacts were obtained for 44\% and findings reproduced for 26\% \citep{stodden2018journalpolicy}; of more than 9,000 R files from replication datasets, 74\% failed initially and 56\% after automatic cleaning \citep{trisovic2022codequality}. Policy without executable verification has a low yield; the protocol arms therefore turn provenance requirements into gates that run.

\subsection{Preregistration and specification discipline}
\label{sec:lit-prereg}

Preregistration separates prediction from postdiction by fixing question and analysis plan before outcomes are observed \citep{nosek2018preregistration}; Registered Reports review and provisionally accept the protocol before data collection \citep{chambers2013registeredreports}, and early evidence indicates they work without being a universal cure \citep{chambers2022registeredreports}. Flexibility in collection, analysis and reporting dramatically inflates false positives \citep{simmons2011falsepositive}; data-dependent choices do so even without intent, the garden of forking paths \citep{gelman2014statisticalcrisis}; and hypotheses can be presented as a priori after the results are known \citep{kerr1998harking}. The TOP guidelines and the reproducibility manifesto translate these concerns into graded standards and measures \citep{nosek2015top,munafo2017manifesto}. The clearest quantitative effect is that the first hypothesis was positive in 96\% of 152 standard reports but 44\% of 71 Registered Reports \citep{scheel2021excess}, against a background in which 100 replications recovered roughly one-third to one-half of original findings \citep{osc2015reproducibility}.

An agent under an open instruction walks the garden of forking paths unobserved; the structured-protocol arm is, functionally, a preregistration imposed on the agent, with phases, ledgers and stopping rules fixed before execution. The analogy is inexact: Registered Report authors self-select; arms here are assigned.

\subsection{Human--AI delegation, automation bias and self-reported efficiency}
\label{sec:lit-delegation}

Human-factors research supplies the vocabulary: use, misuse, disuse and abuse of automation \citep{parasuraman1997automation}; automation of acquisition, analysis, decision selection and implementation along a manual-to-automatic continuum \citep{parasuraman2000levels}; and the irony that the operator is left with what could not be automated and must monitor a system that rarely needs intervention \citep{bainbridge1983ironies}. Automation bias across 74 studies is mediated by cognitive style, system design and experience \citep{goddard2012automationbias}; explanations raise acceptance regardless of correctness \citep{bansal2021whole}; cognitive forcing reduced overreliance among 199 participants \citep{bucinca2021trust}, the principle behind having reviewers execute checks rather than read summaries. People prefer machine-in-the-loop designs to full AI control, with trust the dominant factor \citep{lubars2019delegability}; the design space of such studies is surveyed by \citet{lai2023humanai}; and a preregistered meta-analysis of 106 experiments finds human--AI combinations on average below the best of either alone, with gains only in content creation \citep{vaccaro2024combinations}.

The decisive result for measurement is a randomised trial of 16 experienced developers on 246 tasks: AI tools \emph{increased} completion time by 19\%, although developers forecast a 24\% speed-up and still believed in a 20\% speed-up afterwards \citep{becker2025metr}. Perceived and measured efficiency diverged in sign, which is why the cost of delegation here is read from logged time and resource use, never from self-report. The trial is a preprint; the methodological point stands.

\subsection{Durable execution and resource accounting}
\label{sec:lit-durable}

Long autonomous runs fail like distributed systems. Most outages stem from software and operations rather than hardware, motivating process pairs and transactions \citep{gray1986whystop}; tail-tolerant techniques absorb latency variability \citep{dean2013tail}; retries are safe only under idempotence \citep{helland2012idempotence}; chaos engineering verifies reliability by experimenting with failure \citep{basiri2016chaos}; sampled tracing makes behaviour observable \citep{sigelman2010dapper}; and durable workflows with persistent state have a formal semantics \citep{burckhardt2021durablefunctions}. Agent-era work adds a taxonomy of artifacts to trace \citep{dong2024agentops}, aligned checkpoints for rewind and resume \citep{zhuang2026agentrewind}, evidence over 23,392 episodes that capability and reliability diverge as tasks lengthen \citep{khanal2026reliability}, and cost-of-pass, the expected cost of a correct solution, as the economic unit \citep{erol2025costofpass}; these four are preprints. Orchestrators such as Temporal, Airflow and Argo embody the same principles as software. This study borrows the apparatus---idempotent checkpointed steps, traced runs, cost per correct outcome---and treats recovery after failure as an outcome of the protocol.

\subsection{Claim verification and fabricated references}
\label{sec:lit-claims}

FEVER labels 185,445 claims Supported, Refuted or NotEnoughInfo with Fleiss kappa 0.68 \citep{thorne2018fever}; SciFact pairs 1.4K expert claims with evidence and rationales \citep{wadden2020scifact}; Attributed QA requires generation to cite its support \citep{bohnet2022attributedqa}; RARR attributes and post-edits any generator's output \citep{gao2023rarr}; FActScore scores atomic facts, finding only 58\% of ChatGPT biographical facts supported \citep{min2023factscore}. Self-consistency improves reasoning by sampling and agreement \citep{wang2023selfconsistency}, but models struggle to self-correct without external feedback and can degrade when they try \citep{huang2024selfcorrect}; the gates here are external for that reason. Fabricated references are the most visible failure: 55\% of GPT-3.5 and 18\% of GPT-4 citations in 84 short reviews were fabricated \citep{walters2023fabrication}, as clinical case reports had documented early \citep{alkaissi2023hallucinations}; nearly 300 ACL, NAACL and EMNLP papers of 2024--2025 contain a hallucinated citation and more than 100 were accepted \citep{sakai2026hallucitation}; and an audit of 111 million references conservatively estimates 146,932 hallucinated citations in 2025 \citep{zhao2026hallucinationswild}. The first study used 2023-era models and the last is a preprint with an unknown false-negative rate; both motivate a gate that resolves every reference against a primary metadata record.

\subsection{Positioning}
\label{sec:lit-positioning}

Before naming the design we searched for prior use. An exact-phrase search (web and arXiv titles, 2026-10-29/30) for ``protocol-guided autonomous research'' returned no paper, system or project using it. The nearest hits are XScientist, where ``protocol'' denotes a git-like provenance and hand-off standard \citep{luo2026xscientist}, and ProtoPilot, where it denotes a wet-laboratory procedure for robots; neither uses the phrase and neither compares instruction structures. The acronym ``PGAR'' returned no match in the AI or science-agent context; it is used elsewhere for unrelated entities such as gene and company names.

Table~\ref{tab:prior} places the closest designs beside this study. Systems hold their pipeline fixed, vary human involvement or propose a protocol without an experiment; benchmarks hold the task fixed and vary the model or scaffold; audits inspect traces after the fact. None varies the instruction regime while holding the executor constant and grading with oracles the executor cannot see.

\begin{table}[htbp]
\centering
\small
\setlength{\tabcolsep}{3pt}
\caption{Closest prior systems, benchmarks and audits, and the design of this study. Cells summarise the sources cited in the text; ``n/a'' means not reported in the source consulted.}
\label{tab:prior}
\begin{tabular}{>{\raggedright\arraybackslash}p{1.0in}>{\raggedright\arraybackslash}p{1.1in}>{\raggedright\arraybackslash}p{1.1in}>{\raggedright\arraybackslash}p{0.85in}>{\raggedright\arraybackslash}p{0.75in}>{\raggedright\arraybackslash}p{0.95in}}
\toprule
System & Inputs / workflow & Verification mechanism & Human involvement & Resources reported & What is compared \\
\midrule
AI Scientist \citep{lu2024aiscientist} & Code template; fixed idea-to-paper pipeline & In-system LLM reviewer; code runs & None during run & Under \$15 per paper & Nothing; single pipeline \\
AI Scientist-v2 \citep{yamada2025aiscientistv2} & Topic prompt; agentic tree search & In-system reviewer; one workshop acceptance & Minimal & n/a & Nothing; single pipeline \\
Agent Laboratory \citep{schmidgall2025agentlab} & Human idea; three staged solvers & Human survey; automated review scores & Optional feedback each stage & 84\% lower expenses & Copilot vs autonomous; models \\
Coscientist \citep{boiko2023coscientist} & NL goal; tools incl.\ cloud lab & Physical outcome; author inspection & Supervise execution & n/a & Nothing; six tasks \\
Co-Scientist \citep{gottweis2026coscientist} & Research goal; tournament loop & In-system critique; wet-lab downstream & Scientists specify, validate & n/a & Test-time compute scaling \\
data-to-paper \citep{ifargan2025datatopaper} & Annotated data; stepwise agents & Back-traced numbers; human verification & Optional copilot steps & n/a & Autopilot vs copilot \\
Kosmos \citep{mitchener2025kosmos} & Objective + dataset; world model & Expert reading of reports & None during run & Up to 12 h, 200 rollouts & Nothing; case studies \\
XScientist \citep{luo2026xscientist} & Research state as typed objects & Claim--evidence closure; integrity gates & Fork, inspect, extend & n/a & No experiment reported \\
Science\-Agent\-Bench \citep{chen2025scienceagentbench} & Task + data; program output & Execution metrics; figure judging & Experts validate tasks & API cost per task & Models and frameworks \\
PaperBench \citep{starace2025paperbench} & Paper text; replicate from scratch & Author rubrics; validated LLM judge & Rubric co-development & In paper & Models and scaffolds \\
BLADE \citep{gu2024blade} & Dataset + question & Decisions vs expert analyses & Experts give ground truth & n/a & Models and prompting \\
Pitfalls audits \citep{luo2025pitfalls,beel2025sakana} & Existing open-source systems & Auditors inspect traces, code & Auditors only & n/a & Failure modes present or not \\
\midrule
This study (WP29) & Fixed task; one of four instruction regimes & Hidden oracles on separate host; executable gates & None in loop; interventions ledgered & Logged time and cost per run & Four arms, executor fixed \\
\bottomrule
\end{tabular}
\end{table}

Prior work has established that end-to-end pipelines produce reviewable papers cheaply, that staging plus human feedback helps, that traceable evidence chains can be built in, that such systems commit leakage, metric misuse, post-hoc selection and citation errors visible only in traces, that executable per-task oracles with cost accounting exist, and that a git-like research protocol has been proposed. What this study adds, and should claim no more than, is a controlled four-arm comparison in which the executor---Claude Code 2.1.314 in bare mode, model claude-opus-5-5, isolated on cluster nodes---is held fixed and the instruction regime varied: open instruction (A); a structured protocol with phases, ledgers and stopping and recovery rules (B); the same with controller-enforced executable gates (C); and the same with a separate review-and-repair session (D), across six task families from data audit to integrated mini-projects, with hidden deterministic oracles on a separate host and outcomes defined as scientific validity rather than reviewer score. The novelty is the experimental design and its measurement, not the idea that protocols help, which the systems above already presuppose.