# jof250 review (referee) — Protocol-Guided Autonomous Research with Claude Code

Model claude-opus-5-5, effort high, 2026-11-03T23:14:10+00:00; cost 0.186712 USD. Deterministic audit appended.

# Referee report

Manuscript: "Protocol-Guided Autonomous Research with Claude Code" (WP29, v1.0)

# 1. Summary

The paper asks whether the structure of the instruction file given to a coding agent changes what the agent delivers on bounded scientific tasks, with the agent, tools, inputs and budget held fixed. Each confirmatory task (six families; the family-table caption implies four tasks per family) is run in fresh Claude Code sessions (client 2.1.314, model claude-opus-5-5) under four nested regimes:

A: an open instruction.
B: a structured protocol with phases, ledgers, and stopping and recovery rules.
C: B plus a controller-enforced executable gate on the deliverable contract and reproducibility.
D: C plus a separate review-and-repair session.

Scoring is done on a separate host by deterministic hidden oracles validated on seeded artefacts. Design, tasks and analysis were frozen with hashes beforehand.

The primary endpoint is intervention-free verified completion, with the D−A contrast inferred by task-cluster bootstrap and sign-flip permutation. The headline is a null. The D−A interval runs from about three percentage points against D to about six in its favour, the mechanism contrasts are also null, and violations are rare. Arms C and D achieve full contract compliance at two to three times the inference cost per run.

On the authors' inspection, every non-success under the frozen scoring is an oracle defect. Under a post hoc corrected rubric, every arm completes every task. Two further facts limit the test of the mechanisms:

Arm D's repair loop never fired in the frozen benchmark.
A launcher defect (A-006) meant gate and review feedback never reached executors as prompt text.

The paper adds fault-injection and recovery experiments, a length-matched control (arm L), three transfer mini-projects, and a reflexive account of its own production. It concludes that, for bounded tasks with known answers, protocol structure buys auditability rather than correctness, and that the scarce resource is a trustworthy oracle.

# 2. Contribution and positioning

Strengths.

The design is careful and unusually transparent.
The agent is held fixed while the instruction regime varies.
Oracles are kept off the execution hosts, the design is preregistered and hash-frozen, and there is an amendment ledger.
Deviations (A-004, A-006, RC0159) are reported candidly rather than buried.
The related-work section correctly identifies the gap. Existing autonomous-research systems (AI Scientist, Agent Laboratory, data-to-paper, Kosmos) are single pipelines without control arms. Audits (Luo et al. 2025; Beel et al. 2025; Ding et al. 2026) document failure modes but do not manipulate instructions.
The closing message, "spend the budget on checking the checker", is useful and well supported by the paper's own experience.

Concerns about the contribution. As executed, the confirmatory experiment cannot discriminate between regimes. The authors effectively concede this: "the benchmark … is at the ceiling in every arm", and the corrected rubric yields 100% everywhere. A null at the ceiling is not a "precise null" about protocol value. It shows that the tasks were too easy for this model, so the comparison carries little information about the paper's own question. In addition, two of the three mechanism contrasts (C−B and D−C) were not exercised as designed:

The gate passed at the first cycle in every C/D run.
The review loop never triggered in a completed run.
Feedback text never reached the executor (A-006).

What remains is a well-documented comparison of protocol texts delivered through the system prompt, plus a methodological lesson about oracle construction. The second is, in my view, the paper's most solid contribution, but the framing (abstract, introduction, conclusion) foregrounds the first.

Fit for a finance readership. The JEL codes (C18, C63, C88, O33) and the task families (forecasting under information constraints, CPI growth, estimation and inference) are relevant to empirical finance and economics. However, the manuscript never says why a finance audience should care beyond general research delegation. A paragraph is needed linking the task families to finance-specific validity threats, such as look-ahead bias in real-time forecasting, data-snooping and specification search, or sample-exclusion choices. Without it, the paper reads as a computer-science or meta-science contribution.

# 3. Major comments

# Identification and inference

1. Ceiling effect and the interpretation of the null (Main results, primary endpoint; Table tab:arms; Figure fig:primary).

At or near 100% success in all arms, the paired task-level differences are almost all ties. The bootstrap interval then reflects a handful of oracle-defect tasks rather than sampling variation in agent behaviour. The development pilot (23 of 24 successes) already signalled the ceiling before the freeze.

Calling the result "precise" requires a prespecified smallest effect size of interest and an equivalence test. If the preregistration contains such a margin, report the TOST (equivalence) result against it. If it does not, the result should be described as uninformative at this difficulty, not as a precise null.

Resolution:

(a) State whether the analysis plan contained a power calculation or difficulty-calibration rule, and why the pilot ceiling did not trigger recalibration.
(b) Reframe the headline as "no detectable difference at a ceiling" rather than "precise null".
(c) Ideally, add a confirmatory replication on a harder or longer-horizon task tier calibrated so that arm A succeeds roughly 40–70% of the time. Alternatively, use a weaker model under the same treatments to show the design can detect protocol effects when headroom exists.

2. The mechanism contrasts C−B and D−C were not tested as designed (subsection "What the gates and the reviewer actually did"; amendment A-006).

The C−B and D−C contrasts are meant to identify the value of enforcement and review. In the frozen runs:

the gate never failed;
the only revise verdict occurred in the aborted first attempt of RC0094;
the number of repair sessions and gate-feedback sessions executed is reported only through the unexpanded macros \nRepairSessions{} and \nGateCycleSessions{}, and the "never triggered" narrative implies these are zero;
continuation prompts were truncated to one word.

H3 is acknowledged as not evaluable, and the same logic applies to the enforcement component of C. The paper should state explicitly, in the abstract and introduction, that C−B and D−C measure the effect of additional protocol text describing a gate and a reviewer (plus the reviewer's cost), not the effect of gate feedback or repair. The post-fix verification set (Table tab:postfix) is too small to substitute.

Resolution: either a targeted confirmatory study in which gate or review failures are induced (e.g., seeded deliverable defects) with the corrected launcher, or a narrowing of the claims.

3. Attribution of contract compliance to the gate (same subsection).

The text says "the gate did enforce the deliverable contract". But the gate passed at the first cycle in every C/D run, so no run was ever brought into compliance by enforcement. The compliance gap between A/B and C/D must therefore come from something prior to enforcement. Candidates include:

the C/D instruction text;
exposure to gate_spec.json;
executors invoking the gate program voluntarily before finishing.

The paper reports no voluntary invocation in arms A or B but is silent on C and D.

Resolution:

Report how often C/D executors ran the gate program themselves before ending.
State whether gate_spec.json was visible to them.
Use arm L and B to separate "knowing the contract" from "being checked". For example, report cont

# Manuscript audit — Protocol-Guided Autonomous Research with Claude Code

Generated 2026-11-03T23:13:09+00:00 (jof250 7ef3b9a-dirty). Body words: 15946.

# Sections

Introduction — role introduction, 848 words
Related work — role literature, 2587 words
Conceptual framework: delegation contracts, autonomy and evidence — role theory, 1044 words
Corpus-informed method design — role other, 1573 words
PGAR implementation — role other, 1725 words
Experimental design — role other, 1238 words
Evaluation — role other, 672 words
Main results — role results, 1312 words
Recovery, faults and ablations — role other, 751 words
End-to-end transfer and visual case studies — role other, 912 words
Reflexive analysis of the WP29 production process — role other, 784 words
Limitations, reproducibility and implications — role other, 1001 words
Conclusion — role conclusion, 249 words
AI-use and disclosure statement — role other, 361 words
Task suite and task cards — role other, 8 words
Treatment texts (verbatim) — role other, 57 words
Evaluator validation — role other, 4 words
Statistical details and additional results — role results, 4 words
Environment and model records — role theory, 4 words
Intervention and amendment ledgers — role other, 4 words
Reproduction instructions — role other, 8 words

# Tables (12) and figures (7)

table 1 (tab:prior): no notes block (JoF convention: notes state sample, units, standard errors)
table 2 (tab:contract): no notes block (JoF convention: notes state sample, units, standard errors)
table 3 (tab:corpus): no notes block (JoF convention: notes state sample, units, standard errors)
table 4 (tab:treatments): no notes block (JoF convention: notes state sample, units, standard errors)
table 5 (tab:accounting): no notes block (JoF convention: notes state sample, units, standard errors)
table 6 (tab:arms): no notes block (JoF convention: notes state sample, units, standard errors)
table 7 (tab:contrasts): never referenced in text
table 7 (tab:contrasts): no notes block (JoF convention: notes state sample, units, standard errors)
table 8 (tab:family): no notes block (JoF convention: notes state sample, units, standard errors)
table 9 (tab:cost): no notes block (JoF convention: notes state sample, units, standard errors)
table 10 (tab:postfix): no notes block (JoF convention: notes state sample, units, standard errors)
table 11 (tab:transfer): no notes block (JoF convention: notes state sample, units, standard errors)
table 12 (tab:effort): no notes block (JoF convention: notes state sample, units, standard errors)
figure 3 (fig:quality): never referenced
figure 6 (fig:recovery): never referenced

# Equations (3, 3 labelled)

equation 1 (eq:contract): labelled but never referenced
equation 2 (eq:Y): labelled but never referenced
\ref to undefined label: app:gallery
\ref to undefined label: tab:tasks

# Citations

\cite commands: 110; bib entries: 84 (47 with DOI)
unknown keys: none
DOI check: {'checked': 47, 'resolved': 47, 'unresolved': []}

# Possibly unsupported claims (6)

[increase] What the protocol, the gate and the reviewer did buy was contract compliance (no arm C or D deliverable failed the post-hoc contract check; \nContractFailA{} arm-A and \nContractFailB{} arm-B deliverables did) and a two- to three-fold increase in inference cost per run; the quality--cost frontier is
[improves] These evaluations established that end-to-end pipelines can produce reviewable papers cheaply, that staging plus human feedback improves outcomes and lowers cost, and that traceable evidence chains can be built in.
[show that] The consequence for the continuation mechanisms is more severe: every gate-feedback, review-repair and resumption turn in the frozen runs consisted of one word, and the transcripts show that the resumed sessions replied that the message had been cut off and ended after one or two turns without readi
[shows that] It shows that a mandate of this form can be executed end to end by the client under study, with every intermediate artefact hashed and every deviation logged.
[show that] It does not show that the execution was autonomous in the scientific sense: the mandate was written by a human who knew what a credible study looks like, the task families and the hypotheses were given, and the same organisation controlled the builder, the tasks and the executors.
[improve] \label{sec:conclusion} We asked whether an explicit research protocol, executable verification gates, persistent evidence records and a separate review stage improve a current coding agent's ability to complete bounded scientific tasks autonomously and validly, and what the improvement costs.

spb/wp45_uqo