Protocol-Guided Autonomous Research with Claude Code
UQO Working Paper No. 29 — Scientific Verification, Reproducibility, and the Cost of Delegation: A Controlled Study
A controlled, four-arm experiment in which fresh, isolated Claude Code sessions (bare mode,
claude-opus-5-5) solve bounded scientific tasks under (A) an open instruction, (B) a structured research protocol, (C) the protocol plus controller-enforced executable gates, and (D) the same plus a separate review-and-repair session. Outcomes are scored by deterministic hidden oracles on a separate host. Every number in the paper is regenerated fromruns/state.sqlite.
Status
See reports/phase_status.md (append-only phase log), protocol/amendments.csv and reports/interventions.csv. The complete research mandate is protocol/WP29_FULL_MANDATE.md.
Layout
protocol/ mandate, preregistration, contract, treatments (arms/), evaluation plan, screenshot protocol, amendments
research/ corpus of 11 source mandates (hashed), coding, literature matrix + verified BibTeX
configs/ budgets and per-split node/envelope configuration
tasks/ TASK_AUTHORING_SPEC.md, registry.csv, task_cards/, public_bundles/ (hidden oracles live in ~/wp29-oracles/, outside the repo)
src/pgar/ harness: contracts, state (SQLite), events, launcher (ssh + claude --bare), isolation, budgets, runtime_gates (gate program), recovery, scheduler, evaluation, exports, capture, evidence_viewer, cli
evaluators/ public receipt schema + run_validations.py (seeded evaluator validation)
tests/ infrastructure tests (no provider access)
runs/ state.sqlite + per-run directories (raw streams are not committed)
evidence/ screenshot originals (restricted, not committed) and public copies
analysis/ build_analysis_dataset, primary_analysis, robustness, figures, tables
results/ analysis_data (CSV exports), estimates, figures (PDF), tables (LaTeX)
paper/ LaTeX manuscript (UQO template)
reports/ environment/isolation audits, evaluator validation, pilot report, accounting, final reviewCommands (laptop = controller; see Makefile)
make test # infrastructure tests (state machine, gate, evaluator, bundle hygiene)
make deploy # install the gate program + key file on executor nodes
make probe # sentinel isolation probes on the nodes (tiny live calls)
make smoke-live # 1 task × 4 arms, live
PYTHONPATH=src .venv/bin/python -m pgar.cli assign --split pilot --tasks <ids> --repeats 1 --seed 202
PYTHONPATH=src .venv/bin/python -m pgar.cli run --split pilot --workers 8
make analyze && make paperRequirements: Python 3.12 venv (uv venv .venv --python 3.12 && uv pip install -p .venv/bin/python -e .), SSH access to the executor nodes, ANTHROPIC_API_KEY in ~/wp29/.env on each node, Claude Code 2.1.314 on the nodes.