WP29 — Final report (concise research summary)
Protocol-Guided Autonomous Research with Claude Code: Scientific Verification, Reproducibility, and the Cost of Delegation — A Controlled Study. UQO Working Paper No. 29, Simon-Pierre Boucher, October 2026. Repository: ~/Desktop/Recherche-UQO/methodes/autonomous-research-claude-code/ (spbgit wp29_uqo.git).
1. Question and contributed method
Does an explicit research protocol, with executable verification gates, persistent evidence records and a separate review stage, improve Claude Code's intervention-free, scientifically verified completion of bounded research tasks, and at what cost? Method contributed: a delegation contract P = (Q, Θ, D, A, G, V, B, R, L) serialised into files and enforced by a small auditable harness (src/pgar/: durable SQLite state with validated transitions, isolated bare-mode executors on separate hosts with manifest-verified workspaces, oracle-free runtime gate, controller-enforced gate loop, separate review session, deterministic hidden evaluators on the evaluator host, resource envelope, fault injection, native capture with a manifest), plus a benchmark of 24 confirmatory + 7 development tasks and 3 transfer projects with hidden oracles and seeded validation sets.
2. Experiments actually run (client Claude Code 2.1.314; model claude-opus-5-5 for all executors and reviewers; builder session claude-fable-5-1)
| Stage | Runs | Live sessions | Notes |
|---|---|---|---|
| Smoke | 4 (+4 infra-failed attempts, no inference) | 5 | F2-smoke-01 × A–D |
| Pilot | 24 | 30 | 6 development tasks × A–D; 23/24 successes |
| Main benchmark | 288 (24 tasks × 4 arms × 3 repeats, frozen seed 301) | 362 (incl. 2 of the aborted attempt) | 1 infrastructure error (A-004), rerun; 0 timeouts; 0 budget exhaustions |
| Transfer | 24 (+1 post-fix verification) | 37 | T1–T3 × A–D × 2 |
| Recovery | 15 (R1 12, R3 1, R4 2) + 8 R2 gate checks | 29 | faults injected by the controller |
| Ablation (arm L) | 24 | 24 | length-matched control |
| Post-fix verification | 4 (+1) | 5 (+) | after amendment A-006 |
| Total: 384 registered runs, 503 sessions, 172.69 USD of inference (client list-price estimates), all on the author's cluster nodes M3U96a/M3U96b/M4M64a/M4M64b between 2026-10-30 03:30 and ≈ 07:00 UTC. |
3. Counts
Assigned 288 / launched 288 / completed 288 / evaluated 288 / scientific successes 263 / runs with a critical violation 13 (12 = one evaluator-defect label) / infrastructure attempts 1.
4. Main results (frozen oracles; task-weighted; J = 24 task clusters)
| Arm | p̂ | Violation rate | Mean cost/run (USD) | Cost per success |
|---|---|---|---|---|
| A open | 0.903 | 0.042 | 0.24 | 0.27 |
| B protocol | 0.917 | 0.042 | 0.42 | 0.46 |
| C + gates | 0.917 | 0.042 | 0.45 | 0.49 |
| D + review | 0.917 | 0.056 | 0.64 | 0.70 |
| Primary contrast D−A = +0.014, task-cluster bootstrap 95% CI [−0.028, +0.056], permutation p = 1.00; B−A +0.014 [0.000, +0.042]; C−B 0.000; D−C 0.000 [−0.042, +0.042] (Holm p = 1.00 for all). Rubric-corrected sensitivity (A-005): A 0.986, B 1.000, C 1.000, D 0.972; D−A −0.014 [−0.056, +0.028]. Contract-check failures (post hoc): A 7, B 9, C 0, D 0. |
5. Hypotheses
H1 (structure improves verified completion): not supported (null, tight interval). H2 (gates reduce violations): not supported (violation rates equal; no leakage/fabrication/substitution in any arm). H3 (review improves claim validity at a cost): claim-level unsupported-claim rates 0.14/0.11/0.11/0.06 per run (A–D), differences within noise; cost increase confirmed. H4 (persistent state improves recovery): not supported on short tasks (both arms recovered 100%; D duplicated more work and recorded the interruption no better). H5 (heterogeneity by family): failures concentrated in two tasks (evaluator defects), no family effect of arm. H6 (human preparation/review costs change efficiency): builder effort (≈ 4.5 h wall, ≈ 2.6 M sub-session tokens) dwarfs 173 USD of inference; mandate preparation time unknown.
6. Human effort and costs
Human: mandate preparation (unknown duration), one launch instruction, final manuscript review (pending at build time); zero interventions in executor sessions. Builder: see reports/builder_effort.json, reports/experiment_accounting.md.
7. Screenshots (evidence/manifests/screenshots.csv)
Native live: S002 (terminal tail of RD0002), S003 (builder session window, incidental), S007 (terminal tail of RC0009), S009 (gate rejection — DELIBERATELY INDUCED, labelled stress test). Dashboard live: S008 (main benchmark). Viewer post-run: S005, S006. Failed captures recorded: S001, S004 (blank Safari window captures), S010 (screen locked). No synthetic imagery.
8. Reproducibility
From saved artefacts (no provider access): make test, make analyze, make paper, make replay (clean-copy replay with byte comparison). Fresh live replication requires an API key, executor hosts, hidden oracles (release/WP29_Reproducibility.md).
9. Limitations and validation status
Designer dependence (same organisation/client family authored tasks, oracles, treatments); bounded tasks with a ceiling; one model/version/week; deterministic evaluators with two detected defects (sensitivity reported); prompt-delivery defect A-006 (symmetric; post-fix verification unchanged); no human reviewer; costs are client estimates. Validation status: three adversarial sub-session reviews performed (reports/review_*.md), harness and evaluator defects found and disclosed; final human review of the manuscript by the author pending.
10. Deliverables (paths)
paper/main.pdf (→ release/WP29_Autonomous_Research_Claude_Code_Boucher.pdf), paper/ LaTeX source, release/WP29_Method_and_Implementation.zip, protocol/ (mandate, preregistration v1.0, contract, treatments, evaluation plan, screenshot protocol, amendments, freeze manifests), runs/state.sqlite + runs/<split>/<run>/ (registry, transcripts, deliverables, receipts), evaluators/ + ~/wp29-oracles/ (specs, validation), results/analysis_data/runs.csv (→ release/WP29_Experiment_Results.csv), results/figures, results/tables, release/WP29_Visual_Evidence_Appendix.pdf, evidence/manifests/screenshots.csv, evidence/public_screenshots/, release/WP29_Reproducibility.md, release/WP29_Limitations_and_Claims.md, release/WP29_Human_Interventions_and_Costs.csv, release/WP29_Final_Review.md, this file.