1# WP29 — Final report (concise research summary)23**Protocol-Guided Autonomous Research with Claude Code: Scientific Verification, Reproducibility, and the Cost of Delegation — A Controlled Study.** UQO Working Paper No. 29, Simon-Pierre Boucher, October 2026. Repository: `~/Desktop/Recherche-UQO/methodes/autonomous-research-claude-code/` (spbgit `wp29_uqo.git`).45## 1. Question and contributed method6Does an explicit research protocol, with executable verification gates, persistent evidence records and a separate review stage, improve Claude Code's intervention-free, scientifically verified completion of bounded research tasks, and at what cost? Method contributed: a delegation contract P = (Q, Θ, D, A, G, V, B, R, L) serialised into files and enforced by a small auditable harness (`src/pgar/`: durable SQLite state with validated transitions, isolated bare-mode executors on separate hosts with manifest-verified workspaces, oracle-free runtime gate, controller-enforced gate loop, separate review session, deterministic hidden evaluators on the evaluator host, resource envelope, fault injection, native capture with a manifest), plus a benchmark of 24 confirmatory + 7 development tasks and 3 transfer projects with hidden oracles and seeded validation sets.78## 2. Experiments actually run (client Claude Code 2.1.314; model claude-opus-5-5 for all executors and reviewers; builder session claude-fable-5-1)9| Stage | Runs | Live sessions | Notes |10|---|---|---|---|11| Smoke | 4 (+4 infra-failed attempts, no inference) | 5 | F2-smoke-01 × A–D |12| Pilot | 24 | 30 | 6 development tasks × A–D; 23/24 successes |13| Main benchmark | 288 (24 tasks × 4 arms × 3 repeats, frozen seed 301) | 362 (incl. 2 of the aborted attempt) | 1 infrastructure error (A-004), rerun; 0 timeouts; 0 budget exhaustions |14| Transfer | 24 | 36 | T1–T3 × A–D × 2 (arm D's reviewer never ran on T1: the gate never passed) |15| Recovery | 15 (R1 12, R3 1, R4 2) + 8 R2 gate checks | 29 | faults injected by the controller |16| Ablation (arm L) | 24 | 24 | length-matched control |17| Post-fix verification | 8 (smoke ×4, T1 arms C and D, R1 ×2) | 17 | after amendment A-006; never pooled |18Total: 387 registered runs, 514 sessions, 176.03 USD of inference (client list-price estimates), all on the author's cluster nodes M3U96a/M3U96b/M4M64a/M4M64b between 2026-10-30 03:30 and ≈ 07:00 UTC.1920## 3. Counts21Assigned 288 / launched 288 / completed 288 / evaluated 288 / scientific successes 263 / runs with a critical violation 13 (12 = one evaluator-defect label) / infrastructure attempts 1.2223## 4. Main results (frozen oracles; task-weighted; J = 24 task clusters)24| Arm | p̂ | Violation rate | Mean cost/run (USD) | Cost per success |25|---|---|---|---|---|26| A open | 0.903 | 0.042 | 0.24 | 0.27 |27| B protocol | 0.917 | 0.042 | 0.42 | 0.46 |28| C + gates | 0.917 | 0.042 | 0.45 | 0.49 |29| D + review | 0.917 | 0.056 | 0.64 | 0.70 |30Primary contrast D−A = +0.014, task-cluster bootstrap 95% CI [−0.028, +0.056], permutation p = 1.00; B−A +0.014 [0.000, +0.042]; C−B 0.000; D−C 0.000 [−0.042, +0.042] (Holm p = 1.00 for all). Rubric-corrected sensitivity (A-005 + A-007, four oracle defects): A 1.000, B 1.000, C 1.000, D 1.000; D−A 0.000. Contract-check failures (post hoc): A 7, B 9, C 0, D 0.3132## 5. Hypotheses33H1 (structure improves verified completion): not supported (null, tight interval). H2 (gates reduce violations): not supported (violation rates equal; all recorded violations are evaluator artefacts; no leakage/fabrication/substitution in any arm). H3 (review improves claim validity at a cost): not evaluable — the review-repair loop never triggered in the benchmark (1 revise in an aborted attempt, 1 review without verdict); cost increase confirmed. H4 (persistent state improves recovery): not supported on short tasks (both arms recovered 100%; D duplicated more work and recorded the interruption no better). H5 (heterogeneity by family): failures concentrated in two tasks (evaluator defects), no family effect of arm. H6 (human preparation/review costs change efficiency): builder effort (≈ 4.5 h wall, ≈ 2.6 M sub-session tokens) dwarfs 173 USD of inference; mandate preparation time unknown.3435## 6. Human effort and costs36Human: mandate preparation (unknown duration), one launch instruction, final manuscript review (pending at build time); zero interventions in executor sessions. Builder: see `reports/builder_effort.json`, `reports/experiment_accounting.md`.3738## 7. Screenshots (evidence/manifests/screenshots.csv)39Native live: S002 (terminal tail of RD0002), S003 (builder session window, incidental), S007 (terminal tail of RC0009), S009 (gate rejection — DELIBERATELY INDUCED, labelled stress test). Dashboard live: S008 (main benchmark). Viewer post-run: S005, S006. Failed captures recorded: S001, S004 (blank Safari window captures), S010 (screen locked). No synthetic imagery.4041## 8. Reproducibility42From saved artefacts (no provider access): `make test`, `make analyze`, `make paper`, `make replay` (clean-copy replay with byte comparison). Fresh live replication requires an API key, executor hosts, hidden oracles (`release/WP29_Reproducibility.md`).4344## 9. Limitations and validation status45Designer dependence (same organisation/client family authored tasks, oracles, treatments); bounded tasks with a ceiling; one model/version/week; deterministic evaluators with two detected defects (sensitivity reported); prompt-delivery defect A-006 (one-word user prompt in every frozen session, so the controller feedback loop, the review-repair stage and the resumption instruction were not delivered in any frozen run; symmetric across arms; seven post-fix verification runs behaved as the originals and demonstrated the delivered feedback loop); no human reviewer; costs are client estimates. Validation status: three adversarial sub-session reviews performed (`reports/review_*.md`, responses in `reports/final_review.md`): four oracle defects, three harness defects (A-002, A-004, A-006) and ledger errors (A-008) found and disclosed; release replay identical byte for byte; no human review of the manuscript had taken place at build time (the author's review is the next step).4647## 10. Deliverables (paths)48`paper/main.pdf` (→ `release/WP29_Autonomous_Research_Claude_Code_Boucher.pdf`), `paper/` LaTeX source, `release/WP29_Method_and_Implementation.zip`, `protocol/` (mandate, preregistration v1.0, contract, treatments, evaluation plan, screenshot protocol, amendments, freeze manifests), `runs/state.sqlite` + `runs/<split>/<run>/` (registry, transcripts, deliverables, receipts), `evaluators/` + `~/wp29-oracles/` (specs, validation), `results/analysis_data/runs.csv` (→ `release/WP29_Experiment_Results.csv`), `results/figures`, `results/tables`, `release/WP29_Visual_Evidence_Appendix.pdf`, `evidence/manifests/screenshots.csv`, `evidence/public_screenshots/`, `release/WP29_Reproducibility.md`, `release/WP29_Limitations_and_Claims.md`, `release/WP29_Human_Interventions_and_Costs.csv`, `release/WP29_Final_Review.md`, this file.49