1# Preregistration — WP29 (DRAFT v0.9, written 2026-10-30 before the pilot; to be frozen as v1.0 with hashes before the confirmatory stage)23## 1. Research question4Under controlled information, tool access and resource conditions, does an explicit research protocol, combined with executable verification and persistent evidence records, improve Claude Code's ability to complete scientifically valid research tasks autonomously, and at what cost in compute, time and human effort?56## 2. Design7Paired, four-arm comparison (A open instruction; B structured protocol; C protocol + controller-enforced executable gates; D = C + separate review-and-repair session). Every confirmatory task is run under every arm with R repeats; arm order within a task block and the order of task blocks are randomised (seeded) and interleaved in time across four executor nodes; model, client, tools, envelope and task information are identical (see `protocol/treatment_definitions.md`). Executors are fresh `claude --bare` sessions on hosts that do not hold the oracles.89## 3. Hypotheses (directions are expectations, not guarantees)10- H1 (primary): p_D − p_A > 0 for intervention-free, science-verified completion within the envelope.11- H2: critical integrity violations per run are lower in C than in B (executable gates).12- H3: claim validity (fraction of `supported` claims that are actually supported) is higher in D than in C, at a measurable additional cost (reviewer share of the envelope).13- H4: after a standardised interruption, valid continuation (no duplicated expensive work, no lost artefacts, final scientific success) is more frequent under protocols with persistent state (B/C/D) than under A.14- H5: effects differ by task family (F1–F6) and by task difficulty class.15- H6: accounting for preparation and review effort materially changes efficiency comparisons (descriptive).1617## 4. Primary endpoint and estimand18Y_iar = 1{scientific_success ∧ status = completed ∧ no human intervention}. Task-weighted completion rate p̂_a = (1/J) Σ_j (1/R_j) Σ_r Y_jar; primary contrast Δ̂_DA = p̂_D − p̂_A. Prespecified secondary contrasts: B−A, C−B, D−C (reported with the same uncertainty method; the primary contrast is the only one for which a confirmatory claim is made; secondary contrasts are reported as mechanism evidence with Holm adjustment over the three).1920## 5. Uncertainty21Task-cluster bootstrap (resample tasks with replacement, B = 5,000; percentile 95% intervals of the task-weighted paired difference) and an exact/Monte-Carlo paired permutation test over tasks (sign-flip of task-level paired differences, 20,000 permutations). With J ≤ 24 task clusters, interval widths are expected to be wide; this is reported, not hidden. Secondary: family-level estimates with their own cluster counts; hierarchical logistic model as a sensitivity analysis only.2223## 6. Scale decision rule (fixed before the pilot)24Let c̄ be the mean observed cost per run in the pilot (24 runs, all arms) and t̄ the mean wall time. The confirmatory configuration is the largest of {24 tasks × 4 arms × 3 repeats (288 runs), 24 × 4 × 2 (192), 12 × 4 × 2 (96)} whose projected cost 1.15·c̄·N_runs fits within the remaining study envelope after reserving 150 USD for transfer, recovery and ablations (amendment A-001: 800 USD total), and whose projected wall time at 8 concurrent runs is ≤ 12 hours. If 288 runs fit, repeats R = 3; if only 192 fit, R = 2 with all 24 tasks (coverage over repeats); 12 × 4 × 2 only as a last resort, choosing 2 tasks per family by the registry order (not by pilot difficulty).2526## 7. Exclusions and missingness27None of the assigned runs are excluded. Infrastructure errors (no inference performed) are rerun under the attempt-numbered policy and the original attempt is reported. Runs that time out or exhaust the budget are evaluated and count as Y = 0. Missing evaluator outputs (evaluator crash) are reported as missing, not as failures, and the crash is fixed and re-evaluated only if the fix does not depend on the run's content (the fix and re-evaluation are logged as an amendment).2829## 8. Stopping30The confirmatory stage runs the frozen assignment once; no run is added or removed after looking at outcomes. Exploratory follow-ups are labelled exploratory.3132## 9. Freeze procedure33Before any confirmatory run: hashes of every confirmatory bundle (`bundle.json`), every oracle (`oracle.json` + `evaluate.py`), the four arm texts, the common preamble, the reviewer prompt, the gate program, `configs/*.yaml`, the analysis scripts' interfaces, and this document are recorded in `protocol/freeze_manifest.json` and in `tasks/registry.csv` (`frozen_utc`). Any later change is an amendment (`protocol/amendments.csv`) with before/after-results status.34