# Preregistration — WP29 (v1.0, frozen 2026-10-30 after the pilot and before any confirmatory run; hashes in protocol/freeze_manifest_conf.json)

> Version history: v0.9 drafted 2026-10-30 ≈ 04:00 UTC before the pilot (git 76de75e…2c82656); v1.0 adds §10 (pilot-informed decisions) only. No hypothesis, endpoint, contrast or analysis rule was changed after the pilot.

## 1. Research question
Under controlled information, tool access and resource conditions, does an explicit research protocol, combined with executable verification and persistent evidence records, improve Claude Code's ability to complete scientifically valid research tasks autonomously, and at what cost in compute, time and human effort?

## 2. Design
Paired, four-arm comparison (A open instruction; B structured protocol; C protocol + controller-enforced executable gates; D = C + separate review-and-repair session). Every confirmatory task is run under every arm with R repeats; arm order within a task block and the order of task blocks are randomised (seeded) and interleaved in time across four executor nodes; model, client, tools, envelope and task information are identical (see `protocol/treatment_definitions.md`). Executors are fresh `claude --bare` sessions on hosts that do not hold the oracles.

## 3. Hypotheses (directions are expectations, not guarantees)
- H1 (primary): p_D − p_A > 0 for intervention-free, science-verified completion within the envelope.
- H2: critical integrity violations per run are lower in C than in B (executable gates).
- H3: claim validity (fraction of `supported` claims that are actually supported) is higher in D than in C, at a measurable additional cost (reviewer share of the envelope).
- H4: after a standardised interruption, valid continuation (no duplicated expensive work, no lost artefacts, final scientific success) is more frequent under protocols with persistent state (B/C/D) than under A.
- H5: effects differ by task family (F1–F6) and by task difficulty class.
- H6: accounting for preparation and review effort materially changes efficiency comparisons (descriptive).

## 4. Primary endpoint and estimand
Y_iar = 1{scientific_success ∧ status = completed ∧ no human intervention}. Task-weighted completion rate p̂_a = (1/J) Σ_j (1/R_j) Σ_r Y_jar; primary contrast Δ̂_DA = p̂_D − p̂_A. Prespecified secondary contrasts: B−A, C−B, D−C (reported with the same uncertainty method; the primary contrast is the only one for which a confirmatory claim is made; secondary contrasts are reported as mechanism evidence with Holm adjustment over the three).

## 5. Uncertainty
Task-cluster bootstrap (resample tasks with replacement, B = 5,000; percentile 95% intervals of the task-weighted paired difference) and an exact/Monte-Carlo paired permutation test over tasks (sign-flip of task-level paired differences, 20,000 permutations). With J ≤ 24 task clusters, interval widths are expected to be wide; this is reported, not hidden. Secondary: family-level estimates with their own cluster counts; hierarchical logistic model as a sensitivity analysis only.

## 6. Scale decision rule (fixed before the pilot)
Let c̄ be the mean observed cost per run in the pilot (24 runs, all arms) and t̄ the mean wall time. The confirmatory configuration is the largest of {24 tasks × 4 arms × 3 repeats (288 runs), 24 × 4 × 2 (192), 12 × 4 × 2 (96)} whose projected cost 1.15·c̄·N_runs fits within the remaining study envelope after reserving 150 USD for transfer, recovery and ablations (amendment A-001: 800 USD total), and whose projected wall time at 8 concurrent runs is ≤ 12 hours. If 288 runs fit, repeats R = 3; if only 192 fit, R = 2 with all 24 tasks (coverage over repeats); 12 × 4 × 2 only as a last resort, choosing 2 tasks per family by the registry order (not by pilot difficulty).

## 7. Exclusions and missingness
None of the assigned runs are excluded. Infrastructure errors (no inference performed) are rerun under the attempt-numbered policy and the original attempt is reported. Runs that time out or exhaust the budget are evaluated and count as Y = 0. Missing evaluator outputs (evaluator crash) are reported as missing, not as failures, and the crash is fixed and re-evaluated only if the fix does not depend on the run's content (the fix and re-evaluation are logged as an amendment).

## 8. Stopping
The confirmatory stage runs the frozen assignment once; no run is added or removed after looking at outcomes. Exploratory follow-ups are labelled exploratory.

## 9. Freeze procedure
Before any confirmatory run: hashes of every confirmatory bundle (`bundle.json`), every oracle (`oracle.json` + `evaluate.py`), the four arm texts, the common preamble, the reviewer prompt, the gate program, `configs/*.yaml`, the analysis scripts' interfaces, and this document are recorded in `protocol/freeze_manifest.json` and in `tasks/registry.csv` (`frozen_utc`). Any later change is an amendment (`protocol/amendments.csv`) with before/after-results status.

## 10. Pilot-informed decisions (recorded at freeze, before any confirmatory run)
- Pilot (24 runs, 6 development tasks × 4 arms, 2026-10-30 03:47–04:02 UTC): every run completed within the envelope; 23 of 24 were scientifically successful (the exception: F3-dev-01 under arm A, failed criteria without violations); mean cost 0.39 USD per run, mean wall 115 s. The development tasks therefore show a ceiling; the confirmatory tasks (designed with planted defects, information traps, null truths and false premises before the pilot was run) are used unchanged. We state in advance that a ceiling in the confirmatory stage is a possible, legitimate outcome (no detectable protocol effect at this task difficulty with this model).
- Scale decision (rule §6): projected cost of 24 × 4 × 3 = 288 runs ≈ 1.15 × 0.39 × 288 ≈ 130 USD and ≈ 1.1 h at 8 concurrent runs, within the envelope; **the confirmatory design is 24 tasks × 4 arms × 3 repeats (288 runs)**, assignment seed 301, 4 nodes × 2 concurrent sessions.
- Harness amendment A-002 (evaluator workspace composition) was applied before the freeze; no oracle was modified after its validation except through that harness fix.
- One harness fact learned in the pilot: first-turn input tokens are identical across tasks within an arm (the task statement is read from the workspace, not injected), so arm-level context sizes are A 2,238 / B 3,717 / C 4,111 / D 4,319 tokens.