# WP42 — Research plan (operational design, v1.0, 2026-11-02)

This file turns the mandate (CLAUDE.md) into the concrete design that the code, the scenarios and the preregistration implement. Decisions the mandate left open are listed in §9 with the reason; they are the executing agent's, not the PI's, until confirmed.

# 1. Question and estimands

Primary question: what share of professionally acceptable white-collar deliverables can an autonomous Claude Code worker produce under successive levels of computer access, at what cost, with what reliability, and how much of the gap is closed by tools and by bounded self-repair?

Unit of observation: one run = one worker session (or one session plus its bounded recovery, arm A4) on one scenario under one arm. Outcomes per run (all defined before any run is executed; see protocol/preregistration/PREREGISTRATION.md):

• S acceptance = deterministic completion (every required criterion passes, no critical violation) and blind quality ≥ q (q = 70/100).
• D deterministic completion alone; Q blind quality (mean of the blind evaluators, 0–100); E critical-error indicator; C marginal API cost (USD, from result.total_cost_usd); T wall time (s, censored at the limit); tool calls, turns, tokens, retries, recovery attempts/success, application transitions, abstention/clarification flag.

Primary contrasts (acceptance probability, task-weighted): A1 − A0 (shell premium), A3 − A1 (computer-use premium), A4 − A3 (self-repair premium). Secondary: A2 − A1 (macOS layer), cost per accepted deliverable by arm, quality and critical-error rates, time-to-completion (survival), heterogeneity by occupation / difficulty / dependency depth, perturbation robustness.

# 2. Treatments (agent arms; the human arms H0/H1 are designed in protocol/ethics/ but NOT executed — no participants, no approval)

Model: claude-opus-5-5 in every arm (primary). Sensitivity: claude-sonnet-5-5 and claude-fable-5-1 on arm A3 for the same scenarios (if the budget rule allows). Effort: default. Per-session caps: 4.00 USD, 1,800 s wall, 150 turns. A4 recovery: the same caps minus what was spent.

# 3. Scenarios (benchmark)

Pilot: 10 scenarios, 10 occupations (accounting, corporate finance, real estate appraisal, academic research, executive administration, banking, legal, HR, procurement, business intelligence); see benchmark/SCENARIO_AUTHORING_SPEC.md and benchmark/taxonomy/. Each scenario = public bundle (brief, inputs, portal data) + hidden oracle (truths, deterministic evaluator, rubric) kept in ~/wp42-oracles/ on the controller only. Scenarios embed designed features: planted errors, a prompt-injection document, a missing-information trap requiring clarification/abstention, and a designed dependency depth (number of dependent operations a careful professional must perform). Scale-up (Phase 6) is conditional on the preregistered rule (§5).

# 4. Pipeline per run

1. Controller draws the assignment (seeded randomisation of scenario × arm × repeat order, interleaved over nodes).
2. Workspace <node>:~/wp42/runs/<run_id>/ is created from the public bundle (rsync), plus tools/ (public self-check for A4), plus the arm's adapter documentation; the oracle is never copied.
3. For A3/A4 the portal is started on the node (127.0.0.1:<port>, task's portal.json), credentials are injected in the prompt.
4. claude --bare -p "$(cat PROMPT.md)" --output-format stream-json … runs under the arm's flags; the controller streams events to data/raw/runs/<run_id>/raw_stream.jsonl (immutable), kills the process at the wall limit, and records the result event.
5. out/ is collected, every artefact is SHA-256 hashed, the portal submissions are collected, the workspace is deleted (reset).
6. Deterministic evaluation (hidden evaluate.py) → receipt; blind evaluation (3 LLM judges, rubric, no arm/model information) → quality scores; metrics parsed from the stream; everything stored in SQLite and exported to Parquet (data/validated/).

# 5. Scale decision rule (fixed before the pilot runs)

Pilot design: 10 scenarios × 5 arms × 2 repeats = 100 runs (+ sensitivity models later). Study envelope: 800 USD (§9). After the pilot: if mean cost per run c̄ and the remaining envelope allow 1.15·c̄·N for N = 10 × 5 × 3 additional runs plus 2 model sensitivities (10 × 2 × 2), run them (confirmatory stage with R = 3 repeats per cell after pooling with the pilot only if no scenario or evaluator was modified; otherwise the pilot is reported separately). Scaling to 20 scenarios is conditional on the remaining envelope and on the pilot's evaluator-agreement check (Krippendorff α ≥ 0.6 on quality).

# 6. Analysis (Agent G)

Task-cluster bootstrap (B = 5,000) for task-weighted acceptance differences; paired sign-flip permutation over scenarios; mixed-effects logistic (scenario random intercept) as sensitivity; Kaplan–Meier / Cox for wall time with censoring; cost per accepted deliverable with bootstrap CIs; ALF curves over the budget B and the human-time H; Workflow Dependency Frontier (acceptance vs designed depth / tool calls); Autonomous Recovery Index (A4: P(recovered | first session failed public self-check)); Professional Reliability Frontier (completion × quality × critical errors × cost); evaluator agreement (α, ICC); sensitivity to q, weights, prices, wages.

# 7. Roles → processes (operational separation)

# 8. Phases and status: see PROJECT_STATUS.md.

# 9. Executor decisions pending PI confirmation (logged as amendments in protocol/amendments.csv)

• A-001 Study envelope 800 USD (the mandate gives none; WP29 used the same figure), 4 USD / session.
• A-002 Human arms H0/H1 not executed (no participants, no ethics approval); protocol written for a future wave.
• A-003 A2 defined as the headless macOS automation layer because GUI Apple Events are blocked by TCC over SSH (Phase 0 §0.5).
• A-004 Primary model claude-opus-5-5 (production default of the series' earlier study), Fable/Sonnet as sensitivities.
• A-005 Blind evaluators are LLM judges; the independent human expert review is left to the PI (not performed in-session).