1# WP42 — Research plan (operational design, v1.0, 2026-11-02)23This file turns the mandate (`CLAUDE.md`) into the concrete design that the code, the scenarios and the preregistration implement.4Decisions the mandate left open are listed in §9 with the reason; they are the executing agent's, not the PI's, until confirmed.56## 1. Question and estimands7Primary question: what share of professionally acceptable white-collar deliverables can an autonomous Claude Code worker produce8under successive levels of computer access, at what cost, with what reliability, and how much of the gap is closed by tools and9by bounded self-repair?1011Unit of observation: one **run** = one worker session (or one session plus its bounded recovery, arm A4) on one scenario under one arm.12Outcomes per run (all defined before any run is executed; see `protocol/preregistration/PREREGISTRATION.md`):13- `S` acceptance = deterministic completion (every required criterion passes, no critical violation) **and** blind quality ≥ q (q = 70/100).14- `D` deterministic completion alone; `Q` blind quality (mean of the blind evaluators, 0–100); `E` critical-error indicator;15 `C` marginal API cost (USD, from `result.total_cost_usd`); `T` wall time (s, censored at the limit); tool calls, turns, tokens,16 retries, recovery attempts/success, application transitions, abstention/clarification flag.1718Primary contrasts (acceptance probability, task-weighted): **A1 − A0** (shell premium), **A3 − A1** (computer-use premium),19**A4 − A3** (self-repair premium). Secondary: A2 − A1 (macOS layer), cost per accepted deliverable by arm, quality and critical-error20rates, time-to-completion (survival), heterogeneity by occupation / difficulty / dependency depth, perturbation robustness.2122## 2. Treatments (agent arms; the human arms H0/H1 are designed in `protocol/ethics/` but NOT executed — no participants, no approval)23| Arm | Claude Code tool set (CLI flags) | Extra environment | Nodes |24|---|---|---|---|25| A0 Restricted | `--tools Read,Write,Edit,Glob,Grep` (no shell) | none | any executor |26| A1 Shell | A0 + `Bash`; `--disallowedTools "Bash(osascript:*)" "Bash(open:*)" "Bash(soffice:*)" "Bash(*LibreOffice*)"`; policy text forbids browser/office apps | project venv (python, pandas, openpyxl, python-docx, …) | any |27| A2 macOS | A1 tools + `osascript` and macOS document utilities allowed (adapter doc appended) | same venv | M3U96a, M4M64a (System Events answers) |28| A3 Full computer-use | A2 + synthetic enterprise portal (loopback HTTP, Playwright headless Chromium) + LibreOffice headless (office adapter) | portal started per run; adapter docs appended | any (portal and soffice on all four) |29| A4 Self-repairing | A3 + checkpointing instructions + public self-check tool + **one** bounded controller-side recovery resume if the first session ends without passing the public self-check | same | any |3031Model: `claude-opus-5-5` in every arm (primary). Sensitivity: `claude-sonnet-5-5` and `claude-fable-5-1` on arm A3 for the same scenarios (if the budget rule allows).32Effort: default. Per-session caps: 4.00 USD, 1,800 s wall, 150 turns. A4 recovery: the same caps minus what was spent.3334## 3. Scenarios (benchmark)35Pilot: 10 scenarios, 10 occupations (accounting, corporate finance, real estate appraisal, academic research, executive36administration, banking, legal, HR, procurement, business intelligence); see `benchmark/SCENARIO_AUTHORING_SPEC.md` and37`benchmark/taxonomy/`. Each scenario = public bundle (brief, inputs, portal data) + hidden oracle (truths, deterministic evaluator,38rubric) kept in `~/wp42-oracles/` on the controller only. Scenarios embed designed features: planted errors, a prompt-injection39document, a missing-information trap requiring clarification/abstention, and a designed dependency depth (number of dependent40operations a careful professional must perform). Scale-up (Phase 6) is conditional on the preregistered rule (§5).4142## 4. Pipeline per run431. Controller draws the assignment (seeded randomisation of scenario × arm × repeat order, interleaved over nodes).442. Workspace `<node>:~/wp42/runs/<run_id>/` is created from the public bundle (rsync), plus `tools/` (public self-check for A4),45 plus the arm's adapter documentation; the oracle is never copied.463. For A3/A4 the portal is started on the node (`127.0.0.1:<port>`, task's `portal.json`), credentials are injected in the prompt.474. `claude --bare -p "$(cat PROMPT.md)" --output-format stream-json …` runs under the arm's flags; the controller streams events48 to `data/raw/runs/<run_id>/raw_stream.jsonl` (immutable), kills the process at the wall limit, and records the `result` event.495. `out/` is collected, every artefact is SHA-256 hashed, the portal submissions are collected, the workspace is deleted (reset).506. Deterministic evaluation (hidden `evaluate.py`) → receipt; blind evaluation (3 LLM judges, rubric, no arm/model information) →51 quality scores; metrics parsed from the stream; everything stored in SQLite and exported to Parquet (`data/validated/`).5253## 5. Scale decision rule (fixed before the pilot runs)54Pilot design: 10 scenarios × 5 arms × 2 repeats = 100 runs (+ sensitivity models later). Study envelope: 800 USD (§9).55After the pilot: if mean cost per run c̄ and the remaining envelope allow 1.15·c̄·N for N = 10 × 5 × 3 additional runs plus 2 model56sensitivities (10 × 2 × 2), run them (confirmatory stage with R = 3 repeats per cell after pooling with the pilot only if no57scenario or evaluator was modified; otherwise the pilot is reported separately). Scaling to 20 scenarios is conditional on the58remaining envelope and on the pilot's evaluator-agreement check (Krippendorff α ≥ 0.6 on quality).5960## 6. Analysis (Agent G)61Task-cluster bootstrap (B = 5,000) for task-weighted acceptance differences; paired sign-flip permutation over scenarios;62mixed-effects logistic (scenario random intercept) as sensitivity; Kaplan–Meier / Cox for wall time with censoring; cost per63accepted deliverable with bootstrap CIs; ALF curves over the budget B and the human-time H; Workflow Dependency Frontier64(acceptance vs designed depth / tool calls); Autonomous Recovery Index (A4: P(recovered | first session failed public self-check));65Professional Reliability Frontier (completion × quality × critical errors × cost); evaluator agreement (α, ICC); sensitivity to q,66weights, prices, wages.6768## 7. Roles → processes (operational separation)69| Role | Process | Sees oracle? |70|---|---|---|71| A Director | this controller session + `protocol/` | yes |72| B Scenario generator | one fresh subagent per scenario (writes bundle + oracle) | writes it |73| C Scenario validator | one fresh subagent per scenario, solves from the public bundle first, then compares | after solving |74| D Worker | `claude --bare` on an executor node, fresh workspace | **never** |75| E Deterministic evaluator | `evaluate.py` on the controller | yes |76| F Blind evaluator | `claude -p --json-schema` judges on the controller, rubric + deliverables only | rubric only |77| G Statistician | `analysis/` scripts | validated data |78| H Writer | subagents writing `manuscript/` from `analysis/` outputs | no |79| I Auditor | fresh subagent checking claims vs data at the end | all records |8081## 8. Phases and status: see `PROJECT_STATUS.md`.8283## 9. Executor decisions pending PI confirmation (logged as amendments in `protocol/amendments.csv`)84- A-001 Study envelope 800 USD (the mandate gives none; WP29 used the same figure), 4 USD / session.85- A-002 Human arms H0/H1 not executed (no participants, no ethics approval); protocol written for a future wave.86- A-003 A2 defined as the headless macOS automation layer because GUI Apple Events are blocked by TCC over SSH (Phase 0 §0.5).87- A-004 Primary model claude-opus-5-5 (production default of the series' earlier study), Fable/Sonnet as sensitivities.88- A-005 Blind evaluators are LLM judges; the independent human expert review is left to the PI (not performed in-session).89