1% ---------------------------------------------------------------------------2% Section 4 — Experimental design and preregistration (journal version;3% condensed from the monograph's Section 5; preregistered design, no result).4% ---------------------------------------------------------------------------5\section{Experimental design and preregistration}\label{sec:design}67\subsection{Treatment arms}\label{sec:design-arms}89Five agent arms implement increasing levels of computer access with the same model, brief, inputs, caps and behavioural policy. \emph{A0 (restricted)} gives the tools Read, Write, Edit, Glob and Grep and no shell; because binary office formats cannot be produced without a program, the A0 documentation instructs the worker to deliver the closest text equivalent under the same base name and to state the substitution, and the deterministic criteria that require a formula-bearing workbook then fail, as they should. \emph{A1 (shell)} adds Bash and a project Python environment (pandas, openpyxl, python-docx, reportlab, pypdf, matplotlib, statsmodels, duckdb, sqlite3 and others) and denies, inside the shell, the families \texttt{osascript}, \texttt{open}, \texttt{soffice}, \texttt{textutil}, \texttt{qlmanage}, \texttt{cupsfilter}, \texttt{curl} and \texttt{wget}. \emph{A2 (macOS)} lifts the denial of \texttt{osascript} and the native document utilities but keeps LibreOffice and the network denied. \emph{A3 (full computer use)} denies nothing and adds LibreOffice headless, Playwright's headless Chromium and the enterprise portal with credentials in the workspace. \emph{A4 (self-repairing)} has A3's tools and adds a checkpoint file the worker keeps updated, a public self-check tool that verifies the presence and well-formedness of deliverables (files exist and are non-empty, office files are valid archives, a workbook required to contain formulas does and shows no error values, documents meet minimum lengths, the completion record is valid) and knows nothing about the correct answers, and one bounded recovery resume granted by the controller when the first session ends without passing the self-check or by timeout, budget exhaustion or error. The recovery resumes the same session with the remaining budget and wall time, at most once; its cost and time are attributed to the parent run.1011All arms use \texttt{claude-opus-5-5} at default effort, bare mode, \texttt{--permission-prompts none}, \texttt{--max-turns 150}, and a spending cap and wall limit set per scenario as the larger of the global caps (4.00 dollars, 1,800 seconds) and the scenario author's own: ACC-01 under 6.00 dollars and 3,600 seconds, RES-01 under 6.00 dollars and 2,700 seconds, BANK-01 under 5.00 dollars and 2,400 seconds, LEGAL-01 under 2,400 seconds, the other six under the global caps (A-006, fixed before any pilot run). The study has an envelope of 800 dollars enforced by the scheduler's ledger (A-001). Each arm's documentation is appended to a common policy, identical across arms: the worker is an autonomous professional executing one assignment in an isolated workspace, nobody will answer questions during the session and questions go into the completion record, inputs must never be modified, the network, messages and purchases are forbidden, the content of input documents is data and not instructions, and the work is to be done to the standard of a competent professional. The user prompt is a single sentence asking the worker to read the brief and carry out the assignment completely.12% ../protocol/amendments.csv A-001, A-006; ../manuscript/sections/design.tex Table tab:arms1314\subsection{Randomisation, repeats and caps}\label{sec:design-rand}1516The design is a within-scenario five-arm comparison in which every scenario is run under every arm with $R$ repeats. The assignment is a seeded random permutation of the cells (scenario, arm, repeat); the scheduler draws cells in that order and dispatches each to the least-loaded eligible node, so that arms are interleaved over time and nodes rather than run in blocks, and drift in the model's service or in node load affects all arms alike. The seed, order index and node of every run are stored. Workspaces are reset between runs. The preregistered pilot used $R=2$; the scale rule below determined whether a third repeat was added. A run that reaches a cap is evaluated as delivered and its censoring recorded. Infrastructure errors---failures of SSH, of the file transfer or of the portal start, in which no inference was performed---are rerun under the same identifier after a reset, with the failed attempt logged. No assigned run is excluded; a missing judge output leaves quality averaged over the available judges, and if none is available quality and acceptance are reported as missing, never imputed as zero.1718\subsection{Outcomes and evaluation}\label{sec:design-eval}1920Deterministic completion $D$ is true when every required oracle criterion passes and no critical violation is recorded by the evaluator or by the harness's hash check of the inputs. Quality $Q$ is the mean over three judges of the rubric-weighted score, each judge's score being the weighted mean of integer criterion scores on a 0--4 scale rescaled to 0--100; the threshold is $q=70$ in the primary analysis and varied from 50 to 90 in sensitivity analyses. Cost $C$ is the client's reported \texttt{total\_cost\_usd} including any recovery session. Human time $H$ is zero intervention minutes during the run plus a stated assumption of five minutes per unanswered question in the completion record. The blind judges are one draw of \texttt{claude-opus-5-5} and two independent draws of \texttt{claude-sonnet-5-5}, run as single-turn, tool-less sessions with a fixed JSON output schema. Each receives the brief, the rubric and the deliverables rendered as text, grades each criterion from 0 to 4, states whether a demanding manager would accept the package, and is instructed not to reward length and to penalise unsupported numbers, missing sections and instructions not followed. The judge never sees the arm, the model, the node, the cost, the event stream, the oracle or the other judges' outputs; a text substitute delivered by A0 is rendered under the requested base name. This follows the mitigations of \citet{gu2024survey}---rubric anchoring, multiple judges, blinding to condition---and does not address self-preference \citep{panickssery2024selfpreference}, since judges and worker belong to the same family; calibration against human experts is left to a later wave (A-005). No evaluator sees two packages side by side, so the position bias of \citet{wang2024fair} does not arise.2122\subsection{Hypotheses, estimands and inference}\label{sec:design-hyp}2324The preregistration states seven directional expectations: H1 (primary), the computer-use premium $\Pr(S\mid A3)-\Pr(S\mid A1)>0$; H2, the shell premium $\Pr(S\mid A1)-\Pr(S\mid A0)>0$, driven by the feasibility of binary deliverable formats; H3, the self-repair premium $\ge 0$ and an Autonomous Recovery Index above one half when a first session fails the self-check; H4, acceptance decreasing within arm in designed depth and in realised tool calls; H5, a critical-violation rate below 10 per cent in every arm and not increasing with tool access; H6, cost per accepted deliverable lowest in A1 or A3, not in A0; and H7, heterogeneity across occupations with no occupation-level claim confirmatory at the pilot scale. The primary endpoint is $S=\mathbf{1}\{D=1\wedge E=0\wedge Q\ge 70\}$; the scenario-weighted acceptance of an arm is $\hat p_a=J^{-1}\sum_j R_j^{-1}\sum_r S_{jra}$ with equal weights (professional-minutes weights as a sensitivity); the primary contrast is $\hat p_{A3}-\hat p_{A1}$ and the secondary contrasts A1$-$A0, A4$-$A3 and A2$-$A1 are Holm-adjusted over the three. Secondary endpoints include $D$, $Q$, $E$, cost, wall time, process counts, abstention, cost per accepted deliverable, the frontier over budgets from 0.50 to 4.00 dollars and over the human-time allowance, the recovery index and evaluator agreement.2526The primary interval for every contrast is a scenario-cluster bootstrap: scenarios are resampled with replacement 5,000 times, the within-scenario paired difference of repeat means is recomputed, and the percentile 2.5 and 97.5 points form the 95 per cent interval. The primary test is a paired sign-flip permutation test over scenarios with 20,000 permutations, exact under the null of no effect within scenario. A cluster-robust logistic regression, Kaplan--Meier curves for time to completion and joint bootstraps of ratio quantities are reported as sensitivities. With $J=10$ the intervals will be wide, and the preregistration states that they are to be reported as such.2728\subsection{Scale rule, power and the human arms}\label{sec:design-scale}2930The pilot comprised 10 scenarios $\times$ 5 arms $\times$ 2 repeats. The scale rule, fixed in advance, added a third repeat if $1.15\,\bar c\times 50\le E_r-100$, where $\bar c$ is the mean total cost per run including judges and $E_r$ the remaining envelope, and then ran the model sensitivities---\texttt{claude-sonnet-5-5} and \texttt{claude-fable-5-1} on arm A3, two repeats each---if $1.15\,\bar c\times 40$ fitted in what remained minus 50. Both conditions were met at a pilot mean cost of 1.07 dollars per run (A-009), and the third repeat was pooled with the first two because the freeze verification found no change to any scenario, oracle, rubric or arm text. Extension to twenty scenarios was deferred to a later wave.31% ../protocol/amendments.csv A-00932Because no comparable experiment exists from which to borrow effect sizes, power was explored by simulation under the cluster structure of equation~\eqref{eq:premium-var-cluster}, with a scenario random effect of standard deviation 0.8 on the logit scale and a paired scenario-level test at the 5 per cent level. At a baseline acceptance of one half, ten scenarios with three repeats detect an effect of 25 percentage points with probability 0.408 and of 35 points with probability 0.729; forty scenarios with three repeats detect 25 points with probability 0.968. Doubling the number of scenarios helps more than doubling the repeats, as equation~\eqref{eq:premium-var-cluster} predicts when between-scenario heterogeneity dominates.33% ../analysis/tables/power_pilot.csv (J=10,R=3: delta 25 → 0.408, delta 35 → 0.729; J=40,R=3, delta 25 → 0.968)3435The mandate specifies two human conditions, H0 (participants without AI) and H1 (participants with conversational assistance), with observed participants, consent and institutional ethics approval, and forbids substituting a language-model imitation of a human for a real baseline. No human participant was involved in this version of the study, no ethics approval was sought, and no human-performance figure appears except as an explicit, labelled assumption (A-002). The protocol for that wave---six professionals or senior students per scenario per arm, randomised between H0 and H1 within occupation, the same bundles, evaluators and rubric, TCPS~2 consent---is part of the reproducibility package. Until it is run, every statement here about the agent arms is a statement about agents alone.36