1\section{Experimental design}2\label{sec:design}34\subsection{Tasks}56The benchmark comprises \nConfTasks{} confirmatory tasks, four in each of six families, plus seven development tasks (one smoke task and one development task per family) used only for the smoke test and the pilot. Table~\ref{tab:tasks} lists every confirmatory task with its data type, reference-answer provenance, planted condition and difficulty rationale; the task cards in Appendix~\ref{app:tasks} give the full statements. The families follow the mandate: F1 data provenance and audit (planted duplicates, unit mixes, vintage misalignments, a changing country universe, a final-vintage variable unavailable at origin, a corrupted file reconciled to a control total); F2 estimation and inference (two-way fixed effects, design-based uncertainty under clustering, a memo with an inappropriate i.i.d.\ standard error under serial dependence, Monte Carlo coverage, Okun's law on FRED data); F3 forecasting and temporal information (rolling-origin evaluation on public final-vintage series with release lags, one genuinely real-time task built on ALFRED vintages (ALFRED is the St.\ Louis Fed's archive of historical data releases), a future-informed normalisation trap and overlapping multi-horizon targets); F4 source-grounded evidence synthesis (coefficient extraction with locators, assertion checking, harmonisation of incompatible effect-size definitions, citation verification, all on CC~BY arXiv e-prints and a Statistics Canada release); F5 theory and numerical validation (a Cobb--Douglas exchange economy with a broken aggregate identity, the Marriott--Pope AR(1) bias, a false proposition requiring a counterexample, value-function iteration against a closed form); F6 integrated mini-projects (a Preston curve, a genuine null result on weekday effects in Treasury yields, an invalid-premise event study, Okun's law with a stability test, a planted confounder). Of the \nConfTasks{} confirmatory tasks, \nConfRealData{} use observed public data (FRED, ALFRED, World Bank, Bank of Canada), \nConfSourceDocs{} are source-document tasks, and \nConfNullTruth{} have a null, infeasible or false-premise truth, so that honest recognition rather than completion is the correct answer.78Every task is delivered as an immutable bundle (statement, inputs with provenance, a public oracle-free gate specification, a manifest with the SHA-256 of every file) and has a hidden oracle (reference values, tolerances, required criteria, critical-violation checks, a deterministic evaluator and a seeded validation set). Reference values come from analytical identities, from code written independently from any solution, or from source documents with table and page locators; no reference was produced by a language model solving the task. Every evaluator was validated before the pilot against at least one independently written correct solution and at least two wrong ones (Appendix~\ref{app:evaluators}); the pilot then exposed one harness defect in how evaluators were invoked (Section~\ref{sec:pilot}), which was corrected before the confirmatory stage; the audit after the results found four oracle-level defects (Section~\ref{sec:results}).910\subsection{Treatments}1112Table~\ref{tab:treatments} summarises the four arms. All arms receive the same task statement, inputs, output obligations (a results file with named keys, a six-section report, a replication script that regenerates the results from the inputs, and a claims ledger), the same common preamble with environment facts and integrity rules, the same client, model, delivered tool inventory, permission policy, node without web tools, pinned Python environment and resource envelope (\envUSD{} USD, \envWall{} s of wall clock, \envTurns{} turns per session). The arms differ only in the text appended to the system prompt and in what the controller does after a session ends. (The user prompt intended for every arm, ``Begin now. The task is in \texttt{TASK.md} in the current directory. Work until the deliverables are complete, then stop.'', was truncated by a launcher quoting defect to its first word in every session; see Section~\ref{sec:implementation} and amendment A-006. The defect is symmetric across arms.) Arm A appends one sentence. Arm B appends a six-phase protocol (scope and contract; data audit with an exclusion ledger; baseline; method and replication script; verification; reporting with a claims ledger) with stopping and recovery rules and a phase ledger. Arm C adds a mandatory, executable gate: the executor must run the gate program before declaring completion, and the controller re-runs its own copy after the session and returns failures as resumed turns, at most three times within the envelope. Arm D adds a separate review-and-repair session after the gate passes, drawn from the same envelope. The appended texts are 6, 665, 827 and 921 words long (common preamble 279 words); the first assistant turn of the smoke runs consumed \ctxA, \ctxB, \ctxC{} and \ctxD{} input tokens respectively, the difference being the protocol text only. A length-matched unstructured control (general research advice, 533 words, no phases, ledgers or gates) is used in the ablation of Section~\ref{sec:recovery}.1314\begin{table}[htbp]\centering\small15\caption{Treatments. Common to all arms: task statement and bundle, common preamble (279 words), client 2.1.314 in bare mode, model claude-opus-5-5, delivered tools Bash/Edit/Read, envelope \envUSD{} USD / \envWall{} s / \envTurns{} turns per session, same user prompt. First-turn input tokens measured in the smoke runs.}16\label{tab:treatments}17\begin{tabular}{p{0.9in}p{2.6in}p{1.9in}cc}18\toprule19Arm & Appended system-prompt text & Controller after the session & Words & Tokens \\20\midrule21A open & one sentence & collect; post-hoc contract check; evaluate & 6 & \ctxA \\22B protocol & phases P0--P5, stopping and recovery rules, phase/claims/exclusion ledgers & as A & 665 & \ctxB \\23C + gates & B + mandatory self-run gate; controller gate loop ($\leq 3$ cycles) & controller gate; feedback turn on failure & 827 & \ctxC \\24D + review & C + announced separate review & as C; reviewer session; one repair turn on \emph{revise} & 921 & \ctxD \\25\bottomrule26\end{tabular}27\end{table}2829\subsection{Development pilot}30\label{sec:pilot}31A smoke test (one development task, four arms) and a pilot (six development tasks, four arms, one repeat) preceded the freeze. All 28 runs finished within the envelope and 27 were scientifically successful; the pilot exposed one harness defect (final evaluators invoked on a workspace without the task inputs, which scored four correct source-grounded answers as fabrications; amendment A-002, corrected before the freeze) and fixed the scale decision (mean cost 0.39 USD and 115 s per run, hence the full 24 $\times$ 4 $\times$ 3 design). The ceiling on the development tasks was recorded before the confirmatory runs as a possible, legitimate outcome.3233\subsection{Assignment, blocking and repeats}3435The design is paired: every confirmatory task is run under every arm with \nRepeats{} repeats, for \nMainRuns{} assigned runs. A seeded generator shuffles the order of task blocks and, within each block, the order of the (arm, repeat) cells; runs are dispatched to four executor nodes with two concurrent sessions per node, so that the arms of a task block execute interleaved in time and no arm is systematically earlier than another. Repeats are fresh sessions in fresh workspaces; the client exposes no seed or temperature control, so model stochasticity is not controlled and repeat outcomes are preserved as observed. The assignment manifest and every run's node, order and timestamps are stored before launch.3637\subsection{Endpoints and estimands}3839The primary endpoint for run $(i,a,r)$ is40\begin{equation}41Y_{iar} = \mathbf{1}\{\text{required criteria satisfied} \wedge \text{no critical violation} \wedge \text{completed within the envelope} \wedge \text{no human intervention}\},42\label{eq:Y}43\end{equation}44and the task-weighted completion rate of arm $a$ and the paired contrast are45\begin{equation}46\hat p_a = \frac{1}{J}\sum_{j=1}^{J}\frac{1}{R_j}\sum_{r=1}^{R_j} Y_{jar}, \qquad \hat\Delta_{ab} = \hat p_a - \hat p_b .47\label{eq:phat}48\end{equation}49The primary contrast is D$-$A; B$-$A, C$-$B and D$-$C are prespecified mechanism contrasts with Holm adjustment. Uncertainty uses a task-cluster bootstrap (5,000 resamples of tasks, percentile intervals of the paired difference) and a sign-flip permutation test on the task-level paired differences (20,000 permutations); with $J=\nConfTasks$ task clusters, intervals are wide, and we report them as such. Secondary endpoints are correct completion regardless of the envelope, quality components (correctness, integrity, reproducibility, reporting, baseline; frozen weights 0.40/0.20/0.15/0.15/0.10), critical violations by type (leakage, fabrication, invalid inference, unreported substitution, false completion, provenance), fresh-environment reproducibility, correct recognition of infeasible or null-truth tasks and false blocking on feasible ones, and cost, usage and wall time with full denominators. The analysis plan, the scale decision rule and the freeze procedure were written before the pilot (\texttt{protocol/preregistration.md}) and frozen with hashes before the confirmatory stage; every departure is an amendment in \texttt{protocol/amendments.csv}.50