1\section{Experimental design}2\label{sec:design}34\subsection{Tasks}56The benchmark comprises \nConfTasks{} confirmatory tasks, four in each of six families, plus seven development tasks (one smoke task and one development task per family) used only for the smoke test and the pilot. Table~\ref{tab:tasks} lists every confirmatory task with its data type, reference-answer provenance, planted condition and difficulty rationale; the task cards in Appendix~\ref{app:tasks} give the full statements. The families follow the mandate: F1 data provenance and audit (planted duplicates, unit mixes, vintage misalignments, a changing country universe, a final-vintage variable unavailable at origin, a corrupted file reconciled to a control total); F2 estimation and inference (two-way fixed effects, design-based uncertainty under clustering, a memo with an inappropriate i.i.d.\ standard error under serial dependence, Monte Carlo coverage, Okun's law on FRED data); F3 forecasting and temporal information (rolling-origin evaluation on public series with release lags, real-time vintages, a future-informed normalisation trap and overlapping multi-horizon targets); F4 source-grounded evidence synthesis (coefficient extraction with locators, assertion checking, harmonisation of incompatible effect-size definitions, citation verification, all on CC~BY arXiv e-prints and a Statistics Canada release); F5 theory and numerical validation (a Cobb--Douglas exchange economy with a broken aggregate identity, the Marriott--Pope AR(1) bias, a false proposition requiring a counterexample, value-function iteration against a closed form); F6 integrated mini-projects (a Preston curve, a genuine null result on weekday effects in Treasury yields, an invalid-premise event study, Okun's law with a stability test, a planted confounder). Of the \nConfTasks{} confirmatory tasks, \nConfRealData{} use observed public data (FRED, ALFRED, World Bank, Bank of Canada), \nConfSourceDocs{} are source-document tasks, and \nConfNullTruth{} have a null, infeasible or false-premise truth, so that honest recognition rather than completion is the correct answer.78Every task is delivered as an immutable bundle (statement, inputs with provenance, a public oracle-free gate specification, a manifest with the SHA-256 of every file) and has a hidden oracle (reference values, tolerances, required criteria, critical-violation checks, a deterministic evaluator and a seeded validation set). Reference values come from analytical identities, from code written independently from any solution, or from source documents with table and page locators; no reference was produced by a language model solving the task. Every evaluator was validated before the pilot against at least one independently written correct solution and at least two wrong ones (Appendix~\ref{app:evaluators}); the pilot then exposed one harness defect in how evaluators were invoked (Section~\ref{sec:pilot}), which was corrected before the confirmatory stage.910\subsection{Treatments}1112Table~\ref{tab:arms} summarises the four arms. All arms receive the same task statement, inputs, output obligations (a results file with named keys, a six-section report, a replication script that regenerates the results from the inputs, and a claims ledger), the same common preamble with environment facts and integrity rules, the same client, model, tool inventory, permission policy, offline node, pinned Python environment and resource envelope (\envUSD{} USD, \envWall{} s of wall clock, \envTurns{} turns per session). The arms differ only in the text appended to the system prompt and in what the controller does after a session ends. Arm A appends one sentence. Arm B appends a six-phase protocol (scope and contract; data audit with an exclusion ledger; baseline; method and replication script; verification; reporting with a claims ledger) with stopping and recovery rules and a phase ledger. Arm C adds a mandatory, executable gate: the executor must run the gate program before declaring completion, and the controller re-runs its own copy after the session and returns failures as resumed turns, at most three times within the envelope. Arm D adds a separate review-and-repair session after the gate passes, drawn from the same envelope. The appended texts are 6, 665, 827 and 921 words long (common preamble 279 words); the first assistant turn of the smoke runs consumed \ctxA, \ctxB, \ctxC{} and \ctxD{} input tokens respectively, the difference being the protocol text only. A length-matched unstructured control (general research advice, 533 words, no phases, ledgers or gates) is used in the ablation of Section~\ref{sec:recovery}.1314\subsection{Assignment, blocking and repeats}1516The design is paired: every confirmatory task is run under every arm with \nRepeats{} repeats, for \nMainRuns{} assigned runs. A seeded generator shuffles the order of task blocks and, within each block, the order of the (arm, repeat) cells; runs are dispatched to four executor nodes with two concurrent sessions per node, so that the arms of a task block execute interleaved in time and no arm is systematically earlier than another. Repeats are fresh sessions in fresh workspaces; the client exposes no seed or temperature control, so model stochasticity is not controlled and repeat outcomes are preserved as observed. The assignment manifest and every run's node, order and timestamps are stored before launch.1718\subsection{Endpoints and estimands}1920The primary endpoint for run $(i,a,r)$ is21\begin{equation}22Y_{iar} = \mathbf{1}\{\text{required criteria satisfied} \wedge \text{no critical violation} \wedge \text{completed within the envelope} \wedge \text{no human intervention}\},23\label{eq:Y}24\end{equation}25and the task-weighted completion rate of arm $a$ and the paired contrast are26\begin{equation}27\hat p_a = \frac{1}{J}\sum_{j=1}^{J}\frac{1}{R_j}\sum_{r=1}^{R_j} Y_{jar}, \qquad \hat\Delta_{ab} = \hat p_a - \hat p_b .28\label{eq:phat}29\end{equation}30The primary contrast is D$-$A; B$-$A, C$-$B and D$-$C are prespecified mechanism contrasts with Holm adjustment. Uncertainty uses a task-cluster bootstrap (5,000 resamples of tasks, percentile intervals of the paired difference) and a sign-flip permutation test on the task-level paired differences (20,000 permutations); with $J=\nConfTasks$ task clusters, intervals are wide, and we report them as such. Secondary endpoints are correct completion regardless of the envelope, quality components (correctness, integrity, reproducibility, reporting, baseline; frozen weights 0.40/0.20/0.15/0.15/0.10), critical violations by type (leakage, fabrication, invalid inference, unreported substitution, false completion, provenance), fresh-environment reproducibility, correct recognition of infeasible or null-truth tasks and false blocking on feasible ones, and cost, usage and wall time with full denominators. The analysis plan, the scale decision rule and the freeze procedure were written before the pilot (\texttt{protocol/preregistration.md}) and frozen with hashes before the confirmatory stage; every departure is an amendment in \texttt{protocol/amendments.csv}.31