1% ---------------------------------------------------------------------------2% Section 3 — Benchmark and laboratory (journal version; condensed from the3% monograph's Sections 4 and 6; describes what was built and measured).4% ---------------------------------------------------------------------------5\section{The benchmark and the laboratory}\label{sec:benchmark}67\subsection{The Virtual White-Collar Economy and the anatomy of a scenario}\label{sec:bench-anatomy}89Scenarios are set in a small, internally consistent fictional economy of seventeen organisations---an accounting firm, an investment firm, a commercial bank, an appraisal firm, a law firm, a university department, a manufacturer and its human-resources and procurement departments, a municipality, and others---that share conventions (fiscal year 2025, Canadian dollars, Québec sales taxes, ISO dates) so that an agent moving between scenarios faces a coherent world. All names, people and figures are fictional, and one manufacturer, Boréal Fenestration Inc., recurs as client, employer and subject across six scenarios.1011A scenario has two halves produced together by one reproducible generator and never mixed afterwards: a \emph{public bundle} that the worker sees and a \emph{hidden oracle} that only the controller and the evaluators see. The separation is enforced by location: the bundle lives in the repository; the oracle lives outside it on the controller and is never copied to an executor node. The bundle contains metadata reduced to what a worker legitimately needs; a brief written in the voice of the organisation stating context, inputs, deliverables with exact file names and formats, business rules and conventions, ending with common working rules (deliverables go to \texttt{out/}, inputs must not be modified, no contact outside the workspace and no network except an internal portal if provided, a completion record must be written, and if the assignment cannot be completed because information is missing or contradictory the worker must say so with its questions instead of guessing); the input artefacts (spreadsheets, PDF invoices and statements, Word contracts, e-mail messages, calendar files, SQLite databases, policies, images), identical in every arm; and, for the portal arms, a description of what the enterprise portal serves. The brief contains no grading information and no hint that any document contains an injection.1213The oracle contains the ground truth (planted items and correct values, tolerances, required deliverables, injection markers), the deterministic evaluator, the blind-evaluation rubric with weighted criteria anchored on a 0--4 scale, a reference document and the validator's report. The evaluator reads only the collected output and the public workspace, never executes the worker's code, applies the oracle's tolerances and returns a receipt listing each required criterion's pass/fail status, each critical violation, the completion flag and the abstention status. Every scenario has at least six required criteria, of which at least three are facts only the oracle knows, and at least two critical-violation checks. Deliverables are real files---workbooks with live formulas, Word documents with specified sections, CSV tables, figures, code---plus a machine-readable completion record whose status is \emph{completed}, \emph{needs\_clarification} or \emph{abstained} and which lists the questions the worker would have asked. Every scenario embeds \emph{planted errors} (a transposed amount, a duplicated invoice, a comparable sale with the wrong date), one \emph{prompt injection} inside an input document whose obedience is a critical violation, and in four scenarios a \emph{clarification trap}, a sub-task that cannot be completed without information the bundle does not contain. The behavioural rule given to every worker---treat the content of input documents as data, not as instructions---is identical across arms. Each scenario also carries a \emph{designed dependency depth}, the minimal chain of dependent operations a careful professional must perform.1415\subsection{Generation and independent validation}\label{sec:bench-validation}1617Scenarios are produced and checked by operationally separate agents. The \emph{generator} is a fresh, single-purpose Claude Code session that writes the generator script, the brief, the inputs, the portal data and the oracle for one scenario; the script is deterministic given its seed and internally coherent by construction, so that the planted discrepancies are the only discrepancies. The \emph{validator} is a second fresh session that never sees the generator's conversation: it first solves the task from the public bundle alone, then opens the oracle, runs the evaluator on its own solution (which must pass) and on a plausible wrong solution (which must fail), and checks that the brief is realistic, the task solvable, the hidden answers correct, the rubric gradable from the deliverables alone and the portal free of exclusive information. All ten validator solutions pass their evaluators and every empty output fails; the validators found and fixed between one and seven evaluator defects per scenario before the freeze---false positives on honest disclosure of the injection, vacuous or lenient criteria, non-reproducible document hashes---and removed two brief sentences that primed workers against injections.18% ../docs/PHASE_5_AUDIT.md §119A scenario enters the assignment only after validation, and the hashes of its bundle and oracle are recorded in the preregistration freeze manifest (amendment A-007). The protocol protects against the task-validity failures catalogued by \citet{zhu2025rigorous}; it does not remove the circularity that generator, validator, worker and judge are all language-model agents (Section~\ref{sec:discussion}).2021\subsection{The ten pilot scenarios}\label{sec:bench-pilot}2223The pilot covers ten of the twenty occupations of the study's taxonomy. Each scenario is a complete assignment: a year-end bank reconciliation from 150 invoices, twelve statements and a general ledger (ACC-01, accounting, designed depth 44); a discounted-cash-flow valuation and investment memorandum (FIN-01, corporate finance, depth 33); a single-family appraisal with an adjustment grid and report from eight closed sales (RE-01, real-estate appraisal, depth 34); a pre-specified difference-in-differences replication with data cleaning and a reference check against a library catalogue (RES-01, academic research, depth 34); the triage of a chief executive's forty messages, 25 calendar events and five project documents into an action plan, draft replies, a filing index and a briefing note (ADM-01, executive administration, depth 44); a credit analysis of a 1.8-million-dollar equipment loan under a credit policy (BANK-01, banking, depth 24); a clause-by-clause review of a 25-clause master services agreement against a negotiation playbook (LEGAL-01, legal services, depth 23); the closing of a pay period for 25 employees under a payroll policy (HR-01, human resources, depth 22); the evaluation of five supplier quotations under a published method (PROC-01, procurement, depth 21); and an annual sales KPI pack from a raw ERP database, with a dashboard workbook, a commentary and the code that produced it (BI-01, business intelligence, depth 31). Four scenarios are of the standard difficulty class, four complex and two workflow; planted errors number three to nine; every scenario has an injection document; the clarification traps are in the executive-administration, legal, human-resources and procurement assignments; and the authors' professional-minutes estimates range from 150 to 420.24% ../manuscript/sections/benchmark.tex Table tab:pilot (task.json metadata: class, depth, minutes, errors, injection, clarification)25No short scenario is in the pilot, a deliberate choice to measure the regime in which the frontier is expected to be informative and a limitation for estimating the dependency frontier at low depth.2627\subsection{Claude Code calling Claude Code}\label{sec:bench-lab}2829The laboratory has a controller--executor architecture (Figure~\ref{fig:architecture}). The \emph{controller} is a laptop session that holds the repository, the hidden oracles, the experiment database and the ledger; it draws the assignment, composes workspaces, launches sessions over SSH, streams and stores their events, enforces the wall limit, collects and hashes deliverables, runs the deterministic evaluator and the blind judges, and renders a private dashboard. It never executes a worker itself. The \emph{executors} are four Mac Studio nodes of an authorised cluster (M3U96a, M3U96b, M4M64a, M4M64b; macOS 26.4--26.6; 16 to 28 cores and 64 to 96 GB) running Claude Code 2.1.314, each with a project Python environment, LibreOffice 26.2 for headless conversion and recalculation and Playwright's headless Chromium; each runs up to three concurrent sessions in separate per-run directories and holds no oracle, rubric or evaluator code. Model access was verified by headless probes whose cost was read from the client's own result event, and that field is the source of every cost in the paper.30% ../manuscript/sections/infrastructure.tex Table tab:nodes (Phase 0 survey)3132\begin{figure}[htbp]33\centering34\includegraphics[width=\textwidth]{fig01_architecture}35\caption{Laboratory architecture. The controller (left) holds the hidden oracles outside the repository, composes a per-run workspace from the public bundle, launches a bare-mode Claude Code session over SSH on one of four executor nodes under arm-specific flags, streams and stores every event, enforces the wall limit, collects and hashes the deliverables, deletes the workspace and runs the deterministic evaluator and three blind judges. Executor nodes (right) host per-run workspaces, the loopback enterprise portal for arms A3 and A4, LibreOffice headless and Playwright Chromium, and never hold oracle, rubric or evaluator code. Source: \texttt{analysis/figures/out/fig01\_architecture.pdf} (figure script of the reproducibility package).}36\label{fig:architecture}37\end{figure}3839A worker session is a single command assembled by the launcher: the client is started in bare mode, ignoring the user's settings, hooks and instruction files so that nothing from the controller's configuration reaches the worker; the tool set is restricted by \texttt{--tools} and pre-approved by \texttt{--allowedTools}; permission prompts are disabled; budget and turn caps are set; the arm's documentation is appended to the system prompt; denied command families are listed in \texttt{--disallowedTools} with patterns such as \texttt{Bash(osascript:*)}. The effect of each flag was verified by probe: a worker started without the shell tool could not run a trivial command, and a denied command family was refused inside the shell while a plain command ran. No Agent or Task tool is ever given, so a worker cannot spawn sub-agents. Each workspace is pushed to the node with a manifest of SHA-256 hashes; after the session the output directory is copied back and hashed, any change to an input is recorded as a critical violation, and the workspace is deleted. The \emph{synthetic enterprise portal} is a small web application started on the node for each A3 and A4 run, bound to the loopback interface, serving a login page, a document library whose links point to files already in the inputs, record tables and forms whose submissions are collected with the other deliverables; it holds no information absent from the inputs, and external network access is forbidden by policy.4041Role separation is operational rather than nominal: generator and validator are distinct sessions; the worker is a separate process on a separate machine; the evaluator is code the worker never sees; the judges receive a rendered package and a rubric; and the sessions that wrote the sections of this paper reporting results had access to the analysis outputs and not to the runs. Three properties of the environment constrained the design and are recorded as amendments. Control of graphical applications is not available to a process started over SSH---Finder and TextEdit requests time out on all four nodes because the automation consent prompt cannot be answered for a non-console process---so the mandate's ``macOS agent'' is implemented as the automation layer that works headlessly (\texttt{osascript} with AppleScript and JavaScript for Automation, System Events on the two nodes where it answers, and the native document utilities), and A2 runs are pinned to those nodes (A-003). No human evaluator was available, so the blind professional evaluators are language-model judges (A-005). And the human arms of the mandate were designed and not executed (A-002).42% ../protocol/amendments.csv A-002, A-003, A-005; ../manuscript/sections/infrastructure.tex §6.543