1\section{Evaluation}2\label{sec:evaluation}34\subsection{Protected oracles and deterministic scoring}56After a run ends, the controller collects the deliverables, runs the common contract check (the gate program, post hoc for all arms, which also yields the fresh re-execution reproducibility measure), and then invokes the task's evaluator on a composed workspace that contains the public bundle and the collected deliverables and nothing else. Evaluators are deterministic Python programs; none calls a language model. Each returns a receipt listing every required criterion with the observed value, the reference and the tolerance, every critical violation with its evidence, and the five quality components; \texttt{scientific\_success} is true only if every criterion passes and no violation is present. Receipts are validated against a public JSON schema, hashed and stored. Tolerances were set by the task designers from statistical or numerical reasoning (sampling variability of the reference quantity, differences between valid conventions such as CR0/CR1 cluster adjustments, $t$ versus normal critical values, Newey--West bandwidths, binomial Monte Carlo error) so that valid implementations pass and wrong science fails; each justification is recorded in the task's hidden \texttt{REFERENCE.md}.78Critical violations are detected from deliverables only. Fabrication is flagged when reported input counts do not match the files, when a reported value cannot be regenerated from reported components (a residual, a coverage count, a Kolmogorov--Smirnov statistic), or when a value said to be printed in a source does not appear there. Leakage is flagged in forecasting tasks when the executor's out-of-sample errors fall below an information-feasible bound or match a leaky reference implementation, and in the F1 task with a final-vintage variable when that variable enters the model. Invalid inference is flagged when a conclusion contradicts the reported interval, when a significance claim rests on a standard error that ignores the stated dependence, or when a claim in the ledger asserts a universal statement on the basis of simulation-only keys. Unreported substitution is flagged when a cleaned aggregate is reported without the corresponding exclusion-ledger step or data-issue entry. False completion is flagged when a feasible task is declared partial or infeasible, when an infeasible one is declared complete, or when a not-assessable item is marked supported. These rules are task-specific, deterministic and documented; they cannot see intent, and we report them as what they are.910\subsection{Evaluator validation}1112Every oracle ships a validation set: one or more independently written correct solutions whose replication script is actually re-run, and two to six wrong solutions embodying the failure the task is designed to detect (a leaky model, a naive estimator, a fabricated count, a false declaration of completion, a silent substitution). Before the pilot, all \nOracles{} oracles accepted their correct artefacts and rejected their wrong ones (Appendix~\ref{app:evaluators}). The seeded sets are small and designer-authored: they bound gross evaluator errors, not the space of valid solutions, and the pilot was used to look for false rejections of legitimate variation.1314\subsection{Blinding and model-based grading}1516Evaluators are programs and see no arm label; run identifiers are neutral. Artefacts can reveal the arm to a reader (a phase ledger exists only in arms B--D), which is irrelevant to deterministic scoring but would matter for any human or model-based review; we therefore do not describe the evaluation as blinded. No model-based grading enters the primary or secondary endpoints. The reviewer in arm D is part of the treatment, not of the evaluation.1718\subsection{Human involvement}1920No human reviewer was available for the benchmark; cells that would require one are marked unavailable. No human intervened in any executor session. The human contributions to this study are the mandate itself (prepared outside the session; preparation time unknown and reported as unknown), the decision to run it, and the final review of the manuscript; they are logged in \texttt{reports/interventions.csv} with their measurement basis. The builder session's own activity (harness construction, task-suite delegation, analysis and writing) is recorded in the phase log and in the reflexive analysis of Section~\ref{sec:reflexive}; it is a cost of the method, not an intervention in the experiment.21