% --------------------------------------------------------------------------- % Appendix E — Reproducibility package (Agent H). % Sources: README.md, docs/OPERATIONS.md, scripts/{freeze,verify_freeze}.py, % protocol/preregistration/freeze_manifest.json (counts only), % protocol/amendments.csv (all rows), src/wp42/{metrics,export,scheduler}.py, % analysis/{econometrics,figures,robustness}/*.py, benchmark/taxonomy/registry.csv. % --------------------------------------------------------------------------- \section{Reproducibility package}\label{app:repro} This appendix describes the package from which the study can be re-run and the manuscript's exhibits regenerated: the layout of the repository, the commands that rebuild scenarios and oracles, run a stage, export the data and produce every table and figure, the freeze procedure and the contents of the manifest, the complete amendment log, the data dictionary of the validated run table, what the public package deliberately omits and how a replicator obtains it, and the known limitations of the harness recorded in the amendment log. \subsection{Repository layout}\label{app:repro-layout} The repository follows the structure prescribed by the mandate. Table~\ref{tab:app-layout} lists the directories and what each holds; the hidden oracles and the secrets live outside it. \begin{table}[htbp] \singlespacing \centering \caption{Repository layout} \label{tab:app-layout} \footnotesize \begin{tabularx}{\textwidth}{@{}l >{\raggedright\arraybackslash}X@{}} \toprule Path & Content \\ \midrule \texttt{CLAUDE.md} & the research mandate, verbatim \\ \texttt{README.md}, \path{RESEARCH_PLAN.md}, \path{PROJECT_STATUS.md} & operational pointers, design plan, phase status \\ \path{protocol/preregistration/} & \path{PREREGISTRATION.md} (v0.9 drafted before any pilot run, frozen as v1.0) and \path{freeze_manifest.json} \\ \path{protocol/amendments.csv} & the amendment log (Table~\ref{tab:app-amendments}) \\ \path{protocol/ethics/} & protocol and consent template for the human arms, designed but not executed \\ \path{protocol/power_analysis/} & \path{power_simulation.py} (Table~\ref{tab:power}) \\ \texttt{literature/} & \path{references.bib}, the structured review and the verification log \\ \texttt{benchmark/} & \path{SCENARIO_AUTHORING_SPEC.md}, \path{BRIEF_FOOTER.md}, \path{COMPLETION_SCHEMA.json}, \texttt{taxonomy/} (occupations, organisations, \texttt{registry.csv}), \path{scenarios//} (public bundles with \texttt{generate.py}), \texttt{validators/} (\texttt{common.py}, \path{receipt_schema.json}), \path{tools/selfcheck.py} \\ \texttt{src/wp42/} & the controller: \texttt{config}, \texttt{contracts}, \texttt{store}, \texttt{events}, \texttt{launcher}, \texttt{arms}, \texttt{bundle}, \texttt{portal}, \texttt{office}, \texttt{scheduler}, \texttt{evaluate}, \texttt{judge}, \texttt{metrics}, \texttt{export}, \texttt{dashboard}, \texttt{watchdog}, \texttt{cli} \\ \texttt{scripts/} & \texttt{freeze.py}, \path{verify_freeze.py}, \path{build_registry.py} \\ \texttt{data/raw/} & \texttt{wp42.sqlite} and \path{runs//} (git-ignored; Appendix~\ref{app:infra-artefacts}) \\ \path{data/validated/} & \path{runs_}, \path{judgments_}, \path{receipts_}, \path{events_} as Parquet (and CSV) \\ \path{analysis/econometrics/} & \texttt{estimate.py} (preregistered estimands, tables 1--8, \path{derived/estimates_.json}) \\ \path{analysis/robustness/} & \path{sensitivity_models.py}, \path{recovery_experiment.py}, \path{economic_sensitivity.py}, \path{audit_pilot.py} \\ \path{analysis/figures/} & \path{fig01_architecture.py}, \path{make_figures.py}, \texttt{out/} (PDF and PNG) \\ \path{analysis/tables/}, \path{analysis/derived/} & generated CSV tables and JSON estimates \\ \texttt{manuscript/} & \texttt{main.tex}, \texttt{sections/}, \texttt{appendices/}, \texttt{figures/}, \texttt{tables/} \\ \path{journal_version/} & the journal-length version \\ \texttt{dashboard/} & the private monitoring page \\ \texttt{tests/} & \path{test_core.py} (seven harness unit tests) and fixtures \\ \texttt{docs/} & \path{PHASE_0_DISCOVERY.md}, \path{OPERATIONS.md} \\ \midrule \path{~/wp42-oracles//} & \emph{outside the repository}: \texttt{oracle.json}, \texttt{evaluate.py}, \texttt{rubric.json}, \texttt{REFERENCE.md}, \texttt{validation/} \\ \path{~/Desktop/Cluster/secrets/wp42/.env} & \emph{outside the repository}: the API key (mode 600) \\ \bottomrule \end{tabularx} \end{table} \subsection{Regenerating scenarios and oracles}\label{app:repro-generate} Each public bundle is rebuilt by its generator from its seed; the same command rebuilds the oracle's ground truth and reference document into the directory given by \texttt{--oracle-root}, and the bundle can be rebuilt with no oracle present: \begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt] cd benchmark/scenarios/ python generate.py --seed --oracle-root ~/wp42-oracles \end{Verbatim} The seeds are recorded in \path{benchmark/taxonomy/registry.csv} and in each \texttt{task.json}: ACC-01 4201, FIN-01 4202, RE-01 4203, RES-01 4204, ADM-01 4205, BANK-01 4206, LEGAL-01 4207, HR-01 4208, PROC-01 4209, BI-01 4210. The generators draw from \path{numpy.random.default_rng(seed)} and write the oracle only to the oracle root. Two caveats, both documented by the validators (Appendix~\ref{app:scenarios}): the generator writes \texttt{oracle.json} and \texttt{REFERENCE.md} but \emph{not} \texttt{evaluate.py} or \texttt{rubric.json}, which were written by the generator agent and corrected by the validators and exist only in the oracle directory; and one bundle (FIN-01) contains a Word document whose zip member timestamps vary between regenerations, so that its hash does not reproduce although its content does. The freeze manifest, not regeneration, is therefore the reference for what was run. \subsection{Running a stage}\label{app:repro-run} With the controller prerequisites of Appendix~\ref{app:infra-nodes} in place and \texttt{PYTHONPATH=src}, a stage is run by the following sequence (from \path{docs/OPERATIONS.md}): \begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt] .venv/bin/python -m pytest -q tests # harness unit tests .venv/bin/python scripts/freeze.py --tasks ACC-01 ADM-01 BANK-01 BI-01 FIN-01 HR-01 LEGAL-01 PROC-01 RE-01 RES-01 .venv/bin/python scripts/verify_freeze.py # exit 1 and list if anything changed .venv/bin/python -m wp42.cli sync-tools # portal package -> nodes .venv/bin/python -m wp42.cli assign --tasks --arms A0 A1 A2 A3 A4 --reps 2 --seed 4242 \ --model claude-opus-5-5 --stage pilot .venv/bin/python -m wp42.cli run --where "stage='pilot' AND status='assigned'" --parallel 12 .venv/bin/python -m wp42.cli status | ledger .venv/bin/python -m wp42.cli evaluate --where "status='collected'" # (re)evaluate + judge .venv/bin/python -m wp42.cli reset-run --run-ids # infra error -> rerun (log an amendment) .venv/bin/python -m wp42.cli kill-all # emergency shutdown on every node .venv/bin/python -m wp42.cli export --tag pilot # data/validated/*.parquet|csv .venv/bin/python -m wp42.dashboard --serve 8642 # private dashboard \end{Verbatim} \texttt{assign} draws the seeded permutation of the (scenario, arm, repeat) cells and inserts one \texttt{runs} row per cell with its order index; it is idempotent for existing identifiers, so a third repeat is added by re-running it with \texttt{--reps 3} and a new seed (amendment A-009 used 4243), and the sensitivity stage by \texttt{--stage sensitivity --model --model-tag --arms A3}. The recovery experiment of amendment A-011 uses \texttt{--stage recovery --interrupt-s 120 --seed 4246}. \texttt{run} dispatches in order of the permutation to the least-loaded eligible node with a free slot, up to \texttt{--parallel} sessions in total, stops when the ledger reaches the envelope (\texttt{--envelope} overrides the configured 800~USD), and treats an interrupt signal as a graceful stop. An infrastructure error leaves the run in \path{infra_error}; \texttt{reset-run} returns it to \texttt{assigned} for a rerun under the same identifier, which must be logged in the amendment file (Section~\ref{sec:design-exclusions}). \subsection{Reproducing every table and figure}\label{app:repro-exhibits} All exhibits are produced by scripts from the validated Parquet tables; none is edited by hand. Table~\ref{tab:app-exhibits} maps each script to its outputs. The tag selects the stage (\texttt{pilot}, \texttt{sensitivity}, \texttt{recovery}; the \texttt{smoke} tag is the pre-pilot dry run and is not reported). \begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt] .venv/bin/python analysis/econometrics/estimate.py --tag pilot [--q 70 --B 5000 --perm 20000 --seed 42] .venv/bin/python analysis/figures/make_figures.py --tag pilot .venv/bin/python analysis/figures/fig01_architecture.py .venv/bin/python analysis/robustness/sensitivity_models.py .venv/bin/python analysis/robustness/recovery_experiment.py .venv/bin/python analysis/robustness/economic_sensitivity.py .venv/bin/python analysis/robustness/audit_pilot.py --tag pilot .venv/bin/python protocol/power_analysis/power_simulation.py cd manuscript && latexmk -pdf main.tex \end{Verbatim} \begin{table}[htbp] \singlespacing \centering \caption{Scripts and the exhibits they produce} \label{tab:app-exhibits} \footnotesize \begin{tabularx}{\textwidth}{@{}l >{\raggedright\arraybackslash}X@{}} \toprule Script & Outputs (\path{analysis/tables/}, \path{analysis/figures/out/}, \path{analysis/derived/}) \\ \midrule \path{econometrics/estimate.py} & \path{table1_outcomes_by_arm_.csv}, \path{table2_contrasts_.csv}, \path{table3_occupation_arm_.csv}, \path{table4_dependency_.csv}, \path{table5_alf_.csv}, \path{table6_q_sensitivity_.csv}, \path{table7_judges_.csv}, \path{table8_det_vs_judges_.csv}, \path{derived/estimates_.json} \\ \path{figures/make_figures.py} & \path{fig03_acceptance_by_arm}, \path{fig04_alf_curves}, \path{fig05_cost_quality}, \path{fig06_dependency}, \path{fig07_tool_effects}, \path{fig08_failure_severity}, \path{fig09_recovery}, \path{fig11_time_cdf}, \path{fig12_judge_agreement}, \path{fig14_occupation_heterogeneity} (each \path{_.pdf} and \texttt{.png}) \\ \path{figures/fig01_architecture.py} & \path{fig01_architecture.pdf/.png} (Figure~\ref{fig:architecture}) \\ \path{robustness/sensitivity_models.py} & \path{table9_model_sensitivity.csv}, \path{table9b_model_contrasts.csv}, \path{table9c_model_by_scenario.csv} \\ \path{robustness/recovery_experiment.py} & \path{table10_recovery.csv}, \path{derived/recovery_experiment.json} \\ \path{robustness/economic_sensitivity.py} & \path{table11_economic_sensitivity_pilot.csv}, \path{table12_supervision_pilot.csv}, \path{table13_human_cost_assumption.csv}, \path{fig10_supervision_pilot}, \path{fig13_model_sensitivity}, \path{fig15_economic_sensitivity_pilot} \\ \path{robustness/audit_pilot.py} & \path{audit_a0_outputs}, \path{audit_denied}, \path{audit_det_vs_judge}, \path{audit_failures}, \path{audit_families}, \path{audit_portal}, \path{audit_violations} (each \path{_.csv}) \\ \path{protocol/power_analysis/power_simulation.py} & \path{power_pilot.csv} (Table~\ref{tab:power}) \\ \bottomrule \end{tabularx} \begin{tablenotes} \footnotesize \item \texttt{estimate.py} implements the preregistered inference plan (scenario-cluster bootstrap with $B$ resamples, paired sign-flip permutation test, clustered and mixed-effects logistic sensitivities, Kaplan--Meier and Cox models, count models, the Autonomous Labor Frontier over the budget grid, and evaluator agreement) with the bootstrap and permutation seeds fixed on the command line. Figure numbers follow the mandate's list of proposed figures; figures 10, 13 and 15 are produced by \path{economic_sensitivity.py}, figure 1 by its own script, and figure 2 (benchmark coverage) has no script in the package at the time of writing, Table~\ref{tab:taxonomy} serving in its place. \end{tablenotes} \end{table} \subsection{The freeze procedure and the manifest}\label{app:repro-freeze} Before the first pilot run, \path{scripts/freeze.py} wrote \path{protocol/preregistration/freeze_manifest.json}: for each of the ten scenarios, the SHA-256 of every file of the public bundle (including \texttt{generate.py} and the portal description); for each oracle, the SHA-256 of its four files, the \texttt{validation/} directory excluded; the SHA-256 of the fourteen harness files that affect outcomes (\texttt{arms.py}, \texttt{launcher.py}, \texttt{bundle.py}, \texttt{scheduler.py}, \texttt{judge.py}, \texttt{evaluate.py}, \texttt{metrics.py}, \texttt{config.py}, \texttt{portal.py}, \texttt{office.py}, \texttt{selfcheck.py}, \texttt{common.py}, \path{BRIEF_FOOTER.md}, \path{COMPLETION_SCHEMA.json}); the SHA-256 of three protocol files (\path{PREREGISTRATION.md}, \path{RESEARCH_PLAN.md}, \path{analysis/econometrics/estimate.py}); the UTC time of the freeze; the git head of the repository; and the controller's Claude Code version string. The manifest of the pilot freeze (amendment A-007) contains 371 hashes: 314 public-bundle files (167 for ACC-01, 54 for ADM-01, 22 for RE-01, 13 each for HR-01 and PROC-01, 10 for RES-01, 9 each for BANK-01, BI-01 and FIN-01, 8 for LEGAL-01), 40 oracle files (four per scenario), 14 harness files and 3 protocol files; it records the controller client as version 2.1.316. \path{scripts/verify_freeze.py} recomputes every hash and prints each changed, added or removed path, exiting with status~1 if any differs. It is run before pooling any later runs with the pilot (Section~\ref{sec:design-scale}) and by the reproducibility auditor. At the time of writing, \path{verify_freeze} reports six files changed since the freeze, each tied to an amendment: \path{src/wp42/judge.py} (A-008, prompt sanitisation), \path{src/wp42/launcher.py} and \path{src/wp42/bundle.py} (A-010, collection retries and file-count check; A-012, removal of \texttt{--no-session-persistence}), \path{src/wp42/scheduler.py} (A-010 and A-012, the rerun and recovery logic), \path{src/wp42/metrics.py} and \path{analysis/econometrics/estimate.py} (A-013, status filter, cost rule for killed sessions and version capture). Every scenario bundle, every oracle, \path{arms.py}, \path{config.py}, \path{portal.py}, \path{office.py}, \path{selfcheck.py}, \path{common.py} and the preregistration are unchanged since the freeze, so the information, tools and evaluation rules seen by every run are those that were frozen. \subsection{The amendment log}\label{app:repro-amendments} Table~\ref{tab:app-amendments} reproduces every row of \path{protocol/amendments.csv} with the decision and the reason summarised. The phase column follows the mandate's phases (0 discovery, 2 engineering, 3 pilot benchmark, 4 pilot experiments, 5 scientific audit, 7 analysis). Timestamps in the log were re-stamped after the audit from database and git evidence; all stages ran on 2026-11-03 between 00:32 and 02:26 UTC. Amendments A-001 and A-002 carry the status \emph{pending PI confirmation}; all others are \emph{recorded}. \begin{footnotesize} \setlength{\tabcolsep}{3pt} \begin{singlespace} \begin{longtable}{@{}lcp{6.6cm}p{6.6cm}@{}} \caption{The amendment log (\texttt{protocol/amendments.csv}), decisions and reasons summarised}\label{tab:app-amendments}\\ \toprule Id & Ph. & Decision & Reason \\ \midrule \endfirsthead \toprule Id & Ph. & Decision & Reason \\ \midrule \endhead \midrule \multicolumn{4}{r}{\textit{continued on next page}}\\ \endfoot \bottomrule \endlastfoot A-001 & 0 & Study envelope 800 USD; per-session caps of 4.00 USD, 1,800 s and 150 turns. & The mandate gives no budget; the envelope of the preceding study in the series is reused; the PI may change it. \\ A-002 & 0 & Human arms H0 and H1 designed but not executed. & No participants and no ethics approval exist; the mandate forbids LLM imitations of humans. \\ A-003 & 0 & A2 is the headless macOS automation layer (osascript, System Events, native utilities) without GUI application control. & GUI Apple Events time out over SSH: the consent prompt cannot be answered on headless nodes. \\ A-004 & 0 & Primary worker model claude-opus-5-5; Sonnet 5.5 and Fable 5.1 as sensitivities on arm A3. & Cost: Fable about 2.5 times Opus per token in the probe; Opus was the subject of the preceding study. \\ A-005 & 0 & Blind professional evaluators are three LLM judges per run; human expert review left to the PI. & No human evaluators available in-session. \\ A-006 & 2 & Per-scenario caps are the larger of the global caps and the scenario's values: wall 3,600 s (ACC-01, ADM-01), 2,700 s (RES-01), 2,400 s (BANK-01, HR-01, LEGAL-01, RE-01); budget 6 USD (ACC-01, ADM-01, RES-01), 5 USD (BANK-01, HR-01, RE-01); A4 recovery gets the remainder. & Scenario authors sized the caps to the designed workload; set before any pilot run and before the freeze; list corrected after the audit. \\ A-007 & 3 & Freeze v1.0 of the ten validated scenarios, oracles and harness after the validator fixes (evaluators 1.1, HR-01 at 1.0); five bundles regenerated at the same seed. & Every validator solution passes its evaluator and empty outputs fail; no pilot run before the freeze except one development run on a pre-validation bundle, excluded. \\ A-008 & 4 & Judge prompt sanitisation (null and control bytes removed; 400,000-character cap) after eight pilot runs of one scenario crashed the judge subprocess; those runs judged after the fix. & Harness defect found during the pilot; outcome-neutral for runs whose rendered deliverables contain no null byte; the re-judged runs are flagged for a sensitivity analysis. \\ A-009 & 4 & Scale decision per the preregistration: a third repeat for all fifty cells (seed 4243) and the model sensitivities on arm A3, ten scenarios, two repeats (stage sensitivity). & The preregistered rule is satisfied by the pilot's mean cost per run; no scenario, oracle or arm change since the freeze, only \texttt{judge.py} (A-008) at that point. \\ A-010 & 5 & Collection defect: the copy of \texttt{out/} silently failed for three pilot runs whose sessions had produced deliverables; the runs are reset and rerun under the same identifiers after the fix (retries with back-off, file-count check, loud failure). & Likely SSH connection throttling under twelve concurrent sessions plus collection traffic; the original attempts are kept in the infrastructure-stability statistics. \\ A-011 & 5 & Exploratory addendum: an induced-interruption recovery experiment, ten scenarios, A3 and A4 interrupted at 120 s, two repeats (stage recovery, seed 4246). & The pilot's A4 arm never triggered a recovery, so the Autonomous Recovery Index is unidentified in the main design; labelled exploratory. \\ A-012 & 5 & Harness defect: \texttt{--no-session-persistence} prevented the A4 recovery resume from starting; fixed and verified; the A4 runs of the recovery stage reset and rerun, the A3 interrupted runs kept. & In the pilot no A4 first session failed the public self-check, so pilot A4 outcomes are unaffected, but A4 self-repair was de facto checkpointing plus self-check; stated in the manuscript. \\ A-013 & 7 & Post-freeze changes to \path{estimate.py} (keep collected and evaluated runs; cost of killed sessions: cumulative client total for resumed A4 runs, wall-time imputation for the A3 interrupted control) and \path{metrics.py} (client version from the init event only). & Analysis-code fixes found by the writers and the auditor; no pilot estimate changes (tables 1--8 identical before and after); only the exploratory recovery-stage costs are affected. \\ \end{longtable} \end{singlespace} \end{footnotesize} \subsection{Data dictionary of the validated run table}\label{app:repro-dictionary} \path{python -m wp42.cli export --tag } writes \path{data/validated/runs_.parquet} (and \texttt{.csv}) with one row per run, built from the metrics computed by \path{metrics.compute} at evaluation time, the completion-record and collection fields added by the scheduler, and three fields added by the exporter (\texttt{stage}, \texttt{seed}, \path{human_intervention_min}). Nested fields are serialised as JSON strings. Table~\ref{tab:app-dictionary} lists the 54 columns in the order of the file. Missing values are preserved as missing: a run killed before its result event has no cost, a run without judge output has no quality, a non-A4 run has no self-check flags. The export also writes \path{judgments_} (one row per judge per run: \path{run_id}, \path{judge_idx}, \path{judge_model}, \texttt{score}, \texttt{scores} as JSON with the per-criterion integers and the acceptability vote, \path{cost_usd}, \path{judged_utc}), \path{receipts_} (one row per run: \path{run_id}, \path{det_completion}, \path{det_score}, \path{evaluator_version}, \path{oracle_sha256}, \texttt{criteria} and \texttt{violations} as JSON lists) and \path{events_} (one row per normalised event: \path{run_id}, \texttt{seq}, \path{elapsed_s}, \path{event_type}, \path{tool_name}, \texttt{status}, \path{error_class}, and the first 300 characters of the summary). \begin{footnotesize} \setlength{\tabcolsep}{3pt} \begin{singlespace} \begin{longtable}{@{}p{3.9cm}lp{10.6cm}@{}} \caption{Data dictionary of \texttt{data/validated/runs\_.parquet}}\label{tab:app-dictionary}\\ \toprule Column & Type & Definition \\ \midrule \endfirsthead \toprule Column & Type & Definition \\ \midrule \endhead \midrule \multicolumn{3}{r}{\textit{continued on next page}}\\ \endfoot \bottomrule \endlastfoot \multicolumn{3}{@{}l}{\textit{Identification and assignment}}\\ \path{run_id} & str & Run identifier \path{__r[_]}; primary key. \\ \path{task_id} & str & Scenario identifier (e.g.\ \texttt{ACC-01}). \\ \texttt{occupation} & str & Occupation of the scenario, from \texttt{task.json}. \\ \texttt{arm} & str & Treatment arm, \texttt{A0} to \texttt{A4}. \\ \texttt{rep} & int & Repeat index within the (scenario, arm) cell, from 1. \\ \texttt{model} & str & Worker model identifier assigned to the run. \\ \texttt{node} & str & Executor node on which the session ran. \\ \texttt{stage} & str & Stage tag of the assignment (\texttt{pilot}, \texttt{sensitivity}, \texttt{recovery}, \texttt{dev}, \texttt{smoke}); exporter. \\ \texttt{seed} & int & Seed of the random permutation that ordered the stage's cells; exporter. \\ \path{interrupt_s} & float & Induced interruption of the first session in seconds (recovery experiment); missing otherwise. \\ \multicolumn{3}{@{}l}{\textit{Session termination and time}}\\ \texttt{status} & str & Run status at export (\texttt{evaluated}, \texttt{collected}, \path{infra_error}, \ldots). \\ \path{result_subtype} & str & Subtype of the client's result event (\texttt{success}, budget or turn cap subtypes, \path{error_during_execution}), or \texttt{timeout} when killed by the controller, or \path{no_result} when no result event was received. \\ \path{timed_out} & bool & True when the controller killed the session at the wall limit. \\ \path{wall_s} & float & Wall time of the first session, seconds, as measured by the controller. \\ \path{wall_total_s} & float & \path{wall_s} plus the recovery session's wall time when one was granted ($T$). \\ \path{tool_time_s} & float & Sum over tool calls of the interval between the tool-use event and the following tool-result event, seconds. \\ \path{first_tool_s} & float & Elapsed time of the first tool call, seconds; missing if no tool was called. \\ \path{num_turns} & int & Number of turns reported in the result event. \\ \path{n_events} & int & Number of normalised events stored for the run (both sessions). \\ \multicolumn{3}{@{}l}{\textit{Tool use}}\\ \path{tool_calls} & int & Number of tool-use blocks in assistant events. \\ \path{by_tool} & str (JSON) & Object mapping tool name to its number of calls. \\ \path{by_family} & str (JSON) & Object mapping tool family (\texttt{file}, \texttt{shell}, \texttt{python}, \texttt{sql}, \texttt{office}, \texttt{applescript}, \texttt{browser}) to its number of calls. \\ \path{family_sequence} & str (JSON) & List of tool families in order of use, consecutive repeats collapsed. \\ \path{app_transitions} & int & Length of \path{family_sequence} minus one, floored at zero. \\ \path{tool_errors} & int & Number of tool results flagged as errors by the client. \\ \texttt{retries} & int & Number of tool calls that re-invoked the tool whose previous call had just errored. \\ \texttt{tokens} & str (JSON) & Object with summed \texttt{input}, \texttt{output}, \path{cache_read} and \path{cache_create} tokens over assistant events. \\ \path{model_usage} & str (JSON) & Per-model usage object from the result event. \\ \path{claude_code_version} & str & Client version reported in the system event. \\ \path{model_reported} & str & Model identifier reported in the system event. \\ \multicolumn{3}{@{}l}{\textit{Cost}}\\ \path{cost_usd} & float & \path{total_cost_usd} of the first session's result event; missing if none was received. \\ \path{recovery_cost_usd} & float & Incremental cost of the recovery session; missing when no recovery. \\ \path{total_cost_usd} & float & \path{cost_usd} plus \path{recovery_cost_usd}, missing treated as zero in the sum ($C$). \\ \multicolumn{3}{@{}l}{\textit{Self-repair (arm A4)}}\\ \path{recovery_attempted} & bool & True when a recovery session was launched. \\ \path{recovery_success} & bool & Final public self-check passed after recovery; missing when no recovery. \\ \path{selfcheck_pass_first} & float (0/1) & Controller's self-check on the first session's output; missing outside A4. \\ \path{selfcheck_pass_final} & float (0/1) & Self-check on the final output (after recovery if any); missing outside A4. \\ \multicolumn{3}{@{}l}{\textit{Deterministic evaluation}}\\ \path{det_completion} & bool & Every required criterion passed and no critical violation ($D$). \\ \path{det_score} & float & Weighted share of passed required criteria, 0--1. \\ \path{n_critical_violations} & int & Number of critical violations in the receipt, including the harness integrity check. \\ \path{critical_error} & bool & \path{n_critical_violations} $>0$ ($E$). \\ \path{abstention_status} & str & Status read from the completion record by the evaluator (\texttt{completed}, \path{needs_clarification}, \texttt{abstained}); missing if unreadable. \\ \path{abstention_justified} & bool & True when the status is a clarification or abstention and the oracle says the scenario requires one; false when it does not; missing when not applicable. \\ \multicolumn{3}{@{}l}{\textit{Blind evaluation}}\\ \texttt{quality} & float & Mean of the available judges' rubric-weighted scores, 0--100 ($Q$); missing if no judge output. \\ \path{quality_scores} & str (JSON) & List of the available judges' scores in judge order. \\ \path{judge_acceptable_votes} & str (JSON) & List of the judges' professional-acceptability votes (true/false/null) in judge order. \\ \path{n_judges} & int & Number of judges with a score. \\ \multicolumn{3}{@{}l}{\textit{Human time, completion record and artefacts}}\\ \path{human_intervention_min} & float & Sum of intervention minutes recorded against the run; zero for agent runs ($H$ before the question allowance); exporter. \\ \path{completion_status} & str & \texttt{status} field of the collected \path{COMPLETION.json}; missing if absent or invalid. \\ \path{n_questions} & int & Length of the \texttt{questions} list of the completion record; missing if the record is unreadable. \\ \path{n_issues_found} & int & Length of the \path{issues_found} list. \\ \path{n_assumptions} & int & Length of the \texttt{assumptions} list. \\ \path{n_artifacts} & int & Number of files collected from \texttt{out/}. \\ \path{inputs_changed} & bool & True when any input file's hash differed from the workspace manifest after the session. \\ \end{longtable} \end{singlespace} \end{footnotesize} The primary endpoint $S=D\,(1-E)\,\mathbf{1}\{Q\ge q\}$ and the scenario weights are computed by \texttt{estimate.py} from these columns at analysis time and are not stored in the table; the designed dependency depth and the professional-minutes estimate are joined from the scenarios' \texttt{task.json}. \subsection{What the public package does not contain, and how to obtain it}\label{app:repro-hidden} The public package deliberately omits four things. First, the hidden oracles: \texttt{oracle.json}, \texttt{evaluate.py}, \texttt{rubric.json}, \texttt{REFERENCE.md} and the \texttt{validation/} directory of each scenario. They are withheld because the benchmark is meant to be reused on future models and the mandate forbids exposing hidden answers; their hashes are in the freeze manifest, so a copy obtained from the principal investigator can be verified. A replicator who cannot obtain them can regenerate \texttt{oracle.json} and \texttt{REFERENCE.md} from the public \texttt{generate.py} with the recorded seed (Appendix~\ref{app:repro-generate}); the current generators already contain the validators' corrections to the oracle (ADM-01 keyword list, FIN-01 median and tolerance, HR-01 keywords and leave count, LEGAL-01 patterns, PROC-01 lead-time fields and tolerance), so a regenerated oracle matches the frozen one except for the FIN-01 input hash noted above. \texttt{evaluate.py} and \texttt{rubric.json} are \emph{not} regenerated by any script: they must be obtained from the restricted package, together with the validators' before-and-after copies and diffs that document the version 1.0 to 1.1 changes summarised in Appendix~\ref{app:scenarios}. Second, the API key, held outside the repository on the controller and on each node with restricted permissions and never logged; a replicator supplies their own. Third, the raw per-run directories (\texttt{data/raw/}), which contain the full event streams with the model's text and tool inputs, the collected deliverables and the raw judge outputs; they are git-ignored, archived by the principal investigator, and available on request for audit, while the normalised tables (\path{events_} with truncated summaries, \path{receipts_}, \path{judgments_}, \path{runs_}) are public. Fourth, the private dashboard's rendered pages, which are regenerated from the database. \subsection{Known limitations of the harness}\label{app:repro-limitations} The amendment log records three defects of the harness discovered during or after the pilot, each fixed and each with a stated consequence for the data. (i)~\emph{Judge crash on control bytes} (A-008): rendered deliverables containing a null byte crashed the judge subprocess; sanitisation was added and the affected runs, all of one scenario, were judged after the fix. The change is outcome-neutral for every run whose rendered deliverables contain no null byte; the re-judged runs are flagged for a sensitivity analysis. (ii)~\emph{Silent collection failure} (A-010): under twelve concurrent sessions the copy of \texttt{out/} from a node could fail without error, so that a session which had produced its deliverables was evaluated on an empty directory. Three pilot runs were affected; they were reset and rerun under the same identifiers after retries, a file-count check and a loud failure mode were added, and their first attempts are kept in the infrastructure-stability statistics. The exclusion rule of Section~\ref{sec:design-exclusions} covers this case. (iii)~\emph{Recovery resume never started} (A-012): the launcher passed \texttt{--no-session-persistence}, so the A4 recovery session, which resumes the first session's identifier, could not start and ended immediately with an execution error. The flag was removed and the fix verified with an interrupted run that resumed and passed the self-check; the A4 runs of the recovery stage were reset and rerun. Because no A4 first session of the pilot failed the public self-check, no recovery was ever requested in the pilot and its A4 outcomes are unaffected, but the A4 arm of the pilot is, de facto, A3 plus checkpointing and self-check rather than the bounded self-repair protocol as designed; the Autonomous Recovery Index is therefore estimated only from the exploratory recovery experiment of amendment A-011. A fourth, post-freeze change (A-013) concerns the analysis code rather than the harness: the loader of \path{estimate.py} now keeps runs in the \texttt{collected} and \texttt{evaluated} states, the cost of killed sessions follows a two-case rule (the cumulative client total for resumed A4 runs, a wall-time imputation fitted on completed pilot runs for the A3 interrupted control), and \path{metrics.py} reads the client version from the init event only; the log records that the pilot tables are identical before and after and that only the exploratory recovery-stage costs are affected. Two further limitations are structural rather than defects and are recorded in Sections~\ref{sec:infrastructure-macos} and~\ref{sec:design-human}: GUI application control is not available to a headless session on the executor nodes, so arm A2 is narrower than the mandate's description (A-003), and the human arms were designed but not executed (A-002).