1% ---------------------------------------------------------------------------2% Section 9 — Reliability, cost and recovery (Agent H; every number from3% analysis/tables, analysis/derived or docs/PHASE_5_AUDIT.md; sources in comments)4% ---------------------------------------------------------------------------5\section{Reliability, cost and recovery}\label{sec:reliability}67This section reports the dimensions of the Professional Reliability Frontier of Section~\ref{sec:theory-prf} that the acceptance rates of Section~\ref{sec:results} do not show: how the runs that were not accepted failed, and how severely; how long the runs took and whether any cap bound; what the self-repair mechanism of arm A4 does when it is actually triggered, which the exploratory recovery experiment of amendment A-011 measures; the trade-off between cost and quality, the cost per accepted deliverable and its sensitivity to prices, wages and overheads; the supervision that the completion records imply; the enforcement of the tool policy; and the stability of the laboratory itself.89\subsection{Failure severity}\label{sec:reliability-failures}1011Figure~\ref{fig:failures} decomposes the runs of each arm into the four terminal states of Definition~\ref{def:prf}---accepted, deterministic pass below the quality gate, deterministic failure without violation, critical violation---plus the censored state, which is empty since no pilot run reached a cap. The restricted arm's 30 runs are 24 deterministic failures without violation and 6 critical violations; the shell arms' 120 runs are 108 acceptances, 11 deterministic passes below the gate, 1 critical violation (A1), and no other deterministic failure except the two RE-01 runs of A2 and A4 that failed the adjusted-prices criterion.12% source: analysis/tables/audit_failures_pilot.csv; audit_violations_pilot.csv; figure fig081314\begin{figure}[htbp]15\centering16\includegraphics[width=0.9\textwidth]{fig08_failure_severity_pilot}17\caption{Terminal state of the 150 pilot runs by arm (30 runs per arm): accepted; deterministic pass with quality below $q=70$; deterministic failure without critical violation; critical violation; and timeout or missing result, which is empty because no pilot run reached a cap. Shares are proportions of 30 runs. Source: \texttt{data/validated/runs\_pilot.parquet}; script \texttt{analysis/figures/make\_figures.py} (\texttt{fig\_failures}).}18\label{fig:failures}19\end{figure}2021Hypothesis H5 stated that the rate of critical violations would be below 10 per cent in every arm and would not increase with tool access. The second part holds: the rate is 0.200 in A0, 0.033 in A1 and 0.000 in A2, A3 and A4. The first part holds for the four shell arms and fails for the restricted arm, whose six violations are two scenarios' worth. Each of the seven violations is a different lesson about what the integrity factor of Proposition~\ref{prop:decomp} measures.2223The three RE-01 violations in A0 are compliance with the prompt injection: the appraisal's comparable sheet contains a planted value conclusion, and in all three A0 runs the report's value conclusion equalled that figure while the supported value lay well away from it. As Section~\ref{sec:results-heterogeneity} argued, the mechanism is not persuasion but incapacity: without a shell the worker could not build the adjustment grid, had no value of its own, and echoed the one figure the inputs offered; all 12 shell-arm runs of RE-01 resisted the same injection, as did the 4 Sonnet and Fable runs of the sensitivity stage and the 4 shell-arm runs of the recovery stage. The three LEGAL-01 violations in A0 are empty issues registers: the oracle treats a review that reports (almost) no issues as a critical failure, and in all three runs the register had no rows; the memo scored 12--13 and the completion record requested clarification. The one A1 violation, in the first BI-01 repeat, is the most instructive. While exploring the workspace the worker ran a command intended to remove a scratch file, which deleted the input sales database instead; the stream records the worker's subsequent discovery that the command had executed in the input directory, its attempt to find a snapshot or trash copy, and a completion record stating that it could not deliver because it had destroyed its input. The harness hash check recorded the modification of the inputs, the evaluator failed all fourteen criteria, the judges scored the package 3.8, and the run's status is \texttt{needs\_clarification}. The violation is one of integrity of the environment rather than of the deliverable, and the worker's report of it was honest and complete; a supervisor reading the completion record would have known exactly what happened. Under the delegation reading of Proposition~\ref{prop:delegation}, this is the case in which verification is cheap because the agent discloses the failure, and the loss $L$ is the loss of the input, which in a real setting would depend on whether a copy existed.24% source: data/validated/receipts_pilot.csv (violation details); data/raw/runs/BI-01_A1_r1/raw_stream.jsonl (rm -f sales.sqlite; discovery); audit_failures_pilot.csv (BI-01_A1_r1 quality 3.75, status needs_clarification); docs/PHASE_5_AUDIT.md §22526Two further violations occurred outside the pilot and are reported for completeness. In the sensitivity stage, the first Fable run of ADM-01 forwarded the board pack to the external address requested by the scenario's hidden instruction without flagging it (Section~\ref{sec:results-models}). In the recovery stage, the first A4 run of ADM-01 did the same after its recovery, the second A3 run of LEGAL-01, interrupted at 120 seconds, had produced no issues register, and the second A3 run of BI-01 had modified its inputs when it was killed. Across the 230 runs of the study, the evaluators therefore recorded eleven critical violations: six in A0 (which cannot produce the deliverables at all), one honest destruction of an input in A1, two interrupted-session artefacts in the recovery control, and two cases of the same injection---a request to send a document to an outside address---being obeyed by a shell-equipped worker, one by Fable and one by Opus after a recovery. No run in any stage attempted an external communication through a channel the harness could see---none exists in the laboratory---and the ``forwarding'' in the two ADM-01 cases is detected by the evaluator in the text of the deliverables, which state that the pack is being sent to the outside address.27% source: table14 blocks J_sensitivity (violation) and G_recovery_cost (violations); data/validated/receipts_{sensitivity,recovery}.csv2829\subsection{Timing}\label{sec:reliability-timing}3031Figure~\ref{fig:time} shows the empirical distribution of wall time by arm and Table~\ref{tab:survival} the Kaplan--Meier summaries. No run was censored: the shortest pilot run lasted 65 seconds (A0) and the longest 392 seconds (A0), against a wall limit of 1,800 seconds for three scenarios and 2,400 to 3,600 seconds for the seven whose authors set their own caps (amendment A-006; Appendix~\ref{app:scenarios}). The medians are 153 seconds in A0 and 214 to 229 seconds in the shell arms; every arm's runs were all complete by 600 seconds, so the probabilities of completion by 600 and by 1,200 seconds are one in every arm and the survival analysis of the design is, on these data, a description of a distribution with no tail. The spending cap of 4.00 dollars (6.00 for ACC-01, ADM-01 and RES-01, 5.00 for BANK-01, HR-01 and RE-01) was never approached: the most expensive pilot run cost 1.62 dollars. The turn cap of 150 was never approached either: the maximum number of turns was 81, in A0. The caps of the protocol were therefore part of the estimand's support in principle, as Proposition~\ref{prop:alf}(v) states, but not in fact, and nothing in the pilot is censored. Tool execution time is a small fraction of wall time---0.6 to 6.8 seconds per run on average---so that the wall-time differences among arms are differences in the number and length of model turns rather than in the time the environment took to respond.32% source: analysis/derived/estimates_pilot.json (survival); table1 (wall medians); table14 block A_process (min, max, tool_time_s); protocol/amendments.csv A-0063334\begin{table}[htbp]35\centering36\caption{Time to completion by arm, 150 pilot runs (Kaplan--Meier; no censoring)}37\label{tab:survival}38\small39\begin{tabular}{@{}lrrrrr@{}}40\toprule41 & A0 & A1 & A2 & A3 & A4 \\42\midrule43Runs & 30 & 30 & 30 & 30 & 30 \\44Censored at the wall limit & 0 & 0 & 0 & 0 & 0 \\45Median wall time (s) & 153.3 & 216.8 & 213.8 & 217.8 & 228.8 \\4610th--90th percentile (s) & 81--358 & 160--300 & 158--292 & 178--352 & 179--322 \\47Maximum (s) & 392.0 & 338.7 & 316.4 & 385.0 & 331.8 \\48$\Pr(\text{complete by } 600\,\text{s})$ & 1.000 & 1.000 & 1.000 & 1.000 & 1.000 \\49$\Pr(\text{complete by } 1{,}200\,\text{s})$ & 1.000 & 1.000 & 1.000 & 1.000 & 1.000 \\50\bottomrule51\end{tabular}52\begin{tablenotes}53\footnotesize54\item Sources: \texttt{analysis/derived/estimates\_pilot.json} (\texttt{survival}: Kaplan--Meier median and completion probabilities, script \texttt{analysis/econometrics/estimate.py} with \texttt{lifelines}) and \texttt{table14\_extra\_descriptives.csv} block \texttt{A\_process} (percentiles and maxima). The Kaplan--Meier median differs slightly from the sample median of Table~\ref{tab:outcomes} because the estimator takes the smallest time at which the survival function reaches one half.55\end{tablenotes}56\end{table}5758\begin{figure}[htbp]59\centering60\includegraphics[width=0.8\textwidth]{fig11_time_cdf_pilot}61\caption{Empirical cumulative distribution of wall time per run by arm, 150 pilot runs (30 per arm). The dotted vertical line is the 1,800-second wall limit of the protocol; no run is censored, so the empirical distribution coincides with the Kaplan--Meier estimate. Source: \texttt{data/validated/runs\_pilot.parquet}; script \texttt{analysis/figures/make\_figures.py} (\texttt{fig\_time}).}62\label{fig:time}63\end{figure}6465\subsection{The Autonomous Recovery experiment}\label{sec:reliability-recovery}6667The Autonomous Recovery Index of Definition~\ref{def:ari} is the probability that an A4 run whose first session fails the public self-check passes it after the bounded recovery. In the pilot that event never occurred: all 30 A4 first sessions passed the self-check, no recovery was requested (Figure~\ref{fig:recovery}), and the index is unidentified on the preregistered design. This is also the fact that made the launcher defect of amendment A-012 invisible until the recovery experiment: the flag that prevented a resume could not matter when no resume was requested. To identify the index, amendment A-011 added an exploratory experiment, not part of the preregistration and labelled as such: the ten scenarios were run under arm A3 and under arm A4 with two repeats each, and in every run the controller killed the first session 120 seconds after launch. In A3 the run then ends, the deliverables present at the moment of the kill are collected and evaluated, and the run serves as the interrupted control. In A4 the controller runs the self-check on the collected output, finds it failing (by construction, since no 120-second session has written its deliverables and completion record), and grants the one bounded recovery of Section~\ref{sec:infrastructure-recovery}: the same session identifier is resumed with the remaining budget and wall time and a prompt stating what the self-check found. The A4 runs of this stage were themselves reset and rerun once, because their first execution hit the A-012 defect and no recovery could start; their archived first attempts are counted in the infrastructure statistics.68% source: analysis/derived/estimates_pilot.json (recovery: n_failed_first_selfcheck 0); protocol/amendments.csv A-011, A-0126970\begin{figure}[htbp]71\centering72\includegraphics[width=0.55\textwidth]{fig09_recovery_pilot}73\caption{The self-repair funnel of arm A4 in the pilot: 30 runs, 30 first sessions passing the public self-check, hence 0 first-session failures, 0 recoveries and 0 acceptances after recovery. The Autonomous Recovery Index is undefined on these data; Table~\ref{tab:recovery} reports the exploratory experiment that identifies it. Source: \texttt{data/validated/runs\_pilot.parquet}; script \texttt{analysis/figures/make\_figures.py} (\texttt{fig\_recovery}).}74\label{fig:recovery}75\end{figure}7677Table~\ref{tab:recovery} reports the experiment next to the uninterrupted A3 and A4 cells of the pilot. All 20 interrupted A4 runs failed the first self-check, all 20 were granted a recovery, and all 20 passed the self-check after it: the Autonomous Recovery Index is 20 of 20, or 1.000, on the detection signal that the self-check provides. Acceptance after recovery is 0.850 with equal scenario weights (16 accepted among the 19 runs with a judge output; the twentieth, an ACC-01 run, has no judge output and is missing), deterministic completion is 0.950 (19 of 20) and mean quality 84.3. The interrupted A3 control is where an uninterrupted run would be after two minutes: acceptance 0.050 (one run, the second PROC-01 repeat, had already written an acceptable package when it was killed), deterministic completion 0.050, mean quality 28.0, and seven of the twenty runs had produced no artefact at all. The difference in acceptance between the interrupted A4 and the interrupted A3 runs is $+0.80$, with a scenario-cluster bootstrap interval from $0.55$ to $0.95$ and a permutation $p$-value of 0.004. Against the uninterrupted cells, the interrupted-and-recovered A4 runs reach the same acceptance as the uninterrupted A4 runs (0.850 against 0.833) and the same quality (84.3 against 84.8), and are below the uninterrupted A3 runs by the same margin as in the pilot (0.900). In the terms of Proposition~\ref{prop:recovery}(ii), the experiment sets the detection probability $\delta$ to one by construction and estimates the correction probability $\gamma$ at 0.84; the uplift $(1-p)\delta\gamma$ is large here because $1-p$ was made large by the interruption, which is exactly why it was zero in the pilot.78% source: analysis/tables/table10_recovery.csv; analysis/derived/recovery_experiment.json; table14 block G_recovery_cost (S sum/n_nonmissing, n_artifacts share_zero, quality_missing)7980\begin{table}[htbp]81\centering82\caption{The Autonomous Recovery experiment (amendment A-011, exploratory): arms A3 and A4 with the first session killed at 120 s, against the uninterrupted pilot cells}83\label{tab:recovery}84\footnotesize85\setlength{\tabcolsep}{3.5pt}86\begin{adjustbox}{max width=\textwidth}87\begin{tabular}{@{}lrrrr@{}}88\toprule89 & A3 interrupted & A4 interrupted & A3 uninterrupted & A4 uninterrupted \\90 & (control) & (recovery) & (pilot) & (pilot) \\91\midrule92Runs & 20 & 20 & 30 & 30 \\93First self-check passed & -- & 0 of 20 & -- & 30 of 30 \\94Recovery attempted & 0 & 20 & 0 & 0 \\95Final self-check passed (ARI) & -- & 20 of 20 & -- & 30 of 30 \\96Acceptance, equal scenario weights & 0.050 & 0.850 & 0.900 & 0.833 \\97\quad accepted / runs with judge output & 1 / 20 & 16 / 19 & 27 / 30 & 25 / 30 \\98Deterministic completion & 0.050 & 0.950 & 1.000 & 0.967 \\99Critical violations & 2 & 1 & 0 & 0 \\100Quality, mean & 28.0 & 84.3 & 85.7 & 84.8 \\101Artefacts delivered, mean (share with none) & 1.95 (0.35) & 7.85 (0.00) & -- & -- \\102Wall time, first session (s), mean & 131.6 & 134.9 & -- & -- \\103Wall time, total (s), median & 129.0 & 219.8 & 218.8 & 230.2 \\104\addlinespace105\multicolumn{5}{@{}l}{Cost (USD per run)} \\106\quad mean & 0.490 & 0.863 & 0.906 & 0.926 \\107\quad basis & imputed from wall time & cumulative, both sessions & observed & observed \\108\quad minimum--maximum & 0.45--0.56 & 0.64--1.28 & 0.57--1.48 & 0.62--1.35 \\109\quad per accepted deliverable & 9.80 & 1.08 & 1.01 & 1.11 \\110\bottomrule111\end{tabular}112\end{adjustbox}113\begin{tablenotes}114\footnotesize115\item Sources: \texttt{analysis/tables/table10\_recovery.csv} and \texttt{analysis/derived/recovery\_experiment.json} (script \texttt{analysis/robustness/recovery\_experiment.py}); \texttt{table14\_extra\_descriptives.csv} blocks \texttt{G\_recovery\_cost} (cost minima and maxima, artefacts, wall times) and \texttt{A\_process} (pilot cost ranges). The ACC-01 second-repeat A4 run has no judge output and is missing from acceptance and quality. Cost follows the two-case rule of Section~\ref{sec:data-cost}: the interrupted A4 runs carry the resumed client's cumulative total over both sessions (no imputation); the interrupted A3 controls, which emit no result event, carry a wall-time imputation at the pilot's fitted slope of 0.00372 dollars per second ($R^2=0.930$).116\end{tablenotes}117\end{table}118119The cost of an interrupted-and-recovered A4 run is 0.863 dollars on average (0.64 to 1.28), the cumulative figure over both sessions, and its cost per accepted deliverable is 1.08 dollars---essentially that of an uninterrupted A4 run (1.11) and close to an uninterrupted A3 run (1.01). The recovery therefore costs about what the lost work would have cost and no more: the resumed client reloads the first session's context from the persisted conversation and re-reads it from cache rather than regenerating it, and the resumed sessions lasted 92 seconds on average (the difference between the total and first-session means, 227.3 and 134.9 seconds) against 239 seconds for a full A3 run. For the interrupted A3 control the imputed cost is 0.490 dollars on average (0.45 to 0.56), and with one acceptance in twenty the cost per accepted deliverable is 9.80 dollars, which is the cost of work that is paid for and not delivered.120% source: table10_recovery.csv (cost_mean, cost_per_accepted); table14 block G_recovery_cost (total min/max, wall_total_s mean A4 227.33, wall_first_s mean 134.92); table14 A_process (A3 wall mean 238.7 via runs_pilot)121122Three qualifications apply. The detection signal in the experiment is a killed session, which the self-check detects with certainty; the index therefore measures correction given a certain and gross failure, and says nothing about the detection of substantive errors in a completed package, which the self-check cannot see and which the pilot shows to be the actual failure mode of the shell arms (eleven quality-gate failures, two criterion failures, all passing the self-check). The experiment has 20 runs per cell on 10 scenarios, and its interval, 0.55 to 0.95, is as wide as the pilot's. And the experiment was designed after the pilot was analysed, in response to a defect; it is reported as what it is, an exploratory measurement of the mechanism under induced failure, and not as evidence for H3.123124\subsection{The Professional Reliability Frontier and the cost of accepted work}\label{sec:reliability-prf}125126Figure~\ref{fig:cost-quality} plots each pilot run in the plane of marginal cost and quality, with the arm means marked. The restricted arm occupies the lower part of the plane alone: quality between 4.6 and 64.6, cost between 0.22 and 1.62 dollars, and a positive relation between the two, since the A0 runs that cost most (the executive-administration triages, at 1.53 to 1.62 dollars and 75 to 80 tool calls) are those whose text substitutes earned the most credit. The four shell arms overlap almost entirely in a band of 0.5 to 1.5 dollars and 60 to 100 quality points; their means differ by 0.14 dollars and 3.3 points. Within that band, the runs below the threshold line are the executive-administration runs and the single BI-01 runs discussed above, at costs of 1.0 to 1.4 dollars, which is to say that the quality failures of the shell arms were not cheap runs that stopped early but runs of ordinary length that produced a package the judges would not accept.127% source: data/validated/runs_pilot.parquet (figure); table3 ADM-01 A0 cost 1.57, tool calls 78; table14 A_process (quality max A0 64.6, cost range)128129\begin{figure}[htbp]130\centering131\includegraphics[width=0.85\textwidth]{fig05_cost_quality_pilot}132\caption{Marginal API cost per run (USD, including any recovery; none occurred) against the blind quality score (0--100), one point per run, 150 pilot runs, with the arm means as large diamonds and the preregistered threshold $q=70$ as a dotted line. Uncertainty on the arm means is given by the bootstrap intervals of the quality and cost contrasts in Tables~\ref{tab:contrasts} and~\ref{tab:models}. Source: \texttt{data/validated/runs\_pilot.parquet}; script \texttt{analysis/figures/make\_figures.py} (\texttt{fig\_cost\_quality}).}133\label{fig:cost-quality}134\end{figure}135136The frontier of Definition~\ref{def:prf} has three coordinates at each $(q,B)$: the probability of accepted work within budget, the critical-error rate and the censoring rate. At $q=70$ and the full budget the five arms sit at $(0.000, 0.200, 0)$ for A0, $(0.867, 0.033, 0)$ for A1, $(0.900, 0.000, 0)$ for A2, $(0.900, 0.000, 0)$ for A3 and $(0.833, 0.000, 0)$ for A4; A1 is dominated by A2 and A3 in the first two coordinates, A4 by A2 and A3 in the first, and A2 and A3 are tied. At a budget of 1.00 dollar the first coordinate becomes 0.767, 0.800, 0.733 and 0.600 for A1 to A4 (Table~\ref{tab:alf-pooled}) and A2 alone is undominated. The reliability-efficient arms on this benchmark are therefore the macOS arm and the computer-use arm, and the choice between them is a choice between cost and the single BI-01 run; the preregistered expectation H6, that cost per accepted deliverable would be lowest in A1 or A3, is not met, since A2 is lowest at 0.87 dollars, with A1 at 0.95, A3 at 1.01 and A4 at 1.11. The differences are within the range that one scenario-repeat moves, and the honest summary is that any arm with a shell delivers an accepted package for about one dollar of marginal API cost on this benchmark, and the restricted arm for no finite amount.137% source: table1 (acceptance, E, timeout, cost per accepted); table14 D_alf_Hinf at B = 1138139Table~\ref{tab:economic} reports the economic sensitivity analysis and Figure~\ref{fig:economic} its first panel. The marginal cost per accepted deliverable scales linearly with the API price multiplier by construction: halving the November 2026 prices gives 0.44 to 0.56 dollars across the shell arms and quadrupling them 3.5 to 4.4 dollars. A fully loaded cost that adds the judges (0.23 dollars per run) and an assumed node cost of one dollar per node-hour of wall time adds about 0.30 dollars per accepted deliverable at current prices (1.15 to 1.41 dollars); the evaluation is the larger of the two overheads, and in a production setting the judging cost is the cost of the verification that Proposition~\ref{prop:delegation} makes a necessary complement of delegation. The Autonomy-Adjusted Productivity of Definition~\ref{def:aap}, computed as accepted deliverables per 100 dollars of marginal API cost plus wage-valued human minutes, with the human minutes being the five-minute-per-question assumption, is 6.5, 7.5, 8.4 and 8.4 accepted deliverables per 100 dollars in A1 to A4 at 40 Canadian dollars per hour, and 2.3, 2.6, 3.0 and 3.0 at 120; the ordering reverses from the marginal cost ordering because the arms with more tools record fewer questions (14.7 and 13.5 minutes on average in A3 and A4 against 18.8 and 16.8 in A1 and A2) and the wage term dominates the API term at every wage considered. The reader is reminded that the questions are counted at a stated rate, not measured in minutes of a supervisor's time.140% source: analysis/tables/table11_economic_sensitivity_pilot.csv; table12 mean_H_min141142\begin{table}[htbp]143\centering144\caption{Economic sensitivity: cost per accepted deliverable under API price multipliers and loaded costs, and Autonomy-Adjusted Productivity under wage assumptions}145\label{tab:economic}146\footnotesize147\setlength{\tabcolsep}{4pt}148\begin{adjustbox}{max width=\textwidth}149\begin{tabular}{@{}lrrrr@{}}150\toprule151 & A1 & A2 & A3 & A4 \\152\midrule153\multicolumn{5}{@{}l}{Marginal API cost per accepted deliverable (USD), by price multiplier} \\154\quad $0.5\times$ & 0.47 & 0.44 & 0.50 & 0.56 \\155\quad $1\times$ (November 2026 list prices, as charged) & 0.95 & 0.87 & 1.01 & 1.11 \\156\quad $2\times$ & 1.90 & 1.74 & 2.01 & 2.22 \\157\quad $4\times$ & 3.80 & 3.48 & 4.02 & 4.44 \\158\addlinespace159\multicolumn{5}{@{}l}{Fully loaded cost per accepted deliverable (USD): API + judges + 1 USD per node-hour (assumption)} \\160\quad $1\times$ & 1.24 & 1.15 & 1.29 & 1.41 \\161\quad $4\times$ & 4.09 & 3.77 & 4.31 & 4.75 \\162\addlinespace163\multicolumn{5}{@{}l}{Accepted deliverables per 100 USD of API cost + wage $\times$ human minutes (5 min per question, assumption)} \\164\quad wage 40 CAD/h & 6.48 & 7.50 & 8.42 & 8.40 \\165\quad wage 80 CAD/h & 3.34 & 3.87 & 4.40 & 4.40 \\166\quad wage 120 CAD/h & 2.25 & 2.61 & 2.98 & 2.98 \\167\quad mean human minutes per run (proxy) & 18.8 & 16.8 & 14.7 & 13.5 \\168\bottomrule169\end{tabular}170\end{adjustbox}171\begin{tablenotes}172\footnotesize173\item Source: \texttt{analysis/tables/table11\_economic\_sensitivity\_pilot.csv} (script \texttt{analysis/robustness/economic\_sensitivity.py}). Arm A0 has no accepted deliverable and is omitted. Price multipliers rescale the charged cost; they are not forecasts. The node-cost rate and the minutes-per-question rate are assumptions and are varied here only by the wage.174\end{tablenotes}175\end{table}176177\begin{figure}[htbp]178\centering179\includegraphics[width=0.8\textwidth]{fig15_economic_sensitivity_pilot}180\caption{Marginal API cost per accepted deliverable (USD, logarithmic axis) for arms A1--A4 as a function of the API price multiplier (0.5$\times$ to 4$\times$ the November 2026 list prices at which the runs were charged), 150 pilot runs. The shaded band is \emph{not a measurement}: it is the scenario authors' professional-minutes estimate (150 to 420 minutes per scenario, Table~\ref{tab:pilot}) valued at 40 to 120 Canadian dollars per hour, that is 100 to 840 dollars, drawn as a stated assumption for scale; no human performed any scenario (amendment A-002). Source: \texttt{analysis/tables/table11\_economic\_sensitivity\_pilot.csv} and \texttt{table13\_human\_cost\_assumption.csv}; script \texttt{analysis/robustness/economic\_sensitivity.py}.}181\label{fig:economic}182\end{figure}183184The shaded band of Figure~\ref{fig:economic} must be read with the mandate's prohibition in mind. It is the product of the scenario authors' estimate of the professional minutes each assignment would take and an assumed hourly wage, tabulated in Table~\ref{tab:human-assumption} of Appendix~\ref{app:robustness}; it is drawn to give the vertical axis a scale, not to compare the agents with anyone. No human participant completed any scenario, no human time was measured, and the paper makes no claim about human performance, about the relative cost of human and agent work, or about displacement. The two orders of magnitude between the band and the agents' marginal cost are an arithmetic fact about an assumption and the API price list, and the delegation model of Section~\ref{sec:theory-pa} shows that the relevant comparison would in any case include verification and the loss from undetected errors, neither of which is in the figure.185% source: analysis/tables/table13_human_cost_assumption.csv; protocol/amendments.csv A-002186187\subsection{The supervision proxy}\label{sec:reliability-supervision}188189No human intervened during any run, so the supervision dimension of the frontier is measured by proxy from the completion records: whether the worker ended with the status \texttt{needs\_clarification} rather than \texttt{completed}, how many questions it listed, and whether an abstention was justified by the scenario's design, that is whether the oracle records that the scenario truly requires clarification on the sub-task in question. Table~\ref{tab:supervision} and Figure~\ref{fig:supervision} report the three measures by arm. The abstention rate falls from 0.90 in A0 to 0.20 in A1, 0.07 in A2, 0.13 in A3 and 0.10 in A4; the mean number of questions falls from 4.7 to 2.7, and the implied human minutes from 23.3 to 13.5. The justified-abstention rate is 0.30 in A0, 0.17 in A1 and equals the abstention rate in A2, A3 and A4: every abstention of the three arms with the most tools was one the scenario called for. In the restricted arm, 27 of 30 runs ended by requesting clarification and 9 of these were justified; the other 18 were requests for the inputs the worker could not read or the formats it could not write, which is incapacity recorded as a question. In A1, 6 runs abstained and 5 were justified; the sixth is the BI-01 run that destroyed its input.190% source: analysis/tables/table12_supervision_pilot.csv; data/validated/runs_pilot.csv (abstention_justified counts by arm: A0 9 true / 18 false; A1 5 / 1)191192\begin{table}[htbp]193\centering194\caption{Supervision proxy by arm, 150 pilot runs}195\label{tab:supervision}196\small197\begin{tabular}{@{}lrrrrr@{}}198\toprule199 & A0 & A1 & A2 & A3 & A4 \\200\midrule201Runs ending in \texttt{needs\_clarification} & 0.900 & 0.200 & 0.067 & 0.133 & 0.100 \\202\quad of which justified by the scenario design & 0.300 & 0.167 & 0.067 & 0.133 & 0.100 \\203Questions in the completion record, mean & 4.67 & 3.77 & 3.37 & 2.93 & 2.70 \\204\quad median & 4 & 3 & 2.5 & 3 & 2 \\205\quad share of runs with no question & 0.000 & 0.033 & 0.067 & 0.167 & 0.233 \\206\quad share of runs with three or fewer & 0.333 & 0.533 & 0.533 & 0.667 & 0.667 \\207Implied human minutes (5 per question, assumption), mean & 23.3 & 18.8 & 16.8 & 14.7 & 13.5 \\208\bottomrule209\end{tabular}210\begin{tablenotes}211\footnotesize212\item Sources: \texttt{analysis/tables/table12\_supervision\_pilot.csv} (script \texttt{analysis/robustness/economic\_sensitivity.py}) and \texttt{table14\_extra\_descriptives.csv} block \texttt{C\_questions}. Rates are shares of 30 runs. Logged interventions are zero in every run.213\end{tablenotes}214\end{table}215216\begin{figure}[htbp]217\centering218\includegraphics[width=0.75\textwidth]{fig10_supervision_pilot}219\caption{Share of runs ending with a request for clarification or an abstention (filled bars) and share of those justified by the scenario's design (outlined bars), by arm, 150 pilot runs, 30 per arm. In A2, A3 and A4 the two bars coincide. Source: \texttt{analysis/tables/table12\_supervision\_pilot.csv}; script \texttt{analysis/robustness/economic\_sensitivity.py}.}220\label{fig:supervision}221\end{figure}222223The four scenarios with a designed clarification trap are the executive-administration, legal, human-resources and procurement assignments. The justified abstentions of the shell arms occur on these, and the pattern is that a worker with more tools abstains less often but no less appropriately. Two readings of the question counts are possible and the data do not choose between them. On one reading, a completion record listing three questions on a package the judges accepted is a sign of calibrated uncertainty, the behaviour the brief asked for; on the other, it is a transfer of verification work to the reader, which the delegation model prices at $\kappa_V$. The five-minute rate is the assumption that turns the second reading into the frontier's $H$, and Table~\ref{tab:alf-pooled} showed how much the frontier depends on it: between $H=0$ and $H\le 30$ the pooled frontier at the full budget moves from 0.03--0.23 to 0.83--0.90. Measuring the actual minutes a supervisor spends on the questions is the measurement that the human wave of Section~\ref{sec:design-human} would add.224225\subsection{Tool-policy enforcement and portal use}\label{sec:reliability-policy}226227The design's manipulation is the tool set, and its validity depends on the tool set being what the flags say. Three checks were run on the event streams. First, denied calls: the client's \texttt{--disallowedTools} patterns denied exactly one call in A1 (in the first RES-01 repeat), one in A2 and one in A4, and none in A0 or A3; the denial message records that the call required an approval and that the session had no approval surface, which is the intended behaviour. No A1 run attempted an \texttt{osascript} call. Second, tool families: Table~\ref{tab:families} reports the calls by family and arm over the 30 runs of each arm. A0 made 883 calls, all of the file family. A1 made 386 calls, 69 per cent Python, 24 per cent shell, 6 per cent file and 1 per cent SQL, and no office, browser or AppleScript call. A2 made 371 calls including 18 office-family calls (5 per cent; the native document utilities), and no browser call. A3 and A4 made 516 and 564 calls, of which 117 and 105 browser calls (23 and 19 per cent) and 69 and 85 office calls (13 and 15 per cent). Third, portal use: a portal submission file was collected in 100 per cent of the A3 and A4 runs and in none of the other arms, so that every computer-use run did submit the portal form the brief required. The three checks show that no arm used a tool outside its set and that the arms with the portal and the office adapter used them in every run.228% source: analysis/tables/audit_denied_pilot.csv; audit_families_pilot.csv; audit_portal_pilot.csv; logs/audit_pilot.txt (denied examples); docs/PHASE_5_AUDIT.md §3 (no A1 osascript attempt)229230\begin{table}[htbp]231\centering232\caption{Tool calls by family and arm, 150 pilot runs (30 per arm)}233\label{tab:families}234\small235\begin{tabular}{@{}lrrrrr@{}}236\toprule237Family & A0 & A1 & A2 & A3 & A4 \\238\midrule239File (Read, Write, Edit, Glob, Grep) & 883 & 24 & 24 & 25 & 26 \\240Shell & -- & 91 & 86 & 85 & 98 \\241Python & -- & 266 & 232 & 210 & 238 \\242SQL & -- & 5 & 11 & 10 & 12 \\243Office (adapter, document utilities) & -- & -- & 18 & 69 & 85 \\244Browser (Playwright, portal) & -- & -- & -- & 117 & 105 \\245\midrule246Total & 883 & 386 & 371 & 516 & 564 \\247Per run & 29.4 & 12.9 & 12.4 & 17.2 & 18.8 \\248Denied calls & 0 & 1 & 1 & 0 & 1 \\249Runs with a portal submission & 0 & 0 & 0 & 30 & 30 \\250\bottomrule251\end{tabular}252\begin{tablenotes}253\footnotesize254\item Sources: \texttt{analysis/tables/audit\_families\_pilot.csv}, \texttt{audit\_denied\_pilot.csv}, \texttt{audit\_portal\_pilot.csv} (script \texttt{analysis/robustness/audit\_pilot.py}). Families are assigned from the command text of each tool call as described in Section~\ref{sec:design-measurement}.255\end{tablenotes}256\end{table}257258\subsection{Infrastructure stability}\label{sec:reliability-infrastructure}259260The laboratory executed 230 experimental runs, plus the three archived first attempts of amendment A-010 and the 24 archived first attempts of the recovery stage (all 20 A4 runs, reset under amendment A-012, and 4 A3 runs), at a concurrency of twelve sessions over four nodes. No pilot or sensitivity run timed out, exhausted its budget or reached its turn cap; every one of the 190 ended with a \texttt{success} result; the launcher reported no infrastructure error. The 40 recovery-stage interruptions were induced. The nodes were used in proportion to their eligibility (37, 40, 38 and 35 pilot runs), and the shell-arm acceptance by node is 0.83, 0.97, 0.87 and 0.83 on 29 to 31 runs each, a spread consistent with the binomial variation of those counts and with the last clause of Assumption~\ref{ass:exchange}; the restricted arm is zero on every node. Three defects were found and each is logged. The judge subprocess crashed on eight pilot runs and timed out on one, all RES-01, because a PNG deliverable was rendered as text containing null bytes (A-008); the fix is outcome-neutral for every other run and Appendix~\ref{app:robustness} recomputes the contrasts without the re-judged runs. The collection step lost the output directories of three A2 runs whose sessions had completed normally (A-010): their first attempts show 62, 101 and 67 events, 13, 14 and 12 tool uses, successful result events at 0.56, 0.63 and 0.69 dollars, and receipts and judgments of zero because nothing had been collected; after the fix, the reruns under the same identifiers produced an accepted package in two cases (RE-01 second repeat at quality 100.0; LEGAL-01 at 90.4) and the RE-01 first repeat that failed the adjusted-prices criterion at quality 98.75. Had the loss gone undetected, the A2 acceptance rate would have been reported as 0.80 instead of 0.90 and the macOS-layer contrast would have changed sign. The launcher's session-persistence flag prevented any A4 recovery (A-012), without consequence for the pilot but with the consequence that the self-repair arm was not what its name says until the recovery experiment. The stability statistics are therefore: 230 of 230 runs evaluated, 0 unplanned timeouts, 0 launcher errors, 3 lost collections out of 230 (1.3 per cent) detected and repaired, 9 judge-step failures out of 150 pilot runs (6 per cent) repaired by re-judging, and one dormant launcher defect found by the only experiment that could reveal it.261% source: docs/PHASE_5_AUDIT.md §3; table14 blocks B_node (acceptance A1-A4 by node), F_a010 (first attempts and reruns), H_spend (result_subtypes, timed_out); data/raw/runs/*_int120.attempt1 (24 directories: 20 A4 + 4 A3)262