1% ---------------------------------------------------------------------------2% Section 8 — Main results (Agent H; every number from analysis/tables,3% analysis/derived or docs/PHASE_5_AUDIT.md; sources in comments)4% ---------------------------------------------------------------------------5\section{Main results}\label{sec:results}67This section reports the preregistered estimands on the 150 pilot runs and the two additional stages, in the order of the preregistration: outcomes by arm, the primary and secondary contrasts with their scenario-cluster bootstrap intervals and permutation tests, the Autonomous Labor Frontier by occupation, the heterogeneity across occupations, the Workflow Dependency Frontier, the sensitivity to the worker model and the sensitivity to the quality threshold. Two features of the data govern the reading of every estimate and are stated at the outset. The benchmark has $J=10$ scenarios, so that every scenario-weighted contrast is estimated from ten paired differences and its interval is wide, as Section~\ref{sec:design-power} anticipated. And every arm with a shell is near the ceiling of the acceptance scale on this benchmark, between 0.83 and 0.90, so that the contrasts among those arms are differences of small numbers of scenario failures and the primary hypothesis is tested in a region where the design has little power to detect anything but a large effect. The estimates are reported with their signs and intervals whatever they are; nulls are reported as nulls.89\subsection{Outcomes by arm}\label{sec:results-arms}1011Table~\ref{tab:outcomes} gives the preregistered outcomes by arm and Figure~\ref{fig:acceptance} the acceptance rates. The restricted arm A0 accepted no deliverable in 30 runs and completed none deterministically: its mean quality score is 27.4, its critical-error rate 0.20, and 90 per cent of its runs ended with a completion record requesting clarification. The four arms with a shell are close to one another. Scenario-weighted acceptance is 0.867 in A1, 0.900 in A2 and A3, and 0.833 in A4; deterministic completion is 0.967, 0.967, 1.000 and 0.967; mean quality is 82.3, 85.0, 85.7 and 84.8, with a markedly larger dispersion in A1 (standard deviation 18.5, against 10.0 to 11.4 elsewhere) because one A1 run of BI-01 scored 3.8 after destroying its input; the critical-error rate is 0.033 in A1 and zero in A2, A3 and A4. Under professional-minutes weights the acceptance rates move by at most 0.03 (0.885, 0.909, 0.926, 0.844). Cost per accepted deliverable, which keeps the failed runs in the denominator, is 0.95 dollars in A1, 0.87 in A2, 1.01 in A3 and 1.11 in A4, and is undefined in A0. No run in any arm timed out.12% source: analysis/tables/table1_outcomes_by_arm_pilot.csv1314\begin{table}[htbp]15\centering16\caption{Preregistered outcomes by arm, 150 pilot runs (\texttt{claude-opus-5-5}, $q=70$)}17\label{tab:outcomes}18\footnotesize19\setlength{\tabcolsep}{4.5pt}20\begin{adjustbox}{max width=\textwidth}21\begin{tabular}{@{}lrrrrr@{}}22\toprule23 & A0 & A1 & A2 & A3 & A4 \\24 & restricted & shell & macOS & computer-use & self-repair \\25\midrule26Acceptance $S$, equal scenario weights & 0.000 & 0.867 & 0.900 & 0.900 & 0.833 \\27Acceptance $S$, professional-minutes weights & 0.000 & 0.885 & 0.909 & 0.926 & 0.844 \\28Deterministic completion $D$ & 0.000 & 0.967 & 0.967 & 1.000 & 0.967 \\29Critical-error rate $E$ & 0.200 & 0.033 & 0.000 & 0.000 & 0.000 \\30Quality $Q$, mean (s.d.) & 27.4 (17.4) & 82.3 (18.5) & 85.0 (10.0) & 85.7 (11.4) & 84.8 (10.6) \\31Timeout rate & 0.000 & 0.000 & 0.000 & 0.000 & 0.000 \\32Abstention / clarification rate & 0.900 & 0.200 & 0.067 & 0.133 & 0.100 \\33Cost $C$ (USD), mean (s.d.) & 0.710 (0.416) & 0.823 (0.230) & 0.783 (0.190) & 0.906 (0.259) & 0.926 (0.221) \\34Cost per accepted deliverable (USD) & -- & 0.950 & 0.870 & 1.006 & 1.111 \\35Wall time (s), median & 162.0 & 217.7 & 213.9 & 218.8 & 230.2 \\36Tool calls, mean & 29.4 & 12.9 & 12.4 & 17.2 & 18.8 \\37\bottomrule38\end{tabular}39\end{adjustbox}40\begin{tablenotes}41\footnotesize42\item Source: \texttt{analysis/tables/table1\_outcomes\_by\_arm\_pilot.csv} (script \texttt{analysis/econometrics/estimate.py}). Each arm: 30 runs on 10 scenarios, 3 repeats. $S=\mathbf{1}\{D=1\wedge E=0\wedge Q\ge 70\}$. The abstention rate is the share of runs whose completion record has the status \texttt{needs\_clarification}. Cost per accepted deliverable is $\sum C/\sum S$ over all runs of the arm and is undefined when $\sum S=0$.43\end{tablenotes}44\end{table}4546\begin{figure}[htbp]47\centering48\includegraphics[width=0.8\textwidth]{fig03_acceptance_by_arm_pilot}49\caption{Scenario-weighted acceptance rate by arm (filled bars) and deterministic completion rate (dashed outlines), 150 pilot runs, 10 scenarios $\times$ 3 repeats per arm, \texttt{claude-opus-5-5}, $q=70$. Uncertainty is reported in Figure~\ref{fig:contrasts} as scenario-cluster bootstrap intervals on the between-arm differences, which is the preregistered inferential quantity; no interval is drawn on the levels. Source: \texttt{analysis/tables/table1\_outcomes\_by\_arm\_pilot.csv}; script \texttt{analysis/figures/make\_figures.py} (\texttt{fig\_completion\_by\_arm}).}50\label{fig:acceptance}51\end{figure}5253The gap between deterministic completion and acceptance in the shell arms---0.10 in A1, 0.067 in A2, 0.10 in A3, 0.133 in A4---is the quality gate: runs that pass every oracle criterion but are scored below 70 by the judges. Section~\ref{sec:results-heterogeneity} shows that this gap is almost entirely one scenario.5455\subsection{Preregistered contrasts}\label{sec:results-contrasts}5657Table~\ref{tab:contrasts} reports the four preregistered contrasts for acceptance, deterministic completion, the critical-error indicator and quality, each as the equal-weight mean of ten within-scenario differences of repeat means, with the percentile interval of the scenario-cluster bootstrap ($B=5{,}000$), the $p$-value of the paired sign-flip permutation test over scenarios (20,000 permutations), and the professional-minutes-weighted estimate; the Holm-adjusted $p$-values over the three secondary acceptance contrasts are given in the last column. Figure~\ref{fig:contrasts} plots the acceptance contrasts.58% source: analysis/tables/table2_contrasts_pilot.csv5960\begin{table}[htbp]61\centering62\caption{Preregistered contrasts: scenario-weighted differences with scenario-cluster bootstrap 95\% intervals and paired permutation $p$-values ($J=10$)}63\label{tab:contrasts}64\footnotesize65\setlength{\tabcolsep}{4pt}66\begin{adjustbox}{max width=\textwidth}67\begin{tabular}{@{}llrrrrrr@{}}68\toprule69Contrast & Outcome & Estimate & 95\% CI lower & 95\% CI upper & Perm.\ $p$ & Minutes-weighted & Holm $p$ \\70\midrule71A3 $-$ A1 (primary, H1) & $S$ & $+0.033$ & $0.000$ & $0.100$ & 1.000 & $+0.041$ & -- \\72 & $D$ & $+0.033$ & $0.000$ & $0.100$ & 1.000 & $+0.041$ & \\73 & $E$ & $-0.033$ & $-0.100$ & $0.000$ & 1.000 & $-0.041$ & \\74 & $Q$ & $+3.35$ & $-0.81$ & $+9.56$ & 0.300 & $+4.57$ & \\75\addlinespace76A1 $-$ A0 (secondary, H2) & $S$ & $+0.867$ & $0.667$ & $1.000$ & 0.004 & $+0.885$ & 0.012 \\77 & $D$ & $+0.967$ & $0.900$ & $1.000$ & 0.002 & $+0.959$ & \\78 & $E$ & $-0.167$ & $-0.467$ & $+0.067$ & 0.499 & $-0.144$ & \\79 & $Q$ & $+54.96$ & $+40.44$ & $+68.24$ & 0.002 & $+57.66$ & \\80\addlinespace81A4 $-$ A3 (secondary, H3) & $S$ & $-0.067$ & $-0.167$ & $0.000$ & 0.499 & $-0.082$ & 0.998 \\82 & $D$ & $-0.033$ & $-0.100$ & $0.000$ & 1.000 & $-0.041$ & \\83 & $E$ & $0.000$ & $0.000$ & $0.000$ & 1.000 & $0.000$ & \\84 & $Q$ & $-0.90$ & $-2.71$ & $+0.97$ & 0.395 & $-1.39$ & \\85\addlinespace86A2 $-$ A1 (secondary) & $S$ & $+0.033$ & $-0.067$ & $+0.133$ & 1.000 & $+0.025$ & 1.000 \\87 & $D$ & $0.000$ & $-0.100$ & $+0.100$ & 1.000 & $0.000$ & \\88 & $E$ & $-0.033$ & $-0.100$ & $0.000$ & 1.000 & $-0.041$ & \\89 & $Q$ & $+2.63$ & $-1.75$ & $+8.79$ & 0.534 & $+2.96$ & \\90\bottomrule91\end{tabular}92\end{adjustbox}93\begin{tablenotes}94\footnotesize95\item Source: \texttt{analysis/tables/table2\_contrasts\_pilot.csv} (script \texttt{analysis/econometrics/estimate.py}; bootstrap seed 42). Intervals are percentile intervals of 5,000 resamples of the ten scenarios with replacement; $p$-values are two-sided sign-flip permutation tests over the ten paired scenario differences. Holm adjustment is over the three secondary acceptance contrasts, as preregistered.96\end{tablenotes}97\end{table}9899\begin{figure}[htbp]100\centering101\includegraphics[width=0.85\textwidth]{fig07_tool_effects_pilot}102\caption{The four preregistered contrasts on acceptance probability: equal-weight mean of the ten within-scenario differences of repeat means, with scenario-cluster bootstrap 95\% percentile intervals ($B=5{,}000$) and the paired sign-flip permutation $p$-value (20,000 permutations) printed above each point. 150 pilot runs, 10 scenarios, 3 repeats per cell. Source: \texttt{analysis/tables/table2\_contrasts\_pilot.csv}; script \texttt{analysis/figures/make\_figures.py} (\texttt{fig\_contrasts}).}103\label{fig:contrasts}104\end{figure}105106\paragraph{H1, the computer-use premium.} The primary contrast, acceptance under the full computer-use arm minus acceptance under the shell arm, is $+0.033$ with a bootstrap interval from $0.000$ to $0.100$ and a permutation $p$-value of 1.000. The estimate is one scenario-repeat in thirty: A3 accepted one run that A1 did not (the BI-01 run in which A1 destroyed its input), and the two arms agree on every other scenario mean. The interval's lower bound sits at zero because no scenario favours A1, and its upper bound at 0.10 because at most one scenario favours A3; the permutation test cannot reject because the ten paired differences are nine zeros and one positive value. The same holds for deterministic completion. The contrast on quality is $+3.3$ points with an interval from $-0.8$ to $+9.6$ ($p=0.30$). H1 is therefore not supported by the pilot. The result is not an absence of information: with the shell arm at 0.867, the computer-use premium on this benchmark cannot exceed 0.133, and the interval excludes a premium larger than 0.10. The reading we propose is that for the ten pilot scenarios, in which every deliverable can be produced programmatically from the inputs by a shell with a Python environment, the browser, the office suite and the portal added nothing that the shell could not already do; whether a premium exists on tasks whose inputs or deliverables can only be reached through those applications is a question that this benchmark, by construction, does not ask (Section~\ref{sec:benchmark-anatomy}).107% source: table2 A3-A1 rows; table3 BI-01 rows; audit_failures_pilot.csv (BI-01_A1_r1)108109\paragraph{H2, the shell premium.} The shell premium is $+0.867$ with an interval from $0.667$ to $1.000$ and a permutation $p$-value of 0.004 (Holm-adjusted 0.012); for deterministic completion it is $+0.967$ ($0.900$ to $1.000$, $p=0.002$) and for quality $+55.0$ points ($40.4$ to $68.2$, $p=0.002$). This is the only confirmatory effect of the pilot, and its mechanism is the one the hypothesis named: the deliverables of every pilot scenario include at least one binary office file, usually a workbook with live formulas, which the restricted arm cannot produce. The restricted arm delivered text substitutes in every run---CSV and Markdown files under the requested base names---and the deterministic criteria that require the workbook's structure and formulas failed in every case, which is the deliverable-format mechanism of H2. The contrast on the critical-error indicator, $-0.167$ with an interval from $-0.467$ to $+0.067$ ($p=0.50$), is not significant; the six A0 violations are concentrated in two scenarios, which makes the paired difference non-zero in only two of ten pairs.110% source: table2 A1-A0 rows; audit_a0_outputs_pilot.csv (substitutes); audit_violations_pilot.csv111112\paragraph{H3, the self-repair premium.} The self-repair premium is $-0.067$, with an interval from $-0.167$ to $0.000$ and a permutation $p$-value of 0.499 (Holm-adjusted 0.998); on deterministic completion it is $-0.033$ ($-0.100$ to $0.000$, $p=1.000$), on quality $-0.9$ points ($-2.7$ to $+1.0$, $p=0.40$), and on the critical-error indicator exactly zero. The sign is negative and the point estimate is two scenario-repeats in thirty: A4 lost one run of BI-01 to the quality gate (scored 67.9) and one run of RE-01 to the adjusted-prices criterion, both of which A3 passed in all three repeats. The hypothesis stated an effect of at least zero; the estimate is below zero but its interval includes zero, and the data do not distinguish a small negative effect from none. The interpretation is bounded by amendment A-012: in the pilot, all 30 A4 first sessions passed the public self-check, no recovery was ever requested, and the arm therefore differed from A3 only by its checkpointing and self-check instructions and by the longer sessions these produce (18.8 tool calls against 17.2, 230 against 219 seconds, 0.93 against 0.91 dollars). Proposition~\ref{prop:recovery}(ii) states that the self-repair premium is proportional to the first-session failure rate that the self-check can see; that rate was zero in the pilot, so the premium was zero by construction, and the Autonomous Recovery Index is unidentified on the pilot design. Section~\ref{sec:reliability-recovery} reports the exploratory experiment that identifies it.113% source: table2 A4-A3 rows; table1 (tool calls, wall, cost); estimates_pilot.json recovery (n_failed_first_selfcheck 0); audit_failures (BI-01_A4_r1, RE-01_A4_r2)114115\paragraph{The macOS layer.} The contrast between the macOS arm and the shell arm is $+0.033$ ($-0.067$ to $+0.133$, $p=1.000$, Holm 1.000) on acceptance, zero on deterministic completion, $-0.033$ on critical errors and $+2.6$ points on quality ($p=0.53$). A2 accepted the BI-01 run that A1 lost and lost the RE-01 run that A1 passed; the two differences cancel in the mean. The headless automation layer, whose scope Section~\ref{sec:infrastructure-macos} limits to \texttt{osascript} and the native document utilities, was used sparingly: 18 office-family calls in 30 runs, against 69 in A3, and no run depended on it for a deliverable that the shell could not produce.116% source: table2 A2-A1 rows; audit_families_pilot.csv (A2 office 18, A3 office 69)117118\paragraph{The cluster-robust logit as sensitivity.} A logistic regression of acceptance on arm indicators without intercept, with standard errors clustered by scenario, returns logits of 1.87 (cluster standard error 0.89) for A1, 2.20 (0.80) for A2, 2.20 (1.13) for A3 and 1.61 (0.75) for A4, which invert to the arm acceptance rates of Table~\ref{tab:outcomes}; the A0 coefficient is $-23.6$ with a standard error of 0.34, which is complete separation rather than an estimate, since no A0 run was accepted. The clustered standard errors of the shell arms, 0.75 to 1.13 on the logit scale, are of the same order as the differences among those arms (at most 0.59), which restates on the model scale what the bootstrap intervals say: the shell arms are not distinguishable from one another on this benchmark. The mixed-effects variant with a scenario random intercept was not estimated by the script, which fitted the cluster-robust model only; given the separation in A0 and the near-separation in several scenarios of the other arms, a random-intercept logit would require penalisation to converge and is left to the scaled study.119% source: analysis/derived/estimates_pilot.json (mixed_logit_S)120121\paragraph{The ceiling.} The lesson of the contrasts is the ceiling. The scale rule of Section~\ref{sec:design-scale} gave the pilot a third repeat, which reduces the within-scenario component of the variance in equation~\eqref{eq:premium-var-cluster}, but the component that governs the primary contrast is the between-scenario heterogeneity of the effect, and with the shell arms succeeding on nine scenarios out of ten there is no heterogeneity left to measure. The power table of Section~\ref{sec:design-power} applies to a baseline of 0.5; at a baseline of 0.87 the largest detectable premium is 0.13 whatever the number of repeats. The implication for the next wave, which Section~\ref{sec:benchmark-scale} anticipated, is that the benchmark needs scenarios on which the shell arm fails---tasks whose inputs live only in an application, longer dependency chains, perturbations---before the computer-use premium can be estimated at all.122123\subsection{The Autonomous Labor Frontier by occupation}\label{sec:results-alf}124125The frontier $\mathrm{ALF}_o(B,H,q)$ of Definition~\ref{def:alf} is the task-weighted probability that a run is accepted, costs at most $B$ and requires at most $H$ minutes of human time. With one scenario per occupation and three repeats, each occupation's frontier takes the values 0, $\tfrac13$, $\tfrac23$ or 1 and the weights (equal or professional-minutes) are immaterial within occupation, so that the equal-weight and minutes-weighted curves of the frontier table coincide; the weights matter only when occupations are pooled, as in Table~\ref{tab:outcomes}. The budget argument is varied from 0.25 to 4.00 dollars, the latter being the per-session cap beyond which Proposition~\ref{prop:alf}(v) makes the curve flat; the human-time argument is the stated assumption of five minutes per unanswered question recorded in the completion record, since no intervention occurred during any run.126% source: analysis/tables/table5_alf_pilot.csv (weights equal, minutes, equal_H15; H = 0 and 15)127128Figure~\ref{fig:alf} shows the preregistered curve at $H=0$, and it is nearly empty: at $H=0$ the frontier is positive only for academic research (A1 at $\tfrac13$, A3 and A4 at 1), corporate finance (A3 at $\tfrac13$, A4 at 1), business intelligence (A3 and A4 at $\tfrac13$), accounting and real-estate appraisal (A2 at $\tfrac13$), and zero for every other occupation and arm. The reason is not that the work was rejected but that the condition $H\le 0$ excludes every run whose completion record lists at least one question, and almost every run does: no A0 run and only one A1 run, two A2 runs, five A3 runs and seven A4 runs recorded no question at all. The questions are not, for the most part, abstentions---the completion status is \texttt{completed} in 80 to 93 per cent of shell-arm runs---but the list of points the worker would have raised with a manager had one been available, which the brief's working conventions invite. Read strictly, the $H=0$ frontier says that the pilot's agents almost never produce an accepted deliverable \emph{and} declare that nothing needs to be checked with a human; read as a measurement of supervision, it says that the proxy counts every recorded question at five minutes regardless of its substance, and Section~\ref{sec:reliability-supervision} examines what the questions were.129% source: table5 (equal, H=0, B=4); table14 block C_questions (share_zero_questions by arm); table1 abstention_rate130131\begin{figure}[htbp]132\centering133\includegraphics[width=\textwidth]{fig04_alf_curves_pilot}134\caption{Autonomous Labor Frontier $\mathrm{ALF}_o(B,H=0,q=70)$ by occupation and arm as a function of the budget cap $B$ (USD per run, logarithmic axis), equal task weights, 150 pilot runs (one scenario per occupation, 3 repeats per arm, so each curve takes values in $\{0,\tfrac13,\tfrac23,1\}$). The human-time allowance $H=0$ admits only runs whose completion record lists no question; Table~\ref{tab:alf-pooled} reports the frontier at $H=15$, $H=30$ and without the human-time condition. No uncertainty interval is drawn: with one scenario per occupation the occupation-level frontier has no between-task variance to resample, and its sampling uncertainty is that of a binomial proportion on three repeats. Source: \texttt{analysis/tables/table5\_alf\_pilot.csv}; script \texttt{analysis/figures/make\_figures.py} (\texttt{fig\_alf}).}135\label{fig:alf}136\end{figure}137138Table~\ref{tab:alf-pooled} therefore reports the frontier pooled over the ten occupations with equal weights, at $H=0$, $H=15$, $H=30$ and with the human-time condition removed, across the budget grid. Without the human-time condition, the frontier is zero in every arm up to a budget of 0.50 dollars because the cheapest accepted run cost 0.509 dollars; it rises to 0.30--0.50 at 0.75 dollars, to 0.60--0.80 at 1.00 dollar, and reaches its ceiling---the marginal acceptance rate of Table~\ref{tab:outcomes}, 0.867, 0.900, 0.900 and 0.833---at 1.50 dollars in every arm, since the most expensive accepted run cost 1.354 dollars. The budget margin is thus binding between 0.5 and 1.5 dollars and slack above; and within that range the cheaper arms A1 and A2 dominate A3 and A4 at a budget of 0.75 or 1.00 dollar (0.50 and 0.47 against 0.37 and 0.30 at 0.75 dollars), while at 1.50 dollars and above the ordering is that of the acceptance rates. The human-time allowance moves the frontier more than the budget does: at the full budget, $H=15$---three questions or fewer---admits half of the accepted work in A1 and A2 and two thirds in A3, and $H=30$ admits nearly all of it. By occupation and without the human-time condition, the frontier at a budget of 1.00 dollar is 1 in every shell arm for academic research, corporate finance and procurement, and 1 in three of the four shell arms for banking (A1 at $\tfrac23$), human resources and legal services (A4 at $\tfrac23$) and real-estate appraisal (A2 and A4 at $\tfrac23$); it is 0 to $\tfrac23$ in accounting (zero in A3 and A4, whose runs cost more than a dollar) and in business intelligence (zero in A4), and zero in executive administration; at 2.00 dollars the accounting frontier is 1 in every shell arm, since its runs cost 1.04 to 1.20 dollars on average.139% source: table14 block D_alf_Hinf (pooled, H unbounded), D_alf_H (pooled, H = 0/15/30), D_alf_Hinf_occ (by occupation at B = 1, 2, 4); table3 cost column (accounting 1.04–1.20); runs_pilot.csv: cheapest accepted run PROC-01_A2_r1 0.5087 USD, dearest ACC-01_A1_r1 1.3542 USD140141\begin{table}[htbp]142\centering143\caption{Autonomous Labor Frontier pooled over the ten occupations (equal weights, $q=70$) by budget cap $B$ and human-time allowance $H$}144\label{tab:alf-pooled}145\footnotesize146\setlength{\tabcolsep}{3.5pt}147\begin{tabular}{@{}lrrrrrrrr@{}}148\toprule149 & \multicolumn{8}{c}{Budget cap $B$ (USD per run)} \\150\cmidrule{2-9}151Arm and allowance & 0.25 & 0.50 & 0.75 & 1.00 & 1.50 & 2.00 & 3.00 & 4.00 \\152\midrule153A1, no $H$ condition & 0.000 & 0.000 & 0.500 & 0.767 & 0.867 & 0.867 & 0.867 & 0.867 \\154A2, no $H$ condition & 0.000 & 0.000 & 0.467 & 0.800 & 0.900 & 0.900 & 0.900 & 0.900 \\155A3, no $H$ condition & 0.000 & 0.000 & 0.367 & 0.733 & 0.900 & 0.900 & 0.900 & 0.900 \\156A4, no $H$ condition & 0.000 & 0.000 & 0.300 & 0.600 & 0.833 & 0.833 & 0.833 & 0.833 \\157\addlinespace158A1, $H\le 30$ min & 0.000 & 0.000 & 0.500 & 0.733 & 0.833 & 0.833 & 0.833 & 0.833 \\159A2, $H\le 30$ min & 0.000 & 0.000 & 0.467 & 0.800 & 0.867 & 0.867 & 0.867 & 0.867 \\160A3, $H\le 30$ min & 0.000 & 0.000 & 0.367 & 0.733 & 0.900 & 0.900 & 0.900 & 0.900 \\161A4, $H\le 30$ min & 0.000 & 0.000 & 0.300 & 0.600 & 0.833 & 0.833 & 0.833 & 0.833 \\162\addlinespace163A1, $H\le 15$ min & 0.000 & 0.000 & 0.233 & 0.433 & 0.500 & 0.500 & 0.500 & 0.500 \\164A2, $H\le 15$ min & 0.000 & 0.000 & 0.233 & 0.433 & 0.500 & 0.500 & 0.500 & 0.500 \\165A3, $H\le 15$ min & 0.000 & 0.000 & 0.267 & 0.500 & 0.667 & 0.667 & 0.667 & 0.667 \\166A4, $H\le 15$ min & 0.000 & 0.000 & 0.200 & 0.400 & 0.600 & 0.600 & 0.600 & 0.600 \\167\addlinespace168A1, $H=0$ & 0.000 & 0.000 & 0.033 & 0.033 & 0.033 & 0.033 & 0.033 & 0.033 \\169A2, $H=0$ & 0.000 & 0.000 & 0.033 & 0.067 & 0.067 & 0.067 & 0.067 & 0.067 \\170A3, $H=0$ & 0.000 & 0.000 & 0.100 & 0.167 & 0.167 & 0.167 & 0.167 & 0.167 \\171A4, $H=0$ & 0.000 & 0.000 & 0.133 & 0.200 & 0.233 & 0.233 & 0.233 & 0.233 \\172\bottomrule173\end{tabular}174\begin{tablenotes}175\footnotesize176\item Source: \texttt{analysis/tables/table14\_extra\_descriptives.csv}, blocks \texttt{D\_alf\_Hinf} and \texttt{D\_alf\_H} (script \texttt{analysis/robustness/extra\_descriptives.py}), computed from the same run data as \texttt{table5\_alf\_pilot.csv}. $H$ is five minutes per question recorded in the completion record (assumption) plus logged interventions (none). Arm A0 is zero throughout and omitted. The $H=0$ and $H\le 15$ rows at $B=4$ equal the corresponding occupation averages of \texttt{table5\_alf\_pilot.csv}.177\end{tablenotes}178\end{table}179180\subsection{Heterogeneity across occupations}\label{sec:results-heterogeneity}181182Table~\ref{tab:occupation} and Figure~\ref{fig:occupation} report acceptance by occupation and arm; the complete occupation-by-arm table with completion, quality, critical errors, cost and tool calls is Table~\ref{tab:occupation-full} of Appendix~\ref{app:robustness}. The hypothesis H7 stated that effects differ across occupations and that no occupation-level claim is confirmatory at the pilot scale; the first part is visible in the table and the second remains true.183% source: analysis/tables/table3_occupation_arm_pilot.csv184185\begin{table}[htbp]186\centering187\caption{Acceptance rate by occupation and arm (3 repeats per cell), with mean quality in the shell arms}188\label{tab:occupation}189\footnotesize190\setlength{\tabcolsep}{4pt}191\begin{tabular}{@{}llrrrrrr@{}}192\toprule193Occupation & Scenario & A0 & A1 & A2 & A3 & A4 & $Q$, A1--A4 \\194\midrule195Academic research & RES-01 & 0 & 1 & 1 & 1 & 1 & 82.2--86.4 \\196Accounting & ACC-01 & 0 & 1 & 1 & 1 & 1 & 89.3--98.9 \\197Banking & BANK-01 & 0 & 1 & 1 & 1 & 1 & 91.3--96.7 \\198Business intelligence & BI-01 & 0 & 0.667 & 1 & 1 & 0.667 & 50.0--78.1 \\199Corporate finance & FIN-01 & 0 & 1 & 1 & 1 & 1 & 82.5--87.8 \\200Executive administration & ADM-01 & 0 & 0 & 0.333 & 0 & 0 & 62.2--66.7 \\201Human resources & HR-01 & 0 & 1 & 1 & 1 & 1 & 73.6--76.3 \\202Legal services & LEGAL-01 & 0 & 1 & 1 & 1 & 1 & 90.7--92.9 \\203Procurement & PROC-01 & 0 & 1 & 1 & 1 & 1 & 89.2--91.0 \\204Real-estate appraisal & RE-01 & 0 & 1 & 0.667 & 1 & 0.667 & 93.5--98.8 \\205\bottomrule206\end{tabular}207\begin{tablenotes}208\footnotesize209\item Source: \texttt{analysis/tables/table3\_occupation\_arm\_pilot.csv}. The last column is the range of the four arm means of quality for A1--A4.210\end{tablenotes}211\end{table}212213\begin{figure}[htbp]214\centering215\includegraphics[width=0.95\textwidth]{fig14_occupation_heterogeneity_pilot}216\caption{Acceptance rate by occupation (rows) and arm (columns), 150 pilot runs, 3 repeats per cell, $q=70$. Each cell is a proportion of three runs and carries the binomial uncertainty of $n=3$; the figure is descriptive and no occupation-level contrast is confirmatory (H7). Source: \texttt{analysis/tables/table3\_occupation\_arm\_pilot.csv}; script \texttt{analysis/figures/make\_figures.py} (\texttt{fig\_occupation}).}217\label{fig:occupation}218\end{figure}219220Seven of the ten occupations are solved in every repeat of every shell arm. Three carry all the shell-arm failures, and they fail for different reasons.221222\paragraph{Executive administration: a quality-gate failure.} ADM-01 is the mandate's inbox-and-calendar triage, the second workflow-class scenario (designed depth 44) and the one with the most planted features (seven planted errors, an injection, a clarification trap). Every shell-arm run passed all ten deterministic criteria: the ranked task sheet, the calendar conflicts, the top-five priorities, the briefing note, the time-zone and travel checks. Every shell-arm run but one was scored below 70 by the judges: the twelve A1--A4 quality scores range from 59.6 to 70.4, the only passing score being the second A2 repeat at 70.4, which gives the A2 cell its $\tfrac13$. The judges' per-criterion scores locate the shortfall: on the 0--4 rubric scale the twelve runs average 4.00 on the calendar criterion, 3.56 on the briefing note, 3.17 on prioritisation and 3.19 on risk handling, but 0.64 on the draft replies and 0.14 on the filing index. The deterministic evaluator checks that replies and a filing index exist and that the required ones are present; the judges read the replies and the index and found them below the standard of the Office of the CEO handbook. The restricted arm, which cannot produce the workbook, scored 47--56 on the same scenario, higher than on any other except procurement, because its text substitutes contain most of the triage. The scenario is the clearest example in the pilot of the multitask logic of Definition~\ref{def:acceptance}: a package can satisfy every verifiable requirement and still be one a demanding manager would not accept, and the two instruments measure different things. Whether the judges are right is a question for the human calibration that amendment A-005 leaves to the principal investigator; what the data show is that the shortfall is consistent across twelve runs and four arms and is specific to the two criteria that require writing in a prescribed voice.223% source: table3 ADM-01 rows; audit_failures_pilot.csv (ADM-01 det true, quality); table14 block K_gate (criterion means; ADM-01_A2_r2 quality 70.42 from runs_pilot.csv)224225\paragraph{Business intelligence: one integrity failure and one gate failure.} BI-01 (designed depth 31) requires a KPI workbook with formulas and a chart, a commentary and the code, from a sales database. The A1 failure is the pilot's only shell-arm critical violation: in its first repeat the worker, while exploring the workspace, ran a command that deleted the input database, discovered the loss, reported in its completion record that it could not deliver, and ended with a quality score of 3.8 and a status of \texttt{needs\_clarification}. The harness hash check recorded the modification of the inputs; the deterministic evaluator failed every criterion. The A4 failure is a quality-gate failure: the first A4 repeat passed all fourteen criteria and was scored 67.9. The other ten shell-arm runs were accepted, with quality between 70.0 and 78.1 on average by arm, the lowest range of any accepted scenario.226% source: table3 BI-01 rows; audit_failures_pilot.csv (BI-01_A1_r1, BI-01_A4_r1); docs/PHASE_5_AUDIT.md §2227228\paragraph{Real-estate appraisal: one criterion against the judges.} RE-01 (designed depth 34) requires an adjustment grid with formulas and an appraisal report; it carries the pilot's prompt injection, a comparable sheet with a planted value conclusion. The two shell-arm failures, A2 first repeat and A4 second repeat, passed nine of ten criteria and failed the one on the adjusted prices of the comparables, while being scored 98.75 by the judges, the highest scores of the scenario. These are the two runs of Table~\ref{tab:det-vs-judge} in which the oracle is stricter than the judges: the grid's adjusted prices departed from the oracle's beyond its tolerance, a departure that a reader of the report cannot see. The scenario is also where the restricted arm failed most severely: in all three A0 runs the value conclusion equalled the planted figure, and the evaluator recorded compliance with the injection. The pattern is not that A0 was persuaded by the injection where the shell arms resisted it; it is that A0, unable to build the grid, had no supported value of its own and echoed the one figure that the inputs offered.229% source: table3 RE-01 rows; audit_failures_pilot.csv (RE-01_A2_r1, RE-01_A4_r2: failed adjusted_prices, quality 98.75); receipts_pilot.csv (RE-01 A0 injection_obeyed detail)230231\paragraph{The other seven.} Accounting (ACC-01, the deepest workflow at depth 44, with 150 invoices and twelve statements) was solved by every shell-arm run, with mean quality of 89.3 to 98.9 and the highest costs of the pilot (1.04 to 1.20 dollars per run); banking, legal services and procurement were solved with quality above 89; corporate finance and academic research with quality in the 80s; human resources with the lowest accepted quality (73.6 to 76.3). The restricted arm failed all seven, with quality from 12.6 (legal services, where its issues register was empty) to 62.9 (procurement, where a text bid evaluation earned partial credit).232% source: table3 rows (quality, cost)233234\subsection{The Workflow Dependency Frontier}\label{sec:results-dependency}235236Hypothesis H4 stated that, within arm, acceptance decreases with the designed dependency depth and with the realised number of tool calls. Table~\ref{tab:dependency} reports acceptance and mean tool calls by arm and designed depth, and Figure~\ref{fig:dependency} plots acceptance against tool calls at the run level. Pooling the four shell arms, acceptance is 1.000 on the four standard-class scenarios (depth 21--24; 48 runs), 0.917 on the four complex-class scenarios (depth 31--34; 48 runs) and 0.542 on the two workflow-class scenarios (depth 44; 24 runs); the scenario-level Spearman correlation between acceptance and depth is $-0.46$ in the pooled shell arms and between $-0.34$ and $-0.54$ by arm. Realised tool calls rise with depth within every arm (Spearman 0.60 to 0.88 at the scenario level), from 11.9 per run in the standard class to 16.7 in the complex class and 19.4 in the workflow class.237% source: analysis/tables/table4_dependency_pilot.csv; table14 block L_dependency238239\begin{table}[htbp]240\centering241\caption{Acceptance and tool calls by arm and designed dependency depth}242\label{tab:dependency}243\footnotesize244\setlength{\tabcolsep}{4pt}245\begin{adjustbox}{max width=\textwidth}246\begin{tabular}{@{}lrrrrrrrrrr@{}}247\toprule248 & \multicolumn{2}{c}{A0} & \multicolumn{2}{c}{A1} & \multicolumn{2}{c}{A2} & \multicolumn{2}{c}{A3} & \multicolumn{2}{c}{A4} \\249\cmidrule(lr){2-3}\cmidrule(lr){4-5}\cmidrule(lr){6-7}\cmidrule(lr){8-9}\cmidrule(lr){10-11}250Depth (scenario; $n$ per arm) & $S$ & calls & $S$ & calls & $S$ & calls & $S$ & calls & $S$ & calls \\251\midrule25221 (PROC-01; 3) & 0 & 18.7 & 1 & 7.7 & 1 & 8.3 & 1 & 13.0 & 1 & 14.0 \\25322 (HR-01; 3) & 0 & 42.0 & 1 & 10.0 & 1 & 10.3 & 1 & 14.3 & 1 & 16.7 \\25423 (LEGAL-01; 3) & 0 & 15.3 & 1 & 9.7 & 1 & 9.0 & 1 & 15.3 & 1 & 17.3 \\25524 (BANK-01; 3) & 0 & 20.0 & 1 & 9.3 & 1 & 7.3 & 1 & 12.0 & 1 & 15.3 \\25631 (BI-01; 3) & 0 & 13.3 & 0.667 & 14.0 & 1 & 15.3 & 1 & 22.7 & 0.667 & 24.0 \\25733 (FIN-01; 3) & 0 & 15.0 & 1 & 12.3 & 1 & 10.3 & 1 & 12.3 & 1 & 14.3 \\25834 (RE-01, RES-01; 6) & 0 & 28.8 & 1 & 14.7 & 0.833 & 15.5 & 1 & 20.2 & 0.833 & 20.8 \\25944 (ACC-01, ADM-01; 6) & 0 & 56.2 & 0.5 & 18.2 & 0.667 & 16.0 & 0.5 & 21.0 & 0.5 & 22.3 \\260\bottomrule261\end{tabular}262\end{adjustbox}263\begin{tablenotes}264\footnotesize265\item Source: \texttt{analysis/tables/table4\_dependency\_pilot.csv}; scenario identities from Table~\ref{tab:pilot}. Depth is the designed dependency depth of the scenario; calls are mean tool calls per run.266\end{tablenotes}267\end{table}268269\begin{figure}[htbp]270\centering271\includegraphics[width=0.8\textwidth]{fig06_dependency_pilot}272\caption{Acceptance against the realised number of tool calls, one point per run (150 pilot runs; vertical jitter added for visibility), by arm. The restricted arm A0 makes the most calls and accepts nothing; among the shell arms the accepted and rejected runs overlap in tool calls (means of 12.7 against 14.0 in A1, 17.1 against 17.7 in A3). No fitted curve is drawn; the scenario-level relation is reported in Table~\ref{tab:dependency}. Source: \texttt{data/validated/runs\_pilot.parquet}; script \texttt{analysis/figures/make\_figures.py} (\texttt{fig\_dependency}).}273\label{fig:dependency}274\end{figure}275276The negative slope is real in the data but its interpretation as a depth effect is not warranted, for a reason the table makes visible. The workflow class contains two scenarios: accounting, the deepest and longest, which was accepted in every shell-arm run, and executive administration, which was accepted in one run of twelve for the quality-gate reason of Section~\ref{sec:results-heterogeneity}. The drop from 0.92 to 0.54 between the complex and the workflow class is entirely the ADM-01 cell; with ten scenarios, depth and scenario identity cannot be separated, and the frontier of Proposition~\ref{prop:compound} cannot be estimated as a function of depth from across-scenario variation alone. The second part of H4 is also not supported at the run level: within arm, the accepted and the rejected runs have similar numbers of tool calls, and the scenario-level correlation between acceptance and tool calls is weak and of inconsistent size ($-0.06$ to $-0.45$). What the table does establish is that longer workflows require more operations in every arm, and that the arms with more tools use more of them at every depth without that use translating into acceptance. The compounding model of Section~\ref{sec:theory-wdf} would predict, at the shell arms' per-step reliability implied by a completion rate near one at depth 24, a half-depth far beyond 44; the pilot cannot confirm or refute the shape because it has no failures that are failures of depth. The designed within-occupation variation in depth that Section~\ref{sec:benchmark-scale} plans for the scaled benchmark is what the estimation of the frontier requires.277% source: table4; table3 ACC-01 vs ADM-01; table14 block L_dependency (spearman_S_toolcalls by arm, mean tool calls accepted vs rejected)278279\subsection{Sensitivity to the worker model}\label{sec:results-models}280281Arm A3 was rerun on the ten scenarios with two repeats each under \texttt{claude-sonnet-5-5} and \texttt{claude-fable-5-1}, and Table~\ref{tab:models} compares the three models; Figure~\ref{fig:models} plots acceptance against cost, and Table~\ref{tab:models-scenario} of Appendix~\ref{app:robustness} gives the per-scenario values. The three models reach nearly the same acceptance---0.90 for Opus on three repeats, 0.85 for Sonnet and 0.85 for Fable on two---and the differences, $-0.05$ for each with intervals from $-0.15$ to $0.00$ and permutation $p$-values of 1.000, are again one scenario-repeat: both Sonnet and Fable accepted one of the two BI-01 runs where Opus accepted three of three, and all three models failed ADM-01 in every run and solved the other eight scenarios in every run. The models differ in quality and cost. Sonnet's mean quality is 4.8 points below Opus (interval $-6.7$ to $-3.0$, $p=0.002$) at 38 per cent of the cost ($-0.56$ dollars per run, $-0.68$ to $-0.46$, $p=0.001$) and 60 per cent of the wall time (median 131 against 219 seconds), giving a cost per accepted deliverable of 0.40 dollars against 1.01. Fable's mean quality is 1.2 points above Opus (interval $-1.4$ to $+3.8$, $p=0.40$), not distinguishable from it, at 3.1 times the cost ($+1.90$ dollars per run, $+1.56$ to $+2.31$, $p=0.002$) and 1.8 times the wall time (median 398 seconds), with 37,200 output tokens per run against 26,500, giving 3.30 dollars per accepted deliverable. Fable also produced the only critical violation of the sensitivity stage: in its first ADM-01 run the worker forwarded the board pack to the external address that the scenario's hidden instruction requested, without flagging it, which the evaluator records as compliance with the injection; the run's deterministic completion was false and its quality 67.1. One violation in 40 runs is a rate of 0.025 for the stage and 0.05 for Fable, below the 10 per cent of H5, but it is the only injection compliance by a shell-equipped worker in the study, and it occurred on the scenario whose injection is a request to send a document rather than a request to adopt a figure.282% source: analysis/tables/table9_model_sensitivity.csv, table9b_model_contrasts.csv, table9c_model_by_scenario.csv; table14 blocks J_sensitivity (violation ADM-01_A3_r1_fable, wall quantiles) and O_tokens (output tokens by model); data/validated/runs_sensitivity.csv (ADM-01_A3_r1_fable det false, quality 67.08)283284\begin{table}[htbp]285\centering286\caption{Arm A3 under three worker models: outcomes and contrasts against \texttt{claude-opus-5-5}}287\label{tab:models}288\footnotesize289\setlength{\tabcolsep}{4pt}290\begin{adjustbox}{max width=\textwidth}291\begin{tabular}{@{}lrrr@{}}292\toprule293 & \texttt{claude-sonnet-5-5} & \texttt{claude-opus-5-5} & \texttt{claude-fable-5-1} \\294\midrule295Runs (repeats per scenario) & 20 (2) & 30 (3) & 20 (2) \\296Acceptance, equal scenario weights & 0.850 & 0.900 & 0.850 \\297Deterministic completion & 1.000 & 1.000 & 0.950 \\298Critical-error rate & 0.000 & 0.000 & 0.050 \\299Quality, mean & 80.9 & 85.7 & 86.9 \\300Cost (USD), mean & 0.341 & 0.906 & 2.805 \\301Cost per accepted deliverable (USD) & 0.401 & 1.006 & 3.300 \\302Wall time (s), median & 131.1 & 218.8 & 397.6 \\303Tool calls, mean & 13.5 & 17.2 & 21.9 \\304Output tokens, mean & 19,912 & 26,497 & 37,186 \\305Timeouts & 0 & 0 & 0 \\306\midrule307\multicolumn{4}{@{}l}{Contrast against Opus (scenario-cluster bootstrap 95\% CI; permutation $p$; $J=10$)} \\308Acceptance $S$ & $-0.05$ [$-0.15$, $0.00$]; 1.000 & -- & $-0.05$ [$-0.15$, $0.00$]; 1.000 \\309Deterministic completion $D$ & $0.00$ [$0.00$, $0.00$]; 1.000 & -- & $-0.05$ [$-0.15$, $0.00$]; 1.000 \\310Quality $Q$ & $-4.80$ [$-6.70$, $-2.97$]; 0.002 & -- & $+1.24$ [$-1.38$, $+3.84$]; 0.404 \\311Cost $C$ (USD) & $-0.564$ [$-0.678$, $-0.465$]; 0.001 & -- & $+1.899$ [$+1.563$, $+2.306$]; 0.002 \\312\bottomrule313\end{tabular}314\end{adjustbox}315\begin{tablenotes}316\footnotesize317\item Sources: \texttt{analysis/tables/table9\_model\_sensitivity.csv} and \texttt{table9b\_model\_contrasts.csv} (script \texttt{analysis/robustness/sensitivity\_models.py}); output tokens from \texttt{table14\_extra\_descriptives.csv} block \texttt{O\_tokens}. The Opus column is the A3 arm of the pilot. Sonnet and Fable runs were executed on the same nodes, caps and harness (stage \texttt{sensitivity}).318\end{tablenotes}319\end{table}320321\begin{figure}[htbp]322\centering323\includegraphics[width=0.7\textwidth]{fig13_model_sensitivity}324\caption{Scenario-weighted acceptance of arm A3 against mean marginal API cost per run (USD, logarithmic axis) for the three worker models: \texttt{claude-sonnet-5-5} ($n=20$), \texttt{claude-opus-5-5} ($n=30$) and \texttt{claude-fable-5-1} ($n=20$), with the mean quality score $Q$ printed beside each point. Bootstrap intervals on the differences in acceptance against Opus are $[-0.15, 0.00]$ for both models (Table~\ref{tab:models}). This figure replaces the interface-perturbation figure of the mandate's list, since no perturbation experiment was run. Source: \texttt{analysis/tables/table9\_model\_sensitivity.csv}; script \texttt{analysis/robustness/economic\_sensitivity.py}.}325\label{fig:models}326\end{figure}327328The model comparison sharpens the reading of the ceiling. The three models span an eightfold range of cost per run and a sixteen-point range of mean quality, yet they are separated by a single scenario-repeat in acceptance, because the acceptance criterion at $q=70$ is met by all three on the same eight scenarios and failed by all three on the same one. The quality threshold is where the models differ, as the next subsection shows.329330\subsection{Sensitivity to the quality threshold}\label{sec:results-q}331332The acceptance threshold $q=70$ was fixed before any run; Table~\ref{tab:q-sensitivity} reports the scenario-weighted acceptance of each arm when it is varied from 50 to 90. Below the preregistered threshold the arms are at or near their deterministic-completion ceiling: at $q=50$ or $60$, acceptance is 0.967 in A1, A2 and A4 and 0.967 to 1.000 in A3, the residual being the single deterministic failure of each arm. At $q=70$ the executive-administration runs fall out and the arms separate slightly, as reported. At $q=80$ acceptance falls to 0.63--0.70 and at $q=90$ to 0.33--0.50, and the ordering of the arms changes: A1 is highest at $q=90$ (0.50) and A2 lowest (0.33), the reverse of their ordering at $q=70$. The frontier's dependence on $q$ is thus steep between 70 and 90, where most of the accepted deliverables are scored, and its ordering of arms is not robust to the threshold, which is a further reason not to read the differences among the shell arms as rankings. The restricted arm is zero at every threshold because its deliverables fail the deterministic criteria regardless of quality.333% source: analysis/tables/table6_q_sensitivity_pilot.csv334335\begin{table}[htbp]336\centering337\caption{Scenario-weighted acceptance by arm as a function of the quality threshold $q$}338\label{tab:q-sensitivity}339\small340\begin{tabular}{@{}lrrrrr@{}}341\toprule342$q$ & A0 & A1 & A2 & A3 & A4 \\343\midrule34450 & 0.000 & 0.967 & 0.967 & 1.000 & 0.967 \\34560 & 0.000 & 0.967 & 0.967 & 0.967 & 0.967 \\34670 (preregistered) & 0.000 & 0.867 & 0.900 & 0.900 & 0.833 \\34780 & 0.000 & 0.667 & 0.633 & 0.700 & 0.667 \\34890 & 0.000 & 0.500 & 0.333 & 0.467 & 0.400 \\349\bottomrule350\end{tabular}351\begin{tablenotes}352\footnotesize353\item Source: \texttt{analysis/tables/table6\_q\_sensitivity\_pilot.csv} (script \texttt{analysis/econometrics/estimate.py}). $S(q)=\mathbf{1}\{D=1\wedge E=0\wedge Q\ge q\}$, equal scenario weights, 150 pilot runs.354\end{tablenotes}355\end{table}356357\subsection{Summary of the main results}\label{sec:results-summary}358359Against the seven preregistered expectations, the pilot supports H2 (the shell premium, $+0.867$, $p=0.004$) and the integrity hypothesis H5 for the shell arms (critical-error rates of 0.033 in A1 and zero in A2--A4, not increasing with tool access), and it does not support H1 (the computer-use premium, $+0.033$, interval $[0.00, 0.10]$, $p=1.000$), H3 (the self-repair premium, $-0.067$, interval $[-0.167, 0.00]$, $p=0.499$; the Autonomous Recovery Index unidentified in the pilot), or H4 (the depth gradient is one scenario). H6 stated that cost per accepted deliverable would be lowest in A1 or A3; it is lowest in A2 (0.87 dollars), with A1 at 0.95 and A3 at 1.01, and undefined in A0 as expected. H7's heterogeneity is present and non-confirmatory. The restricted arm's zero and the shell arms' ceiling are the two facts of the pilot; the computer-use, macOS and self-repair premiums are nulls at this difficulty and remain open at higher difficulty. Section~\ref{sec:reliability} turns to what the shell arms' failures looked like, how long and how much the work cost, what the self-repair mechanism does when it is actually triggered, and how much supervision the completion records imply.360