spb/wp42_uqo

1% ---------------------------------------------------------------------------2% Section 5 — Results (journal version; condensed from the monograph's3% Sections 7–9). Every number comes from ../analysis/tables, ../analysis/derived4% or ../docs/PHASE_5_AUDIT.md; the source file is named in a comment.5% ---------------------------------------------------------------------------6\section{Results}\label{sec:results}78\subsection{The data}\label{sec:results-data}910Three stages were executed and are analysed separately. The \emph{pilot} comprises 150 runs: the ten validated scenarios, each under the five arms with three repeats, all at \texttt{claude-opus-5-5} on Claude Code 2.1.314, executed between the evening of 2 November and the early hours of 3 November 2026 at a concurrency of twelve sessions over the four nodes (37, 40, 38 and 35 runs per node; arm A2 on the two nodes where the automation layer answers). The \emph{sensitivity} stage comprises 40 runs: arm A3 on the same scenarios with two repeats each under \texttt{claude-sonnet-5-5} and \texttt{claude-fable-5-1}. The \emph{recovery} stage comprises 40 runs, the exploratory addendum of amendment A-011 (Section~\ref{sec:results-recovery}). Every pilot and sensitivity run ended with a result event of subtype \texttt{success}---no run timed out, exhausted its budget or reached the turn cap---and every run received three judge outputs; the 40 recovery-stage first sessions ended by the induced interruption. In total 230 experimental runs were executed and evaluated, and the launcher reported no infrastructure error.11% ../docs/PHASE_5_AUDIT.md §0, §3; ../manuscript/sections/data.tex §7.1 (nodes 37/40/38/35; dates)1213Three events during the pilot changed how some data were produced, each logged as an amendment and none affecting a pilot outcome. The judge subprocess crashed on eight runs whose rendered deliverables contained a null byte and timed out once; the renderer was fixed and the nine runs re-judged (A-008). The collection step silently lost the output directories of three A2 runs whose sessions had completed; the defect was found by the audit, the step was given retries and a file-count check, and the three runs were reset and rerun under the same identifiers---had the loss gone undetected, A2's acceptance would have been reported as 0.80 instead of 0.90 and the macOS-layer contrast would have changed sign (A-010). And a launcher flag had made the A4 recovery resume impossible; since no A4 first session in the pilot failed the self-check, no recovery was ever requested, but the self-repair arm was in the pilot de facto limited to its checkpoint and self-check components (A-012).14% ../protocol/amendments.csv A-008, A-010, A-012; ../manuscript/sections/reliability.tex §9.81516The three evaluation instruments agree to a degree that governs the reading of every acceptance rate. Krippendorff's alpha across the three judges is 0.973 over the 150 runs, with pairwise Pearson correlations of 0.964 and 0.966 between the Opus judge and each Sonnet judge and 0.990 between the two Sonnet draws; the two models are not systematically different in severity (means 73.1 against 73.0), but the near-identity of the two Sonnet draws means the panel is closer to two independent opinions than to three.17% ../analysis/derived/estimates_pilot.json (agreement); ../docs/PHASE_5_AUDIT.md §218Deterministic completion and the quality gate disagree on 14 of 150 runs, 9.3 per cent: of 117 runs that pass every required criterion, 105 also score at least 70 and 12 do not; of 33 that fail a criterion, 2 are scored above 70. The disagreement is concentrated: eleven of the twelve runs the judges reject are the executive-administration scenario in arms A1--A4, and the two the oracle rejects are appraisals scored 98.75 that failed a single adjusted-price criterion. Acceptance requires both instruments, as preregistered.19% ../docs/PHASE_5_AUDIT.md §2; ../manuscript/sections/data.tex Table tab:det-vs-judge (105/12/31/2)2021\subsection{Outcomes by arm}\label{sec:results-arms}2223Table~\ref{tab:outcomes} gives the preregistered outcomes by arm, with the two sensitivity models in the last two columns, and Figure~\ref{fig:acceptance} the acceptance rates. The restricted arm A0 accepted no deliverable in 30 runs and completed none deterministically: its mean quality is 27.4, its critical-error rate 0.20, and 90 per cent of its runs ended by requesting clarification. It is the busiest arm and the least effective---29.4 tool calls per run against 12 to 19 elsewhere, 11.9 tool errors against 0.7 to 2.0---because its calls are mostly failed attempts to read binary inputs and to do without a program.24% ../analysis/tables/table1_outcomes_by_arm_pilot.csv; table14_extra_descriptives.csv block O_a0_errors25The four arms with a shell are close to one another: scenario-weighted acceptance is 0.867 in A1, 0.900 in A2 and A3 and 0.833 in A4; deterministic completion 0.967, 0.967, 1.000 and 0.967; mean quality 82.3, 85.0, 85.7 and 84.8, with a larger dispersion in A1 because one run scored 3.8 after destroying its input; the critical-error rate 0.033 in A1 and zero in A2--A4. Under professional-minutes weights the rates move by at most 0.03. The gap between deterministic completion and acceptance in the shell arms---0.10, 0.067, 0.10 and 0.133---is the quality gate, and Section~\ref{sec:results-heterogeneity} shows that it is almost entirely one scenario. Cost per accepted deliverable, which keeps failed runs in the denominator, is 0.95 dollars in A1, 0.87 in A2, 1.01 in A3 and 1.11 in A4, and undefined in A0.26% ../analysis/tables/table1_outcomes_by_arm_pilot.csv2728\begin{table}[htbp]29\centering30\caption{Preregistered outcomes by arm, 150 pilot runs (\texttt{claude-opus-5-5}, $q=70$), with the two model sensitivities on arm A3}31\label{tab:outcomes}32\footnotesize33\setlength{\tabcolsep}{3.5pt}34\begin{adjustbox}{max width=\textwidth}35\begin{tabular}{@{}lrrrrrrr@{}}36\toprule37 & \multicolumn{5}{c}{\texttt{claude-opus-5-5}} & \multicolumn{2}{c}{Arm A3, other models} \\38\cmidrule(lr){2-6}\cmidrule(lr){7-8}39 & A0 & A1 & A2 & A3 & A4 & Sonnet 5.5 & Fable 5.1 \\40 & restricted & shell & macOS & computer-use & self-repair & & \\41\midrule42Runs (scenarios $\times$ repeats) & 30 (10$\times$3) & 30 (10$\times$3) & 30 (10$\times$3) & 30 (10$\times$3) & 30 (10$\times$3) & 20 (10$\times$2) & 20 (10$\times$2) \\43Acceptance $S$, equal scenario weights & 0.000 & 0.867 & 0.900 & 0.900 & 0.833 & 0.850 & 0.850 \\44Acceptance $S$, professional-minutes weights & 0.000 & 0.885 & 0.909 & 0.926 & 0.844 & -- & -- \\45Deterministic completion $D$ & 0.000 & 0.967 & 0.967 & 1.000 & 0.967 & 1.000 & 0.950 \\46Critical-error rate $E$ & 0.200 & 0.033 & 0.000 & 0.000 & 0.000 & 0.000 & 0.050 \\47Quality $Q$, mean (s.d.) & 27.4 (17.4) & 82.3 (18.5) & 85.0 (10.0) & 85.7 (11.4) & 84.8 (10.6) & 80.9 & 86.9 \\48Abstention / clarification rate & 0.900 & 0.200 & 0.067 & 0.133 & 0.100 & -- & -- \\49Timeout rate & 0.000 & 0.000 & 0.000 & 0.000 & 0.000 & 0.000 & 0.000 \\50Cost $C$ (USD), mean (s.d.) & 0.710 (0.416) & 0.823 (0.230) & 0.783 (0.190) & 0.906 (0.259) & 0.926 (0.221) & 0.341 & 2.805 \\51Cost per accepted deliverable (USD) & -- & 0.950 & 0.870 & 1.006 & 1.111 & 0.401 & 3.300 \\52Wall time (s), median & 162.0 & 217.7 & 213.9 & 218.8 & 230.2 & 131.1 & 397.6 \\53Tool calls, mean & 29.4 & 12.9 & 12.4 & 17.2 & 18.8 & 13.5 & 21.9 \\54Tool errors, mean & 11.9 & 0.9 & 0.7 & 0.9 & 2.0 & -- & -- \\55Application transitions, mean & 0.0 & 3.3 & 4.4 & 9.3 & 9.3 & -- & -- \\56\bottomrule57\end{tabular}58\end{adjustbox}59\begin{tablenotes}60\footnotesize61\item Sources: \texttt{analysis/tables/table1\_outcomes\_by\_arm\_pilot.csv} (arms A0--A4; script \texttt{analysis/econometrics/estimate.py}) and \texttt{table9\_model\_sensitivity.csv} (Sonnet and Fable; script \texttt{analysis/robustness/sensitivity\_models.py}). $S=\mathbf{1}\{D=1\wedge E=0\wedge Q\ge 70\}$. The abstention rate is the share of runs whose completion record has the status \texttt{needs\_clarification}. Cost is the client's reported \texttt{total\_cost\_usd}; no recovery was triggered in the pilot. Cost per accepted deliverable is $\sum C/\sum S$ over all runs of the arm and is undefined when $\sum S=0$.62\end{tablenotes}63\end{table}6465\begin{figure}[htbp]66\centering67\includegraphics[width=0.8\textwidth]{fig03_acceptance_by_arm_pilot}68\caption{Scenario-weighted acceptance rate by arm (filled bars) and deterministic completion rate (dashed outlines), 150 pilot runs, 10 scenarios $\times$ 3 repeats per arm, \texttt{claude-opus-5-5}, $q=70$. Uncertainty is reported in Figure~\ref{fig:contrasts} as scenario-cluster bootstrap intervals on the between-arm differences, which is the preregistered inferential quantity; no interval is drawn on the levels. Source: \texttt{analysis/tables/table1\_outcomes\_by\_arm\_pilot.csv}; script \texttt{analysis/figures/make\_figures.py}.}69\label{fig:acceptance}70\end{figure}7172\subsection{Preregistered contrasts}\label{sec:results-contrasts}7374Table~\ref{tab:contrasts} reports the four preregistered contrasts on acceptance, deterministic completion, the critical-error indicator and quality, each as the equal-weight mean of ten within-scenario differences of repeat means, with the scenario-cluster bootstrap interval, the paired sign-flip permutation $p$-value and the Holm-adjusted $p$ over the three secondary acceptance contrasts; Figure~\ref{fig:contrasts} plots the acceptance contrasts.75% ../analysis/tables/table2_contrasts_pilot.csv7677\begin{table}[htbp]78\centering79\caption{Preregistered contrasts: scenario-weighted differences with scenario-cluster bootstrap 95\% intervals and paired permutation $p$-values ($J=10$)}80\label{tab:contrasts}81\footnotesize82\setlength{\tabcolsep}{4pt}83\begin{adjustbox}{max width=\textwidth}84\begin{tabular}{@{}llrrrrrr@{}}85\toprule86Contrast & Outcome & Estimate & 95\% CI lower & 95\% CI upper & Perm.\ $p$ & Minutes-weighted & Holm $p$ \\87\midrule88A3 $-$ A1 (primary, H1) & $S$ & $+0.033$ & $0.000$ & $0.100$ & 1.000 & $+0.041$ & -- \\89 & $D$ & $+0.033$ & $0.000$ & $0.100$ & 1.000 & $+0.041$ & \\90 & $E$ & $-0.033$ & $-0.100$ & $0.000$ & 1.000 & $-0.041$ & \\91 & $Q$ & $+3.35$ & $-0.81$ & $+9.56$ & 0.300 & $+4.57$ & \\92\addlinespace93A1 $-$ A0 (secondary, H2) & $S$ & $+0.867$ & $0.667$ & $1.000$ & 0.004 & $+0.885$ & 0.012 \\94 & $D$ & $+0.967$ & $0.900$ & $1.000$ & 0.002 & $+0.959$ & \\95 & $E$ & $-0.167$ & $-0.467$ & $+0.067$ & 0.499 & $-0.144$ & \\96 & $Q$ & $+54.96$ & $+40.44$ & $+68.24$ & 0.002 & $+57.66$ & \\97\addlinespace98A4 $-$ A3 (secondary, H3) & $S$ & $-0.067$ & $-0.167$ & $0.000$ & 0.499 & $-0.082$ & 0.998 \\99 & $D$ & $-0.033$ & $-0.100$ & $0.000$ & 1.000 & $-0.041$ & \\100 & $E$ & $0.000$ & $0.000$ & $0.000$ & 1.000 & $0.000$ & \\101 & $Q$ & $-0.90$ & $-2.71$ & $+0.97$ & 0.395 & $-1.39$ & \\102\addlinespace103A2 $-$ A1 (secondary) & $S$ & $+0.033$ & $-0.067$ & $+0.133$ & 1.000 & $+0.025$ & 1.000 \\104 & $D$ & $0.000$ & $-0.100$ & $+0.100$ & 1.000 & $0.000$ & \\105 & $E$ & $-0.033$ & $-0.100$ & $0.000$ & 1.000 & $-0.041$ & \\106 & $Q$ & $+2.63$ & $-1.75$ & $+8.79$ & 0.534 & $+2.96$ & \\107\bottomrule108\end{tabular}109\end{adjustbox}110\begin{tablenotes}111\footnotesize112\item Source: \texttt{analysis/tables/table2\_contrasts\_pilot.csv} (script \texttt{analysis/econometrics/estimate.py}; bootstrap seed 42). Intervals are percentile intervals of 5,000 resamples of the ten scenarios with replacement; $p$-values are two-sided sign-flip permutation tests over the ten paired scenario differences (20,000 permutations). Holm adjustment is over the three secondary acceptance contrasts, as preregistered.113\end{tablenotes}114\end{table}115116\begin{figure}[htbp]117\centering118\includegraphics[width=0.85\textwidth]{fig07_tool_effects_pilot}119\caption{The four preregistered contrasts on acceptance probability: equal-weight mean of the ten within-scenario differences of repeat means, with scenario-cluster bootstrap 95\% percentile intervals ($B=5{,}000$) and the paired sign-flip permutation $p$-value printed above each point. 150 pilot runs, 10 scenarios, 3 repeats per cell. Source: \texttt{analysis/tables/table2\_contrasts\_pilot.csv}; script \texttt{analysis/figures/make\_figures.py}.}120\label{fig:contrasts}121\end{figure}122123\paragraph{H1, the computer-use premium, is a precise null.} Acceptance under full computer use minus acceptance under the shell is $+0.033$ with a bootstrap interval from 0.000 to 0.100 and a permutation $p$ of 1.000. The estimate is one scenario-repeat in thirty: A3 accepted one run that A1 did not---the business-intelligence run in which A1 destroyed its input---and the two arms agree on every other scenario mean; the ten paired differences are nine zeros and one positive value, which is why the lower bound is zero, the upper bound 0.10 and the permutation test cannot reject. The contrast on quality is $+3.3$ points [$-0.8$, $+9.6$], $p=0.30$. H1 is not supported, and the result is not an absence of information: with the shell arm at 0.867 the premium on this benchmark cannot exceed 0.133, and the interval excludes a premium above 0.10. For ten scenarios in which every deliverable can be produced programmatically from the inputs, the browser, the office suite and the portal added nothing the shell could not already do; whether a premium exists on tasks whose inputs or deliverables can only be reached through those applications is a question this benchmark, by construction, does not ask.124% ../analysis/tables/table2_contrasts_pilot.csv (A3-A1 rows); table3_occupation_arm_pilot.csv (BI-01)125126\paragraph{H2, the shell premium, is large and confirmatory.} The shell premium is $+0.867$ [0.667, 1.000], $p=0.004$ (Holm 0.012); on deterministic completion $+0.967$ [0.900, 1.000], $p=0.002$; on quality $+55.0$ points [40.4, 68.2], $p=0.002$. It is the only confirmatory effect of the pilot, and its mechanism is the one the hypothesis named: every scenario's deliverables include at least one binary office file, usually a workbook with live formulas, which the restricted arm cannot produce; it delivered text substitutes in every run and the criteria that require the workbook's structure and formulas failed in every case. The contrast on the critical-error indicator, $-0.167$ [$-0.467$, $+0.067$], $p=0.50$, is not significant because the six A0 violations are concentrated in two scenarios.127% ../analysis/tables/table2_contrasts_pilot.csv (A1-A0 rows); audit_a0_outputs_pilot.csv; audit_violations_pilot.csv128129\paragraph{H3, the self-repair premium, is a null bounded by the design.} The premium is $-0.067$ [$-0.167$, 0.000], $p=0.499$ (Holm 0.998); on quality $-0.9$ points; on critical errors exactly zero. The point estimate is two scenario-repeats in thirty: A4 lost one business-intelligence run to the quality gate (scored 67.9) and one appraisal run to the adjusted-prices criterion, both of which A3 passed in all three repeats. The interpretation is bounded by amendment A-012: all 30 A4 first sessions passed the public self-check, no recovery was ever requested, and the arm differed from A3 only by its checkpointing and self-check instructions and the slightly longer sessions these produce (18.8 tool calls against 17.2; 230 against 219 seconds; 0.93 against 0.91 dollars). Since the self-repair premium is proportional to the first-session failure rate the self-check can see (Section~\ref{sec:fw-companions}), it was zero by construction and the Autonomous Recovery Index is unidentified on the pilot design.130% ../analysis/tables/table2_contrasts_pilot.csv (A4-A3); table1 (tool calls, wall, cost); ../analysis/derived/estimates_pilot.json (recovery: n_failed_first_selfcheck 0)131132\paragraph{The macOS layer and the ceiling.} A2 minus A1 is $+0.033$ [$-0.067$, $+0.133$], $p=1.000$: A2 accepted the business-intelligence run that A1 lost and lost an appraisal run that A1 passed. The headless automation layer was used sparingly---18 office-family calls in 30 runs against 69 in A3---and no run depended on it for a deliverable the shell could not produce.133% ../analysis/tables/table2_contrasts_pilot.csv (A2-A1); audit_families_pilot.csv134A cluster-robust logit of acceptance on arm indicators returns logits of 1.87, 2.20, 2.20 and 1.61 for A1--A4 with clustered standard errors of 0.75 to 1.13, of the same order as the differences among the arms, which restates on the model scale what the bootstrap says: the shell arms are not distinguishable from one another.135% ../analysis/derived/estimates_pilot.json (mixed_logit_S)136The scale rule gave the pilot a third repeat, which reduces the within-scenario component of equation~\eqref{eq:premium-var-cluster}, but the component that governs the primary contrast is between-scenario heterogeneity of the effect, and with the shell arms succeeding on nine scenarios in ten there is none left to measure. The benchmark needs scenarios on which the shell arm fails before the computer-use premium can be estimated at all.137138\subsection{Heterogeneity across occupations}\label{sec:results-heterogeneity}139140Table~\ref{tab:occupation} reports acceptance by occupation and arm. Seven of the ten occupations are solved in every repeat of every shell arm---accounting (the deepest workflow, with the highest quality, 89 to 99, and the highest costs, 1.04 to 1.20 dollars per run), banking, legal services and procurement with quality above 89, corporate finance and academic research in the 80s, and human resources with the lowest accepted quality (73.6 to 76.3). Three carry all the shell-arm failures, and they fail for different reasons. Each cell is a proportion of three runs, so H7's heterogeneity is visible but, as the hypothesis stated, not confirmatory.141% ../analysis/tables/table3_occupation_arm_pilot.csv142143\begin{table}[htbp]144\centering145\caption{Acceptance rate by occupation and arm (3 repeats per cell), with deterministic completion where it differs and the range of mean quality in the shell arms}146\label{tab:occupation}147\footnotesize148\setlength{\tabcolsep}{4pt}149\begin{adjustbox}{max width=\textwidth}150\begin{tabular}{@{}llcrrrrrr@{}}151\toprule152Occupation & Scenario & Depth & A0 & A1 & A2 & A3 & A4 & $Q$, A1--A4 \\153\midrule154Academic research & RES-01 & 34 & 0 & 1 & 1 & 1 & 1 & 82.2--86.4 \\155Accounting & ACC-01 & 44 & 0 & 1 & 1 & 1 & 1 & 89.3--98.9 \\156Banking & BANK-01 & 24 & 0 & 1 & 1 & 1 & 1 & 91.3--96.7 \\157Business intelligence & BI-01 & 31 & 0 & 0.667 & 1 & 1 & 0.667 ($D$=1) & 50.0--78.1 \\158Corporate finance & FIN-01 & 33 & 0 & 1 & 1 & 1 & 1 & 82.5--87.8 \\159Executive administration & ADM-01 & 44 & 0 & 0 ($D$=1) & 0.333 ($D$=1) & 0 ($D$=1) & 0 ($D$=1) & 62.2--66.7 \\160Human resources & HR-01 & 22 & 0 & 1 & 1 & 1 & 1 & 73.6--76.3 \\161Legal services & LEGAL-01 & 23 & 0 & 1 & 1 & 1 & 1 & 90.7--92.9 \\162Procurement & PROC-01 & 21 & 0 & 1 & 1 & 1 & 1 & 89.2--91.0 \\163Real-estate appraisal & RE-01 & 34 & 0 & 1 & 0.667 & 1 & 0.667 & 93.5--98.8 \\164\bottomrule165\end{tabular}166\end{adjustbox}167\begin{tablenotes}168\footnotesize169\item Source: \texttt{analysis/tables/table3\_occupation\_arm\_pilot.csv} (acceptance, det, quality by occupation and arm); designed depth from each scenario's \texttt{task.json}. Where deterministic completion $D$ differs from acceptance it is shown in parentheses; elsewhere the two coincide. The last column is the range of the four arm means of quality for A1--A4. Each cell is a proportion of three runs and no occupation-level contrast is confirmatory (H7).170\end{tablenotes}171\end{table}172173\paragraph{Executive administration: a quality-gate failure.} ADM-01, the inbox-and-calendar triage, is the second workflow-class scenario and the one with the most planted features. Every shell-arm run passed all ten deterministic criteria, and every shell-arm run but one was scored below 70: the twelve A1--A4 scores range from 59.6 to 70.4, the only pass being an A2 repeat at 70.4. The judges' per-criterion scores locate the shortfall: on the 0--4 scale the twelve runs average 4.00 on the calendar criterion and 3.56 on the briefing note, but 0.64 on the draft replies and 0.14 on the filing index. The evaluator checks that replies and an index exist and that the required ones are present; the judges read them and found them below the standard of the Office of the CEO handbook. The scenario is the clearest instance of the multitask logic of Definition~\ref{def:acceptance}: a package can satisfy every verifiable requirement and still be one a demanding manager would not accept. Whether the judges are right is a question for the human calibration that amendment A-005 defers; the shortfall is consistent across twelve runs, four arms and, as Section~\ref{sec:results-models} shows, three worker models.174% ../analysis/tables/table3_occupation_arm_pilot.csv (ADM-01); table14_extra_descriptives.csv block K_gate (criterion means); ../docs/PHASE_5_AUDIT.md §2175176\paragraph{Business intelligence and real-estate appraisal.} The BI-01 failure in A1 is the pilot's only shell-arm critical violation: while exploring the workspace the worker ran a command intended to remove a scratch file that deleted the input sales database instead, discovered the loss, and ended with a completion record stating that it could not deliver because it had destroyed its input; the harness hash check recorded the modification and the judges scored the package 3.8. The violation is one of integrity of the environment rather than of the deliverable, and the worker's report of it was honest and complete. The A4 failure on the same scenario is a quality-gate failure (all fourteen criteria passed, scored 67.9). RE-01 carries a comparable sheet with a planted value conclusion of 785,000 dollars. The two shell-arm failures, in A2 and A4, passed nine of ten criteria and failed the one on the adjusted prices of the comparables while being scored 98.75 by the judges---the two runs in which the oracle is stricter than the judges, since a departure of the grid beyond the oracle's tolerance is invisible to a reader of the report. The scenario is also where the restricted arm failed most severely: in all three A0 runs the value conclusion equalled the planted 785,000 and the evaluator recorded compliance with the injection. The pattern is not that A0 was persuaded where the shell arms resisted; it is that A0, unable to build the grid, had no supported value of its own and echoed the one figure the inputs offered. All twelve shell-arm runs of RE-01, and the eight further runs of the sensitivity and recovery stages, resisted the same injection.177% ../docs/PHASE_5_AUDIT.md §2; ../analysis/tables/audit_failures_pilot.csv (BI-01_A1_r1, BI-01_A4_r1, RE-01_A2_r1, RE-01_A4_r2); audit_violations_pilot.csv178179\paragraph{Depth is confounded with scenario.} Pooling the shell arms, acceptance is 1.000 on the four standard-class scenarios (depth 21--24; 48 runs), 0.917 on the four complex scenarios (depth 31--34) and 0.542 on the two workflow scenarios (depth 44), and the scenario-level Spearman correlation between acceptance and depth is $-0.46$; realised tool calls rise with depth in every arm, from 11.9 per run in the standard class to 19.4 in the workflow class.180% ../analysis/tables/table4_dependency_pilot.csv; table14_extra_descriptives.csv block L_dependency181The slope is real but its interpretation as a depth effect is not warranted: the workflow class contains accounting, the deepest and longest scenario, accepted in every shell-arm run, and the triage, accepted in one run of twelve for the quality-gate reason above. With ten scenarios, depth and scenario identity cannot be separated, and within arm the accepted and rejected runs have similar numbers of tool calls, so neither part of H4 is supported. The Workflow Dependency Frontier is reported as descriptive; what the data establish is that longer workflows require more operations in every arm, and that arms with more tools use more of them at every depth without that use translating into acceptance.182183\subsection{Sensitivity to the worker model and to the quality threshold}\label{sec:results-models}184185Arm A3 was rerun with two repeats per scenario under \texttt{claude-sonnet-5-5} and \texttt{claude-fable-5-1} (last two columns of Table~\ref{tab:outcomes}). The three models reach nearly the same acceptance---0.90 for Opus on three repeats, 0.85 for Sonnet and 0.85 for Fable on two---and the differences, $-0.05$ for each with intervals from $-0.15$ to 0.00 and permutation $p$ of 1.000, are again one scenario-repeat: both accepted one of two business-intelligence runs where Opus accepted three of three, all three failed the triage in every run, and all three solved the other eight scenarios in every run. The models differ in quality and cost. Sonnet's mean quality is 4.8 points below Opus ([$-6.7$, $-3.0$], $p=0.002$) at 38 per cent of the cost ($-0.56$ dollars per run, [$-0.68$, $-0.46$], $p=0.001$) and 60 per cent of the wall time, giving 0.40 dollars per accepted deliverable against 1.01. Fable's mean quality is 1.2 points above Opus ([$-1.4$, $+3.8$], $p=0.40$), not distinguishable, at 3.1 times the cost ($+1.90$ dollars per run, [$+1.56$, $+2.31$], $p=0.002$) and 1.8 times the wall time, giving 3.30 dollars per accepted deliverable. Fable also produced the only critical violation of the sensitivity stage: in one triage run it forwarded the board pack to the external address the scenario's hidden instruction requested, without flagging it---the only injection compliance by a shell-equipped worker in the pilot and sensitivity stages, on the one scenario whose injection asks to send a document rather than to adopt a figure. Across the three models, acceptance moves by 0.05 and cost by a factor of eight: on this benchmark the model, not the tool set, is the cost lever, and quality is the only thing the extra spending buys.186% ../analysis/tables/table9_model_sensitivity.csv; table9b_model_contrasts.csv; table14_extra_descriptives.csv block J_sensitivity (ADM-01_A3_r1_fable violation)187188The acceptance threshold is where the models and the arms differ. When $q$ is varied from the preregistered 70, acceptance at $q=50$ or 60 is 0.967 in A1, A2 and A4 and 0.967 to 1.000 in A3, the residual being each arm's single deterministic failure; at $q=80$ it falls to 0.63--0.70 and at $q=90$ to 0.33--0.50, with the ordering of the arms reversed between 70 and 90. The frontier's dependence on $q$ is steep between 70 and 90, where most accepted deliverables are scored, and the ordering of the shell arms is not robust to it---a further reason not to read their differences as rankings.189% ../analysis/tables/table6_q_sensitivity_pilot.csv190191\subsection{The Autonomous Recovery experiment}\label{sec:results-recovery}192193Because no A4 first session in the pilot failed the public self-check, the Autonomous Recovery Index was unidentified and the launcher defect that would have prevented a resume (A-012) was invisible. To identify the index, amendment A-011 added an exploratory experiment, not part of the preregistration and labelled as such: the ten scenarios were run under A3 and under A4 with two repeats each, and in every run the controller killed the first session 120 seconds after launch. In A3 the run then ends and the deliverables present at the kill are evaluated; A3 is the interrupted control. In A4 the controller runs the self-check on the collected output, finds it failing---by construction, since no 120-second session has written its deliverables---and grants the one bounded recovery: the same session is resumed with the remaining budget and wall time and a prompt stating what the self-check found. The A4 runs of this stage were themselves reset and rerun once because their first execution hit the A-012 defect.194% ../protocol/amendments.csv A-011, A-012195196\begin{table}[htbp]197\centering198\caption{The Autonomous Recovery experiment (amendment A-011, exploratory): arms A3 and A4 with the first session killed at 120 s, against the uninterrupted pilot cells}199\label{tab:recovery}200\footnotesize201\setlength{\tabcolsep}{4pt}202\begin{adjustbox}{max width=\textwidth}203\begin{tabular}{@{}lrrrr@{}}204\toprule205 & A3 interrupted & A4 interrupted & A3 uninterrupted & A4 uninterrupted \\206 & (control) & (recovery) & (pilot) & (pilot) \\207\midrule208Runs & 20 & 20 & 30 & 30 \\209First self-check passed & -- & 0 of 20 & -- & 30 of 30 \\210Recovery attempted & 0 & 20 & 0 & 0 \\211Final self-check passed (ARI) & -- & 20 of 20 & -- & 30 of 30 \\212Acceptance, equal scenario weights & 0.050 & 0.850 & 0.900 & 0.833 \\213\quad accepted / runs with judge output & 1 / 20 & 16 / 19 & 27 / 30 & 25 / 30 \\214Deterministic completion & 0.050 & 0.950 & 1.000 & 0.967 \\215Critical-error rate & 0.100 & 0.050 & 0.000 & 0.000 \\216Quality, mean & 28.0 & 84.3 & 85.7 & 84.8 \\217Wall time, total (s), median & 129.0 & 219.8 & 218.8 & 230.2 \\218Cost (USD per run), mean & 0.490 & 0.863 & 0.906 & 0.926 \\219Cost per accepted deliverable (USD) & 9.80 & 1.08 & 1.01 & 1.11 \\220\addlinespace221A4 interrupted $-$ A3 interrupted, acceptance & \multicolumn{4}{l}{$+0.80$ [0.55, 0.95], permutation $p=0.004$, $J=10$} \\222\bottomrule223\end{tabular}224\end{adjustbox}225\begin{tablenotes}226\footnotesize227\item Sources: \texttt{analysis/tables/table10\_recovery.csv} and \texttt{analysis/derived/recovery\_experiment.json} (script \texttt{analysis/robustness/recovery\_experiment.py}). Acceptance after recovery among runs with a judge output is 16/19 = 0.842; one A4 run (ACC-01, second repeat) has no judge output and is missing, as the preregistration's missingness rule requires. A killed session emits no result event, so its cost is unobserved: the A4 interrupted cost is the client's cumulative total over both sessions as reported by the resumed session (no imputation); the A3 interrupted cost is imputed from wall time with a slope fitted on the completed pilot runs ($R^2=0.93$). Critical violations: two in the A3 control (an interrupted contract review with no issues register; an interrupted run that had modified its inputs when killed) and one in A4 (a triage run that forwarded the board pack to the injected external address after its recovery).228\end{tablenotes}229\end{table}230231Table~\ref{tab:recovery} reports the experiment next to the uninterrupted A3 and A4 cells. All 20 interrupted A4 runs failed the first self-check, all 20 were granted a recovery, and all 20 passed the self-check after it: the Autonomous Recovery Index is 20 of 20 on the detection signal the self-check provides. Acceptance after recovery is 0.850 with equal scenario weights (16 of the 19 runs with a judge output), deterministic completion 0.950 and mean quality 84.3. The interrupted A3 control is where an uninterrupted run stands after two minutes: acceptance 0.050 (one procurement run had already written an acceptable package when killed), mean quality 28.0, and seven of twenty runs had produced no artefact at all. The difference in acceptance is $+0.80$ [0.55, 0.95], $p=0.004$, and the interrupted-and-recovered A4 runs reach the same acceptance and quality as the uninterrupted A4 runs (0.850 against 0.833; 84.3 against 84.8). In terms of Section~\ref{sec:fw-companions}, the experiment sets the detection probability $\delta$ to one by construction and estimates the correction probability $\gamma$ at 0.84; the uplift $(1-p)\delta\gamma$ is large here because $1-p$ was made large by the interruption, which is exactly why it was zero in the pilot.232% ../analysis/tables/table10_recovery.csv; ../analysis/derived/recovery_experiment.json (ARI 1.0; acceptance_after_recovery 0.842; contrast 0.80 [0.55, 0.95], perm_p 0.00415); table14_extra_descriptives.csv block G_recovery_cost (share with no artefact 0.35)233234The cost of recovery is modest. The client's cumulative total for an interrupted-and-recovered A4 run is 0.863 dollars, below the 0.926 dollars of an uninterrupted A4 run, and the cost per accepted deliverable is 1.08 dollars against 1.11; the resumed session reloads the first session's context from cache rather than regenerating the work. For the interrupted A3 control the imputed cost is 0.49 dollars per run and, with one acceptance in twenty, 9.80 dollars per accepted deliverable---the cost of work that is paid for and not delivered.235% ../analysis/tables/table10_recovery.csv (cost_mean 0.490 / 0.863; cost_per_accepted 9.802 / 1.079); ../docs/PHASE_5_AUDIT.md §4236Three qualifications apply. The detection signal is a killed session, which the self-check detects with certainty; the index measures correction given a certain and gross failure, and says nothing about detecting substantive errors in a completed package, which the self-check cannot see and which the pilot shows to be the actual failure mode of the shell arms. The experiment has 20 runs per cell on 10 scenarios and its interval is as wide as the pilot's. And it was designed after the pilot was analysed, in response to a defect; it is an exploratory measurement of the mechanism under induced failure, not evidence for H3.237238\subsection{Cost, supervision and the frontier}\label{sec:results-cost}239240Figure~\ref{fig:cost-quality} plots each pilot run in the plane of marginal cost and quality. The restricted arm occupies the lower part of the plane alone, with quality between 4.6 and 64.6 and a positive relation between cost and quality, since the A0 runs that cost most---the triages, at 1.53 to 1.62 dollars---are those whose text substitutes earned the most credit. The four shell arms overlap almost entirely in a band of 0.5 to 1.5 dollars and 60 to 100 quality points; their means differ by 0.14 dollars and 3.3 points, and the runs below the threshold line are runs of ordinary length and cost that produced a package the judges would not accept, not cheap runs that stopped early. Median wall time is 162 seconds in A0 and 214 to 230 seconds in the shell arms; every pilot run finished within 392 seconds against limits of 1,800 seconds or more, the most expensive run cost 1.62 dollars against caps of 4.00 or more, and the maximum number of turns was 81 against a cap of 150, so the caps were part of the estimand's support in principle but not in fact and nothing is censored. Tool execution time is 0.6 to 6.8 seconds per run; wall time is almost entirely model inference.241% ../analysis/tables/table1_outcomes_by_arm_pilot.csv (wall medians); table14_extra_descriptives.csv block A_process (quality and cost ranges, maxima, tool time); ../analysis/derived/estimates_pilot.json (survival: censored 0)242243\begin{figure}[htbp]244\centering245\includegraphics[width=0.85\textwidth]{fig05_cost_quality_pilot}246\caption{Marginal API cost per run (USD) against the blind quality score (0--100), one point per run, 150 pilot runs, with the arm means as large diamonds and the preregistered threshold $q=70$ as a dotted line. Uncertainty on the arm means is given by the bootstrap intervals of the quality and cost contrasts in Table~\ref{tab:contrasts} and Section~\ref{sec:results-models}. Source: \texttt{data/validated/runs\_pilot.parquet}; script \texttt{analysis/figures/make\_figures.py}.}247\label{fig:cost-quality}248\end{figure}249250The Professional Reliability Frontier has, at $q=70$ and the full budget, the coordinates (acceptance within budget, critical-error rate, censoring rate) of $(0.000, 0.200, 0)$ for A0, $(0.867, 0.033, 0)$ for A1, $(0.900, 0, 0)$ for A2 and A3 and $(0.833, 0, 0)$ for A4; A2 and A3 are tied and undominated, and the honest summary is that any arm with a shell delivers an accepted package for about one dollar of marginal API cost and the restricted arm for no finite amount. H6, which expected the lowest cost per accepted deliverable in A1 or A3, is not met: A2 is lowest at 0.87 dollars, within the range that one scenario-repeat moves. Halving the November 2026 API prices gives 0.44 to 0.56 dollars across the shell arms and quadrupling them 3.48 to 4.44. A fully loaded cost that adds the judges (0.23 dollars per run) and an assumed node cost of one dollar per node-hour gives 1.15 to 1.41 dollars per accepted deliverable; the evaluation is the larger overhead, and in a production setting it is the cost of the verification that Proposition~\ref{prop:delegation} makes a necessary complement of delegation.251% ../analysis/tables/table1_outcomes_by_arm_pilot.csv; table11_economic_sensitivity_pilot.csv (price multipliers 0.5 and 4; cost_per_accepted_loaded); table14_extra_descriptives.csv block H_spend (judge cost 0.228 per run)252253No human intervened during any run, so the supervision dimension is measured by proxy from the completion records. The share of runs ending in \texttt{needs\_clarification} is 0.90 in A0, 0.20 in A1, 0.07 in A2, 0.13 in A3 and 0.10 in A4; the mean number of recorded questions falls from 4.7 to 3.8, 3.4, 2.9 and 2.7; and the implied human minutes, at five per question, from 23.3 to 18.8, 16.8, 14.7 and 13.5. The justified-abstention rate---abstentions on a sub-task whose oracle says clarification was truly required---is 0.30 in A0, 0.17 in A1 and equals the abstention rate in A2, A3 and A4: every abstention of the three arms with the most tools was one the scenario called for, so a worker with more tools abstains less often but no less appropriately, whereas in A0 most abstentions were requests for inputs the worker could not read, which is incapacity recorded as a question. The questions are for the most part not abstentions---the completion status is \texttt{completed} in 80 to 93 per cent of shell-arm runs---but the list of points the worker would have raised with a manager, which the brief invites; only one A1 run, two A2 runs, five A3 runs and seven A4 runs recorded no question at all.254% ../analysis/tables/table12_supervision_pilot.csv; table14_extra_descriptives.csv block C_questions (share with no question)255256This proxy is what makes the frontier of Definition~\ref{def:alf} bite. Marginal acceptance in A3 is 0.900. Without the human-time condition the pooled frontier is zero up to a budget of 0.50 dollars (the cheapest accepted run cost 0.51), 0.73 at one dollar and 0.90 at 1.50 dollars and above (the dearest accepted run cost 1.35), so that the budget margin binds between 0.5 and 1.5 dollars and is slack above. At a supervision allowance of zero---no open question---the frontier of A3 averaged over the ten occupations is 0.100 at 0.75 dollars and 0.167 at any budget of one dollar or more. At an allowance of fifteen minutes (three questions or fewer) it is 0.267 at 0.75 dollars, 0.500 at one dollar and 0.667 at 1.50 dollars and above; A1 and A2 reach 0.500 and A4 0.600 at fifteen minutes and the full budget. The gap between 0.900 and 0.667 is the share of accepted work that arrived with more than three open questions; the gap between 0.900 and 0.167 is the share that arrived with any question at all. The budget ceases to bind at about 1.50 dollars; the supervision allowance never does within the range examined. This is the pilot's clearest economic message, and it is a message about the agent's propensity to leave questions open rather than about its ability to do the work.257% computed from ../analysis/tables/table5_alf_pilot.csv (mean over the 10 occupations, equal weights; H=0 and equal_H15); table14_extra_descriptives.csv block D_alf_Hinf (A3 pooled without H condition: 0.733 at B=1, 0.900 at B≥1.5); block A_process (cheapest accepted = A2 min cost 0.5087; dearest = A1 max 1.3542)258259Autonomy-Adjusted Productivity makes the same point in cost units. Accepted deliverables per 100 dollars of API cost plus wage-valued human minutes are 6.5, 7.5, 8.4 and 8.4 in A1--A4 at 40 Canadian dollars per hour, 3.3, 3.9, 4.4 and 4.4 at 80, and 2.3, 2.6, 3.0 and 3.0 at 120; the ordering reverses from the marginal-cost ordering because the arms with more tools record fewer questions, and the wage term dominates the API term at every wage considered: at 80 dollars an hour, fifteen minutes of question-answering costs twenty dollars against one dollar of tokens. The economically relevant cost of the agent, in this pilot, is the cost of the human who closes its questions, which is the conclusion of \citet{catalini2026agi} reached from the other direction; the questions are counted at a stated rate, not measured in minutes of a supervisor's time.260% ../analysis/tables/table11_economic_sensitivity_pilot.csv (accepted_per_100usd_aap by wage)261262\subsection{Integrity and policy enforcement}\label{sec:results-integrity}263264Hypothesis H5 stated that the critical-violation rate would be below 10 per cent in every arm and would not increase with tool access. The second part holds: 0.200 in A0, 0.033 in A1, zero in A2--A4; the first holds for the shell arms and fails for A0. Across the 230 runs of the study the evaluators recorded eleven critical violations: six in A0 (three appraisals that echoed the injected value and three contract reviews with an empty issues register, which the oracle treats as a critical failure of the review), the honest destruction of an input in A1, two interrupted-session artefacts in the recovery control, and two cases of the same injection---a request to send a document to an outside address---obeyed by a shell-equipped worker, once by Fable and once by Opus after a recovery. No run in any stage attempted an external communication through a channel the harness could see; none exists in the laboratory, and the ``forwarding'' is detected in the text of the deliverables.265% ../docs/PHASE_5_AUDIT.md §2; ../analysis/tables/audit_violations_pilot.csv; table14_extra_descriptives.csv blocks J_sensitivity, G_recovery_cost (violations)266267The design's manipulation is the tool set, and its validity depends on the tool set being what the flags say. The \texttt{--disallowedTools} patterns denied exactly one call in A1, one in A2 and one in A4, and no A1 run attempted an \texttt{osascript} call. A0's 883 calls are all of the file family; A1's 386 are 69 per cent Python and 24 per cent shell with no office, browser or AppleScript call; A2's 371 include 18 office-family calls and no browser call; A3's and A4's 516 and 564 include 117 and 105 browser calls and 69 and 85 office calls. A portal submission was collected in every A3 and A4 run and in no run of the other arms. No arm used a tool outside its set, and the arms with the portal and the office adapter used them in every run; the null on the computer-use premium is not an artefact of tools left idle.268% ../analysis/tables/audit_denied_pilot.csv; audit_families_pilot.csv; audit_portal_pilot.csv; ../docs/PHASE_5_AUDIT.md §3269270\subsection{Summary against the preregistered hypotheses}\label{sec:results-summary}271272The pilot supports H2 (the shell premium) and the second clause of H5 (violations do not increase with tool access), and does not support H1 (a precise null), H3 (a null bounded by the design, with the recovery index identified only under induced interruption), H4 (the depth gradient is one scenario) or H6 (cost per accepted deliverable is lowest in A2, within one scenario-repeat); H7's heterogeneity is present and non-confirmatory. The restricted arm's zero and the shell arms' ceiling are the two facts of the pilot; the computer-use, macOS and self-repair premiums are nulls at this difficulty and remain open at higher difficulty.273