SPB Git

spb/wp42_uqo

Public
32.4 KB · 230 lines latex
1% ---------------------------------------------------------------------------2% Section 3 — A task-based framework for autonomous white-collar work3% (Agent H; theoretical definitions only, no empirical result)4% ---------------------------------------------------------------------------5% Theorem environments: defined here only if the preamble has not defined them.6\ifcsname proposition\endcsname\else\newtheorem{proposition}{Proposition}\fi7\ifcsname lemma\endcsname\else\newtheorem{lemma}[proposition]{Lemma}\fi8\ifcsname corollary\endcsname\else\newtheorem{corollary}[proposition]{Corollary}\fi9\ifcsname definition\endcsname\else\theoremstyle{definition}\newtheorem{definition}{Definition}\fi10\ifcsname assumption\endcsname\else\theoremstyle{definition}\newtheorem{assumption}{Assumption}\fi11\ifcsname remark\endcsname\else\theoremstyle{remark}\newtheorem{remark}{Remark}\fi1213\section{A task-based framework for autonomous white-collar work}\label{sec:theory}1415This section states the measurement framework that the experiment of Section~\ref{sec:design} implements. It is deliberately modest in its theoretical ambition: following the positioning of Section~\ref{sec:lit-positioning}, the constructs below operationalise the task model of \citet{acemoglu2018race} and \citet{acemoglu2025macroeconomics} and the cost-effectiveness logic of \citet{svanberg2024costeffective} for a setting in which the ``technology'' is an autonomous agent whose output on a given task is a random variable. The purpose of the formalism is to fix, before any run is executed, what is being estimated, under which assumptions it is identified, and how its components can be decomposed. Proofs are collected in Appendix~\ref{app:math}.1617\subsection{Occupations, tasks and runs}\label{sec:theory-setup}1819Let $\mathcal{O}$ be a finite set of occupations. Occupation $o\in\mathcal{O}$ is represented by a finite set of tasks $j=1,\dots,J_o$, each a complete professional assignment with a brief, input artefacts and required deliverables, and by a weight vector $w_o=(w_{o1},\dots,w_{oJ_o})$ with $w_{oj}\ge 0$ and $\sum_j w_{oj}=1$. The weight $w_{oj}$ is the economically justified importance of task $j$ within occupation $o$; in the primary analysis the weights are equal across the tasks of an occupation, and in the sensitivity analysis they are proportional to the professional-minutes estimate recorded in each scenario's metadata (Section~\ref{sec:benchmark}). The mandate requires that weights come from defensible occupational evidence or be presented explicitly as sensitivity assumptions; Proposition~\ref{prop:alf}(vi) below bounds the consequence of a change of weights.2021An \emph{arm} $a\in\mathcal{A}=\{A0,A1,A2,A3,A4\}$ is a configuration of the computer tools available to the agent, holding the model, the brief, the inputs and the caps fixed (Section~\ref{sec:design-arms}). A \emph{run} is one execution of task $j$ under arm $a$, indexed by its repeat $r=1,\dots,R_{oj}$. The run is a random experiment because the language model samples its tokens stochastically and because the environment---file system, portal, office applications---responds to the sequence of actions it chooses. A run produces the vector22\begin{equation}\label{eq:run-vector}23Y_{ojr}(a)=\bigl(D,\,Q,\,E,\,C,\,T,\,H,\,X\bigr)_{ojr}(a),24\end{equation}25whose components are defined as follows. $D\in\{0,1\}$ is \emph{deterministic completion}: every required criterion of the hidden oracle passes and the deterministic evaluator records no critical violation. $Q\in[0,100]$ is the \emph{professional quality score}, the mean over blind evaluators of a rubric-weighted score. $E\in\{0,1\}$ is the \emph{critical-error indicator}, equal to one when at least one critical violation is recorded (fabricated figures, modified inputs, compliance with an injected instruction, an attempted external communication, or a scenario-specific violation listed in the oracle). $C\ge 0$ is the \emph{marginal execution cost} in United States dollars as reported by the API, including the cost of any recovery session attributed to the run. $T\in(0,\bar T]$ is \emph{wall-clock time}, right-censored at the session limit $\bar T$; $\delta^T=\mathbf{1}\{T<\bar T\}$ is the censoring indicator. $H\ge 0$ is the \emph{human intervention time} in minutes that the run requires; in the agent arms of this version of the study no intervention is permitted during a run, so $H$ is the time implied by the questions the agent records as unanswered, valued at a stated assumption per question, plus any logged post-run intervention (Section~\ref{sec:design-outcomes}). $X$ collects auxiliary process measurements (tool calls, tool errors, retries, application transitions, tokens, recovery attempts) that the mandate requires to be recorded.2627\begin{definition}[Acceptance]\label{def:acceptance}28For a quality threshold $q\in[0,100]$, the \emph{acceptance} indicator of a run is29\begin{equation}\label{eq:acceptance}30S_{ojr}(a;q)=D_{ojr}(a)\,\bigl(1-E_{ojr}(a)\bigr)\,\mathbf{1}\{Q_{ojr}(a)\ge q\}.31\end{equation}32A deliverable package is accepted when it is deterministically complete, free of critical violations, and judged to be of at least the threshold quality. The \emph{acceptance probability} of task $j$ under arm $a$ is $p_{oj}(a;q)=\Pr\{S_{ojr}(a;q)=1\}$, which does not depend on $r$ under Assumption~\ref{ass:exchange} below.33\end{definition}3435The three factors in \eqref{eq:acceptance} are not redundant. Deterministic completion certifies facts that only the oracle knows; the quality gate certifies what a demanding professional would notice but a test cannot encode; and the integrity factor certifies the absence of behaviours that would make an otherwise acceptable deliverable unusable. The multitask logic of \citet{holmstrom1991multitask} is the reason for keeping them separate: an agent optimising against any one of them alone can pass it while failing the others.3637\subsection{The Autonomous Labor Frontier}\label{sec:theory-alf}3839\begin{definition}[Autonomous Labor Frontier]\label{def:alf}40For occupation $o$, arm $a$, budget $B\ge 0$, human-time allowance $H\ge 0$ and quality threshold $q$, the Autonomous Labor Frontier is41\begin{equation}\label{eq:alf}42\mathrm{ALF}_o(B,H,q;a)=\sum_{j=1}^{J_o} w_{oj}\,\Pr\bigl\{S_{oj}(a;q)=1,\; C_{oj}(a)\le B,\; H_{oj}(a)\le H\bigr\}.43\end{equation}44\end{definition}4546The frontier is the task-weighted probability that a randomly drawn run of a randomly drawn task of the occupation produces an accepted deliverable \emph{and} does so within the budget \emph{and} within the supervision allowance. It is a joint probability, not a product of marginals: an arm that succeeds cheaply on easy tasks and fails expensively on hard ones is distinguished from one with the same marginal success and cost distributions but the opposite dependence. When the arm is clear from context the argument $a$ is suppressed. The frontier is indexed by three thresholds because the economic question has three margins---whether the work is good enough, whether it is cheap enough, and whether it is autonomous enough---and because a single number would hide the trade-offs among them. Its basic properties follow directly from the definition.4748\begin{proposition}[Properties of the frontier]\label{prop:alf}49Fix $o$ and $a$. Let $p_{oj}(q)=\Pr\{S_{oj}(q)=1\}$ and let $\rho_o(q)=\sum_j w_{oj}\,p_{oj}(q)$ denote the task-weighted marginal acceptance rate. Then:50\begin{enumerate}[label=(\roman*)]51\item $0\le \mathrm{ALF}_o(B,H,q)\le 1$ for all $(B,H,q)$.52\item $\mathrm{ALF}_o$ is non-decreasing and right-continuous in $B$ and in $H$, and non-increasing in $q$.53\item $\mathrm{ALF}_o(B,H,q)\le \rho_o(q)$ for all $B,H$, and $\lim_{B\to\infty,\,H\to\infty}\mathrm{ALF}_o(B,H,q)=\rho_o(q)$. The marginal acceptance rate is therefore the ceiling of the frontier; the gap $\rho_o(q)-\mathrm{ALF}_o(B,H,q)$ is the share of accepted work that is infeasible at the stated budget or supervision allowance.54\item (Fr\'echet bounds.) With $F^C_{oj}(B)=\Pr\{C_{oj}\le B\}$ and $F^H_{oj}(H)=\Pr\{H_{oj}\le H\}$,55\begin{align*}56\mathrm{ALF}_o(B,H,q)&\;\ge\;\sum_j w_{oj}\max\bigl\{0,\;p_{oj}(q)+F^C_{oj}(B)+F^H_{oj}(H)-2\bigr\},\\57\mathrm{ALF}_o(B,H,q)&\;\le\;\sum_j w_{oj}\min\bigl\{p_{oj}(q),\,F^C_{oj}(B),\,F^H_{oj}(H)\bigr\}.58\end{align*}59\item If the per-session budget cap is $\bar B$ and the run is evaluated as delivered when the cap binds, then $\mathrm{ALF}_o(B,H,q)=\mathrm{ALF}_o(\bar B,H,q)$ for all $B\ge\bar B$: the frontier is flat beyond the cap and the experiment cannot inform the region $B>\bar B$.60\item (Weight sensitivity.) For two weight vectors $w_o$ and $w_o'$ on the same tasks, $\bigl|\mathrm{ALF}_o^{w}-\mathrm{ALF}_o^{w'}\bigr|\le \mathrm{TV}(w_o,w_o')\,\bigl(\max_j \pi_{oj}-\min_j \pi_{oj}\bigr)$, where $\pi_{oj}$ is the joint probability in \eqref{eq:alf} for task $j$ and $\mathrm{TV}$ is the total-variation distance $\tfrac12\sum_j|w_{oj}-w'_{oj}|$.61\end{enumerate}62\end{proposition}6364Item (v) is a reminder that the caps of the experimental protocol are part of the estimand's support, not nuisance parameters: a budget curve reported over $[0,\bar B]$ is informative, and an extrapolation beyond $\bar B$ is not. Item (vi) gives the maximal effect of replacing equal weights by professional-minutes weights: it is small when tasks within an occupation have similar joint probabilities and can be large otherwise, which is why both weightings are reported.6566The next proposition decomposes the joint probability into interpretable factors. Each factor corresponds to an element of the evaluation pipeline and can be estimated separately, which makes the decomposition a diagnostic: it tells us \emph{where} an arm loses acceptable work.6768\begin{proposition}[Decomposition]\label{prop:decomp}69For every task $j$ with $\Pr\{D_{oj}=1\}>0$,70\begin{equation}\label{eq:decomp}71\begin{split}72\Pr\{S_{oj}(q)=1,\,C_{oj}\le B,\,H_{oj}\le H\}73={}&\underbrace{\Pr\{D_{oj}=1\}}_{\text{deterministic completion}}74\;\times\;\underbrace{\Pr\{Q_{oj}\ge q\mid D_{oj}=1\}}_{\text{quality gate}}\\75&\times\;\underbrace{\Pr\{E_{oj}=0\mid D_{oj}=1,\,Q_{oj}\ge q\}}_{\text{integrity}}\\76&\times\;\underbrace{\Pr\{C_{oj}\le B,\,H_{oj}\le H\mid S_{oj}(q)=1\}}_{\text{budget feasibility}},77\end{split}78\end{equation}79with the convention that a factor whose conditioning event has probability zero is set to zero. The frontier is the task-weighted sum of these products.80\end{proposition}8182Because $E=1$ already implies $D=0$ in the deterministic evaluator (a critical violation blocks completion), the integrity factor in \eqref{eq:decomp} equals one by construction when $D$ is defined as in Section~\ref{sec:theory-setup}; it is kept in the decomposition so that the same identity applies when, in sensitivity analyses, completion is redefined on the required criteria alone and integrity is reinstated as a separate gate. The budget-feasibility factor is conditional on acceptance: it measures whether the \emph{accepted} work was cheap and autonomous, which is the economically relevant condition, rather than whether all runs were.8384\subsection{The Computer-Use Tool Premium}\label{sec:theory-premium}8586The second construct is a causal contrast. Each run has a potential outcome under every arm. Write $S_{jr}(a)$ for the acceptance indicator that repeat $r$ of task $j$ \emph{would} produce under arm $a$, and similarly for the other components of \eqref{eq:run-vector}. The design assigns every task to every arm, so the contrast is identified within task from the randomised interleaving of runs rather than from a between-task comparison.8788\begin{assumption}[Exchangeable repeats and no interference]\label{ass:exchange}89For each task $j$ and arm $a$, the run vectors $Y_{j1}(a),\dots,Y_{jR_j}(a)$ are independent and identically distributed; the vector of a run does not depend on the arm, the order, or the node assigned to any other run; and the distribution of $Y_{jr}(a)$ does not depend on the position of the run in the execution order or on the executor node.90\end{assumption}9192\begin{assumption}[Information constancy]\label{ass:info}93The brief, the input artefacts, the deliverable specification, the model, the client version, the caps and the behavioural policy are identical across arms; arms differ only in the tool set, the documentation of that tool set, and, where applicable, the presence of the enterprise portal and of the self-repair protocol.94\end{assumption}9596Assumption~\ref{ass:exchange} is what the engineering of Section~\ref{sec:infrastructure} is built to deliver: fresh workspaces, independent sessions, no shared state, no oracle on the executor nodes, and a seeded random permutation of the execution order interleaved over nodes so that any drift in the API or in node load is orthogonal to the arm. Assumption~\ref{ass:info} makes the contrast interpretable as the effect of computer access rather than of information: the enterprise portal of arms A3 and A4 contains a subset of the input artefacts and never information absent from them (Section~\ref{sec:benchmark-anatomy}).9798\begin{definition}[Computer-Use Tool Premium]\label{def:premium}99For arms $a,a'\in\mathcal{A}$ and a set of tasks $\mathcal{J}$ with weights $w_j$, the tool premium is the task-weighted average treatment effect on acceptance,100\begin{equation}\label{eq:premium}101\tau(a,a';q)=\sum_{j\in\mathcal{J}} w_j\,\bigl[\,p_j(a;q)-p_j(a';q)\,\bigr],102\qquad p_j(a;q)=\mathbb{E}\bigl[S_{jr}(a;q)\bigr].103\end{equation}104The \emph{shell premium} is $\tau(A1,A0)$, the \emph{computer-use premium} is $\tau(A3,A1)$, the \emph{macOS-layer premium} is $\tau(A2,A1)$ and the \emph{self-repair premium} is $\tau(A4,A3)$.105\end{definition}106107The natural estimator is the task-weighted mean of within-task differences of repeat means,108\begin{equation}\label{eq:premium-hat}109\hat\tau(a,a')=\sum_{j\in\mathcal{J}} w_j\bigl[\bar S_j(a)-\bar S_j(a')\bigr],\qquad \bar S_j(a)=\frac{1}{R_j}\sum_{r=1}^{R_j}S_{jr}(a).110\end{equation}111112\begin{proposition}[Identification and finite-task variance]\label{prop:premium}113Under Assumptions~\ref{ass:exchange} and~\ref{ass:info}, and treating the task set $\mathcal{J}$ as fixed, $\mathbb{E}[\hat\tau(a,a')]=\tau(a,a')$ and114\begin{equation}\label{eq:premium-var}115\operatorname{Var}\bigl[\hat\tau(a,a')\bigr]=\sum_{j\in\mathcal{J}} w_j^2\,\frac{p_j(a)\{1-p_j(a)\}+p_j(a')\{1-p_j(a')\}}{R_j}.116\end{equation}117If instead the tasks are an independent sample of size $J$ from an occupation's task population with equal weights $w_j=1/J$, then with $\Delta_j=p_j(a)-p_j(a')$ and $\sigma_j^2(a)=p_j(a)\{1-p_j(a)\}$,118\begin{equation}\label{eq:premium-var-cluster}119\operatorname{Var}\bigl[\hat\tau(a,a')\bigr]=\frac{1}{J}\Bigl[\operatorname{Var}_j(\Delta_j)+\mathbb{E}_j\Bigl(\frac{\sigma_j^2(a)+\sigma_j^2(a')}{R_j}\Bigr)\Bigr].120\end{equation}121\end{proposition}122123Equation \eqref{eq:premium-var-cluster} is the reason the primary inference clusters at the task: between-task heterogeneity in the treatment effect, $\operatorname{Var}_j(\Delta_j)$, is not reduced by adding repeats, only by adding tasks, while within-task stochastic variation is reduced by repeats. It also governs the power simulation of Section~\ref{sec:design-power}. The premium is an ablation in the sense of \citet{xie2024osworld} and \citet{trivedi2024appworld}; what the design adds is that the ablation is randomised on the same tasks across occupations and reported with inference that respects \eqref{eq:premium-var-cluster} \citep{abadie2023clustering,miller2024errorbars}.124125Two caveats on interpretation are stated now. First, under Assumption~\ref{ass:info} the premium is the effect of a \emph{bundle}---tools plus their documentation plus, for A3 and A4, the portal---not of a single tool; disentangling the bundle would require further arms. Second, arm A2 can only be executed on the executor nodes where the operating-system automation layer answers (Section~\ref{sec:infrastructure-macos}), so $\tau(A2,A1)$ is identified conditional on those nodes; under the last clause of Assumption~\ref{ass:exchange} the restriction is innocuous, and the pilot records node identity so that the clause can be checked rather than assumed.126127\subsection{The Workflow Dependency Frontier and the Autonomous Recovery Index}\label{sec:theory-wdf}128129The third construct relates acceptance to the length of the chain of dependent operations a task requires. Each scenario carries a \emph{designed dependency depth} $d_j$, the minimal number of dependent operations---open or convert a document, extract a figure, match records, compute, cross-check, write a deliverable section---that a careful professional must perform (Section~\ref{sec:benchmark-anatomy}). The Workflow Dependency Frontier is the function $d\mapsto p(a;d)$, the acceptance probability of arm $a$ as a function of depth, estimated across scenarios and, as a secondary measure, as a function of the realised number of tool calls. A simple model of compounding error provides the benchmark shape against which the estimated frontier is read.130131Consider a task whose completion requires $d$ dependent steps, and let step $k$ succeed with probability $\pi_k\in(0,1]$ conditional on all previous steps having succeeded, failure being absorbing. Then the probability of completing the task is132\begin{equation}\label{eq:compound}133p(d)=\prod_{k=1}^{d}\pi_k .134\end{equation}135136\begin{proposition}[Compounding reliability]\label{prop:compound}137Let $p(d)$ be as in \eqref{eq:compound}.138\begin{enumerate}[label=(\roman*)]139\item If $\pi_k=\pi$ for all $k$, then $p(d)=\pi^d$, $\log p(d)$ is linear in $d$ with slope $\log\pi<0$, and the \emph{half-depth} at which the acceptance probability falls to one half is $d_{1/2}=\log 2/\log(1/\pi)$.140\item For heterogeneous reliabilities with arithmetic mean $\bar\pi=d^{-1}\sum_k\pi_k$, $p(d)\le\bar\pi^{\,d}$, with equality if and only if all $\pi_k$ are equal. Heterogeneity of step reliability lowers task reliability at a given mean.141\item If the step reliabilities are non-increasing in $k$ (self-conditioning in the sense of \citealp{sinha2025horizon}), then $-\log p(d)$ is convex in $d$: the frontier falls faster than exponentially.142\end{enumerate}143\end{proposition}144145Item (i) connects the frontier to the time-horizon construct of \citet{kwa2025horizon}: the half-depth plays the role of the 50-per-cent horizon with depth in place of human time, and a per-step reliability of $\pi$ implies a half-depth of roughly $0.69/(1-\pi)$ for $\pi$ close to one. Item (ii) implies that two arms with the same average step reliability can have different frontiers if one of them concentrates its failures on a few operations (for instance, producing a binary office file in an arm without a shell), which is a reason to report the frontier by arm rather than to pool. Item (iii) gives a testable shape restriction: a log-acceptance curve that is concave in depth is inconsistent with self-conditioning under this model.146147Recovery changes the model in two ways, corresponding to two mechanisms that the self-repairing arm A4 contains. Within a session, an agent may detect and correct an error at a step before proceeding; across sessions, the controller resumes a session that ended without passing a public self-check (Section~\ref{sec:design-arms}).148149\begin{proposition}[Recovery]\label{prop:recovery}150\begin{enumerate}[label=(\roman*)]151\item (Per-step recovery.) Suppose a failed step is detected with probability $\delta$ and, if detected, corrected with probability $\gamma$, once. With $\rho=\delta\gamma$, the effective step reliability is $\pi'=\pi+(1-\pi)\rho\ge\pi$, the task reliability is $p_\rho(d)=\bigl(\pi+(1-\pi)\rho\bigr)^d$, and the ratio $p_\rho(d)/p(d)=\bigl(1+\rho(1-\pi)/\pi\bigr)^d$ grows geometrically in $d$. Recovery therefore matters more, not less, for deeper workflows.152\item (One-shot end-of-run recovery.) Suppose a run that would fail under arm A3 fails the public self-check with probability $\delta$ (detection), and that a resumed session converts a detected failure into an accepted deliverable with probability $\gamma$ (correction). If the first session of A4 is distributed as a run of A3, then153\[154p(A4;d)=p(A3;d)+\bigl(1-p(A3;d)\bigr)\,\delta\,\gamma,155\]156so the self-repair premium is $\tau(A4,A3;d)=(1-p(A3;d))\,\delta\gamma\ge 0$, increasing in the first-session failure rate.157\end{enumerate}158\end{proposition}159160\begin{definition}[Autonomous Recovery Index]\label{def:ari}161In arm A4, let $F_1$ be the event that the first session ends without passing the public self-check and $P_2$ the event that the deliverables pass the public self-check after the bounded recovery session. The Autonomous Recovery Index is162\begin{equation}\label{eq:ari}163\mathrm{ARI}=\Pr\{P_2\mid F_1\}.164\end{equation}165\end{definition}166167Three remarks fix the interpretation. First, the ARI is a \emph{correction} probability conditional on \emph{detection} by a specific signal: the public self-check verifies presence, format and well-formedness of deliverables and the validity of the completion record, and knows nothing about the correct answers. Detection of substantive errors---a wrong reconciliation total, a missed planted discrepancy---is outside its reach, and the decomposition of Proposition~\ref{prop:recovery}(ii) therefore refers to the subset of failures the signal can see. Second, by the imperfect-verifier result of \citet{stroebl2024flaws}, passing the self-check is not acceptance; the acceptance uplift of A4 over A3 is bounded above by $\Pr\{F_1\}\cdot\mathrm{ARI}$ and may be much smaller. Third, the index is defined on a design in which the recovery session shares the tools, the model, the remaining budget and the remaining wall time of the interrupted session, so that its cost and time are attributed to the same run; an ARI estimated under a different attribution rule is not comparable. The distinction between intrinsic self-correction, which \citet{huang2023selfcorrect} find ineffective, and correction against an external signal, which \citet{shinn2023reflexion} and \citet{chen2023selfdebug} find effective, is thus built into the definition: the ARI measures the second.168169\subsection{Autonomy-Adjusted Productivity}\label{sec:theory-aap}170171The fourth construct converts the frontier into a productivity measure that is net of supervision. Let $\omega$ be the wage rate per human minute used to value intervention time, and let $\kappa$ be a fully loaded operating cost per run (compute allocation, software, overhead) that is reported separately from the marginal API charge $C$, as the mandate requires.172173\begin{definition}[Autonomy-Adjusted Productivity]\label{def:aap}174For arm $a$, task set $\mathcal{J}$ and quality threshold $q$, the Autonomy-Adjusted Productivity is the expected number of accepted deliverables per unit of fully loaded cost,175\begin{equation}\label{eq:aap}176\mathrm{AAP}(a;q,\omega,\kappa)=\frac{\sum_j w_j\,\mathbb{E}[S_{j}(a;q)]}{\sum_j w_j\,\mathbb{E}\bigl[C_j(a)+\omega H_j(a)+\kappa\bigr]} ,177\end{equation}178and its reciprocal is the \emph{cost per accepted deliverable}. The \emph{marginal} version sets $\kappa=0$.179\end{definition}180181The ratio is computed over all runs, accepted or not: an arm pays for its failures, and the mandate's instruction to report cost per \emph{accepted} deliverable rather than per attempted task is precisely the instruction to keep the failed runs in the denominator. The sample analogue $\sum_{jr}S_{jr}/\sum_{jr}(C_{jr}+\omega H_{jr}+\kappa)$ is a ratio estimator; it is consistent but biased in finite samples, and its uncertainty is obtained by bootstrapping the joint distribution of numerator and denominator over tasks (Appendix~\ref{app:math}). Because $\mathrm{AAP}$ is a function of $(q,\omega,\kappa)$ and of the token prices embedded in $C$, it is reported as a sensitivity surface rather than as a number, following \citet{kapoor2024matter} and \citet{svanberg2024costeffective}. The construct is a label for an accounting convention; the precedents listed in Table~\ref{tab:precedents} already measure time and quality jointly in assistive settings, and the measured-versus-perceived divergence of \citet{becker2025metr} is the reason the human-time term is measured rather than reported by the agent.182183\subsection{The Professional Reliability Frontier}\label{sec:theory-prf}184185The fifth construct treats the end of a run as one of several competing terminal events. A run terminates in exactly one of four states: \emph{accepted} ($K=\mathrm{A}$), \emph{critical error} ($K=\mathrm{F}$, a critical violation is recorded, whatever else was delivered), \emph{censored} ($K=\mathrm{X}$, the wall limit, the budget cap or the turn cap stopped the session before a completion record was written), or \emph{finished without acceptance} ($K=\mathrm{N}$, the agent stopped on its own but the package fails completion or the quality gate without a critical violation). Let $T$ be the time at which the terminal state is reached and $K$ its cause. The cumulative incidence function of cause $k$ is $F_k(t)=\Pr\{T\le t,\,K=k\}$, and the four functions sum, as $t\to\bar T$, to one.186187\begin{definition}[Professional Reliability Frontier]\label{def:prf}188For arm $a$, the Professional Reliability Frontier is the map189\begin{equation}\label{eq:prf}190(q,B)\;\longmapsto\;\Bigl(\Pr\{S(a;q)=1,\,C(a)\le B\},\;\Pr\{E(a)=1\},\;\Pr\{K=\mathrm{X}\}\Bigr),191\end{equation}192evaluated task-weighted over the benchmark; the \emph{reliability-efficient} arms at a given $(q,B)$ are those not dominated in all three coordinates.193\end{definition}194195The competing-risks framing \citep{fine1999competing} matters for two reasons. A critical error is a terminal event, not missing data: a run that fabricated a total and then timed out is a failure of integrity, not a censored observation, and treating it as censored would bias the completion curve upward. And the frontier makes the trade-off visible that a single success rate hides: an arm may raise acceptance while raising the critical-error rate, which, in a professional context where a single fabricated figure can void a deliverable, is not an improvement. The jagged-frontier result of \citet{dellacqua2023jagged} is the assistive-setting precedent for this trade-off. Estimation uses the Kaplan--Meier curve for time to any terminal event \citep{kaplan1958nonparametric}, the Cox model for the hazard of acceptance with arm and depth as covariates \citep{cox1972regression}, and the Fine--Gray subdistribution hazard for the cause-specific incidence of acceptance and of critical error.196197\subsection{Delegation with verification: a principal--agent reading}\label{sec:theory-pa}198199The constructs above are measurement conventions. This subsection shows how they enter the decision of a principal---a manager, a partner, a client---who must decide whether to delegate a task to an autonomous agent, and in doing so explains why the frontier has the three thresholds it has. The model is the simplest one that produces the relevant threshold; it draws on the contracting treatments of \citet{sreenivas2026contracting} and \citet{catalini2026agi} and on the classical moral-hazard structure of \citet{holmstrom1979moral}.200201A task yields value $v>0$ to the principal when an acceptable deliverable is used. The principal can produce it in-house at cost $c_H$ (wage times professional minutes), or delegate it to an agent at marginal cost $c_A$, in which case the deliverable is acceptable with probability $p$ and contains a critical error with probability $e$. An unacceptable deliverable that is nevertheless used causes a loss $\ell\ge 0$; a critical error that passes unnoticed causes a loss $L\ge\ell$. The principal may verify the deliverable at cost $\kappa_V$ (human minutes times wage), in which case failures are detected and the task is completed in-house.202203\begin{proposition}[Delegation thresholds]\label{prop:delegation}204\begin{enumerate}[label=(\roman*)]205\item Delegation with verification yields a higher expected payoff than in-house production if and only if206\[207p\;\ge\;\frac{c_A+\kappa_V}{c_H}.208\]209\item Verification is worth its cost, relative to delegating without verification, if and only if210\[211\kappa_V\;\le\;(1-p)\,(\ell-c_H)+e\,L .212\]213\item Consequently, delegation without verification is optimal only when $\kappa_V>(1-p)(\ell-c_H)+eL$ and $c_A+(1-p)\ell+eL\le c_H$; as $L\to\infty$ the second condition fails for any $e>0$, so verification becomes a necessary complement of delegation whenever critical errors are possible.214\end{enumerate}215\end{proposition}216217Item (i) is the frontier's budget condition in decision form: a task is worth delegating when its acceptance probability exceeds the ratio of the agent's cost plus the verification cost to the human cost. The frontier's condition $C\le B$ corresponds to $c_A\le B$, and its condition $H\le H$ corresponds to the human time that verification and completion require; the acceptance probability $p$ that appears in (i) is the $p_{oj}(q)$ of Definition~\ref{def:acceptance}. Item (iii) formalises the claim of \citet{catalini2026agi} that verification is the binding constraint: when the cost of execution $c_A$ collapses, the delegation decision is governed by $\kappa_V$, that is, by how cheaply a human can check the work, and the measurability gap between what the agent does and what the human can verify determines the economically usable share of its output. It is also why the critical-error indicator $E$ is measured separately from acceptance: $e$ and $L$ enter the verification decision through a term that acceptance alone does not capture. The model abstracts from the agent's own incentives, which are those of a software system rather than of a utility-maximising employee; the agency problem is on the principal's side, in observing the quality of what was delivered, which is the ``measurable versus unmeasurable dimensions'' problem of \citet{holmstrom1991multitask} and the scalable-oversight problem of \citet{bowman2022oversight}. The parameters of Proposition~\ref{prop:delegation} other than $p$ and $e$ are not estimated in this study; they are reported as sensitivity assumptions in Section~\ref{sec:results}.218219\subsection{What the frontier is not}\label{sec:theory-not}220221The framework invites misreadings that the mandate explicitly forbids, and it is better to list them here than to qualify every later sentence.222223The frontier is not an estimate of job displacement. It measures the probability that an agent completes a \emph{task} of an \emph{occupation} to a professional standard under stated constraints; it says nothing about whether the occupation's remaining tasks are separable from the automated ones, whether the saved time is reallocated to new tasks, or whether the demand for the occupation's output changes---the displacement, productivity and reinstatement effects of \citet{acemoglu2019automation} and the bottleneck logic of \citet{restrepo2025missed} are outside the measurement. A high frontier value for an occupation is a statement about the feasibility and cost of automating its tasks in a laboratory, and the step from feasibility to adoption and from adoption to employment involves adjustment costs, complementarities and organisational change that \citet{brynjolfsson2018machines}, \citet{hampole2025labor} and \citet{humlum2025still} document as slow and partial.224225A task is not an occupation. The benchmark represents each occupation by a small number of representative assignments chosen to be complete and verifiable; the weights $w_{oj}$ are equal or proportional to professional-minutes estimates, not derived from a census of the occupation's activities, and the mandate requires that this be presented as a sensitivity assumption. The frontier therefore describes the benchmark's tasks, and the extent to which the benchmark covers the occupation is a question of content validity that no estimate from the benchmark can answer.226227The scenarios are synthetic. Organisations, people, documents and figures are generated to be internally coherent and realistic, with planted errors and traps that a real file might or might not contain. Synthetic data make a hidden oracle possible and remove any risk of exposing real records, but they also mean that a frontier estimated here is a frontier on \emph{designed} difficulty, and that the realism of the scenarios is a judgement of their validators, not a measured property. The analysis distinguishes designed features (depth, planted errors, injection, clarification traps) from incidental ones, and reports results on both, so that the reader can see how much of the frontier is shaped by the designer.228229Finally, the frontier is estimated for one family of models and one client version at one date, under the caps of one protocol. Proposition~\ref{prop:alf}(v) states the consequence of the caps; the dependence on the model and on the date is the reason the design includes model sensitivities and records every version identifier, and the reason the names of the constructs are provisional until the evidence of Section~\ref{sec:results} has been examined.230
Files · main
Menu

Theme