SPB Git

spb/wp42_uqo

Public
7.6 KB · 29 lines latex
1% ---------------------------------------------------------------------------2% Section 11 — Conclusion (Agent H; verified results only; sources in comments).3% ---------------------------------------------------------------------------4\section{Conclusion}\label{sec:conclusion}56This paper set out to measure, rather than to estimate from exposure ratings or usage logs, how much complete white-collar work an autonomous computer-using agent can deliver to a professional standard, at what cost, with what reliability and with how much human help. To do so it built a laboratory in which Claude Code launches Claude Code: a controller that composes isolated workspaces, starts bounded sessions on four Mac nodes under five randomised tool configurations, records every event, collects and hashes every deliverable, and evaluates the result against a hidden oracle and three blind judges. It authored and validated ten scenarios in ten occupations inside a synthetic economy, preregistered its design, ran 150 pilot sessions, 40 model sensitivities and 40 recovery sessions for 261.89 dollars, audited itself, and reports the outcome against each hypothesis whatever its sign.7% docs/PHASE_5_AUDIT.md §0, §389The outcome has three parts. The shell premium is large and confirmatory: acceptance rises from 0.000 without a program-execution tool to 0.867 with one, with a scenario-cluster interval of $[0.667, 1.000]$ and a permutation $p$ of 0.004, because the deliverables of professional work are workbooks and documents that cannot be produced by editing text.10% analysis/tables/table2_contrasts_pilot.csv, A1-A0 / S11The computer-use premium, the preregistered primary contrast, is a precise null: $+0.033$ $[0.000, 0.100]$, $p=1.0$, with every arm that has a shell at a ceiling of 0.833 to 0.900, so that the browser, the office adapter and the enterprise portal had nothing left to add on scenarios that were, by their authoring specification, solvable from the shell.12% table2, A3-A1 / S; table1 acceptance_taskw13And bounded recovery, which the pilot never triggered because no first session failed its self-check, converts an interrupted session into an accepted deliverable in an exploratory experiment: acceptance 0.85 against 0.05 without recovery, $+0.80$ $[0.55, 0.95]$, $p=0.004$, with twenty resumes out of twenty passing the public self-check.14% analysis/tables/table10_recovery.csv; analysis/derived/recovery_experiment.json15Around these sit the descriptive facts: seven critical violations in 150 runs, six of them in the arm without a shell and none in the three richest arms; one dollar of marginal API cost per accepted deliverable, with the model rather than the tool set as the cost lever; judges that agree with one another at an alpha of 0.973 and with the oracle in 91 per cent of runs; and a frontier that, at a supervision allowance of zero, is 0.167 against a ceiling of 0.900, rising to 0.667 when fifteen minutes of question-answering are allowed.16% docs/PHASE_5_AUDIT.md §2; table1 cost_per_accepted; estimates_pilot.json agreement; computed from table5_alf_pilot.csv1718The economic reading we draw is deliberately narrow. On designed-solvable professional assignments of the kind in the benchmark, an agent with a shell completes the work nine times in ten for about a dollar, rarely does anything dangerous, and leaves about three questions for a supervisor; the cost that matters is the supervisor's, not the agent's, and the quantity that decides whether delegation pays is the cost of verifying the package rather than the cost of producing it. The Autonomous Labor Frontier makes that structure visible by reporting acceptance jointly with a budget and a supervision allowance, and the pilot's clearest result is that the supervision margin binds long before the budget does. None of this is a statement about occupations, employment or wages; it is a statement about tasks, in a laboratory, for one family of agents, on one date.1920What the study did not do is as much a part of its record as what it did. The human arms were designed and not run; the scenarios were written and validated by agents and are solvable by construction; the judges are models of the workers' family; there are ten scenarios, one model family, one client version, no interface perturbation and no graphical-application condition; the recovery result is exploratory; and three harness defects were found and repaired during the study. The agenda that follows from these limits is concrete---a human wave with ethics approval, a harder tier of twenty to forty scenarios with 30-to-60-step workflows and perturbations, cross-vendor agents, human expert judges, held-out scenarios and natural failures for the recovery index---and it is affordable in API terms, since the pilot used a third of its envelope. Its binding constraint, like the agent's, is human time.2122A word on reliability, in the engineering sense, is also owed, because the mandate asked for a laboratory and not a script. The controller launched 230 sessions in one evening on four nodes at twelve concurrent sessions; none timed out except the 40 recovery-stage sessions interrupted by design, none reached a budget or turn cap, and every uninterrupted session returned a success result.23% docs/PHASE_5_AUDIT.md §324Three defects were nonetheless found by the audit rather than by the run: a judge process that crashed on a deliverable containing null bytes, a transfer that silently lost three runs' deliverables under load, and a flag that would have prevented the self-repair arm from resuming had it ever needed to. Each was repaired, logged as an amendment and, where outcomes could have been affected, rerun or flagged for sensitivity analysis. The record is stated here because a laboratory that reports only its successes is not one whose frontier can be trusted, and because the defects are of the kind that a second wave will meet again at larger scale.2526Two further points belong in a conclusion because they are easy to lose among the numbers. The first is that the laboratory is reusable. Every element that produced the pilot---the authoring specification, the generator-and-validator protocol, the launcher with its verified flags, the per-run isolation and hashing, the deterministic receipts, the blind judging with structured output, the ledger, the dashboard and the analysis scripts---is parameterised by the scenario, the arm and the model, and none of it is specific to the ten scenarios or to the models used. A new scenario is a bundle and an oracle; a new arm is a set of flags and a documentation file; a new agent is a launch command. The marginal cost of the next reading on the instrument is therefore the cost of writing and validating scenarios, which the pilot measured in the human time of the sessions that did it and not in dollars of inference. The second is that the precise null on the primary contrast is a result about benchmark design as much as about agents. A benchmark on which every configuration with a shell reaches nine in ten cannot rank richer configurations, and a frontier estimated on it is a frontier of the benchmark's ceiling. The literature's move from single-step questions to long-horizon, multi-application tasks \citep{kwa2025horizon,xu2024agentcompany,yuan2026osworld2} is the move the next tier of this benchmark must make, and the pilot gives the reason in its own numbers.2728The mandate asked for a measurement of the economic frontier of autonomous knowledge work. The pilot delivers the instrument, a first reading on it, and a demonstration that the reading is informative mainly about where the instrument must be pointed next: at tasks the shell cannot finish, at people doing the same tasks, and at the questions the agent leaves behind.29
Files · main
Menu

Theme