SPB Git

spb/wp42_uqo

Public
3.9 KB · 13 lines latex
1% ---------------------------------------------------------------------------2% Section 7 — Conclusion (journal version; sources in comments).3% ---------------------------------------------------------------------------4\section{Conclusion}\label{sec:conclusion}56This paper set out to measure, rather than to estimate from exposure ratings or usage logs, how much complete white-collar work an autonomous computer-using agent can deliver to a professional standard, at what cost, with what reliability and with how much human help. It built a laboratory in which Claude Code launches Claude Code, authored and validated ten scenarios in ten occupations inside a synthetic economy, preregistered its design, ran 150 pilot sessions, 40 model sensitivities and 40 recovery sessions for 261.89 dollars, audited itself, and reports the outcome against each hypothesis whatever its sign.7% ../docs/PHASE_5_AUDIT.md §0, §389The outcome has three parts. The shell premium is large and confirmatory: acceptance rises from 0.000 without a program-execution tool to 0.867 with one ([0.667, 1.000], $p=0.004$), because the deliverables of professional work are workbooks and documents that cannot be produced by editing text. The computer-use premium, the preregistered primary contrast, is a precise null ($+0.033$ [0.000, 0.100], $p=1.0$), with every arm that has a shell at a ceiling of 0.833 to 0.900, so that the browser, the office adapter and the enterprise portal had nothing left to add on scenarios that were, by their authoring specification, solvable from the shell. And bounded recovery, which the pilot never triggered, converts an interrupted session into an accepted deliverable in an exploratory experiment (0.85 against 0.05, $+0.80$ [0.55, 0.95], $p=0.004$) at essentially the cost of an uninterrupted run. Around these sit the descriptive facts: seven critical violations in 150 runs, six of them in the arm without a shell and none in the three richest arms; about one dollar of marginal API cost per accepted deliverable, with the model rather than the tool set as the cost lever; judges that agree with one another at an alpha of 0.973 and with the oracle in 91 per cent of runs; and a frontier that, at a supervision allowance of zero, is 0.167 against a ceiling of 0.900, rising to 0.667 when fifteen minutes of question-answering are allowed.10% ../analysis/tables/table2_contrasts_pilot.csv; table1_outcomes_by_arm_pilot.csv; table10_recovery.csv; ../docs/PHASE_5_AUDIT.md §2; ../analysis/derived/estimates_pilot.json (agreement); computed from table5_alf_pilot.csv1112The economic reading is deliberately narrow. On designed-solvable professional assignments of the kind in the benchmark, an agent with a shell completes the work nine times in ten for about a dollar, rarely does anything dangerous, and leaves about three questions for a supervisor; the cost that matters is the supervisor's, not the agent's, and the quantity that decides whether delegation pays is the cost of verifying the package rather than the cost of producing it. The Autonomous Labor Frontier makes that structure visible by reporting acceptance jointly with a budget and a supervision allowance, and the pilot's clearest result is that the supervision margin binds long before the budget does. None of this is a statement about occupations, employment or wages; it is a statement about tasks, in a laboratory, for one family of agents, on one date. The human arms were designed and not run; the scenarios were written and validated by agents and are solvable by construction; the judges are models of the workers' family; there are ten scenarios; and the recovery result is exploratory. The laboratory is reusable---a new scenario is a bundle and an oracle, a new arm a set of flags, a new agent a launch command---and the pilot's reading is informative mainly about where the instrument must be pointed next: at tasks the shell cannot finish, at people doing the same tasks, and at the questions the agent leaves behind.13
Files · e883970b49fd91449e9df61ba1881564aeb053c6
Menu

Theme