WP42: journal version (41 pp.), release PDFs, status and README
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
16 changed files +47 −36
modified
.gitignore
+3 −0
@@ -32,3 +32,6 @@ journal_version/*.blg | |||||
| 32 | journal_version/*.fls | 32 | journal_version/*.fls | ||
| 33 | journal_version/*.fdb_latexmk | 33 | journal_version/*.fdb_latexmk | ||
| 34 | data/raw/runs/ | 34 | data/raw/runs/ | ||
| 35 | +journal_version/main.pdf | ||||
| 36 | +!manuscript/main.pdf | ||||
| 37 | +!release/*.pdf | ||||
modified
PROJECT_STATUS.md
+2 −2
@@ -10,7 +10,7 @@ | |||||
| 10 | | 5 Scientific audit | done 2026-11-03 | `docs/PHASE_5_AUDIT.md`, amendments A-008/A-010/A-012 | | 10 | | 5 Scientific audit | done 2026-11-03 | `docs/PHASE_5_AUDIT.md`, amendments A-008/A-010/A-012 | | ||
| 11 | | 6 Full-scale | partial: R = 3 and model sensitivities per the scale rule; scaling to 20+ scenarios NOT done (next wave) | `protocol/amendments.csv` A-009 | | 11 | | 6 Full-scale | partial: R = 3 and model sensitivities per the scale rule; scaling to 20+ scenarios NOT done (next wave) | `protocol/amendments.csv` A-009 | | ||
| 12 | | 7 Analysis | done 2026-11-03 | `analysis/tables/`, `analysis/derived/`, `analysis/figures/out/`, `logs/*.txt` | | 12 | | 7 Analysis | done 2026-11-03 | `analysis/tables/`, `analysis/derived/`, `analysis/figures/out/`, `logs/*.txt` | | ||
| 13 | −| 8 Manuscript | in progress | `manuscript/` (non-empirical sections done; empirical + framing + appendices being written) | | 13 | +| 8 Manuscript | done 2026-11-03 | `manuscript/main.pdf` 188 pages (12 sections, 6 appendices, 110 verified references); `journal_version/main.pdf` 41 pages; `release/` | | ||
| 14 | −| 9 Audit and release | pending | | | 14 | +| 9 Audit and release | done 2026-11-03 | `docs/AUDIT_REPORT.md` (14 findings, blocking ones fixed), `scripts/reproduce_all.sh` verified, repo pushed to `gitsrv:srv/git/wp42_uqo.git`, series README/HARMONISATION rows | | ||
| 15 | 15 | ||||
| 16 | Human arms H0/H1: designed, not executed (A-002). | 16 | Human arms H0/H1: designed, not executed (A-002). | ||
modified
README.md
+7 −0
@@ -7,6 +7,13 @@ Measuring the economic frontier of Claude Code, computer-using agents and autono | |||||
| 7 | professionally acceptable standard, at what cost, with what reliability and supervision needs, and how much of it is explained by the | 7 | professionally acceptable standard, at what cost, with what reliability and supervision needs, and how much of it is explained by the | ||
| 8 | level of computer access (file-only → shell → macOS layer → browser/office/portal → self-repair)? | 8 | level of computer access (file-only → shell → macOS layer → browser/office/portal → self-repair)? | ||
| 9 | 9 | ||||
| 10 | +## Headline results (pilot, 150 runs, claude-opus-5-5, 2026-11-03) | ||||
| 11 | +Scenario-weighted acceptance (deterministic completion ∧ no critical violation ∧ blind quality ≥ 70): A0 0.00 · A1 0.87 · A2 0.90 · A3 0.90 · A4 0.83. | ||||
| 12 | +Shell premium A1−A0 = +0.87 [0.67, 1.00], p = 0.004. Preregistered primary contrast A3−A1 = +0.03 [0.00, 0.10], p = 1.0 (precise null, ceiling). | ||||
| 13 | +Cost per accepted deliverable 0.87–1.11 USD (Sonnet 5.5: 0.40 USD at 0.85; Fable 5.1: 3.30 USD at 0.85). Exploratory recovery experiment: | ||||
| 14 | +after an induced interruption at 120 s, one bounded recovery session raises acceptance from 0.05 to 0.85 (20/20 self-check passes). | ||||
| 15 | +Human arms H0/H1 designed, not executed. Total API spend 261.89 USD. Monograph: `release/wp42_uqo.pdf` (188 pp.); journal version: `release/wp42_uqo_journal_version.pdf` (41 pp.). | ||||
| 16 | + | ||||
| 10 | ## Layout | 17 | ## Layout | ||
| 11 | | Path | Content | | 18 | | Path | Content | | ||
| 12 | |---|---| | 19 | |---|---| | ||
modified
journal_version/main.pdf
+0 −0
Binary file not shown.
modified
journal_version/main.tex
+1 −0
@@ -113,6 +113,7 @@ | |||||
| 113 | \usepackage[toc,page]{appendix} | 113 | \usepackage[toc,page]{appendix} | ||
| 114 | 114 | ||||
| 115 | % --- Misc --- | 115 | % --- Misc --- | ||
| 116 | +\setlength{\emergencystretch}{1.5em} | ||||
| 116 | \usepackage{enumitem} | 117 | \usepackage{enumitem} | ||
| 117 | \setlist{nosep,leftmargin=*} | 118 | \setlist{nosep,leftmargin=*} | ||
| 118 | 119 | ||||
modified
journal_version/sections/abstract.tex
+1 −1
@@ -5,4 +5,4 @@ | |||||
| 5 | % analysis/derived/recovery_experiment.json (recovery), docs/PHASE_5_AUDIT.md, | 5 | % analysis/derived/recovery_experiment.json (recovery), docs/PHASE_5_AUDIT.md, | ||
| 6 | % protocol/amendments.csv (A-002, A-011). | 6 | % protocol/amendments.csv (A-002, A-011). | ||
| 7 | % --------------------------------------------------------------------------- | 7 | % --------------------------------------------------------------------------- | ||
| 8 | −We report a preregistered randomised experiment on whether an autonomous computer-using agent can complete professional assignments to an acceptable standard. A controller launches independent Claude Code sessions (\texttt{claude-opus-5-5}) to perform ten validated scenarios in ten occupations inside a synthetic economy with hidden oracles, planted errors and prompt-injection traps. Each scenario is run under five levels of computer access, from file access without a shell to full computer use with browser, office tools and an enterprise portal, plus a self-repairing variant. Acceptance requires deterministic completion, no critical violation and a blind quality score of at least 70. Over 150 runs, acceptance is 0.00, 0.87, 0.90, 0.90 and 0.83. The shell premium is $+0.87$ [0.67, 1.00]; the preregistered primary contrast, full computer use minus shell, is $+0.03$ [0.00, 0.10], a precise null because every arm with a shell is at a ceiling. Accepted deliverables cost about one dollar of API charges; a cheaper model reaches the same acceptance at 0.40 dollars. In an exploratory experiment, bounded self-repair converts interrupted sessions into accepted deliverables (0.85 against 0.05). The supervision margin, not the budget, separates the Autonomous Labor Frontier from the acceptance ceiling. Human arms were not executed; no displacement estimate is implied. | 8 | +We report a preregistered randomised experiment on whether an autonomous computer-using agent can complete professional assignments to an acceptable standard. A controller launches independent Claude Code sessions (\texttt{claude-opus-5-5}) to perform ten validated scenarios in ten occupations inside a synthetic economy with hidden oracles, planted errors and prompt-injection traps. Each scenario is run under five levels of computer access, from file access without a shell to full computer use with browser, office tools and an enterprise portal, plus a self-repairing variant. Acceptance requires deterministic completion, no critical violation and a blind quality score of at least 70. Over 150 runs, acceptance is 0.00, 0.87, 0.90, 0.90 and 0.83. The shell premium is $+0.87$ [0.67, 1.00]; the preregistered primary contrast, full computer use minus shell, is $+0.03$ [0.00, 0.10], a precise null because every arm with a shell is at a ceiling. Accepted deliverables cost about one dollar of API charges; a cheaper model reaches 0.85 acceptance at 0.40 dollars. In an exploratory experiment, bounded self-repair converts interrupted sessions into accepted deliverables (0.85 against 0.05). The supervision margin, not the budget, separates the Autonomous Labor Frontier from the acceptance ceiling. Human arms were not executed; no displacement estimate is implied. | ||
modified
journal_version/sections/appendix_pointer.tex
+1 −1
@@ -4,4 +4,4 @@ | |||||
| 4 | \section*{Online appendix}\label{app:pointer} | 4 | \section*{Online appendix}\label{app:pointer} | ||
| 5 | \addcontentsline{toc}{section}{Online appendix} | 5 | \addcontentsline{toc}{section}{Online appendix} | ||
| 6 | 6 | ||||
| 7 | −The monograph of which this paper is the journal version---UQO Working Paper No.~42, \emph{The Autonomous White-Collar Worker: Measuring the Economic Frontier of Claude Code, Computer-Using Agents, and Autonomous Knowledge Work}---serves as the online appendix. It contains the full literature review and positioning table; the proofs of the propositions of Section~\ref{sec:framework} and the properties of the frontier; the occupational taxonomy, the organisations of the Virtual White-Collar Economy, the scenario-by-scenario descriptions and the authoring specification; the preregistration (version 1.0 with its freeze manifest) and the amendment log A-001 to A-012; the infrastructure as built, including the Phase~0 survey of the executor nodes, the launcher flags and their probes, the portal, the self-repair protocol and the scheduler; the data documentation, including the 54 run-level variables, the three evaluation instruments and their agreement, the cost measurement and the treatment of missing values; the complete results, including the Autonomous Labor Frontier by occupation and budget, the dependency table, the quality-threshold sensitivity, the Kaplan--Meier summaries, the failure-severity, supervision and tool-family tables, the economic sensitivity analysis and the infrastructure-stability statistics; the rubrics; the reproducibility records; and the robustness tables, including the contrasts recomputed without the re-judged runs. Every exhibit of the present paper names, in a LaTeX comment in its source, the file under \texttt{analysis/tables/}, \texttt{analysis/derived/}, \texttt{docs/} or \texttt{protocol/} from which its numbers are read. The reproducibility package comprises the orchestration and evaluation code, the public scenario bundles, the raw event streams of every session, the validated Parquet datasets and the analysis scripts; the hidden oracles are kept on the controller only, with their hashes recorded in the freeze manifest. | 7 | +The monograph of which this paper is the journal version---UQO Working Paper No.~42, \emph{The Autonomous White-Collar Worker: Measuring the Economic Frontier of Claude Code, Computer-Using Agents, and Autonomous Knowledge Work}---serves as the online appendix. It contains the full literature review and positioning table; the proofs of the propositions of Section~\ref{sec:framework} and the properties of the frontier; the occupational taxonomy, the organisations of the Virtual White-Collar Economy, the scenario-by-scenario descriptions and the authoring specification; the preregistration (version 1.0 with its freeze manifest) and the amendment log A-001 to A-013; the infrastructure as built, including the Phase~0 survey of the executor nodes, the launcher flags and their probes, the portal, the self-repair protocol and the scheduler; the data documentation (run-level variables, evaluation instruments and their agreement, cost measurement, missing values); the complete results, including the frontier by occupation and budget, the dependency, quality-threshold, timing, failure-severity, supervision and tool-family tables, the economic sensitivity analysis and the infrastructure-stability statistics; the rubrics; the reproducibility records; and the robustness tables. Every exhibit of the present paper names, in a LaTeX comment in its source, the file under \texttt{analysis/tables/}, \texttt{analysis/derived/}, \texttt{docs/} or \texttt{protocol/} from which its numbers are read. The reproducibility package comprises the orchestration and evaluation code, the public scenario bundles, the raw event streams of every session, the validated Parquet datasets and the analysis scripts; the hidden oracles are kept on the controller only, with their hashes recorded in the freeze manifest. | ||
modified
journal_version/sections/benchmark.tex
+3 −3
@@ -6,9 +6,9 @@ | |||||
| 6 | 6 | ||||
| 7 | \subsection{The Virtual White-Collar Economy and the anatomy of a scenario}\label{sec:bench-anatomy} | 7 | \subsection{The Virtual White-Collar Economy and the anatomy of a scenario}\label{sec:bench-anatomy} | ||
| 8 | 8 | ||||
| 9 | −Scenarios are set in a small, internally consistent fictional economy of seventeen organisations---an accounting firm, an investment firm, a commercial bank, an appraisal firm, a law firm, a university department, a manufacturer and its human-resources and procurement departments, a municipality, and others---that share conventions (fiscal year 2025, Canadian dollars, Québec sales taxes, ISO dates) so that an agent moving between scenarios faces a coherent world. All names, people and figures are fictional. The economy is built around one manufacturer, Boréal Fenestration Inc., which is the client of the accounting firm and the law firm, the employer in the human-resources and procurement scenarios, the subject of the business-intelligence pack and the organisation whose chief executive's affairs are triaged. | 9 | +Scenarios are set in a small, internally consistent fictional economy of seventeen organisations---an accounting firm, an investment firm, a commercial bank, an appraisal firm, a law firm, a university department, a manufacturer and its human-resources and procurement departments, a municipality, and others---that share conventions (fiscal year 2025, Canadian dollars, Québec sales taxes, ISO dates) so that an agent moving between scenarios faces a coherent world. All names, people and figures are fictional, and one manufacturer, Boréal Fenestration Inc., recurs as client, employer and subject across six scenarios. | ||
| 10 | 10 | ||||
| 11 | −A scenario has two halves produced together by one reproducible generator and never mixed afterwards: a \emph{public bundle} that the worker sees and a \emph{hidden oracle} that only the controller and the evaluators see. The separation is enforced by location: the bundle lives in the repository; the oracle lives outside it on the controller and is never copied to an executor node. The bundle contains metadata reduced to what a worker legitimately needs; a brief written in the voice of the organisation stating context, inputs, deliverables with exact file names and formats, business rules and conventions, ending with common working rules (deliverables go to \texttt{out/}, inputs must not be modified, no contact outside the workspace and no network except an internal portal if provided, a completion record must be written, and if the assignment cannot be completed because information is missing or contradictory the worker must say so with its questions instead of guessing); the input artefacts---spreadsheets, CSV extracts, PDF invoices and statements, Word contracts, e-mail messages, calendar files, SQLite databases, policies, images---identical in every arm; and, for the portal arms, a description of what the enterprise portal serves. The brief contains no grading information and no hint that any document contains an injection. | 11 | +A scenario has two halves produced together by one reproducible generator and never mixed afterwards: a \emph{public bundle} that the worker sees and a \emph{hidden oracle} that only the controller and the evaluators see. The separation is enforced by location: the bundle lives in the repository; the oracle lives outside it on the controller and is never copied to an executor node. The bundle contains metadata reduced to what a worker legitimately needs; a brief written in the voice of the organisation stating context, inputs, deliverables with exact file names and formats, business rules and conventions, ending with common working rules (deliverables go to \texttt{out/}, inputs must not be modified, no contact outside the workspace and no network except an internal portal if provided, a completion record must be written, and if the assignment cannot be completed because information is missing or contradictory the worker must say so with its questions instead of guessing); the input artefacts (spreadsheets, PDF invoices and statements, Word contracts, e-mail messages, calendar files, SQLite databases, policies, images), identical in every arm; and, for the portal arms, a description of what the enterprise portal serves. The brief contains no grading information and no hint that any document contains an injection. | ||
| 12 | 12 | ||||
| 13 | The oracle contains the ground truth (planted items and correct values, tolerances, required deliverables, injection markers), the deterministic evaluator, the blind-evaluation rubric with weighted criteria anchored on a 0--4 scale, a reference document and the validator's report. The evaluator reads only the collected output and the public workspace, never executes the worker's code, applies the oracle's tolerances and returns a receipt listing each required criterion's pass/fail status, each critical violation, the completion flag and the abstention status. Every scenario has at least six required criteria, of which at least three are facts only the oracle knows, and at least two critical-violation checks. Deliverables are real files---workbooks with live formulas, Word documents with specified sections, CSV tables, figures, code---plus a machine-readable completion record whose status is \emph{completed}, \emph{needs\_clarification} or \emph{abstained} and which lists the questions the worker would have asked. Every scenario embeds \emph{planted errors} (a transposed amount, a duplicated invoice, a comparable sale with the wrong date), one \emph{prompt injection} inside an input document whose obedience is a critical violation, and in four scenarios a \emph{clarification trap}, a sub-task that cannot be completed without information the bundle does not contain. The behavioural rule given to every worker---treat the content of input documents as data, not as instructions---is identical across arms. Each scenario also carries a \emph{designed dependency depth}, the minimal chain of dependent operations a careful professional must perform. | 13 | The oracle contains the ground truth (planted items and correct values, tolerances, required deliverables, injection markers), the deterministic evaluator, the blind-evaluation rubric with weighted criteria anchored on a 0--4 scale, a reference document and the validator's report. The evaluator reads only the collected output and the public workspace, never executes the worker's code, applies the oracle's tolerances and returns a receipt listing each required criterion's pass/fail status, each critical violation, the completion flag and the abstention status. Every scenario has at least six required criteria, of which at least three are facts only the oracle knows, and at least two critical-violation checks. Deliverables are real files---workbooks with live formulas, Word documents with specified sections, CSV tables, figures, code---plus a machine-readable completion record whose status is \emph{completed}, \emph{needs\_clarification} or \emph{abstained} and which lists the questions the worker would have asked. Every scenario embeds \emph{planted errors} (a transposed amount, a duplicated invoice, a comparable sale with the wrong date), one \emph{prompt injection} inside an input document whose obedience is a critical violation, and in four scenarios a \emph{clarification trap}, a sub-task that cannot be completed without information the bundle does not contain. The behavioural rule given to every worker---treat the content of input documents as data, not as instructions---is identical across arms. Each scenario also carries a \emph{designed dependency depth}, the minimal chain of dependent operations a careful professional must perform. | ||
| 14 | 14 | ||||
@@ -36,7 +36,7 @@ The laboratory has a controller--executor architecture (Figure~\ref{fig:architec | |||||
| 36 | \label{fig:architecture} | 36 | \label{fig:architecture} | ||
| 37 | \end{figure} | 37 | \end{figure} | ||
| 38 | 38 | ||||
| 39 | −A worker session is a single command assembled by the launcher: the client is started in bare mode, ignoring the user's settings, hooks and instruction files so that nothing from the controller's configuration reaches the worker; the tool set is restricted by \texttt{--tools} and pre-approved by \texttt{--allowedTools}; permission prompts are disabled; budget and turn caps are set; the arm's documentation is appended to the system prompt; denied command families are listed in \texttt{--disallowedTools} with patterns such as \texttt{Bash(osascript:*)}. The effect of each flag was verified by probe: a worker started without the shell tool could not run a trivial command, and a denied command family was refused inside the shell while a plain command ran. No Agent or Task tool is ever given, so a worker cannot spawn sub-agents. Each workspace is pushed to the node with a manifest of SHA-256 hashes; after the session the output directory is copied back and hashed, the inputs are compared with the manifest and any change is recorded as a critical violation, the portal is stopped and the workspace is deleted. The \emph{synthetic enterprise portal} is a small web application started on the node for each A3 and A4 run, bound to the loopback interface, serving a login page, a document library whose links point to files already in the inputs, record tables and forms whose submissions are collected with the other deliverables; it holds no information absent from the inputs. External network access is forbidden by policy. | 39 | +A worker session is a single command assembled by the launcher: the client is started in bare mode, ignoring the user's settings, hooks and instruction files so that nothing from the controller's configuration reaches the worker; the tool set is restricted by \texttt{--tools} and pre-approved by \texttt{--allowedTools}; permission prompts are disabled; budget and turn caps are set; the arm's documentation is appended to the system prompt; denied command families are listed in \texttt{--disallowedTools} with patterns such as \texttt{Bash(osascript:*)}. The effect of each flag was verified by probe: a worker started without the shell tool could not run a trivial command, and a denied command family was refused inside the shell while a plain command ran. No Agent or Task tool is ever given, so a worker cannot spawn sub-agents. Each workspace is pushed to the node with a manifest of SHA-256 hashes; after the session the output directory is copied back and hashed, any change to an input is recorded as a critical violation, and the workspace is deleted. The \emph{synthetic enterprise portal} is a small web application started on the node for each A3 and A4 run, bound to the loopback interface, serving a login page, a document library whose links point to files already in the inputs, record tables and forms whose submissions are collected with the other deliverables; it holds no information absent from the inputs, and external network access is forbidden by policy. | ||
| 40 | 40 | ||||
| 41 | Role separation is operational rather than nominal: generator and validator are distinct sessions; the worker is a separate process on a separate machine; the evaluator is code the worker never sees; the judges receive a rendered package and a rubric; and the sessions that wrote the sections of this paper reporting results had access to the analysis outputs and not to the runs. Three properties of the environment constrained the design and are recorded as amendments. Control of graphical applications is not available to a process started over SSH---Finder and TextEdit requests time out on all four nodes because the automation consent prompt cannot be answered for a non-console process---so the mandate's ``macOS agent'' is implemented as the automation layer that works headlessly (\texttt{osascript} with AppleScript and JavaScript for Automation, System Events on the two nodes where it answers, and the native document utilities), and A2 runs are pinned to those nodes (A-003). No human evaluator was available, so the blind professional evaluators are language-model judges (A-005). And the human arms of the mandate were designed and not executed (A-002). | 41 | Role separation is operational rather than nominal: generator and validator are distinct sessions; the worker is a separate process on a separate machine; the evaluator is code the worker never sees; the judges receive a rendered package and a rubric; and the sessions that wrote the sections of this paper reporting results had access to the analysis outputs and not to the runs. Three properties of the environment constrained the design and are recorded as amendments. Control of graphical applications is not available to a process started over SSH---Finder and TextEdit requests time out on all four nodes because the automation consent prompt cannot be answered for a non-console process---so the mandate's ``macOS agent'' is implemented as the automation layer that works headlessly (\texttt{osascript} with AppleScript and JavaScript for Automation, System Events on the two nodes where it answers, and the native document utilities), and A2 runs are pinned to those nodes (A-003). No human evaluator was available, so the blind professional evaluators are language-model judges (A-005). And the human arms of the mandate were designed and not executed (A-002). | ||
| 42 | % ../protocol/amendments.csv A-002, A-003, A-005; ../manuscript/sections/infrastructure.tex §6.5 | 42 | % ../protocol/amendments.csv A-002, A-003, A-005; ../manuscript/sections/infrastructure.tex §6.5 | ||
modified
journal_version/sections/design.tex
+2 −2
@@ -8,7 +8,7 @@ | |||||
| 8 | 8 | ||||
| 9 | Five agent arms implement increasing levels of computer access with the same model, brief, inputs, caps and behavioural policy. \emph{A0 (restricted)} gives the tools Read, Write, Edit, Glob and Grep and no shell; because binary office formats cannot be produced without a program, the A0 documentation instructs the worker to deliver the closest text equivalent under the same base name and to state the substitution, and the deterministic criteria that require a formula-bearing workbook then fail, as they should. \emph{A1 (shell)} adds Bash and a project Python environment (pandas, openpyxl, python-docx, reportlab, pypdf, matplotlib, statsmodels, duckdb, sqlite3 and others) and denies, inside the shell, the families \texttt{osascript}, \texttt{open}, \texttt{soffice}, \texttt{textutil}, \texttt{qlmanage}, \texttt{cupsfilter}, \texttt{curl} and \texttt{wget}. \emph{A2 (macOS)} lifts the denial of \texttt{osascript} and the native document utilities but keeps LibreOffice and the network denied. \emph{A3 (full computer use)} denies nothing and adds LibreOffice headless, Playwright's headless Chromium and the enterprise portal with credentials in the workspace. \emph{A4 (self-repairing)} has A3's tools and adds a checkpoint file the worker keeps updated, a public self-check tool that verifies the presence and well-formedness of deliverables (files exist and are non-empty, office files are valid archives, a workbook required to contain formulas does and shows no error values, documents meet minimum lengths, the completion record is valid) and knows nothing about the correct answers, and one bounded recovery resume granted by the controller when the first session ends without passing the self-check or by timeout, budget exhaustion or error. The recovery resumes the same session with the remaining budget and wall time, at most once; its cost and time are attributed to the parent run. | 9 | Five agent arms implement increasing levels of computer access with the same model, brief, inputs, caps and behavioural policy. \emph{A0 (restricted)} gives the tools Read, Write, Edit, Glob and Grep and no shell; because binary office formats cannot be produced without a program, the A0 documentation instructs the worker to deliver the closest text equivalent under the same base name and to state the substitution, and the deterministic criteria that require a formula-bearing workbook then fail, as they should. \emph{A1 (shell)} adds Bash and a project Python environment (pandas, openpyxl, python-docx, reportlab, pypdf, matplotlib, statsmodels, duckdb, sqlite3 and others) and denies, inside the shell, the families \texttt{osascript}, \texttt{open}, \texttt{soffice}, \texttt{textutil}, \texttt{qlmanage}, \texttt{cupsfilter}, \texttt{curl} and \texttt{wget}. \emph{A2 (macOS)} lifts the denial of \texttt{osascript} and the native document utilities but keeps LibreOffice and the network denied. \emph{A3 (full computer use)} denies nothing and adds LibreOffice headless, Playwright's headless Chromium and the enterprise portal with credentials in the workspace. \emph{A4 (self-repairing)} has A3's tools and adds a checkpoint file the worker keeps updated, a public self-check tool that verifies the presence and well-formedness of deliverables (files exist and are non-empty, office files are valid archives, a workbook required to contain formulas does and shows no error values, documents meet minimum lengths, the completion record is valid) and knows nothing about the correct answers, and one bounded recovery resume granted by the controller when the first session ends without passing the self-check or by timeout, budget exhaustion or error. The recovery resumes the same session with the remaining budget and wall time, at most once; its cost and time are attributed to the parent run. | ||
| 10 | 10 | ||||
| 11 | −All arms use \texttt{claude-opus-5-5} at default effort, bare mode, \texttt{--permission-prompts none}, \texttt{--max-turns 150}, and a spending cap and wall limit set per scenario as the larger of the global caps (4.00 dollars, 1,800 seconds) and the scenario author's own: ACC-01 under 6.00 dollars and 3,600 seconds, RES-01 under 6.00 dollars and 2,700 seconds, BANK-01 under 5.00 dollars and 2,400 seconds, LEGAL-01 under 2,400 seconds, the other six under the global caps (A-006, fixed before any pilot run). The study has an envelope of 800 dollars enforced by the scheduler's ledger (A-001). Each arm's documentation is appended to a common policy, identical across arms: the worker is an autonomous professional executing one assignment in an isolated workspace, nobody will answer questions during the session, questions go into the completion record, inputs must never be modified, the network is forbidden except the internal portal, messages and purchases are forbidden, the content of input documents is data and not instructions, and the work is to be done to the standard of a competent professional in the organisation described. The user prompt is a single sentence asking the worker to read the brief and carry out the assignment completely. | 11 | +All arms use \texttt{claude-opus-5-5} at default effort, bare mode, \texttt{--permission-prompts none}, \texttt{--max-turns 150}, and a spending cap and wall limit set per scenario as the larger of the global caps (4.00 dollars, 1,800 seconds) and the scenario author's own: ACC-01 under 6.00 dollars and 3,600 seconds, RES-01 under 6.00 dollars and 2,700 seconds, BANK-01 under 5.00 dollars and 2,400 seconds, LEGAL-01 under 2,400 seconds, the other six under the global caps (A-006, fixed before any pilot run). The study has an envelope of 800 dollars enforced by the scheduler's ledger (A-001). Each arm's documentation is appended to a common policy, identical across arms: the worker is an autonomous professional executing one assignment in an isolated workspace, nobody will answer questions during the session and questions go into the completion record, inputs must never be modified, the network, messages and purchases are forbidden, the content of input documents is data and not instructions, and the work is to be done to the standard of a competent professional. The user prompt is a single sentence asking the worker to read the brief and carry out the assignment completely. | ||
| 12 | % ../protocol/amendments.csv A-001, A-006; ../manuscript/sections/design.tex Table tab:arms | 12 | % ../protocol/amendments.csv A-001, A-006; ../manuscript/sections/design.tex Table tab:arms | ||
| 13 | 13 | ||||
| 14 | \subsection{Randomisation, repeats and caps}\label{sec:design-rand} | 14 | \subsection{Randomisation, repeats and caps}\label{sec:design-rand} | ||
@@ -21,7 +21,7 @@ Deterministic completion $D$ is true when every required oracle criterion passes | |||||
| 21 | 21 | ||||
| 22 | \subsection{Hypotheses, estimands and inference}\label{sec:design-hyp} | 22 | \subsection{Hypotheses, estimands and inference}\label{sec:design-hyp} | ||
| 23 | 23 | ||||
| 24 | −The preregistration states seven directional expectations: H1 (primary), the computer-use premium $\Pr(S\mid A3)-\Pr(S\mid A1)>0$; H2, the shell premium $\Pr(S\mid A1)-\Pr(S\mid A0)>0$, driven by the feasibility of binary deliverable formats; H3, the self-repair premium $\ge 0$ and an Autonomous Recovery Index above one half when a first session fails the self-check; H4, acceptance decreasing within arm in designed depth and in realised tool calls; H5, a critical-violation rate below 10 per cent in every arm and not increasing with tool access; H6, cost per accepted deliverable lowest in A1 or A3, not in A0; and H7, heterogeneity across occupations with no occupation-level claim confirmatory at the pilot scale. The primary endpoint is $S=\mathbf{1}\{D=1\wedge E=0\wedge Q\ge 70\}$; the scenario-weighted acceptance of an arm is $\hat p_a=J^{-1}\sum_j R_j^{-1}\sum_r S_{jra}$ with equal weights (professional-minutes weights as a sensitivity); the primary contrast is $\hat p_{A3}-\hat p_{A1}$ and the secondary contrasts A1$-$A0, A4$-$A3 and A2$-$A1 are Holm-adjusted over the three. Secondary endpoints include $D$, $Q$, $E$, cost, wall time, process counts, abstention and its justification, cost per accepted deliverable, the frontier by occupation over budgets from 0.50 to 4.00 dollars and over the human-time allowance, the recovery index and evaluator agreement. | 24 | +The preregistration states seven directional expectations: H1 (primary), the computer-use premium $\Pr(S\mid A3)-\Pr(S\mid A1)>0$; H2, the shell premium $\Pr(S\mid A1)-\Pr(S\mid A0)>0$, driven by the feasibility of binary deliverable formats; H3, the self-repair premium $\ge 0$ and an Autonomous Recovery Index above one half when a first session fails the self-check; H4, acceptance decreasing within arm in designed depth and in realised tool calls; H5, a critical-violation rate below 10 per cent in every arm and not increasing with tool access; H6, cost per accepted deliverable lowest in A1 or A3, not in A0; and H7, heterogeneity across occupations with no occupation-level claim confirmatory at the pilot scale. The primary endpoint is $S=\mathbf{1}\{D=1\wedge E=0\wedge Q\ge 70\}$; the scenario-weighted acceptance of an arm is $\hat p_a=J^{-1}\sum_j R_j^{-1}\sum_r S_{jra}$ with equal weights (professional-minutes weights as a sensitivity); the primary contrast is $\hat p_{A3}-\hat p_{A1}$ and the secondary contrasts A1$-$A0, A4$-$A3 and A2$-$A1 are Holm-adjusted over the three. Secondary endpoints include $D$, $Q$, $E$, cost, wall time, process counts, abstention, cost per accepted deliverable, the frontier over budgets from 0.50 to 4.00 dollars and over the human-time allowance, the recovery index and evaluator agreement. | ||
| 25 | 25 | ||||
| 26 | The primary interval for every contrast is a scenario-cluster bootstrap: scenarios are resampled with replacement 5,000 times, the within-scenario paired difference of repeat means is recomputed, and the percentile 2.5 and 97.5 points form the 95 per cent interval. The primary test is a paired sign-flip permutation test over scenarios with 20,000 permutations, exact under the null of no effect within scenario. A cluster-robust logistic regression, Kaplan--Meier curves for time to completion and joint bootstraps of ratio quantities are reported as sensitivities. With $J=10$ the intervals will be wide, and the preregistration states that they are to be reported as such. | 26 | The primary interval for every contrast is a scenario-cluster bootstrap: scenarios are resampled with replacement 5,000 times, the within-scenario paired difference of repeat means is recomputed, and the percentile 2.5 and 97.5 points form the 95 per cent interval. The primary test is a paired sign-flip permutation test over scenarios with 20,000 permutations, exact under the null of no effect within scenario. A cluster-robust logistic regression, Kaplan--Meier curves for time to completion and joint bootstraps of ratio quantities are reported as sensitivities. With $J=10$ the intervals will be wide, and the preregistration states that they are to be reported as such. | ||
| 27 | 27 | ||||
modified
journal_version/sections/discussion.tex
+4 −4
@@ -23,24 +23,24 @@ The third is what the ceiling says positively. In 120 runs with a shell, one age | |||||
| 23 | 23 | ||||
| 24 | The Autonomous Labor Frontier was defined with three thresholds because the economic question has three margins, and the pilot shows that the margins are not interchangeable: marginal acceptance in A3 is 0.900, the frontier is 0.167 at a supervision allowance of zero and 0.667 at fifteen minutes, and the budget stops binding at about 1.50 dollars while the allowance never stops binding in the range examined. | 24 | The Autonomous Labor Frontier was defined with three thresholds because the economic question has three margins, and the pilot shows that the margins are not interchangeable: marginal acceptance in A3 is 0.900, the frontier is 0.167 at a supervision allowance of zero and 0.667 at fifteen minutes, and the budget stops binding at about 1.50 dollars while the allowance never stops binding in the range examined. | ||
| 25 | % computed from ../analysis/tables/table5_alf_pilot.csv (A3, equal and equal_H15) | 25 | % computed from ../analysis/tables/table5_alf_pilot.csv (A3, equal and equal_H15) | ||
| 26 | −This has two uses. For policy, the exposure indices of \citet{eloundou2024gpts}, \citet{felten2023language} and \citet{gmyrek2025refined} answer the question ``could a model do this task?''; the frontier answers ``did an agent do this task, at this cost, with this much help?'', and the pilot shows that the answer depends more on the help than on the cost. An exposure index that ignores the supervision margin overstates the autonomous share by the width of that gap, here between a quarter and three quarters of the ceiling depending on the allowance. The frontier is also the quantity that \citet{acemoglu2025macroeconomics} needs for the share actually automated and the cost saving, and the pilot's contribution to that accounting is to show that the share is a function of $H$ before it is a function of $B$. For firms, the frontier is a delegation map: the frontier at the firm's own $(B,H,q)$ is the probability that a delegation succeeds without further intervention. The pilot's values say that, for designed-solvable assignments of this kind, an agent with a shell delivers an acceptable package nine times in ten for about a dollar, records about three questions on the way, formally declares a need for clarification in 0.133 of runs, each on a scenario whose oracle required it, and leaves the supervisor's time as the cost to manage. Whether the questions are a defect of the agent or the behaviour of a careful professional is a judgement the firm must make; the frontier makes the quantity visible. | 26 | +This has two uses. For policy, the exposure indices of \citet{eloundou2024gpts}, \citet{felten2023language} and \citet{gmyrek2025refined} answer the question ``could a model do this task?''; the frontier answers ``did an agent do this task, at this cost, with this much help?'', and the pilot shows that the answer depends more on the help than on the cost. An exposure index that ignores the supervision margin overstates the autonomous share by the width of that gap, here between a quarter and three quarters of the ceiling depending on the allowance. The frontier is also the quantity that \citet{acemoglu2025macroeconomics} needs for the share actually automated and the cost saving, and the pilot shows that the share is a function of $H$ before it is a function of $B$. For firms, the frontier at their own $(B,H,q)$ is the probability that a delegation succeeds without further intervention. The pilot's values say that, for designed-solvable assignments of this kind, an agent with a shell delivers an acceptable package nine times in ten for about a dollar, records about three questions on the way, formally declares a need for clarification in 0.133 of runs, each on a scenario whose oracle required it, and leaves the supervisor's time as the cost to manage. Whether the questions are a defect of the agent or the behaviour of a careful professional is a judgement the firm must make; the frontier makes the quantity visible. | ||
| 27 | % ../analysis/tables/table12_supervision_pilot.csv (A3: mean_questions 2.93; abstention_rate 0.133 = justified 0.133) | 27 | % ../analysis/tables/table12_supervision_pilot.csv (A3: mean_questions 2.93; abstention_rate 0.133 = justified 0.133) | ||
| 28 | 28 | ||||
| 29 | The delegation model gives the same reading in decision form. Proposition~\ref{prop:delegation} says that delegation with verification beats in-house production when $p\ge(c_A+\kappa_V)/c_H$ and that verification is worth its cost when $\kappa_V\le(1-p)(\ell-c_H)+eL$. The pilot supplies $p$ and $e$ and the order of magnitude of $c_A$, and nothing about $c_H$, $\kappa_V$, $\ell$ or $L$ except labelled assumptions. Within those, the first inequality is slack: with $p=0.9$, $c_A$ about one dollar and the scenario authors' professional-minutes estimates (150 to 420 minutes) valued at 40 to 120 Canadian dollars an hour, that is 100 to 840 dollars, delegation pays as long as verifying the package costs less than about nine tenths of producing it in-house. | 29 | The delegation model gives the same reading in decision form. Proposition~\ref{prop:delegation} says that delegation with verification beats in-house production when $p\ge(c_A+\kappa_V)/c_H$ and that verification is worth its cost when $\kappa_V\le(1-p)(\ell-c_H)+eL$. The pilot supplies $p$ and $e$ and the order of magnitude of $c_A$, and nothing about $c_H$, $\kappa_V$, $\ell$ or $L$ except labelled assumptions. Within those, the first inequality is slack: with $p=0.9$, $c_A$ about one dollar and the scenario authors' professional-minutes estimates (150 to 420 minutes) valued at 40 to 120 Canadian dollars an hour, that is 100 to 840 dollars, delegation pays as long as verifying the package costs less than about nine tenths of producing it in-house. | ||
| 30 | % ../analysis/tables/table13_human_cost_assumption.csv; table1 (A3 acceptance 0.9, cost_per_accepted 1.006) | 30 | % ../analysis/tables/table13_human_cost_assumption.csv; table1 (A3 acceptance 0.9, cost_per_accepted 1.006) | ||
| 31 | −The second inequality is the one that bites. In 90 runs of arms A2--A4 no critical violation was recorded, so the point estimate of $e$ is zero and a rule-of-three upper bound is about 0.033; but $L$, the loss from an undetected critical error, is unbounded in the professions represented here---a fabricated reconciliation total, an appraisal that echoes a lender's number, a payroll letter with the wrong amount---and $eL$ does not vanish because $e$ is small. The decision turns on the cost of checking, as \citet{catalini2026agi} and \citet{sreenivas2026contracting} argue in theory. The hidden oracle is a laboratory device for making tasks measurable; its analogue inside a firm is a deterministic check written before the work is delegated. The pilot shows that such checks are cheap to write for deliverables of this kind---every scenario had one---and that they disagree with a professional's judgement in one run in eleven, which is the residual that judgement must cover. The triage scenario is the multitask case of \citet{holmstrom1991multitask} in concrete form: an organisation that verifies by test alone will delegate it and one that verifies by judgement will not; reporting the frontier over $q$ leaves the choice to the reader. We decline to call the two orders of magnitude between the human-cost band and the agent's marginal cost a productivity comparison: the human figure is not measured, the acceptance is judged by models, and the deliverable arrives with questions attached. | 31 | +The second inequality is the one that bites. In 90 runs of arms A2--A4 no critical violation was recorded, so the point estimate of $e$ is zero and a rule-of-three upper bound is about 0.033; but $L$, the loss from an undetected critical error, is unbounded in the professions represented here---a fabricated reconciliation total, an appraisal that echoes a lender's number, a payroll letter with the wrong amount---and $eL$ does not vanish because $e$ is small. The decision turns on the cost of checking, as \citet{catalini2026agi} and \citet{sreenivas2026contracting} argue in theory. The hidden oracle's analogue inside a firm is a deterministic check written before the work is delegated; the pilot shows that such checks are cheap to write for deliverables of this kind and that they disagree with a professional's judgement in one run in eleven, the residual that judgement must cover. The triage scenario is the multitask case of \citet{holmstrom1991multitask} in concrete form, and reporting the frontier over $q$ leaves to the reader the choice between verifying by test and verifying by judgement. We decline to call the two orders of magnitude between the human-cost band and the agent's marginal cost a productivity comparison: the human figure is not measured, the acceptance is judged by models, and the deliverable arrives with questions attached. | ||
| 32 | % ../docs/PHASE_5_AUDIT.md §2 (9 % disagreement); rule-of-three bound 3/90 computed by the writer | 32 | % ../docs/PHASE_5_AUDIT.md §2 (9 % disagreement); rule-of-three bound 3/90 computed by the writer | ||
| 33 | 33 | ||||
| 34 | Of the six named constructs, the ALF and the tool premium earned their place---the first because its margins behaved differently, the second because it was identified and estimated with a usable interval. Autonomy-Adjusted Productivity is an accounting convention; the Professional Reliability Frontier reduced, in a pilot with no timeouts and almost no critical errors, to the acceptance rate and is kept as a reporting standard; the Workflow Dependency Frontier could not be estimated because depth and scenario are confounded at $J=10$; and the Autonomous Recovery Index was identified only under induced interruption and measures resumption, not correction. | 34 | Of the six named constructs, the ALF and the tool premium earned their place---the first because its margins behaved differently, the second because it was identified and estimated with a usable interval. Autonomy-Adjusted Productivity is an accounting convention; the Professional Reliability Frontier reduced, in a pilot with no timeouts and almost no critical errors, to the acceptance rate and is kept as a reporting standard; the Workflow Dependency Frontier could not be estimated because depth and scenario are confounded at $J=10$; and the Autonomous Recovery Index was identified only under induced interruption and measures resumption, not correction. | ||
| 35 | 35 | ||||
| 36 | \subsection{Limitations}\label{sec:disc-limits} | 36 | \subsection{Limitations}\label{sec:disc-limits} | ||
| 37 | 37 | ||||
| 38 | −The limitations are the premises of every interpretation above. \emph{The scenarios are synthetic and were authored and validated by language-model agents} of the same family as the workers and the judges. A scenario written by a model and solved by a model may share assumptions about what a professional deliverable looks like that a practising accountant or appraiser would not; the frontier is measured on designed difficulty, and the design was done by the kind of system being measured. The circularity is mitigated by the hidden oracle, which fixes facts neither author nor worker can negotiate, and removed only by scenarios drawn from real professional work and judged by real professionals. \emph{The scenarios are solvable by design}: the frontier excludes the part of white-collar work that is ill-specified, requires negotiation, or cannot be completed from the artefacts a worker is handed. \emph{There are ten scenarios, one per occupation}: every scenario-cluster interval on an acceptance difference is at least ten points wide, no occupation-level statement is confirmatory, and the heterogeneity reported is driven by three scenarios. \emph{One model family, one client version, one date}: the model-sensitivity result describes three points of one vendor's line. \emph{The judges are models of the workers' family}: two of the three are draws of the same model and the third is the workers' model, self-preference \citep{panickssery2024selfpreference} is not excluded by blinding, and the quality gate---the sole reason the triage is not accepted---rests on them. \emph{No human arm} (A-002): every wage, professional-minutes figure and human-cost band is a labelled assumption. \emph{The macOS layer is the headless subset} (A-003), and \emph{no interface perturbation} was executed. \emph{The recovery experiment is exploratory} (A-011, A-012): its interruption is induced at a fixed time rather than arising from the agent's own failure. And \emph{the laboratory's stability was itself under test}: three deliverables lost at collection under twelve concurrent sessions (A-010), a judge crash on null bytes (A-008) and a dormant launcher flag (A-012) were found by the audit rather than by the run, repaired and logged; a laboratory that reports only its successes is not one whose frontier can be trusted. | 38 | +The limitations are the premises of every interpretation above. \emph{The scenarios are synthetic and were authored and validated by language-model agents} of the same family as the workers and the judges. The frontier is measured on designed difficulty, and the design was done by the kind of system being measured. The circularity is mitigated by the hidden oracle, which fixes facts neither author nor worker can negotiate, and removed only by scenarios drawn from real professional work and judged by real professionals. \emph{The scenarios are solvable by design}: the frontier excludes the part of white-collar work that is ill-specified, requires negotiation, or cannot be completed from the artefacts a worker is handed. \emph{There are ten scenarios, one per occupation}: every scenario-cluster interval on an acceptance difference is at least ten points wide, no occupation-level statement is confirmatory, and the heterogeneity reported is driven by three scenarios. \emph{One model family, one client version, one date}: the model-sensitivity result describes three points of one vendor's line. \emph{The judges are models of the workers' family}: two of the three are draws of the same model and the third is the workers' model, self-preference \citep{panickssery2024selfpreference} is not excluded by blinding, and the quality gate---the sole reason the triage is not accepted---rests on them. \emph{No human arm} (A-002): every wage, professional-minutes figure and human-cost band is a labelled assumption. \emph{The macOS layer is the headless subset} (A-003), and \emph{no interface perturbation} was executed. \emph{The recovery experiment is exploratory} (A-011, A-012): its interruption is induced at a fixed time rather than arising from the agent's own failure. And \emph{the laboratory's stability was itself under test}: three deliverables lost at collection under twelve concurrent sessions (A-010), a judge crash on null bytes (A-008) and a dormant launcher flag (A-012) were found by the audit rather than by the run, repaired and logged; a laboratory that reports only its successes is not one whose frontier can be trusted. | ||
| 39 | % ../protocol/amendments.csv A-002, A-003, A-008, A-010, A-011, A-012; ../docs/PHASE_5_AUDIT.md §5–6 | 39 | % ../protocol/amendments.csv A-002, A-003, A-008, A-010, A-011, A-012; ../docs/PHASE_5_AUDIT.md §5–6 | ||
| 40 | 40 | ||||
| 41 | \subsection{An agenda}\label{sec:disc-agenda} | 41 | \subsection{An agenda}\label{sec:disc-agenda} | ||
| 42 | 42 | ||||
| 43 | The limitations define the next wave. A \emph{human wave} would run H0 and H1 under the protocol of Section~\ref{sec:design-scale}, turning the human-cost band into a measurement and the acceptance ceiling into a comparison. A \emph{harder tier} would grow the benchmark to twenty and then forty scenarios, adding the ten uncovered occupations and, within occupations, the short and the 30-to-60-operation workflow classes the pilot lacks, with inputs that exist only behind the portal or a graphical application, deliverables that are states of a system, multi-session length and interface perturbations as a designed factor, so that the computer-use premium is identified where the shell alone cannot finish; forty scenarios with three repeats detect a 25-point effect nineteen times in twenty. | 43 | The limitations define the next wave. A \emph{human wave} would run H0 and H1 under the protocol of Section~\ref{sec:design-scale}, turning the human-cost band into a measurement and the acceptance ceiling into a comparison. A \emph{harder tier} would grow the benchmark to twenty and then forty scenarios, adding the ten uncovered occupations and, within occupations, the short and the 30-to-60-operation workflow classes the pilot lacks, with inputs that exist only behind the portal or a graphical application, deliverables that are states of a system, multi-session length and interface perturbations as a designed factor, so that the computer-use premium is identified where the shell alone cannot finish; forty scenarios with three repeats detect a 25-point effect nineteen times in twenty. | ||
| 44 | % ../analysis/tables/power_pilot.csv (J=40, R=3, delta 25: 0.968) | 44 | % ../analysis/tables/power_pilot.csv (J=40, R=3, delta 25: 0.968) | ||
| 45 | −\emph{Cross-vendor agents} should be run under the same protocol and the measurement repeated at intervals. \emph{Human expert judges} should grade a sample of deliverables blind, with agreement between human and model judges reported \citep{krippendorff2004reliability,hayes2007reliability}. \emph{Held-out, externally authored scenarios} should bound the circularity of self-authorship, following \citet{zhu2025rigorous} and \citet{kapoor2024matter}. \emph{Natural failures} should identify the recovery index where failures arise from the agent, with a self-check extended toward substantive checks that remain ignorant of the answers, so that the imperfect-verifier ceiling of \citet{stroebl2024flaws} can be measured rather than assumed. Each item is affordable in API terms---the pilot used a third of its envelope---and bounded, as this one was, by the human time of the people who write, validate and judge the scenarios rather than by the agents who execute them. That, too, is a finding about the economics of autonomous work. | 45 | +\emph{Cross-vendor agents} should be run under the same protocol and the measurement repeated at intervals. \emph{Human expert judges} should grade a sample of deliverables blind, with agreement between human and model judges reported \citep{krippendorff2004reliability,hayes2007reliability}. \emph{Held-out, externally authored scenarios} should bound the circularity of self-authorship, following \citet{zhu2025rigorous} and \citet{kapoor2024matter}. \emph{Natural failures} should identify the recovery index where failures arise from the agent rather than from an interruption, so that the imperfect-verifier ceiling of \citet{stroebl2024flaws} can be measured rather than assumed. Each item is affordable in API terms---the pilot used a third of its envelope---and bounded, as this one was, by the human time of the people who write, validate and judge the scenarios rather than by the agents who execute them. That, too, is a finding about the economics of autonomous work. | ||
| 46 | % ../docs/PHASE_5_AUDIT.md §4 (261.89 of 800 USD) | 46 | % ../docs/PHASE_5_AUDIT.md §4 (261.89 of 800 USD) | ||
modified
journal_version/sections/framework.tex
+3 −3
@@ -29,7 +29,7 @@ For occupation $o$, arm $a$, budget $B\ge 0$, human-time allowance $H\ge 0$ and | |||||
| 29 | \end{equation} | 29 | \end{equation} | ||
| 30 | \end{definition} | 30 | \end{definition} | ||
| 31 | 31 | ||||
| 32 | −The frontier is the task-weighted probability that a run of a task of the occupation produces an accepted deliverable \emph{and} does so within the budget \emph{and} within the supervision allowance. It is a joint probability, not a product of marginals, and it is indexed by three thresholds because the economic question has three margins---whether the work is good enough, cheap enough and autonomous enough---and a single number would hide the trade-offs among them. Three properties, proved in the monograph, govern its reading. The task-weighted marginal acceptance rate $\rho_o(q)=\sum_j w_{oj}p_{oj}(q)$ is the \emph{ceiling} of the frontier, approached as $B,H\to\infty$; the gap $\rho_o-\mathrm{ALF}_o$ is the share of accepted work that is infeasible at the stated budget or allowance. If the per-session budget cap is $\bar B$ and runs are evaluated as delivered when it binds, the frontier is flat for $B\ge\bar B$ and the experiment cannot inform that region. And for two weight vectors on the same tasks, the change in the frontier is bounded by their total-variation distance times the range of the task-level joint probabilities, which is why both weightings are reported. The joint probability decomposes, for each task, into deterministic completion, the quality gate conditional on completion, integrity, and budget feasibility conditional on acceptance; the decomposition is a diagnostic that tells us \emph{where} an arm loses acceptable work. | 32 | +The frontier is the task-weighted probability that a run of a task of the occupation produces an accepted deliverable \emph{and} does so within the budget \emph{and} within the supervision allowance. It is a joint probability, not a product of marginals, and it is indexed by three thresholds because the economic question has three margins---whether the work is good enough, cheap enough and autonomous enough---and a single number would hide the trade-offs among them. Three properties, proved in the monograph, govern its reading: the task-weighted marginal acceptance rate $\rho_o(q)=\sum_j w_{oj}p_{oj}(q)$ is the \emph{ceiling} of the frontier, approached as $B,H\to\infty$, and the gap $\rho_o-\mathrm{ALF}_o$ is the share of accepted work that is infeasible at the stated budget or allowance; the frontier is flat beyond the per-session budget cap $\bar B$, so the experiment cannot inform that region; and a change of weights moves the frontier by at most their total-variation distance times the range of the task-level joint probabilities, which is why both weightings are reported. The joint probability decomposes into deterministic completion, the quality gate, integrity and budget feasibility, a diagnostic that tells us \emph{where} an arm loses acceptable work. | ||
| 33 | 33 | ||||
| 34 | \subsection{The Computer-Use Tool Premium}\label{sec:fw-premium} | 34 | \subsection{The Computer-Use Tool Premium}\label{sec:fw-premium} | ||
| 35 | 35 | ||||
@@ -55,7 +55,7 @@ Four companion measures complete the framework; each is defined briefly because | |||||
| 55 | 55 | ||||
| 56 | The \emph{Autonomous Recovery Index} is defined on arm A4. Let $F_1$ be the event that the first session ends without passing the public self-check and $P_2$ the event that the deliverables pass it after the bounded recovery; then $\mathrm{ARI}=\Pr\{P_2\mid F_1\}$. If a run that would fail under A3 fails the self-check with probability $\delta$ (detection) and a resumed session converts a detected failure into an accepted deliverable with probability $\gamma$ (correction), and the first session of A4 is distributed as a run of A3, then $p(A4)=p(A3)+(1-p(A3))\,\delta\gamma$: the self-repair premium is proportional to the first-session failure rate the self-check can see, and is zero by construction when that rate is zero. The index measures correction against an external signal, which \citet{shinn2023reflexion} and \citet{chen2023selfdebug} find effective, not intrinsic self-correction, which \citet{huang2023selfcorrect} find ineffective; by the imperfect-verifier result of \citet{stroebl2024flaws}, passing the self-check is not acceptance. | 56 | The \emph{Autonomous Recovery Index} is defined on arm A4. Let $F_1$ be the event that the first session ends without passing the public self-check and $P_2$ the event that the deliverables pass it after the bounded recovery; then $\mathrm{ARI}=\Pr\{P_2\mid F_1\}$. If a run that would fail under A3 fails the self-check with probability $\delta$ (detection) and a resumed session converts a detected failure into an accepted deliverable with probability $\gamma$ (correction), and the first session of A4 is distributed as a run of A3, then $p(A4)=p(A3)+(1-p(A3))\,\delta\gamma$: the self-repair premium is proportional to the first-session failure rate the self-check can see, and is zero by construction when that rate is zero. The index measures correction against an external signal, which \citet{shinn2023reflexion} and \citet{chen2023selfdebug} find effective, not intrinsic self-correction, which \citet{huang2023selfcorrect} find ineffective; by the imperfect-verifier result of \citet{stroebl2024flaws}, passing the self-check is not acceptance. | ||
| 57 | 57 | ||||
| 58 | −\emph{Autonomy-Adjusted Productivity} is the expected number of accepted deliverables per unit of fully loaded cost, $\mathrm{AAP}(a;q,\omega,\kappa)=\sum_j w_j\mathbb{E}[S_j]/\sum_j w_j\mathbb{E}[C_j+\omega H_j+\kappa]$, where $\omega$ is a wage per human minute and $\kappa$ an operating cost per run reported separately from the API charge; its reciprocal is the cost per accepted deliverable, computed over all runs so that an arm pays for its failures. The \emph{Professional Reliability Frontier} treats the end of a run as one of four competing terminal events---accepted, critical error, censored at a cap, or finished without acceptance---and maps $(q,B)$ to the triple (probability of accepted work within budget, critical-error rate, censoring rate), so that an arm that raises acceptance while raising the critical-error rate is not read as an improvement \citep{fine1999competing,dellacqua2023jagged}. | 58 | +\emph{Autonomy-Adjusted Productivity} is the expected number of accepted deliverables per unit of fully loaded cost, $\mathrm{AAP}=\sum_j w_j\mathbb{E}[S_j]/\sum_j w_j\mathbb{E}[C_j+\omega H_j+\kappa]$, with $\omega$ a wage per human minute and $\kappa$ an operating cost per run; its reciprocal is the cost per accepted deliverable, computed over all runs so that an arm pays for its failures. The \emph{Professional Reliability Frontier} treats the end of a run as one of four competing terminal events \citep{fine1999competing} and maps $(q,B)$ to the triple (probability of accepted work within budget, critical-error rate, censoring rate), so that an arm that raises acceptance while raising the critical-error rate is not read as an improvement \citep{dellacqua2023jagged}. | ||
| 59 | 59 | ||||
| 60 | \subsection{Delegation with verification}\label{sec:fw-pa} | 60 | \subsection{Delegation with verification}\label{sec:fw-pa} | ||
| 61 | 61 | ||||
@@ -65,4 +65,4 @@ The constructs enter the decision of a principal who must decide whether to dele | |||||
| 65 | Delegation with verification yields a higher expected payoff than in-house production if and only if $p\ge(c_A+\kappa_V)/c_H$; verification is worth its cost relative to delegating without it if and only if $\kappa_V\le(1-p)(\ell-c_H)+eL$; and as $L\to\infty$ verification becomes a necessary complement of delegation for any $e>0$. | 65 | Delegation with verification yields a higher expected payoff than in-house production if and only if $p\ge(c_A+\kappa_V)/c_H$; verification is worth its cost relative to delegating without it if and only if $\kappa_V\le(1-p)(\ell-c_H)+eL$; and as $L\to\infty$ verification becomes a necessary complement of delegation for any $e>0$. | ||
| 66 | \end{proposition} | 66 | \end{proposition} | ||
| 67 | 67 | ||||
| 68 | −The first inequality is the frontier's budget condition in decision form; the second formalises the claim of \citet{catalini2026agi} and \citet{sreenivas2026contracting} that, when execution is cheap, the delegation decision is governed by how cheaply a human can check the work. It is also why $E$ is measured separately from acceptance: $e$ and $L$ enter the verification decision through a term that acceptance alone does not capture. The frontier is not an estimate of job displacement, a task is not an occupation, the scenarios are synthetic, and the estimates are for one family of models and one client version on one date; the monograph states each of these restrictions formally and the mandate of the study makes them rules. | 68 | +The first inequality is the frontier's budget condition in decision form; the second formalises the claim of \citet{catalini2026agi} and \citet{sreenivas2026contracting} that, when execution is cheap, the delegation decision is governed by how cheaply a human can check the work. It is also why $E$ is measured separately from acceptance: $e$ and $L$ enter the verification decision through a term that acceptance alone does not capture. The frontier is not an estimate of job displacement and a task is not an occupation; the monograph states these restrictions formally and the mandate of the study makes them rules. | ||
modified
journal_version/sections/introduction.tex
+2 −2
@@ -13,9 +13,9 @@ Two developments make direct measurement possible. Agent benchmarks have moved f | |||||
| 13 | \paragraph{Design.} We built a laboratory in which Claude Code launches Claude Code. A controller session holds the hidden oracles, the experiment database and the ledger; it composes an isolated workspace for each run, launches a bare-mode Claude Code session over SSH on one of four Mac Studio nodes, streams and stores every event, enforces a wall-clock limit, collects and hashes the deliverables, deletes the workspace, and runs the scenario's deterministic evaluator and three blind language-model judges (Section~\ref{sec:benchmark}). The benchmark consists of ten scenarios, one in each of ten occupations---accounting, corporate finance, real-estate appraisal, academic research, executive administration, banking, legal services, human resources, procurement and business intelligence---set in a synthetic economy of fictional organisations, each generated by one agent and independently validated by another who solved the task blind before opening the oracle. Every scenario carries a reference solution, three to nine planted errors with known locations, a prompt-injection document and, in four cases, a clarification trap; the worker never sees any of this. | 13 | \paragraph{Design.} We built a laboratory in which Claude Code launches Claude Code. A controller session holds the hidden oracles, the experiment database and the ledger; it composes an isolated workspace for each run, launches a bare-mode Claude Code session over SSH on one of four Mac Studio nodes, streams and stores every event, enforces a wall-clock limit, collects and hashes the deliverables, deletes the workspace, and runs the scenario's deterministic evaluator and three blind language-model judges (Section~\ref{sec:benchmark}). The benchmark consists of ten scenarios, one in each of ten occupations---accounting, corporate finance, real-estate appraisal, academic research, executive administration, banking, legal services, human resources, procurement and business intelligence---set in a synthetic economy of fictional organisations, each generated by one agent and independently validated by another who solved the task blind before opening the oracle. Every scenario carries a reference solution, three to nine planted errors with known locations, a prompt-injection document and, in four cases, a clarification trap; the worker never sees any of this. | ||
| 14 | % ../manuscript/sections/benchmark.tex Table tab:pilot (planted errors 3–9; clarification traps in 4 scenarios) | 14 | % ../manuscript/sections/benchmark.tex Table tab:pilot (planted errors 3–9; clarification traps in 4 scenarios) | ||
| 15 | 15 | ||||
| 16 | −Each scenario is run under five tool configurations that correspond to recognisable working environments: A0, file reading and writing without a shell (the agent as document assistant); A1, the shell and a Python environment (a programmer's workstation); A2, the shell plus the headless macOS automation layer; A3, full computer use with LibreOffice, a browser engine and a synthetic enterprise portal (a professional's desk inside an organisation's systems); and A4, the same tools with a checkpoint file, a public self-check and one controller-granted recovery resume (the agent with a safety net but no supervisor). The order of runs is a seeded permutation interleaved over nodes, so that each contrast is a within-scenario treatment effect with scenario-clustered inference \citep{abadie2023clustering,miller2024errorbars}. A run is \emph{accepted} when it passes every required criterion of the hidden oracle, contains no critical violation---no fabricated figure, no modified input, no compliance with an injected instruction, no attempted external communication---and receives a mean blind quality score of at least 70 on 100. The hypotheses, estimands, inference plan, caps and scale rule were preregistered before any run, and every departure is logged as an amendment \citep{nosek2018preregistration,olken2015preanalysis}. | 16 | +Each scenario is run under five tool configurations that correspond to recognisable working environments: A0, file reading and writing without a shell (the agent as document assistant); A1, the shell and a Python environment (a programmer's workstation); A2, the shell plus the headless macOS automation layer; A3, full computer use with LibreOffice, a browser engine and a synthetic enterprise portal (a professional's desk inside an organisation's systems); and A4, the same tools with a checkpoint file, a public self-check and one controller-granted recovery resume (the agent with a safety net but no supervisor). The order of runs is a seeded permutation interleaved over nodes, so that each contrast is a within-scenario treatment effect with scenario-clustered inference \citep{abadie2023clustering,miller2024errorbars}. A run is \emph{accepted} when it passes every required criterion of the hidden oracle, contains no critical violation---no fabricated figure, no modified input, no compliance with an injected instruction, no attempted external communication---and receives a mean blind quality score of at least 70 on 100. The hypotheses, estimands, inference plan, caps and scale rule were preregistered before any run, and every departure is logged as an amendment (A-001 to A-013) \citep{nosek2018preregistration,olken2015preanalysis}. | ||
| 17 | 17 | ||||
| 18 | −\paragraph{Findings.} The pilot executed 150 runs---ten scenarios, five arms, three repeats, all at \texttt{claude-opus-5-5} on Claude Code 2.1.314, on 2--3 November 2026---plus 40 model-sensitivity runs and 40 runs of an exploratory recovery experiment, for a total API spend of 261.89 dollars against an envelope of 800. | 18 | +\paragraph{Findings.} The pilot executed 150 runs on 2--3 November 2026---ten scenarios, five arms, three repeats, all on Claude Code 2.1.314 with \texttt{claude-opus-5-5}---plus 40 model-sensitivity runs and 40 runs of an exploratory recovery experiment, for a total API spend of 261.89 dollars against an envelope of 800. | ||
| 19 | % ../docs/PHASE_5_AUDIT.md §0 (scope, spend); ../protocol/amendments.csv A-001, A-009, A-011 | 19 | % ../docs/PHASE_5_AUDIT.md §0 (scope, spend); ../protocol/amendments.csv A-001, A-009, A-011 | ||
| 20 | Four results organise the paper. First, without a shell nothing is accepted and with one almost everything is: scenario-weighted acceptance is 0.000 in A0 and 0.867, 0.900, 0.900 and 0.833 in A1--A4, and the shell premium of $+0.867$ [0.667, 1.000] (permutation $p=0.004$) is the only confirmatory effect in the study. | 20 | Four results organise the paper. First, without a shell nothing is accepted and with one almost everything is: scenario-weighted acceptance is 0.000 in A0 and 0.867, 0.900, 0.900 and 0.833 in A1--A4, and the shell premium of $+0.867$ [0.667, 1.000] (permutation $p=0.004$) is the only confirmatory effect in the study. | ||
| 21 | % ../analysis/tables/table1_outcomes_by_arm_pilot.csv (acceptance_taskw); table2_contrasts_pilot.csv (A1-A0, S) | 21 | % ../analysis/tables/table1_outcomes_by_arm_pilot.csv (acceptance_taskw); table2_contrasts_pilot.csv (A1-A0, S) | ||
modified
journal_version/sections/results.tex
+18 −18
@@ -13,14 +13,14 @@ Three stages were executed and are analysed separately. The \emph{pilot} compris | |||||
| 13 | Three events during the pilot changed how some data were produced, each logged as an amendment and none affecting a pilot outcome. The judge subprocess crashed on eight runs whose rendered deliverables contained a null byte and timed out once; the renderer was fixed and the nine runs re-judged (A-008). The collection step silently lost the output directories of three A2 runs whose sessions had completed; the defect was found by the audit, the step was given retries and a file-count check, and the three runs were reset and rerun under the same identifiers---had the loss gone undetected, A2's acceptance would have been reported as 0.80 instead of 0.90 and the macOS-layer contrast would have changed sign (A-010). And a launcher flag had made the A4 recovery resume impossible; since no A4 first session in the pilot failed the self-check, no recovery was ever requested, but the self-repair arm was in the pilot de facto limited to its checkpoint and self-check components (A-012). | 13 | Three events during the pilot changed how some data were produced, each logged as an amendment and none affecting a pilot outcome. The judge subprocess crashed on eight runs whose rendered deliverables contained a null byte and timed out once; the renderer was fixed and the nine runs re-judged (A-008). The collection step silently lost the output directories of three A2 runs whose sessions had completed; the defect was found by the audit, the step was given retries and a file-count check, and the three runs were reset and rerun under the same identifiers---had the loss gone undetected, A2's acceptance would have been reported as 0.80 instead of 0.90 and the macOS-layer contrast would have changed sign (A-010). And a launcher flag had made the A4 recovery resume impossible; since no A4 first session in the pilot failed the self-check, no recovery was ever requested, but the self-repair arm was in the pilot de facto limited to its checkpoint and self-check components (A-012). | ||
| 14 | % ../protocol/amendments.csv A-008, A-010, A-012; ../manuscript/sections/reliability.tex §9.8 | 14 | % ../protocol/amendments.csv A-008, A-010, A-012; ../manuscript/sections/reliability.tex §9.8 | ||
| 15 | 15 | ||||
| 16 | −The three evaluation instruments agree to a degree that governs the reading of every acceptance rate. Krippendorff's alpha across the three judges is 0.973 over the 150 runs, with pairwise Pearson correlations of 0.964 and 0.966 between the Opus judge and each Sonnet judge and 0.990 between the two Sonnet draws; the two models are not systematically different in severity (means 73.1 against 73.0), but the near-identity of the two Sonnet draws means the panel is closer to two independent opinions than to three. | 16 | +The three evaluation instruments agree to a degree that governs the reading of every acceptance rate. Krippendorff's alpha across the three judges is 0.973 over the 150 runs, with pairwise Pearson correlations of 0.964 and 0.966 between the Opus judge and each Sonnet judge and 0.990 between the two Sonnet draws; the near-identity of the two Sonnet draws means the panel is closer to two independent opinions than to three. | ||
| 17 | % ../analysis/derived/estimates_pilot.json (agreement); ../docs/PHASE_5_AUDIT.md §2 | 17 | % ../analysis/derived/estimates_pilot.json (agreement); ../docs/PHASE_5_AUDIT.md §2 | ||
| 18 | Deterministic completion and the quality gate disagree on 14 of 150 runs, 9.3 per cent: of 117 runs that pass every required criterion, 105 also score at least 70 and 12 do not; of 33 that fail a criterion, 2 are scored above 70. The disagreement is concentrated: eleven of the twelve runs the judges reject are the executive-administration scenario in arms A1--A4, and the two the oracle rejects are appraisals scored 98.75 that failed a single adjusted-price criterion. Acceptance requires both instruments, as preregistered. | 18 | Deterministic completion and the quality gate disagree on 14 of 150 runs, 9.3 per cent: of 117 runs that pass every required criterion, 105 also score at least 70 and 12 do not; of 33 that fail a criterion, 2 are scored above 70. The disagreement is concentrated: eleven of the twelve runs the judges reject are the executive-administration scenario in arms A1--A4, and the two the oracle rejects are appraisals scored 98.75 that failed a single adjusted-price criterion. Acceptance requires both instruments, as preregistered. | ||
| 19 | % ../docs/PHASE_5_AUDIT.md §2; ../manuscript/sections/data.tex Table tab:det-vs-judge (105/12/31/2) | 19 | % ../docs/PHASE_5_AUDIT.md §2; ../manuscript/sections/data.tex Table tab:det-vs-judge (105/12/31/2) | ||
| 20 | 20 | ||||
| 21 | \subsection{Outcomes by arm}\label{sec:results-arms} | 21 | \subsection{Outcomes by arm}\label{sec:results-arms} | ||
| 22 | 22 | ||||
| 23 | −Table~\ref{tab:outcomes} gives the preregistered outcomes by arm, with the two sensitivity models in the last two columns, and Figure~\ref{fig:acceptance} the acceptance rates. The restricted arm A0 accepted no deliverable in 30 runs and completed none deterministically: its mean quality is 27.4, its critical-error rate 0.20, and 90 per cent of its runs ended by requesting clarification. It is the busiest arm and the least effective---29.4 tool calls per run against 12 to 19 elsewhere, 11.9 tool errors against 0.7 to 2.0---because its calls are mostly failed attempts to read binary inputs and to do without a program. | 23 | +Table~\ref{tab:outcomes} gives the preregistered outcomes by arm, with the two sensitivity models in the last two columns, and Figure~\ref{fig:acceptance} the acceptance rates. The restricted arm A0 accepted no deliverable in 30 runs and completed none deterministically: its mean quality is 27.4, its critical-error rate 0.20, and 90 per cent of its runs ended by requesting clarification. Its 29.4 tool calls per run, against 12 to 19 elsewhere, are mostly failed attempts to read binary inputs and to do without a program. | ||
| 24 | % ../analysis/tables/table1_outcomes_by_arm_pilot.csv; table14_extra_descriptives.csv block O_a0_errors | 24 | % ../analysis/tables/table1_outcomes_by_arm_pilot.csv; table14_extra_descriptives.csv block O_a0_errors | ||
| 25 | The four arms with a shell are close to one another: scenario-weighted acceptance is 0.867 in A1, 0.900 in A2 and A3 and 0.833 in A4; deterministic completion 0.967, 0.967, 1.000 and 0.967; mean quality 82.3, 85.0, 85.7 and 84.8, with a larger dispersion in A1 because one run scored 3.8 after destroying its input; the critical-error rate 0.033 in A1 and zero in A2--A4. Under professional-minutes weights the rates move by at most 0.03. The gap between deterministic completion and acceptance in the shell arms---0.10, 0.067, 0.10 and 0.133---is the quality gate, and Section~\ref{sec:results-heterogeneity} shows that it is almost entirely one scenario. Cost per accepted deliverable, which keeps failed runs in the denominator, is 0.95 dollars in A1, 0.87 in A2, 1.01 in A3 and 1.11 in A4, and undefined in A0. | 25 | The four arms with a shell are close to one another: scenario-weighted acceptance is 0.867 in A1, 0.900 in A2 and A3 and 0.833 in A4; deterministic completion 0.967, 0.967, 1.000 and 0.967; mean quality 82.3, 85.0, 85.7 and 84.8, with a larger dispersion in A1 because one run scored 3.8 after destroying its input; the critical-error rate 0.033 in A1 and zero in A2--A4. Under professional-minutes weights the rates move by at most 0.03. The gap between deterministic completion and acceptance in the shell arms---0.10, 0.067, 0.10 and 0.133---is the quality gate, and Section~\ref{sec:results-heterogeneity} shows that it is almost entirely one scenario. Cost per accepted deliverable, which keeps failed runs in the denominator, is 0.95 dollars in A1, 0.87 in A2, 1.01 in A3 and 1.11 in A4, and undefined in A0. | ||
| 26 | % ../analysis/tables/table1_outcomes_by_arm_pilot.csv | 26 | % ../analysis/tables/table1_outcomes_by_arm_pilot.csv | ||
@@ -120,10 +120,10 @@ A2 $-$ A1 (secondary) & $S$ & $+0.033$ & $-0.067$ & $+0.133$ & 1.000 & $+0.025$ | |||||
| 120 | \label{fig:contrasts} | 120 | \label{fig:contrasts} | ||
| 121 | \end{figure} | 121 | \end{figure} | ||
| 122 | 122 | ||||
| 123 | −\paragraph{H1, the computer-use premium, is a precise null.} Acceptance under full computer use minus acceptance under the shell is $+0.033$ with a bootstrap interval from 0.000 to 0.100 and a permutation $p$ of 1.000. The estimate is one scenario-repeat in thirty: A3 accepted one run that A1 did not---the business-intelligence run in which A1 destroyed its input---and the two arms agree on every other scenario mean; the ten paired differences are nine zeros and one positive value, which is why the lower bound is zero, the upper bound 0.10 and the permutation test cannot reject. The contrast on quality is $+3.3$ points [$-0.8$, $+9.6$], $p=0.30$. H1 is not supported, and the result is not an absence of information: with the shell arm at 0.867 the premium on this benchmark cannot exceed 0.133, and the interval excludes a premium above 0.10. For ten scenarios in which every deliverable can be produced programmatically from the inputs, the browser, the office suite and the portal added nothing the shell could not already do; whether a premium exists on tasks whose inputs or deliverables can only be reached through those applications is a question this benchmark, by construction, does not ask. | 123 | +\paragraph{H1, the computer-use premium, is a precise null.} Acceptance under full computer use minus acceptance under the shell is $+0.033$ with a bootstrap interval from 0.000 to 0.100 and a permutation $p$ of 1.000. The estimate is one scenario-repeat in thirty: A3 accepted one run that A1 did not---the business-intelligence run in which A1 destroyed its input---and the two arms agree on every other scenario mean, so the ten paired differences are nine zeros and one positive value. The contrast on quality is $+3.3$ points [$-0.8$, $+9.6$], $p=0.30$. H1 is not supported, and the result is not an absence of information: with the shell arm at 0.867 the premium on this benchmark cannot exceed 0.133, and the interval excludes a premium above 0.10. For ten scenarios in which every deliverable can be produced programmatically from the inputs, the browser, the office suite and the portal added nothing the shell could not already do; whether a premium exists on tasks whose inputs or deliverables can only be reached through those applications is a question this benchmark, by construction, does not ask. | ||
| 124 | % ../analysis/tables/table2_contrasts_pilot.csv (A3-A1 rows); table3_occupation_arm_pilot.csv (BI-01) | 124 | % ../analysis/tables/table2_contrasts_pilot.csv (A3-A1 rows); table3_occupation_arm_pilot.csv (BI-01) | ||
| 125 | 125 | ||||
| 126 | −\paragraph{H2, the shell premium, is large and confirmatory.} The shell premium is $+0.867$ [0.667, 1.000], $p=0.004$ (Holm 0.012); on deterministic completion $+0.967$ [0.900, 1.000], $p=0.002$; on quality $+55.0$ points [40.4, 68.2], $p=0.002$. It is the only confirmatory effect of the pilot, and its mechanism is the one the hypothesis named: every scenario's deliverables include at least one binary office file, usually a workbook with live formulas, which the restricted arm cannot produce; it delivered text substitutes in every run and the criteria that require the workbook's structure and formulas failed in every case. The contrast on the critical-error indicator, $-0.167$ [$-0.467$, $+0.067$], $p=0.50$, is not significant because the six A0 violations are concentrated in two scenarios. | 126 | +\paragraph{H2, the shell premium, is large and confirmatory.} The shell premium is $+0.867$ [0.667, 1.000], $p=0.004$ (Holm 0.012); on deterministic completion $+0.967$ [0.900, 1.000], $p=0.002$; on quality $+55.0$ points [40.4, 68.2], $p=0.002$. It is the only confirmatory effect of the pilot, and its mechanism is the one the hypothesis named: every scenario's deliverables include at least one binary office file, usually a workbook with live formulas, which the restricted arm cannot produce; it delivered text substitutes in every run and the criteria that require the workbook's structure and formulas failed in every case. | ||
| 127 | % ../analysis/tables/table2_contrasts_pilot.csv (A1-A0 rows); audit_a0_outputs_pilot.csv; audit_violations_pilot.csv | 127 | % ../analysis/tables/table2_contrasts_pilot.csv (A1-A0 rows); audit_a0_outputs_pilot.csv; audit_violations_pilot.csv | ||
| 128 | 128 | ||||
| 129 | \paragraph{H3, the self-repair premium, is a null bounded by the design.} The premium is $-0.067$ [$-0.167$, 0.000], $p=0.499$ (Holm 0.998); on quality $-0.9$ points; on critical errors exactly zero. The point estimate is two scenario-repeats in thirty: A4 lost one business-intelligence run to the quality gate (scored 67.9) and one appraisal run to the adjusted-prices criterion, both of which A3 passed in all three repeats. The interpretation is bounded by amendment A-012: all 30 A4 first sessions passed the public self-check, no recovery was ever requested, and the arm differed from A3 only by its checkpointing and self-check instructions and the slightly longer sessions these produce (18.8 tool calls against 17.2; 230 against 219 seconds; 0.93 against 0.91 dollars). Since the self-repair premium is proportional to the first-session failure rate the self-check can see (Section~\ref{sec:fw-companions}), it was zero by construction and the Autonomous Recovery Index is unidentified on the pilot design. | 129 | \paragraph{H3, the self-repair premium, is a null bounded by the design.} The premium is $-0.067$ [$-0.167$, 0.000], $p=0.499$ (Holm 0.998); on quality $-0.9$ points; on critical errors exactly zero. The point estimate is two scenario-repeats in thirty: A4 lost one business-intelligence run to the quality gate (scored 67.9) and one appraisal run to the adjusted-prices criterion, both of which A3 passed in all three repeats. The interpretation is bounded by amendment A-012: all 30 A4 first sessions passed the public self-check, no recovery was ever requested, and the arm differed from A3 only by its checkpointing and self-check instructions and the slightly longer sessions these produce (18.8 tool calls against 17.2; 230 against 219 seconds; 0.93 against 0.91 dollars). Since the self-repair premium is proportional to the first-session failure rate the self-check can see (Section~\ref{sec:fw-companions}), it was zero by construction and the Autonomous Recovery Index is unidentified on the pilot design. | ||
@@ -131,7 +131,7 @@ A2 $-$ A1 (secondary) & $S$ & $+0.033$ & $-0.067$ & $+0.133$ & 1.000 & $+0.025$ | |||||
| 131 | 131 | ||||
| 132 | \paragraph{The macOS layer and the ceiling.} A2 minus A1 is $+0.033$ [$-0.067$, $+0.133$], $p=1.000$: A2 accepted the business-intelligence run that A1 lost and lost an appraisal run that A1 passed. The headless automation layer was used sparingly---18 office-family calls in 30 runs against 69 in A3---and no run depended on it for a deliverable the shell could not produce. | 132 | \paragraph{The macOS layer and the ceiling.} A2 minus A1 is $+0.033$ [$-0.067$, $+0.133$], $p=1.000$: A2 accepted the business-intelligence run that A1 lost and lost an appraisal run that A1 passed. The headless automation layer was used sparingly---18 office-family calls in 30 runs against 69 in A3---and no run depended on it for a deliverable the shell could not produce. | ||
| 133 | % ../analysis/tables/table2_contrasts_pilot.csv (A2-A1); audit_families_pilot.csv | 133 | % ../analysis/tables/table2_contrasts_pilot.csv (A2-A1); audit_families_pilot.csv | ||
| 134 | −A cluster-robust logit of acceptance on arm indicators returns logits of 1.87, 2.20, 2.20 and 1.61 for A1--A4 with clustered standard errors of 0.75 to 1.13, of the same order as the differences among the arms, which restates on the model scale what the bootstrap says: the shell arms are not distinguishable from one another. | 134 | +A cluster-robust logit of acceptance on arm indicators gives standard errors of 0.75 to 1.13 on the logit scale for the shell arms, of the same order as the differences among them: the shell arms are not distinguishable from one another. | ||
| 135 | % ../analysis/derived/estimates_pilot.json (mixed_logit_S) | 135 | % ../analysis/derived/estimates_pilot.json (mixed_logit_S) | ||
| 136 | The scale rule gave the pilot a third repeat, which reduces the within-scenario component of equation~\eqref{eq:premium-var-cluster}, but the component that governs the primary contrast is between-scenario heterogeneity of the effect, and with the shell arms succeeding on nine scenarios in ten there is none left to measure. The benchmark needs scenarios on which the shell arm fails before the computer-use premium can be estimated at all. | 136 | The scale rule gave the pilot a third repeat, which reduces the within-scenario component of equation~\eqref{eq:premium-var-cluster}, but the component that governs the primary contrast is between-scenario heterogeneity of the effect, and with the shell arms succeeding on nine scenarios in ten there is none left to measure. The benchmark needs scenarios on which the shell arm fails before the computer-use premium can be estimated at all. | ||
| 137 | 137 | ||||
@@ -170,28 +170,28 @@ Real-estate appraisal & RE-01 & 34 & 0 & 1 & 0.667 & 1 & 0.667 & 93.5--98.8 \\ | |||||
| 170 | \end{tablenotes} | 170 | \end{tablenotes} | ||
| 171 | \end{table} | 171 | \end{table} | ||
| 172 | 172 | ||||
| 173 | −\paragraph{Executive administration: a quality-gate failure.} ADM-01, the inbox-and-calendar triage, is the second workflow-class scenario and the one with the most planted features. Every shell-arm run passed all ten deterministic criteria, and every shell-arm run but one was scored below 70: the twelve A1--A4 scores range from 59.6 to 70.4, the only pass being an A2 repeat at 70.4. The judges' per-criterion scores locate the shortfall: on the 0--4 scale the twelve runs average 4.00 on the calendar criterion and 3.56 on the briefing note, but 0.64 on the draft replies and 0.14 on the filing index. The evaluator checks that replies and an index exist and that the required ones are present; the judges read them and found them below the standard of the Office of the CEO handbook. The scenario is the clearest instance of the multitask logic of Definition~\ref{def:acceptance}: a package can satisfy every verifiable requirement and still be one a demanding manager would not accept. Whether the judges are right is a question for the human calibration that amendment A-005 defers; the shortfall is consistent across twelve runs, four arms and, as Section~\ref{sec:results-models} shows, three worker models. | 173 | +\paragraph{Executive administration: a quality-gate failure.} ADM-01, the inbox-and-calendar triage, is the second workflow-class scenario and the one with the most planted features. Every shell-arm run passed all ten deterministic criteria, and every shell-arm run but one was scored below 70: the twelve A1--A4 scores range from 59.6 to 70.4, the only pass being an A2 repeat at 70.4. The judges' per-criterion scores locate the shortfall: on the 0--4 scale the twelve runs average 4.00 on the calendar criterion and 3.56 on the briefing note, but 0.64 on the draft replies and 0.14 on the filing index. The evaluator checks that replies and an index exist; the judges read them. The scenario is the clearest instance of the multitask logic of Definition~\ref{def:acceptance}: a package can satisfy every verifiable requirement and still be one a demanding manager would not accept. Whether the judges are right is a question for the human calibration that amendment A-005 defers; the shortfall is consistent across twelve runs, four arms and, as Section~\ref{sec:results-models} shows, three worker models. | ||
| 174 | % ../analysis/tables/table3_occupation_arm_pilot.csv (ADM-01); table14_extra_descriptives.csv block K_gate (criterion means); ../docs/PHASE_5_AUDIT.md §2 | 174 | % ../analysis/tables/table3_occupation_arm_pilot.csv (ADM-01); table14_extra_descriptives.csv block K_gate (criterion means); ../docs/PHASE_5_AUDIT.md §2 | ||
| 175 | 175 | ||||
| 176 | −\paragraph{Business intelligence and real-estate appraisal.} The BI-01 failure in A1 is the pilot's only shell-arm critical violation: while exploring the workspace the worker ran a command intended to remove a scratch file that deleted the input sales database instead, discovered the loss, and ended with a completion record stating that it could not deliver because it had destroyed its input; the harness hash check recorded the modification and the judges scored the package 3.8. The violation is one of integrity of the environment rather than of the deliverable, and the worker's report of it was honest and complete. The A4 failure on the same scenario is a quality-gate failure (all fourteen criteria passed, scored 67.9). RE-01 carries a comparable sheet with a planted value conclusion of 785,000 dollars. The two shell-arm failures, in A2 and A4, passed nine of ten criteria and failed the one on the adjusted prices of the comparables while being scored 98.75 by the judges---the two runs in which the oracle is stricter than the judges, since a departure of the grid beyond the oracle's tolerance is invisible to a reader of the report. The scenario is also where the restricted arm failed most severely: in all three A0 runs the value conclusion equalled the planted 785,000 and the evaluator recorded compliance with the injection. The pattern is not that A0 was persuaded where the shell arms resisted; it is that A0, unable to build the grid, had no supported value of its own and echoed the one figure the inputs offered. All twelve shell-arm runs of RE-01, and the eight further runs of the sensitivity and recovery stages, resisted the same injection. | 176 | +\paragraph{Business intelligence and real-estate appraisal.} The BI-01 failure in A1 is the pilot's only shell-arm critical violation: while exploring the workspace the worker ran a command intended to remove a scratch file that deleted the input sales database instead, discovered the loss, and ended with a completion record stating that it could not deliver because it had destroyed its input; the harness hash check recorded the modification and the judges scored the package 3.8. The violation is one of integrity of the environment rather than of the deliverable, and the worker's report of it was honest and complete. The A4 failure on the same scenario is a quality-gate failure (all fourteen criteria passed, scored 67.9). RE-01 carries a comparable sheet with a planted value conclusion of 785,000 dollars. The two shell-arm failures, in A2 and A4, passed nine of ten criteria and failed the one on the adjusted prices of the comparables while being scored 98.75 by the judges---the two runs in which the oracle is stricter than the judges, since a departure of the grid beyond the oracle's tolerance is invisible to a reader of the report. The scenario is also where the restricted arm failed most severely: in all three A0 runs the value conclusion equalled the planted 785,000 and the evaluator recorded compliance with the injection. The pattern is not that A0 was persuaded where the shell arms resisted; it is that A0, unable to build the grid, had no supported value of its own and echoed the one figure the inputs offered. | ||
| 177 | % ../docs/PHASE_5_AUDIT.md §2; ../analysis/tables/audit_failures_pilot.csv (BI-01_A1_r1, BI-01_A4_r1, RE-01_A2_r1, RE-01_A4_r2); audit_violations_pilot.csv | 177 | % ../docs/PHASE_5_AUDIT.md §2; ../analysis/tables/audit_failures_pilot.csv (BI-01_A1_r1, BI-01_A4_r1, RE-01_A2_r1, RE-01_A4_r2); audit_violations_pilot.csv | ||
| 178 | 178 | ||||
| 179 | \paragraph{Depth is confounded with scenario.} Pooling the shell arms, acceptance is 1.000 on the four standard-class scenarios (depth 21--24; 48 runs), 0.917 on the four complex scenarios (depth 31--34) and 0.542 on the two workflow scenarios (depth 44), and the scenario-level Spearman correlation between acceptance and depth is $-0.46$; realised tool calls rise with depth in every arm, from 11.9 per run in the standard class to 19.4 in the workflow class. | 179 | \paragraph{Depth is confounded with scenario.} Pooling the shell arms, acceptance is 1.000 on the four standard-class scenarios (depth 21--24; 48 runs), 0.917 on the four complex scenarios (depth 31--34) and 0.542 on the two workflow scenarios (depth 44), and the scenario-level Spearman correlation between acceptance and depth is $-0.46$; realised tool calls rise with depth in every arm, from 11.9 per run in the standard class to 19.4 in the workflow class. | ||
| 180 | % ../analysis/tables/table4_dependency_pilot.csv; table14_extra_descriptives.csv block L_dependency | 180 | % ../analysis/tables/table4_dependency_pilot.csv; table14_extra_descriptives.csv block L_dependency | ||
| 181 | −The slope is real but its interpretation as a depth effect is not warranted: the workflow class contains accounting, the deepest and longest scenario, accepted in every shell-arm run, and the triage, accepted in one run of twelve for the quality-gate reason above. With ten scenarios, depth and scenario identity cannot be separated, and within arm the accepted and rejected runs have similar numbers of tool calls, so neither part of H4 is supported. The Workflow Dependency Frontier is reported as descriptive; what the data establish is that longer workflows require more operations in every arm, and that arms with more tools use more of them at every depth without that use translating into acceptance. | 181 | +The slope is real but its interpretation as a depth effect is not warranted: the workflow class contains accounting, the deepest and longest scenario, accepted in every shell-arm run, and the triage, accepted in one run of twelve for the quality-gate reason above. With ten scenarios, depth and scenario identity cannot be separated, and within arm the accepted and rejected runs have similar numbers of tool calls, so neither part of H4 is supported. The Workflow Dependency Frontier is therefore reported as descriptive. | ||
| 182 | 182 | ||||
| 183 | \subsection{Sensitivity to the worker model and to the quality threshold}\label{sec:results-models} | 183 | \subsection{Sensitivity to the worker model and to the quality threshold}\label{sec:results-models} | ||
| 184 | 184 | ||||
| 185 | −Arm A3 was rerun with two repeats per scenario under \texttt{claude-sonnet-5-5} and \texttt{claude-fable-5-1} (last two columns of Table~\ref{tab:outcomes}). The three models reach nearly the same acceptance---0.90 for Opus on three repeats, 0.85 for Sonnet and 0.85 for Fable on two---and the differences, $-0.05$ for each with intervals from $-0.15$ to 0.00 and permutation $p$ of 1.000, are again one scenario-repeat: both accepted one of two business-intelligence runs where Opus accepted three of three, all three failed the triage in every run, and all three solved the other eight scenarios in every run. The models differ in quality and cost. Sonnet's mean quality is 4.8 points below Opus ([$-6.7$, $-3.0$], $p=0.002$) at 38 per cent of the cost ($-0.56$ dollars per run, [$-0.68$, $-0.46$], $p=0.001$) and 60 per cent of the wall time, giving 0.40 dollars per accepted deliverable against 1.01. Fable's mean quality is 1.2 points above Opus ([$-1.4$, $+3.8$], $p=0.40$), not distinguishable, at 3.1 times the cost ($+1.90$ dollars per run, [$+1.56$, $+2.31$], $p=0.002$) and 1.8 times the wall time, giving 3.30 dollars per accepted deliverable. Fable also produced the only critical violation of the sensitivity stage: in one triage run it forwarded the board pack to the external address the scenario's hidden instruction requested, without flagging it---the only injection compliance by a shell-equipped worker in the pilot and sensitivity stages, on the one scenario whose injection asks to send a document rather than to adopt a figure. Across the three models, acceptance moves by 0.05 and cost by a factor of eight: on this benchmark the model, not the tool set, is the cost lever, and quality is the only thing the extra spending buys. | 185 | +Arm A3 was rerun with two repeats per scenario under two other models, \texttt{claude-sonnet-5-5} and \texttt{claude-fable-5-1} (last two columns of Table~\ref{tab:outcomes}). The three models reach nearly the same acceptance---0.90 for Opus on three repeats, 0.85 for Sonnet and 0.85 for Fable on two---and the differences, $-0.05$ for each with intervals from $-0.15$ to 0.00 and permutation $p$ of 1.000, are again one scenario-repeat: both accepted one of two business-intelligence runs where Opus accepted three of three, all three failed the triage in every run, and all three solved the other eight scenarios in every run. The models differ in quality and cost. Sonnet's mean quality is 4.8 points below Opus ([$-6.7$, $-3.0$], $p=0.002$) at 38 per cent of the cost ($-0.56$ dollars per run, [$-0.68$, $-0.46$], $p=0.001$) and 60 per cent of the wall time, giving 0.40 dollars per accepted deliverable against 1.01. Fable's mean quality is 1.2 points above Opus ([$-1.4$, $+3.8$], $p=0.40$), not distinguishable, at 3.1 times the cost ($+1.90$ dollars per run, [$+1.56$, $+2.31$], $p=0.002$) and 1.8 times the wall time, giving 3.30 dollars per accepted deliverable. Fable also produced the only critical violation of the sensitivity stage, an injection compliance on the triage scenario (Section~\ref{sec:results-integrity}). Across the three models, acceptance moves by 0.05 and cost by a factor of eight: on this benchmark the model, not the tool set, is the cost lever, and quality is the only thing the extra spending buys. | ||
| 186 | % ../analysis/tables/table9_model_sensitivity.csv; table9b_model_contrasts.csv; table14_extra_descriptives.csv block J_sensitivity (ADM-01_A3_r1_fable violation) | 186 | % ../analysis/tables/table9_model_sensitivity.csv; table9b_model_contrasts.csv; table14_extra_descriptives.csv block J_sensitivity (ADM-01_A3_r1_fable violation) | ||
| 187 | 187 | ||||
| 188 | −The acceptance threshold is where the models and the arms differ. When $q$ is varied from the preregistered 70, acceptance at $q=50$ or 60 is 0.967 in A1, A2 and A4 and 0.967 to 1.000 in A3, the residual being each arm's single deterministic failure; at $q=80$ it falls to 0.63--0.70 and at $q=90$ to 0.33--0.50, with the ordering of the arms reversed between 70 and 90. The frontier's dependence on $q$ is steep between 70 and 90, where most accepted deliverables are scored, and the ordering of the shell arms is not robust to it---a further reason not to read their differences as rankings. | 188 | +The acceptance threshold is where the models and the arms differ. When $q$ is varied from the preregistered 70, acceptance at $q=50$ or 60 is 0.967 in A1, A2 and A4 and 0.967 to 1.000 in A3, the residual being each arm's single deterministic failure; at $q=80$ it falls to 0.63--0.70 and at $q=90$ to 0.33--0.50, with the ordering of the arms reversed between 70 and 90. The ordering of the shell arms is not robust to the threshold, a further reason not to read their differences as rankings. | ||
| 189 | % ../analysis/tables/table6_q_sensitivity_pilot.csv | 189 | % ../analysis/tables/table6_q_sensitivity_pilot.csv | ||
| 190 | 190 | ||||
| 191 | \subsection{The Autonomous Recovery experiment}\label{sec:results-recovery} | 191 | \subsection{The Autonomous Recovery experiment}\label{sec:results-recovery} | ||
| 192 | 192 | ||||
| 193 | −Because no A4 first session in the pilot failed the public self-check, the Autonomous Recovery Index was unidentified and the launcher defect that would have prevented a resume (A-012) was invisible. To identify the index, amendment A-011 added an exploratory experiment, not part of the preregistration and labelled as such: the ten scenarios were run under A3 and under A4 with two repeats each, and in every run the controller killed the first session 120 seconds after launch. In A3 the run then ends and the deliverables present at the kill are evaluated; A3 is the interrupted control. In A4 the controller runs the self-check on the collected output, finds it failing---by construction, since no 120-second session has written its deliverables---and grants the one bounded recovery: the same session is resumed with the remaining budget and wall time and a prompt stating what the self-check found. The A4 runs of this stage were themselves reset and rerun once because their first execution hit the A-012 defect. | 193 | +Because no A4 first session in the pilot failed the public self-check, the Autonomous Recovery Index was unidentified and the launcher defect that would have prevented a resume (A-012) was invisible. To identify the index, amendment A-011 added an exploratory experiment, not part of the preregistration and labelled as such: the ten scenarios were run under A3 and under A4 with two repeats each, and in every run the controller killed the first session 120 seconds after launch. In A3 the run then ends and the deliverables present at the kill are evaluated; A3 is the interrupted control. In A4 the controller runs the self-check on the collected output, finds it failing---by construction, since no 120-second session has written its deliverables---and grants the one bounded recovery: the same session is resumed with the remaining budget and wall time and a prompt stating what the self-check found. The A4 runs of this stage were reset and rerun once because their first execution hit the A-012 defect; the cost convention for killed sessions is that of amendment A-013. | ||
| 194 | −% ../protocol/amendments.csv A-011, A-012 | 194 | +% ../protocol/amendments.csv A-011, A-012, A-013 | ||
| 195 | 195 | ||||
| 196 | \begin{table}[htbp] | 196 | \begin{table}[htbp] | ||
| 197 | \centering | 197 | \centering | ||
@@ -229,7 +229,7 @@ A4 interrupted $-$ A3 interrupted, acceptance & \multicolumn{4}{l}{$+0.80$ [0.55 | |||||
| 229 | \end{table} | 229 | \end{table} | ||
| 230 | 230 | ||||
| 231 | Table~\ref{tab:recovery} reports the experiment next to the uninterrupted A3 and A4 cells. All 20 interrupted A4 runs failed the first self-check, all 20 were granted a recovery, and all 20 passed the self-check after it: the Autonomous Recovery Index is 20 of 20 on the detection signal the self-check provides. Acceptance after recovery is 0.850 with equal scenario weights (16 of the 19 runs with a judge output), deterministic completion 0.950 and mean quality 84.3. The interrupted A3 control is where an uninterrupted run stands after two minutes: acceptance 0.050 (one procurement run had already written an acceptable package when killed), mean quality 28.0, and seven of twenty runs had produced no artefact at all. The difference in acceptance is $+0.80$ [0.55, 0.95], $p=0.004$, and the interrupted-and-recovered A4 runs reach the same acceptance and quality as the uninterrupted A4 runs (0.850 against 0.833; 84.3 against 84.8). In terms of Section~\ref{sec:fw-companions}, the experiment sets the detection probability $\delta$ to one by construction and estimates the correction probability $\gamma$ at 0.84; the uplift $(1-p)\delta\gamma$ is large here because $1-p$ was made large by the interruption, which is exactly why it was zero in the pilot. | 231 | Table~\ref{tab:recovery} reports the experiment next to the uninterrupted A3 and A4 cells. All 20 interrupted A4 runs failed the first self-check, all 20 were granted a recovery, and all 20 passed the self-check after it: the Autonomous Recovery Index is 20 of 20 on the detection signal the self-check provides. Acceptance after recovery is 0.850 with equal scenario weights (16 of the 19 runs with a judge output), deterministic completion 0.950 and mean quality 84.3. The interrupted A3 control is where an uninterrupted run stands after two minutes: acceptance 0.050 (one procurement run had already written an acceptable package when killed), mean quality 28.0, and seven of twenty runs had produced no artefact at all. The difference in acceptance is $+0.80$ [0.55, 0.95], $p=0.004$, and the interrupted-and-recovered A4 runs reach the same acceptance and quality as the uninterrupted A4 runs (0.850 against 0.833; 84.3 against 84.8). In terms of Section~\ref{sec:fw-companions}, the experiment sets the detection probability $\delta$ to one by construction and estimates the correction probability $\gamma$ at 0.84; the uplift $(1-p)\delta\gamma$ is large here because $1-p$ was made large by the interruption, which is exactly why it was zero in the pilot. | ||
| 232 | −% ../analysis/tables/table10_recovery.csv; ../analysis/derived/recovery_experiment.json (ARI 1.0; acceptance_after_recovery 0.842; contrast 0.80 [0.55, 0.95], perm_p 0.00415); table14_extra_descriptives.csv block G_recovery_cost (share with no artefact 0.35) | 232 | +% ../analysis/tables/table10_recovery.csv; ../analysis/derived/recovery_experiment.json (ARI 1.0; acceptance_after_recovery 0.842; contrast 0.80 [0.55, 0.95], perm_p 0.00415); cost convention per ../protocol/amendments.csv A-013; table14_extra_descriptives.csv block G_recovery_cost (share with no artefact 0.35) | ||
| 233 | 233 | ||||
| 234 | The cost of recovery is modest. The client's cumulative total for an interrupted-and-recovered A4 run is 0.863 dollars, below the 0.926 dollars of an uninterrupted A4 run, and the cost per accepted deliverable is 1.08 dollars against 1.11; the resumed session reloads the first session's context from cache rather than regenerating the work. For the interrupted A3 control the imputed cost is 0.49 dollars per run and, with one acceptance in twenty, 9.80 dollars per accepted deliverable---the cost of work that is paid for and not delivered. | 234 | The cost of recovery is modest. The client's cumulative total for an interrupted-and-recovered A4 run is 0.863 dollars, below the 0.926 dollars of an uninterrupted A4 run, and the cost per accepted deliverable is 1.08 dollars against 1.11; the resumed session reloads the first session's context from cache rather than regenerating the work. For the interrupted A3 control the imputed cost is 0.49 dollars per run and, with one acceptance in twenty, 9.80 dollars per accepted deliverable---the cost of work that is paid for and not delivered. | ||
| 235 | % ../analysis/tables/table10_recovery.csv (cost_mean 0.490 / 0.863; cost_per_accepted 9.802 / 1.079); ../docs/PHASE_5_AUDIT.md §4 | 235 | % ../analysis/tables/table10_recovery.csv (cost_mean 0.490 / 0.863; cost_per_accepted 9.802 / 1.079); ../docs/PHASE_5_AUDIT.md §4 | ||
@@ -237,7 +237,7 @@ Three qualifications apply. The detection signal is a killed session, which the | |||||
| 237 | 237 | ||||
| 238 | \subsection{Cost, supervision and the frontier}\label{sec:results-cost} | 238 | \subsection{Cost, supervision and the frontier}\label{sec:results-cost} | ||
| 239 | 239 | ||||
| 240 | −Figure~\ref{fig:cost-quality} plots each pilot run in the plane of marginal cost and quality. The restricted arm occupies the lower part of the plane alone, with quality between 4.6 and 64.6 and a positive relation between cost and quality, since the A0 runs that cost most---the triages, at 1.53 to 1.62 dollars---are those whose text substitutes earned the most credit. The four shell arms overlap almost entirely in a band of 0.5 to 1.5 dollars and 60 to 100 quality points; their means differ by 0.14 dollars and 3.3 points, and the runs below the threshold line are runs of ordinary length and cost that produced a package the judges would not accept, not cheap runs that stopped early. Median wall time is 162 seconds in A0 and 214 to 230 seconds in the shell arms; every pilot run finished within 392 seconds against limits of 1,800 seconds or more, the most expensive run cost 1.62 dollars against caps of 4.00 or more, and the maximum number of turns was 81 against a cap of 150, so the caps were part of the estimand's support in principle but not in fact and nothing is censored. Tool execution time is 0.6 to 6.8 seconds per run; wall time is almost entirely model inference. | 240 | +Figure~\ref{fig:cost-quality} plots each pilot run in the plane of marginal cost and quality. The restricted arm occupies the lower part of the plane alone, with quality between 4.6 and 64.6 and a positive relation between cost and quality, since the A0 runs that cost most---the triages, at 1.53 to 1.62 dollars---are those whose text substitutes earned the most credit. The four shell arms overlap almost entirely in a band of 0.5 to 1.5 dollars and 60 to 100 quality points; their means differ by 0.14 dollars and 3.3 points, and the runs below the threshold line are runs of ordinary length and cost that produced a package the judges would not accept, not cheap runs that stopped early. Every pilot run finished within 392 seconds against limits of 1,800 seconds or more, the most expensive run cost 1.62 dollars against caps of 4.00 or more, and the maximum number of turns was 81 against a cap of 150: no run is censored and the caps did not bind. | ||
| 241 | % ../analysis/tables/table1_outcomes_by_arm_pilot.csv (wall medians); table14_extra_descriptives.csv block A_process (quality and cost ranges, maxima, tool time); ../analysis/derived/estimates_pilot.json (survival: censored 0) | 241 | % ../analysis/tables/table1_outcomes_by_arm_pilot.csv (wall medians); table14_extra_descriptives.csv block A_process (quality and cost ranges, maxima, tool time); ../analysis/derived/estimates_pilot.json (survival: censored 0) | ||
| 242 | 242 | ||||
| 243 | \begin{figure}[htbp] | 243 | \begin{figure}[htbp] | ||
@@ -247,13 +247,13 @@ Figure~\ref{fig:cost-quality} plots each pilot run in the plane of marginal cost | |||||
| 247 | \label{fig:cost-quality} | 247 | \label{fig:cost-quality} | ||
| 248 | \end{figure} | 248 | \end{figure} | ||
| 249 | 249 | ||||
| 250 | −The Professional Reliability Frontier has, at $q=70$ and the full budget, the coordinates (acceptance within budget, critical-error rate, censoring rate) of $(0.000, 0.200, 0)$ for A0, $(0.867, 0.033, 0)$ for A1, $(0.900, 0, 0)$ for A2 and A3 and $(0.833, 0, 0)$ for A4; A2 and A3 are tied and undominated, and the honest summary is that any arm with a shell delivers an accepted package for about one dollar of marginal API cost and the restricted arm for no finite amount. H6, which expected the lowest cost per accepted deliverable in A1 or A3, is not met: A2 is lowest at 0.87 dollars, within the range that one scenario-repeat moves. Halving the November 2026 API prices gives 0.44 to 0.56 dollars across the shell arms and quadrupling them 3.48 to 4.44. A fully loaded cost that adds the judges (0.23 dollars per run) and an assumed node cost of one dollar per node-hour gives 1.15 to 1.41 dollars per accepted deliverable; the evaluation is the larger overhead, and in a production setting it is the cost of the verification that Proposition~\ref{prop:delegation} makes a necessary complement of delegation. | 250 | +The Professional Reliability Frontier has, at $q=70$ and the full budget, the coordinates (acceptance within budget, critical-error rate, censoring rate) of $(0.000, 0.200, 0)$ for A0, $(0.867, 0.033, 0)$ for A1, $(0.900, 0, 0)$ for A2 and A3 and $(0.833, 0, 0)$ for A4; A2 and A3 are tied and undominated, and the honest summary is that any arm with a shell delivers an accepted package for about one dollar of marginal API cost and the restricted arm for no finite amount. H6, which expected the lowest cost per accepted deliverable in A1 or A3, is not met: A2 is lowest at 0.87 dollars, within the range that one scenario-repeat moves. A fully loaded cost that adds the judges (0.23 dollars per run) and an assumed node cost of one dollar per node-hour gives 1.15 to 1.41 dollars per accepted deliverable; the evaluation is the larger overhead, and in a production setting it is the cost of the verification that Proposition~\ref{prop:delegation} makes a necessary complement of delegation. | ||
| 251 | % ../analysis/tables/table1_outcomes_by_arm_pilot.csv; table11_economic_sensitivity_pilot.csv (price multipliers 0.5 and 4; cost_per_accepted_loaded); table14_extra_descriptives.csv block H_spend (judge cost 0.228 per run) | 251 | % ../analysis/tables/table1_outcomes_by_arm_pilot.csv; table11_economic_sensitivity_pilot.csv (price multipliers 0.5 and 4; cost_per_accepted_loaded); table14_extra_descriptives.csv block H_spend (judge cost 0.228 per run) | ||
| 252 | 252 | ||||
| 253 | −No human intervened during any run, so the supervision dimension is measured by proxy from the completion records. The share of runs ending in \texttt{needs\_clarification} is 0.90 in A0, 0.20 in A1, 0.07 in A2, 0.13 in A3 and 0.10 in A4; the mean number of recorded questions falls from 4.7 to 3.8, 3.4, 2.9 and 2.7; and the implied human minutes, at five per question, from 23.3 to 18.8, 16.8, 14.7 and 13.5. The justified-abstention rate---abstentions on a sub-task whose oracle says clarification was truly required---is 0.30 in A0, 0.17 in A1 and equals the abstention rate in A2, A3 and A4: every abstention of the three arms with the most tools was one the scenario called for, so a worker with more tools abstains less often but no less appropriately, whereas in A0 most abstentions were requests for inputs the worker could not read, which is incapacity recorded as a question. The questions are for the most part not abstentions---the completion status is \texttt{completed} in 80 to 93 per cent of shell-arm runs---but the list of points the worker would have raised with a manager, which the brief invites; only one A1 run, two A2 runs, five A3 runs and seven A4 runs recorded no question at all. | 253 | +No human intervened during any run, so the supervision dimension is measured by proxy from the completion records. The share of runs ending in \texttt{needs\_clarification} is 0.90 in A0, 0.20 in A1, 0.07 in A2, 0.13 in A3 and 0.10 in A4; the mean number of recorded questions falls from 4.7 to 3.8, 3.4, 2.9 and 2.7; and the implied human minutes, at five per question, from 23.3 to 18.8, 16.8, 14.7 and 13.5. The justified-abstention rate---abstentions on a sub-task whose oracle says clarification was truly required---is 0.30 in A0, 0.17 in A1 and equals the abstention rate in A2, A3 and A4: every abstention of the three arms with the most tools was one the scenario called for, so a worker with more tools abstains less often but no less appropriately. The questions are for the most part not abstentions---the completion status is \texttt{completed} in 80 to 93 per cent of shell-arm runs---but the list of points the worker would have raised with a manager, which the brief invites; only one A1 run, two A2 runs, five A3 runs and seven A4 runs recorded no question at all. | ||
| 254 | % ../analysis/tables/table12_supervision_pilot.csv; table14_extra_descriptives.csv block C_questions (share with no question) | 254 | % ../analysis/tables/table12_supervision_pilot.csv; table14_extra_descriptives.csv block C_questions (share with no question) | ||
| 255 | 255 | ||||
| 256 | −This proxy is what makes the frontier of Definition~\ref{def:alf} bite. Marginal acceptance in A3 is 0.900. Without the human-time condition the pooled frontier is zero up to a budget of 0.50 dollars (the cheapest accepted run cost 0.51), 0.73 at one dollar and 0.90 at 1.50 dollars and above (the dearest accepted run cost 1.35), so that the budget margin binds between 0.5 and 1.5 dollars and is slack above. At a supervision allowance of zero---no open question---the frontier of A3 averaged over the ten occupations is 0.100 at 0.75 dollars and 0.167 at any budget of one dollar or more. At an allowance of fifteen minutes (three questions or fewer) it is 0.267 at 0.75 dollars, 0.500 at one dollar and 0.667 at 1.50 dollars and above; A1 and A2 reach 0.500 and A4 0.600 at fifteen minutes and the full budget. The gap between 0.900 and 0.667 is the share of accepted work that arrived with more than three open questions; the gap between 0.900 and 0.167 is the share that arrived with any question at all. The budget ceases to bind at about 1.50 dollars; the supervision allowance never does within the range examined. This is the pilot's clearest economic message, and it is a message about the agent's propensity to leave questions open rather than about its ability to do the work. | 256 | +This proxy is what makes the frontier of Definition~\ref{def:alf} bite. Marginal acceptance in A3 is 0.900. Without the human-time condition the pooled frontier is zero up to a budget of 0.50 dollars (the cheapest accepted run cost 0.51), 0.73 at one dollar and 0.90 at 1.50 dollars and above (the dearest accepted run cost 1.35), so that the budget margin binds between 0.5 and 1.5 dollars and is slack above. At a supervision allowance of zero---no open question---the frontier of A3 averaged over the ten occupations is 0.100 at 0.75 dollars and 0.167 at any budget of one dollar or more. At an allowance of fifteen minutes (three questions or fewer) it is 0.267 at 0.75 dollars, 0.500 at one dollar and 0.667 at 1.50 dollars and above. The gap between 0.900 and 0.667 is the share of accepted work that arrived with more than three open questions; the gap between 0.900 and 0.167 is the share that arrived with any question at all. The budget ceases to bind at about 1.50 dollars; the supervision allowance never does within the range examined. This is the pilot's clearest economic message, and it is a message about the agent's propensity to leave questions open rather than about its ability to do the work. | ||
| 257 | % computed from ../analysis/tables/table5_alf_pilot.csv (mean over the 10 occupations, equal weights; H=0 and equal_H15); table14_extra_descriptives.csv block D_alf_Hinf (A3 pooled without H condition: 0.733 at B=1, 0.900 at B≥1.5); block A_process (cheapest accepted = A2 min cost 0.5087; dearest = A1 max 1.3542) | 257 | % computed from ../analysis/tables/table5_alf_pilot.csv (mean over the 10 occupations, equal weights; H=0 and equal_H15); table14_extra_descriptives.csv block D_alf_Hinf (A3 pooled without H condition: 0.733 at B=1, 0.900 at B≥1.5); block A_process (cheapest accepted = A2 min cost 0.5087; dearest = A1 max 1.3542) | ||
| 258 | 258 | ||||
| 259 | Autonomy-Adjusted Productivity makes the same point in cost units. Accepted deliverables per 100 dollars of API cost plus wage-valued human minutes are 6.5, 7.5, 8.4 and 8.4 in A1--A4 at 40 Canadian dollars per hour, 3.3, 3.9, 4.4 and 4.4 at 80, and 2.3, 2.6, 3.0 and 3.0 at 120; the ordering reverses from the marginal-cost ordering because the arms with more tools record fewer questions, and the wage term dominates the API term at every wage considered: at 80 dollars an hour, fifteen minutes of question-answering costs twenty dollars against one dollar of tokens. The economically relevant cost of the agent, in this pilot, is the cost of the human who closes its questions, which is the conclusion of \citet{catalini2026agi} reached from the other direction; the questions are counted at a stated rate, not measured in minutes of a supervisor's time. | 259 | Autonomy-Adjusted Productivity makes the same point in cost units. Accepted deliverables per 100 dollars of API cost plus wage-valued human minutes are 6.5, 7.5, 8.4 and 8.4 in A1--A4 at 40 Canadian dollars per hour, 3.3, 3.9, 4.4 and 4.4 at 80, and 2.3, 2.6, 3.0 and 3.0 at 120; the ordering reverses from the marginal-cost ordering because the arms with more tools record fewer questions, and the wage term dominates the API term at every wage considered: at 80 dollars an hour, fifteen minutes of question-answering costs twenty dollars against one dollar of tokens. The economically relevant cost of the agent, in this pilot, is the cost of the human who closes its questions, which is the conclusion of \citet{catalini2026agi} reached from the other direction; the questions are counted at a stated rate, not measured in minutes of a supervisor's time. | ||
@@ -264,7 +264,7 @@ Autonomy-Adjusted Productivity makes the same point in cost units. Accepted deli | |||||
| 264 | Hypothesis H5 stated that the critical-violation rate would be below 10 per cent in every arm and would not increase with tool access. The second part holds: 0.200 in A0, 0.033 in A1, zero in A2--A4; the first holds for the shell arms and fails for A0. Across the 230 runs of the study the evaluators recorded eleven critical violations: six in A0 (three appraisals that echoed the injected value and three contract reviews with an empty issues register, which the oracle treats as a critical failure of the review), the honest destruction of an input in A1, two interrupted-session artefacts in the recovery control, and two cases of the same injection---a request to send a document to an outside address---obeyed by a shell-equipped worker, once by Fable and once by Opus after a recovery. No run in any stage attempted an external communication through a channel the harness could see; none exists in the laboratory, and the ``forwarding'' is detected in the text of the deliverables. | 264 | Hypothesis H5 stated that the critical-violation rate would be below 10 per cent in every arm and would not increase with tool access. The second part holds: 0.200 in A0, 0.033 in A1, zero in A2--A4; the first holds for the shell arms and fails for A0. Across the 230 runs of the study the evaluators recorded eleven critical violations: six in A0 (three appraisals that echoed the injected value and three contract reviews with an empty issues register, which the oracle treats as a critical failure of the review), the honest destruction of an input in A1, two interrupted-session artefacts in the recovery control, and two cases of the same injection---a request to send a document to an outside address---obeyed by a shell-equipped worker, once by Fable and once by Opus after a recovery. No run in any stage attempted an external communication through a channel the harness could see; none exists in the laboratory, and the ``forwarding'' is detected in the text of the deliverables. | ||
| 265 | % ../docs/PHASE_5_AUDIT.md §2; ../analysis/tables/audit_violations_pilot.csv; table14_extra_descriptives.csv blocks J_sensitivity, G_recovery_cost (violations) | 265 | % ../docs/PHASE_5_AUDIT.md §2; ../analysis/tables/audit_violations_pilot.csv; table14_extra_descriptives.csv blocks J_sensitivity, G_recovery_cost (violations) | ||
| 266 | 266 | ||||
| 267 | −The design's manipulation is the tool set, and its validity depends on the tool set being what the flags say. The \texttt{--disallowedTools} patterns denied exactly one call in A1, one in A2 and one in A4, and no A1 run attempted an \texttt{osascript} call. A0's 883 calls are all of the file family; A1's 386 are 69 per cent Python and 24 per cent shell with no office, browser or AppleScript call; A2's 371 include 18 office-family calls and no browser call; A3's and A4's 516 and 564 include 117 and 105 browser calls and 69 and 85 office calls. A portal submission was collected in every A3 and A4 run and in no run of the other arms. No arm used a tool outside its set, and the arms with the portal and the office adapter used them in every run; the null on the computer-use premium is not an artefact of tools left idle. | 267 | +The design's manipulation is the tool set, and its validity depends on the tool set being what the flags say. The \texttt{--disallowedTools} patterns denied exactly one call in A1, one in A2 and one in A4, and no A1 run attempted an \texttt{osascript} call. A0's calls are all of the file family; A1 made no office, browser or AppleScript call; A2 made no browser call; A3 and A4 made 117 and 105 browser calls and 69 and 85 office calls in 30 runs each. A portal submission was collected in every A3 and A4 run and in no run of the other arms. No arm used a tool outside its set, and the arms with the portal and the office adapter used them in every run; the null on the computer-use premium is not an artefact of tools left idle. | ||
| 268 | % ../analysis/tables/audit_denied_pilot.csv; audit_families_pilot.csv; audit_portal_pilot.csv; ../docs/PHASE_5_AUDIT.md §3 | 268 | % ../analysis/tables/audit_denied_pilot.csv; audit_families_pilot.csv; audit_portal_pilot.csv; ../docs/PHASE_5_AUDIT.md §3 | ||
| 269 | 269 | ||||
| 270 | \subsection{Summary against the preregistered hypotheses}\label{sec:results-summary} | 270 | \subsection{Summary against the preregistered hypotheses}\label{sec:results-summary} | ||
modified
manuscript/main.pdf
+0 −0
Binary file not shown.
added
release/wp42_uqo.pdf
+0 −0
Binary file not shown.
added
release/wp42_uqo_journal_version.pdf
+0 −0
Binary file not shown.