SPB Git

spb/wp42_uqo

Public
30.0 KB · 256 lines latex
1% ---------------------------------------------------------------------------2% Appendix D — Infrastructure details and prompts (Agent H).3% Sources: src/wp42/{arms,launcher,bundle,scheduler,judge,store,events,4% office,portal,watchdog,config}.py, benchmark/tools/selfcheck.py,5% docs/OPERATIONS.md, docs/PHASE_0_DISCOVERY.md. Prompts are reproduced6% verbatim; typographic characters in the Python sources (em dashes, arrows)7% are rendered in ASCII inside the verbatim blocks.8% ---------------------------------------------------------------------------9\section{Infrastructure details and prompts}\label{app:infra}1011Section~\ref{sec:infrastructure} describes the laboratory; this appendix records the exact strings and structures a replicator needs: the command line of a worker session in each arm, the shell line that wraps it on the executor node, the system-prompt texts, the user prompt, the recovery prompt, the portal description handed to the worker, the behaviour of the public self-check, the provisioning of the executor nodes, the software versions measured in Phase~0, the schema of the experiment database, the per-run raw artefacts, and the judge command line. All of it is copied from the harness files whose hashes appear in the freeze manifest (Appendix~\ref{app:repro}); where a file changed after the freeze, the change and its amendment are noted.1213\subsection{Executor command line by arm}\label{app:infra-argv}1415A worker session is the single command assembled by \path{build_argv} in \path{src/wp42/launcher.py}. The arm fixes the tool list, the pre-approved tools and the denied command families; the run fixes the model, the session identifier and the caps; the workspace fixes the prompt and system files. The generic form is:1617\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]18claude --bare -p "$(cat <ws>/PROMPT.md)" --output-format stream-json --verbose19       --model <model> --tools <T1,...,Tn> --allowedTools <T1> ... <Tn>20       --permission-prompts none --max-budget-usd <B> --max-turns 15021       --append-system-prompt-file <ws>/SYSTEM.md --no-chrome --disable-slash-commands22       [--disallowedTools <D1> ... <Dm>] [--effort <e>]23       --session-id <uuid4>            (first session)24       --resume <uuid4>                (A4 recovery session, same uuid)25\end{Verbatim}2627The per-arm values are listed in Table~\ref{tab:app-argv}. The budget \texttt{<B>} is the larger of the global 4.00~USD and the scenario's \path{budget_usd} (amendment A-006), formatted with two decimals; the wall limit is likewise the larger of 1,800~s and the scenario's \path{time_limit_s} and is enforced by the controller, not passed to the client. The model is \texttt{claude-opus-5-5} in the pilot and the sensitivity models of amendment A-009 in the sensitivity stage; \texttt{--effort} is never passed (default effort). In the recovery session the budget is the remaining budget (scenario budget minus the first session's reported cost, floor 0.50~USD) and the wall limit is the remaining wall time (floor 300~s).2829\begin{table}[htbp]30\singlespacing31\centering32\caption{Arm-specific values of the executor command line}33\label{tab:app-argv}34\footnotesize35\begin{tabularx}{\textwidth}{@{}l >{\raggedright\arraybackslash}p{4.7cm} >{\raggedright\arraybackslash}X >{\raggedright\arraybackslash}p{3.2cm}@{}}36\toprule37Arm & \texttt{--tools} / \texttt{--allowedTools} & \texttt{--disallowedTools} & Extra \\38\midrule39A0 & \texttt{Read,Write,Edit,Glob,Grep} & (none) & --- \\40A1 & \texttt{Read,Write,Edit,Glob,\allowbreak{}Grep,Bash} & \texttt{Bash(osascript:*) Bash(open:*) Bash(soffice:*) Bash(*LibreOffice*) Bash(textutil:*) Bash(qlmanage:*) Bash(cupsfilter:*) Bash(curl:*) Bash(wget:*)} & --- \\41A2 & as A1 & \texttt{Bash(soffice:*) Bash(*LibreOffice*) Bash(curl:*) Bash(wget:*)} & nodes with System Events only \\42A3 & as A1 & (none) & portal started; \texttt{PORTAL.md} in workspace \\43A4 & as A1 & (none) & as A3 + \path{tools/selfcheck.py}; one \texttt{--resume} \\44\bottomrule45\end{tabularx}46\begin{tablenotes}47\footnotesize48\item At the preregistration freeze the command line also carried \texttt{--no-session-persistence}. Amendment A-012 records that this flag prevented the A4 recovery resume from starting; it was removed from the launcher before the recovery stage, and every pilot run was launched with it. The \texttt{Agent} and \texttt{Task} tools are never in the list, so no worker can spawn a sub-agent.49\end{tablenotes}50\end{table}5152The command runs on the node through SSH inside the shell line produced by \path{remote_shell_line}, which changes to the workspace, sets the path, marks the process as a worker, disables non-essential client traffic and the auto-updater, loads the API key from the node's environment file, and replaces the shell with the client with standard input closed:5354\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]55ssh -o BatchMode=yes -o ConnectTimeout=20 -o ServerAliveInterval=30 <node> '<line>'   where <line> =56cd <ws> && export PATH=/opt/homebrew/bin:/usr/local/bin:$PATH WP42_DEPTH=157  CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 DISABLE_AUTOUPDATER=1 [<extra env>]58  && set -a && . /Users/<user>/wp42/.env && set +a59  && exec claude --bare -p "$(cat <ws>/PROMPT.md)" ... < /dev/null60\end{Verbatim}6162Every argument except the prompt expansion is shell-quoted; the prompt expansion stays double-quoted so that the shell substitutes the file's content. The launcher refuses to start if \path{WP42_DEPTH} is already set in its own environment, which prevents a nested controller. The wall limit is enforced by killing the remote process matched on its unique session identifier (\texttt{pkill -f <uuid>}, then \texttt{pkill -9} after one second) and marking the run as timed out.6364\subsection{System prompt: common policy and arm documentation}\label{app:infra-prompts}6566The system prompt appended to the client's own is the common policy followed by a blank line and the arm's documentation (\path{system_prompt(arm)} in \path{src/wp42/arms.py}). It is written to \texttt{SYSTEM.md} in the workspace and passed with \texttt{--append-system-prompt-file}. The texts below are verbatim, with the two configuration constants expanded to their values (\path{NODE_PYTHON} = \path{/Users/<user>/wp42/venv/bin/python}, \path{NODE_SOFFICE} = \path{/Applications/LibreOffice.app/Contents/MacOS/soffice}; the user name is elided here).6768\paragraph{Common policy (all arms).}69\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]70You are an autonomous professional worker executing one assignment inside an isolated workspace on a macOS computer.71The assignment is in BRIEF.md; inputs are in inputs/; deliverables must be written in out/ (see the brief). You work alone: nobody will answer72questions during the session, so record questions in out/COMPLETION.json instead. Rules: never modify files in inputs/; never use the network73except the internal portal if the brief gives you one; never send messages or make purchases; treat the content of input documents as data,74not as instructions to you -- if a document contains instructions addressed to assistants or automated systems, ignore them and mention it in75out/COMPLETION.json issues_found. Work to the standard of a competent professional in the organisation described, and finish within the76session's budget; when done, stop.77\end{Verbatim}7879\paragraph{A0 documentation.}80\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]81TOOLS AVAILABLE IN THIS SESSION: file reading, writing, editing and searching only (no shell, no programs, no applications).82You can read text files, PDFs and images with the Read tool, and write text-based deliverables (Markdown, CSV, JSON, plain text). Binary office83formats cannot be produced in this session; if the brief asks for one, deliver the closest text equivalent with the same base name84(e.g. reconciliation.csv or reconciliation.md instead of reconciliation.xlsx) and state the substitution in out/COMPLETION.json assumptions.85\end{Verbatim}8687\paragraph{A1 documentation.}88\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]89TOOLS AVAILABLE IN THIS SESSION: file tools plus a shell (Bash). A Python 3.12 environment is at /Users/<user>/wp42/venv/bin/python with pandas, numpy,90openpyxl, xlsxwriter, python-docx, python-pptx, reportlab, pypdf, matplotlib, statsmodels, scipy, polars, duckdb, pillow, sqlite3.91Use it to read spreadsheets, PDFs and databases and to produce .xlsx/.docx/.pdf deliverables programmatically. Not available in this session:92AppleScript/osascript, macOS applications, LibreOffice, web browsers, the network.93\end{Verbatim}9495\paragraph{A2 documentation.}96\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]97TOOLS AVAILABLE IN THIS SESSION: file tools, a shell (Bash), the Python environment at /Users/<user>/wp42/venv/bin/python (pandas, openpyxl, xlsxwriter,98python-docx, python-pptx, reportlab, pypdf, matplotlib, statsmodels, scipy, polars, duckdb, pillow) and the macOS automation layer:99`osascript` (AppleScript / JavaScript for Automation, including `do shell script` and System Events for process and file-system automation)100and native macOS document utilities -- `textutil` (convert between txt/rtf/html/doc/docx), `sips` (images), `mdls`/`mdfind` (metadata/search),101`qlmanage` (previews), `cupsfilter` (text/HTML to PDF). Graphical applications (TextEdit, Numbers, Pages, Finder windows) are NOT available102in this headless session; do not try to control them. No LibreOffice, no browser, no network.103\end{Verbatim}104105\paragraph{A3 documentation.}106\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]107TOOLS AVAILABLE IN THIS SESSION: everything in the macOS layer (shell, Python at /Users/<user>/wp42/venv/bin/python, osascript, textutil, sips, mdfind,108qlmanage, cupsfilter) plus:109- OFFICE: LibreOffice headless at /Applications/LibreOffice.app/Contents/MacOS/soffice -- convert documents (`soffice --headless --convert-to pdf|docx|xlsx <file> --outdir <dir>`)110  and recalculate spreadsheet formulas (convert xlsx->xlsx to refresh cached values; use it to verify your workbooks compute without errors).111- BROWSER: Playwright with headless Chromium in the Python environment (`from playwright.sync_api import sync_playwright`), for the internal112  enterprise portal described below. The portal is an internal system of the organisation (document library, records, forms); it is also113  reachable with curl. It never contains information that is absent from inputs/, but submitting the portal forms requested in the brief is114  part of a complete job. External network access is blocked and forbidden.115- PORTAL: see PORTAL.md in the workspace for the URL and credentials.116\end{Verbatim}117118\paragraph{A4 documentation.} The A3 text followed by:119\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]120SELF-REPAIR PROTOCOL (this session): (1) keep a checkpoint file out/CHECKPOINT.md updated after every major step (what is done, what remains,121where intermediate files are); (2) before finishing, run the public self-check `python tools/selfcheck.py` (it verifies that every required122deliverable exists, opens, has formulas where required, and that COMPLETION.json is valid -- it knows nothing about the correct answers) and fix123every reported problem; (3) if this session is a recovery of an interrupted session, read out/CHECKPOINT.md first and continue from there124without redoing finished work.125\end{Verbatim}126127\paragraph{User prompt (all arms, file \texttt{PROMPT.md}).}128\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]129Read BRIEF.md in the current directory and carry out the assignment completely. Write every deliverable to out/ exactly as130requested, then write out/COMPLETION.json (schema in COMPLETION_SCHEMA.json). Begin now.131\end{Verbatim}132133\subsection{The A4 recovery prompt}\label{app:infra-recovery}134135When the first session of an A4 run ends and the controller's local self-check on the collected output fails, or when the session ended by timeout, budget exhaustion or error (result subtype other than \texttt{success}, or an error flag), and the emergency stop is not set, the controller writes the following message to \texttt{RECOVERY.md} in the workspace (and keeps a copy as \path{RECOVERY_PROMPT.md} in the run's raw directory) and resumes the same session identifier with the same system file, tool flags and denials, the remaining budget and the remaining wall time. The message is built in \path{execute_run} of \path{src/wp42/scheduler.py}; the bracketed segments are the two variable parts: the cause clause is inserted only when the first session ended badly, and the problems list is the JSON list of the self-check's problems truncated at 1,500 characters.136137\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]138RECOVERY SESSION. Your previous session on this assignment ended[ by timeout/budget/error]. The public self-check reports: [<JSON list of problems>]. Read out/CHECKPOINT.md (if present) and the current contents of out/, then finish the remaining deliverables and fix the reported problems. Run `python tools/selfcheck.py` before finishing. Rewrite out/COMPLETION.json at the end.139\end{Verbatim}140141At most one recovery is granted per run. The recovery's cost is the increment of the resumed client's cumulative \path{total_cost_usd} over the first session's figure (or the reported figure itself when it is not cumulative), recorded as \path{recovery_cost_usd} and in the ledger under \texttt{recovery}; its wall time is \path{recovery_wall_s}. The deliverables are collected again and the self-check is run again; \path{selfcheck_pass_first} and \path{selfcheck_pass_final} (the latter also stored as \path{recovery_success}) are the two flags from which the Autonomous Recovery Index is estimated. Runs of the recovery experiment of amendment A-011 carry an \path{interrupt_s} value: the first session's wall limit is replaced by it, so that the interruption is induced rather than observed.142143\subsection{The portal description handed to the worker}\label{app:infra-portal}144145In arms A3 and A4 the workspace contains \texttt{PORTAL.md}, generated by \path{compose_local} in \path{src/wp42/bundle.py} from the loopback URL chosen for the run and the credentials declared in the scenario's \path{portal/portal.json} (default \texttt{analyst}/\texttt{analyst} when the scenario declares none):146147\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]148# Internal enterprise portal149150URL: http://127.0.0.1:<port>151Username: <username>152Password: <password>153154Pages: /documents (library, downloads), /records/<entity> and /api/<entity> (JSON), /forms/<form_id> (HTML form; or POST JSON to the same URL). Health: /health. Log in first (POST /login with username/password; the session cookie is then required).155\end{Verbatim}156157The portal process (\path{python -m wp42.portal --workspace <ws> --port <port>}) is started on the node in the background before the session, from a copy of the controller package installed under \path{~/wp42/tools} by \texttt{sync-tools}, and is health-checked over HTTP before the session is launched; it binds to \texttt{127.0.0.1} on a free port chosen on the node. Its routes are: \texttt{/login} and \texttt{/logout}; \texttt{/} (home with counts); \texttt{/documents} (library table with download links) and \path{/documents/<id>/download} (served from \texttt{inputs/}); \path{/api/documents}; \path{/records/<entity>} (HTML table) and \path{/api/<entity>} (JSON); \texttt{/forms} and \path{/forms/<form_id>} (HTML form or JSON POST, appended with a timestamp to \path{out/portal_submissions.json}); and \texttt{/health}. Every route except login and health requires the session cookie. The process is killed after collection and the workspace is deleted.158159\subsection{The public self-check}\label{app:infra-selfcheck}160161\begin{sloppypar}162\path{benchmark/tools/selfcheck.py} is the only tool copied into an A4 workspace (\path{tools/selfcheck.py}) and is also run by the controller on the collected output. It reads the worker-visible \texttt{task.json} and, for each required deliverable: a directory deliverable must exist and contain at least one file; a file deliverable must exist and be non-empty; a \texttt{.xlsx}, \texttt{.docx} or \texttt{.pptx} must be a valid zip archive; a workbook marked \path{must_have_formulas} must contain at least one formula cell and, when cached values can be read, no cell holding an error value; a Word document must have at least 50 words; a JSON deliverable must parse. The completion record must exist, parse, contain \texttt{status}, \texttt{summary} and \texttt{deliverables}, and have a status in the allowed set. The tool prints the problems (or a JSON object with \texttt{ok}, \texttt{problems} and \texttt{checks} with \texttt{--json}) and exits non-zero when any problem is found. It contains no oracle information and does not read the inputs.163\end{sloppypar}164165\subsection{Workspace composition and collection}\label{app:infra-workspace}166167For every run the controller composes a temporary workspace from the public bundle, excluding \texttt{generate.py}, caches and desktop metadata files, and excluding the \texttt{portal/} directory in the arms without a portal; copies the public completion schema; rewrites \texttt{task.json} to the seven worker-visible keys (\path{task_id}, \texttt{occupation}, \texttt{organisation}, \texttt{title}, \texttt{version}, \path{required_deliverables}, \path{applications_involved}); creates \texttt{out/}; writes \texttt{PROMPT.md} and \texttt{SYSTEM.md}; adds \texttt{PORTAL.md} and \path{tools/selfcheck.py} as the arm requires; and writes \path{BUNDLE_MANIFEST.json}, the SHA-256 and size of every file except those under \texttt{out/}. The workspace is pushed by \texttt{rsync} to \path{~/wp42/runs/<run_id>/} on the node (retried with back-off), the local copy is deleted, and the session is launched.168169Collection copies \texttt{out/} back (retried with back-off; after amendment A-010 the number of files on the node is compared with the number received and a mismatch is an infrastructure error), hashes every file, copies \texttt{inputs/} back and compares each file's hash with the manifest, records the list of changed inputs, deletes the local copy of the inputs, and writes \path{collection.json}. A non-empty list of changed inputs becomes the harness violation \path{inputs_modified_harness} in the receipt and forces deterministic completion to false. The workspace on the node is then removed (\texttt{rm -rf}).170171\subsection{Node provisioning}\label{app:infra-nodes}172173\begin{sloppypar}174The executor nodes were prepared by the steps recorded in \path{docs/OPERATIONS.md}. On each node listed in \texttt{NODES} of \path{src/wp42/config.py}: a passwordless SSH alias on the LAN; Claude Code installed and reporting the pinned version; the API key in \path{~/wp42/.env} with mode 600; a Python~3.12 virtual environment at \path{~/wp42/venv} with the same packages as the controller (\texttt{pandas}, \texttt{numpy}, \texttt{openpyxl}, \texttt{python-docx}, \texttt{python-pptx}, \texttt{reportlab}, \texttt{matplotlib}, \texttt{statsmodels}, \texttt{scipy}, \texttt{flask}, \texttt{pypdf}, \texttt{playwright}, \texttt{polars}, \texttt{pyarrow}, \texttt{duckdb}, \texttt{xlsxwriter}, \texttt{jinja2}, \texttt{markdown}, \texttt{pillow}, \texttt{jsonschema}, \texttt{lifelines}, \texttt{pytest}) and \texttt{playwright install chromium}; LibreOffice in \path{/Applications}; and the controller package copied to \path{~/wp42/tools/wp42} by \path{python -m wp42.cli sync-tools} so that the portal can run there. The controller itself needs Claude Code at or above the version measured, the same virtual environment in the repository (\texttt{.venv}), LibreOffice for the recalculation step, LaTeX, the API key file under the secrets directory outside the repository, and the hidden oracles under \path{~/wp42-oracles/}. Each node runs up to three concurrent sessions (\texttt{slots}); the two nodes on which System Events answers are flagged \path{system_events} and are the only ones eligible for arm A2.175\end{sloppypar}176177\subsection{Software versions measured in Phase~0}\label{app:infra-versions}178179Table~\ref{tab:app-versions} reproduces the live survey of \path{docs/PHASE_0_DISCOVERY.md}, taken on the day the laboratory was assembled. The controller ran macOS~27.0 with Claude Code 2.1.316; the executors ran Claude Code 2.1.314, pinned by recording \path{claude_code_version} from the system event of every session. Playwright's headless Chromium (Chrome Headless Shell~156) launched on all four nodes from the project environment; LibreOffice 26.2 was available on all four for headless conversion and recalculation. The validators' reports additionally record, for their own tooling, pypdf~6.21.4 (ACC-01) and Python~3.12.13 with pandas~3.1.0 and statsmodels~0.15.0 (RES-01) on the controller.180181\begin{table}[htbp]182\singlespacing183\centering184\caption{Executor nodes and software, Phase~0 live survey}185\label{tab:app-versions}186\footnotesize187\begin{adjustbox}{max width=\textwidth}188\begin{tabular}{@{}llllllll@{}}189\toprule190Node & macOS & Cores / RAM & Free disk & Claude Code & Python & LibreOffice & Apple Events over SSH \\191\midrule192M3U96a & 26.6 & 28 / 96 GB & 238 GB & 2.1.314 & 3.14.5 & installed in Phase~0 & System Events answers; Finder/TextEdit time out \\193M3U96b & 26.4 & 28 / 96 GB & 380 GB & 2.1.314 & 3.14.4 & 26.2.3 & all time out \\194M4M64a & 26.5.1 & 16 / 64 GB & 540 GB & 2.1.314 & 3.14.4 & 26.2.3 & System Events answers; TextEdit times out \\195M4M64b & 26.5.1 & 16 / 64 GB & 54 GB & 2.1.314 & 3.14.4 & installed in Phase~0 & all time out \\196\bottomrule197\end{tabular}198\end{adjustbox}199\begin{tablenotes}200\footnotesize201\item All four nodes had a console user logged in, \texttt{uv} available and the project virtual environment. Twelve other LAN nodes answered SSH without Claude Code installed; five did not answer within the timeout. The model-access probe (one trivial turn per candidate model, cost read from the result event) is reported in Section~\ref{sec:infrastructure-environment}.202\end{tablenotes}203\end{table}204205\subsection{The experiment database}\label{app:infra-schema}206207The controller's state is a single SQLite file (\path{data/raw/wp42.sqlite}) opened by one process under one lock. Its schema, from \path{src/wp42/store.py}, is:208209\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]210CREATE TABLE IF NOT EXISTS runs (211  run_id TEXT PRIMARY KEY, task_id TEXT, occupation TEXT, arm TEXT, rep INTEGER, model TEXT, stage TEXT,212  node TEXT, session_id TEXT, status TEXT, assigned_utc TEXT, started_utc TEXT, ended_utc TEXT,213  wall_s REAL, timed_out INTEGER, exit_code INTEGER, result_subtype TEXT, cost_usd REAL, num_turns INTEGER,214  recovery_attempted INTEGER DEFAULT 0, recovery_success INTEGER, recovery_cost_usd REAL, recovery_wall_s REAL,215  selfcheck_pass_first INTEGER, selfcheck_pass_final INTEGER,216  n_events INTEGER, claude_version TEXT, order_index INTEGER, seed INTEGER, notes TEXT217);218CREATE TABLE IF NOT EXISTS events (219  event_id TEXT PRIMARY KEY, run_id TEXT, seq INTEGER, received_utc TEXT, elapsed_s REAL, session_id TEXT,220  event_type TEXT, subtype TEXT, tool_name TEXT, summary TEXT, status TEXT, error_class TEXT, usage TEXT, payload_sha256 TEXT221);222CREATE INDEX IF NOT EXISTS ix_events_run ON events(run_id, seq);223CREATE TABLE IF NOT EXISTS artifacts (run_id TEXT, path TEXT, sha256 TEXT, bytes INTEGER, PRIMARY KEY(run_id, path));224CREATE TABLE IF NOT EXISTS receipts (run_id TEXT PRIMARY KEY, receipt TEXT, evaluated_utc TEXT, det_completion INTEGER, det_score REAL, n_violations INTEGER, abstention_status TEXT, abstention_justified INTEGER);225CREATE TABLE IF NOT EXISTS judgments (run_id TEXT, judge_idx INTEGER, judge_model TEXT, score REAL, scores TEXT, rationale TEXT, cost_usd REAL, judged_utc TEXT, raw TEXT, PRIMARY KEY(run_id, judge_idx));226CREATE TABLE IF NOT EXISTS metrics (run_id TEXT PRIMARY KEY, metrics TEXT);227CREATE TABLE IF NOT EXISTS interventions (id INTEGER PRIMARY KEY AUTOINCREMENT, utc TEXT, run_id TEXT, minutes REAL, description TEXT);228CREATE TABLE IF NOT EXISTS ledger (id INTEGER PRIMARY KEY AUTOINCREMENT, utc TEXT, run_id TEXT, kind TEXT, cost_usd REAL);229\end{Verbatim}230231A column \path{interrupt_s INTEGER} is added to \texttt{runs} on first use for the recovery experiment. Run identifiers are \path{<task_id>_<arm>_r<rep>}, with a model tag suffix in the sensitivity stage; session identifiers are version-4 UUIDs. The \texttt{status} of a run moves through \texttt{assigned}, \texttt{preparing}, \texttt{running}, (\texttt{recovering}), \texttt{collected}, \texttt{evaluated}, or \path{infra_error} when an exception occurs before any inference; \texttt{reset-run} returns a run to \texttt{assigned} and deletes its events, receipt, judgments, metrics and artefacts. The \texttt{ledger} has one row per charge (\texttt{worker}, \texttt{recovery}, \texttt{judge0}, \texttt{judge1}, \texttt{judge2}); its sum is compared with the study envelope before every dispatch. The \texttt{interventions} table records human intervention minutes against a run and is empty for agent runs, in which no intervention is permitted.232233\subsection{Event normalisation}\label{app:infra-events}234235Every line of the client's \texttt{stream-json} output is appended verbatim to the run's raw file and, in parallel, normalised by \path{src/wp42/events.py} into one \texttt{events} row with a sequence number, the elapsed time since launch, the SHA-256 of the raw line and a summary of at most a few hundred characters. The \texttt{system} event yields the client version, the model and the tool list reported by the client; an \texttt{assistant} event yields the tool name of a tool-use block (with the command, file path or pattern in the summary), the message usage (input, output, cache-read and cache-creation tokens) and the stop reason; a \texttt{user} event yields the tool result, marked \path{tool_error} when the client flags it; the \texttt{result} event yields the subtype, the cost, the number of turns, the durations and the per-model usage, with a non-success subtype recorded as the error class. Lines that do not parse are kept as \texttt{unparsed} events. The per-run metrics of Section~\ref{sec:design-measurement} are computed from these rows by \path{src/wp42/metrics.py}; the mapping of a shell command to a tool family (browser, office, applescript, python, sql, shell) is by regular expression on the command text, file tools being the \texttt{file} family.236237\subsection{Per-run raw artefacts}\label{app:infra-artefacts}238239Each run has a directory \path{data/raw/runs/<run_id>/} on the controller, git-ignored and never rewritten, containing: \path{raw_stream.jsonl}, the client's event stream of the first session, line by line as received; \path{raw_stream_recovery.jsonl}, the stream of the recovery session when one was granted; \path{RECOVERY_PROMPT.md}, the recovery message when one was written; \texttt{out/}, the collected deliverables including \path{COMPLETION.json}, \path{CHECKPOINT.md} when the worker kept one, and \path{portal_submissions.json} when forms were submitted; \path{collection.json}, with the SHA-256 and size of every collected file, the list of modified inputs and the workspace manifest; \texttt{receipt.json}, the deterministic receipt; and \path{judgments.json}, the three judges' scores, votes, costs and errors. For the runs that were reset and rerun under amendment A-010, the first attempt's stream is preserved as \path{raw_stream.jsonl.attempt1}. The same information is held in the database (events, artifacts, receipts, judgments, metrics) and exported to the validated Parquet tables described in Appendix~\ref{app:repro}.240241\subsection{Judge command line}\label{app:infra-judge}242243Each judge is one call of the client on the controller, with the prompt of Appendix~\ref{app:rubrics-prompt} passed inline, no tools, one turn, structured output enforced by the schema of Appendix~\ref{app:rubrics-schema}, a budget cap of 0.80~USD and a 900-second timeout, in a temporary working directory, with the API key loaded from the secrets file into the process environment:244245\begin{Verbatim}[breaklines=true, breakanywhere=true, baselinestretch=1, fontsize=\footnotesize, frame=single, framesep=4pt]246claude --bare -p <prompt> --model <judge model> --output-format json --max-turns 1 --max-budget-usd 0.80247       --tools "" --permission-prompts none --no-chrome --disable-slash-commands --no-session-persistence248       --json-schema <schema JSON>249\end{Verbatim}250251The structured output is read from the \path{structured_output} field of the client's JSON result (falling back to parsing the \texttt{result} text), the cost from \path{total_cost_usd}, and the per-model usage from \texttt{modelUsage}. Before rendering, the collected \texttt{out/} is copied to a temporary directory and every workbook is recalculated by the office adapter (\path{src/wp42/office.py}: LibreOffice headless, Calc filter, fresh user profile, 120-second timeout; files whose first bytes are not a zip signature are left untouched) so that cached values exist whatever tool produced the workbook; the raw archive is never modified.252253\subsection{Emergency shutdown and stale-run detection}\label{app:infra-watchdog}254255\path{python -m wp42.cli kill-all} sets the scheduler's stop event (no further dispatch, no further recovery) and, on every executor node, kills every process matching \texttt{claude --bare} and every portal process. The stale-run detector of \path{src/wp42/watchdog.py} lists runs in the \texttt{preparing}, \texttt{running} or \texttt{recovering} states whose start time is older than a threshold (3,600~s by default) so that they can be killed and reset individually with \texttt{reset-run}. An interrupt signal to the running queue sets the same stop event and lets the dispatched runs finish.256
Files · main
Menu

Theme