SPB Git

spb/wp29_uqo

Public
9.1 KB · 40 lines latex
1\section{PGAR implementation}2\label{sec:implementation}34The reference implementation is a small Python harness (about 1,300 lines in \texttt{src/pgar/}, standard library plus \texttt{numpy}, \texttt{pandas}, \texttt{pyyaml} and \texttt{jsonschema} on the controller; the gate program is standard-library only so that it runs with any interpreter on an executor node). Figure~\ref{fig:architecture} shows the components and the trust boundaries between them. We describe the roles, the isolation mechanism, the launcher, the gate, the review stage, the state store and the capture subsystem, and we state explicitly what each mechanism does not do.56\begin{figure}[htbp]7\centering8\includegraphics[width=\textwidth]{figures/fig_architecture.pdf}9\caption{Architecture of the reference implementation (diagram, not a screenshot). The controller, the state store, the oracles and the evaluators live on the evaluator host; executor sessions run on separate cluster nodes in fresh workspaces that contain only the public task bundle. Arrows crossing the dashed boundary carry only the bundle, the instructions, the stream of events and the deliverables; reference answers never cross it.}10\label{fig:architecture}11\end{figure}1213\subsection{Roles and trust}1415Six roles are distinguished. The \emph{builder} is the Claude Code session that developed this study (model \texttt{claude-fable-5-1}, client 2.1.314) under the researcher's mandate; it wrote the harness, the protocols and the smoke task, and it delegated the authoring of the task suite and the literature review to fresh sub-sessions. The \emph{executors} are fresh Claude Code sessions (client 2.1.314, model \texttt{claude-opus-5-5}) launched in bare mode on four cluster nodes. The \emph{runtime verifier} is the gate program, a deterministic script without reference answers. The \emph{final evaluator} is a deterministic task-specific script executed only on the evaluator host against a hidden oracle. The \emph{reviewer} (arm D only) is a separate Claude Code session with its own system prompt. The \emph{reporting pipeline} consists of the export, analysis, table and figure scripts that produce every number in this paper from the state store.1617The builder's involvement in task design is a limitation that no amount of session freshness removes: the same organisation (the researcher's Claude Code account, two model identities from the same vendor) authored the tasks, the oracles and the executors' instructions. We mitigate it in the ways that are available (reference values from analytical identities, independently written code or source documents with locators; validation of every evaluator against seeded correct and incorrect artefacts; a frozen interface written before any task), and we report, for every task, who could access the reference answers at each stage.1819\subsection{Isolation}2021Three mechanisms separate executors from hidden material. First, a physical boundary: executors run on cluster nodes that hold no copy of the repository, the oracles or the earlier working papers, and the nodes cannot open an SSH connection to the evaluator host (verified by a port probe). Second, a workspace boundary: each run receives a fresh directory containing exactly the public bundle, verified before launch by comparing a SHA-256 manifest computed on the node with the local one; a mismatch aborts the run as an infrastructure error. Third, an instruction boundary: executors are started with \texttt{--bare}, which the client documents as skipping \texttt{CLAUDE.md} discovery, hooks, plugins, MCP servers, auto-memory and skills. We did not take the documentation on trust. Sentinel probes with an innocuous project \texttt{CLAUDE.md} showed that an ordinary print-mode session on the laptop loaded about 49,000 tokens of context, including the researcher's cluster instructions, while a bare session loaded 144 tokens and reported no sentinel; the same probe on each executor node reported no sentinel and no cluster instructions with 175--177 input tokens (Section~\ref{sec:design}). The treatment text is injected explicitly with \texttt{--append-system-prompt-file}, and every run records a hash of the effective context (common preamble, arm text, task statement and user prompt).2223What isolation does not do: executors share a user account on each node and could, with the shell tool, read sibling workspaces or the transcript of another session; the gate program is world-readable on the node and arms A and B were not told about it. We do not prevent these behaviours; we \emph{audit} them by scanning every tool invocation in the verbatim event stream for paths outside the workspace and for invocations of the gate, and we report the counts per run.2425\subsection{Launcher and resource envelope}2627A session is launched over SSH as \texttt{claude --bare -p} with a pre-generated session identifier, structured streaming output, a fixed tool inventory (\texttt{Bash}, \texttt{Read}, \texttt{Write}, \texttt{Edit}, \texttt{Glob}, \texttt{Grep}; no web access and no sub-agents), the listed tools pre-approved and every other permission request denied, a model identifier, a budget cap and a turn limit. The job is written to a ledger \emph{before} submission, which is what makes duplicate submissions detectable. Each line of the output stream is appended verbatim to a raw file and normalised into an event record (type, tool, status, usage, hash of the payload) as it arrives. If the wall-clock limit is reached, the controller kills the remote process by its unique session identifier and records the run as a timeout; the artefacts produced so far are still collected and evaluated. A session can be resumed with the client's \texttt{--resume} flag, which the harness uses for gate feedback (arms C and D), review feedback (D) and the recovery experiment. The envelope is shared: a continuation is launched with a cap equal to the remaining budget, and reviewer sessions draw from the same envelope as the executor they review. The client reports costs as list-price estimates, which can slightly exceed the cap; the harness records both the cap and the observed estimate and never substitutes an invented figure. No temperature or seed control exists in the client; repeats are therefore genuine fresh sessions.2829\subsection{The gate}3031The gate program checks the output contract of a task without knowing the answer. Given the public \texttt{gate\_spec.json}, it verifies that the required files exist, that \texttt{results.json} parses and contains every required key with the declared type and nullability and finite numbers, that the report has the six required sections, that the claims ledger is a non-empty list whose entries reference existing result keys, that an exclusion ledger (if present) sums correctly, that a phase ledger (if present) has no phase marked \emph{passed} without recorded checks and no non-terminal phase, that confidence intervals contain their estimates, that standard errors are positive, that declared date pairs respect chronology and, finally, that the replication script regenerates \texttt{results.json} in a clean copy within a relative tolerance of $10^{-6}$. It writes a receipt with every check and its detail. In arms C and D the executor is told to run the gate before declaring completion, and the controller runs its own pristine copy after the session ends; a failing receipt is returned to the executor as a resumed turn, at most three times, within the envelope. In arms A and B the same program runs once after the session, post hoc and silently, to provide the common contract check and the fresh re-run reproducibility measure for all arms.3233\subsection{Separate review}3435In arm D, once the controller gate passes, a new session is started in the same workspace with a reviewer system prompt. The reviewer is told that it has not seen how the deliverables were produced and must not look for the transcript; it may re-run code; it has no reference answers; it must write a structured verdict (\emph{accept} or \emph{revise}) with located issues. If the verdict is \emph{revise}, the executor session is resumed once with the review, after which the gate runs again. The review is ``separate'' in the sense of a fresh context and a different instruction, not in the sense of an independent judgment: it is the same model family as the executor, and we make no claim of independence \citep{panickssery2024selfpreference}.3637\subsection{State store, evidence and capture}3839All state lives in a single SQLite database opened in write-ahead-logging mode with explicit immediate transactions and serialised access from the controller's worker threads. Eleven tables record runs and their validated transitions, sessions (with the exact command, the cost reported by the client and the first-turn input size), events, gate receipts, reviews, evaluations, screenshots, faults, jobs and interventions. Analysis-ready CSV exports are regenerated from the database by a single script, and a read-only HTTP viewer renders the same rows for inspection and for dashboard captures. Screenshots are taken with the operating system's native capture utility on a window identified through the window server, never synthesised; each capture is logged with its surface, mode, timestamp, hash and the run or event it documents.40
Files · 4f59ed568641ceda6f39552d5cc3a441aa812088
Menu

Theme