1# Adversarial systems and isolation review — WP29 (PGAR)23**Reviewer identity.** This review was produced by a Claude Code sub-session (model `claude-fable-5-1`, same client family as the builder session), launched as a fresh context with no stake in the result. It is **not** an independent human reviewer, and it shares the vendor, client and account of the builder; treat it as an internal adversarial check, not as external validation. No file was modified except the creation of this report; no Claude Code session and no inference was run; `pgar.cli run` was not invoked. Everything below was obtained from read-only commands on the laptop, read-only SQL on `runs/state.sqlite`, the unit tests, and one read-only SSH inspection of M3U96a.45**State of the repository at review time (2026-10-30 ≈ 06:00–06:40 UTC).** The builder was still active: `src/pgar/launcher.py`, `paper/sections/implementation.tex`, `protocol/amendments.csv` (A-006) and the development split (RD0005–RD0008) changed on disk while this review ran; commit `452964e` landed during the review. Findings refer to the state observed; where the builder's concurrent fix is relevant it is noted. DB totals at review time: 384 runs (288 confirmatory, 24 pilot, 24 ablation, 24+1 transfer, 15 recovery, 8 development), 501 sessions, ≈ 172 USD of list-price inference.67---89## 1. Findings, ranked by severity1011### CRITICAL1213**C1 — No continuation session ever received its prompt; the gate-feedback loop of arms C/D was never delivered, and three sections of the paper describe it as if it had been.**14Evidence. `launcher.py` (pre-fix, line 23/43) placed `$(cat prompt_file)` unquoted on the remote shell line; the node's shell word-split it and the client took only the first word as the prompt. Transcript census over all `runs/*/*/sessions/*.jsonl`: first user turn = `'Begin'` × 379, `'Review'` × 89, `'The'` × 15 (every gate-feedback cycle), `'Your'` × 12 (every resume-after-fault); the full prompt appears only in the 5 post-fix development sessions. What the 15 gate-feedback sessions did (raw streams of RR0015, RT0009, RT0010, RT0014, RT0015): 1–2 turns, 0–1 trivial tool calls (`ls`), cost 0.011–0.034 USD, and every one answered in the style of *"Your message came through as just 'The'… could you resend it?"* None read `out/state/gate_receipts/` or `out/review/`. The builder discovered the same defect concurrently (amendment A-006, commit 452964e, unit test `tests/unit/test_launcher_quoting.py`, launcher now double-quotes the expansion), **but the remediation text is itself unsupported**: A-006 and `implementation.tex:27` now state that "continuation feedback … reached the executor only through files the controller had written into the workspace … we treat the feedback mechanism as delivered via files". The transcripts show it reached no executor by any channel.15Consequences. (i) `implementation.tex:31` ("a failing receipt is returned to the executor as a resumed turn, at most three times") was never realised. (ii) `transfer.tex:18` — "in arms C and D the controller returned it to the executor three times. The executors did not fabricate a chronology …: they re-examined their ledgers, explained in their reports why the dates were correct, and resubmitted unchanged" — is false as a description of those sessions; the four T1 C/D runs (RT0009/10/14/15) spent three one-turn sessions asking for the missing message, and any explanation of the dates in their reports was written in the initial session. (iii) `recovery.tex:8` — "resumed once with a neutral prompt stating that the interruption was an infrastructure event" — the resume prompt was the word "Your"; 2 of 12 executors (RR0003, RR0012) explicitly said so; the other 10 inferred from workspace state and continued. (iv) `limitations.tex:14` "the gate and the reviewer could only act through files" — they did not act at all through continuations. (v) The design claim that arms C/D include a controller-enforced repair loop is, for this dataset, a claim about code, not about delivered treatment; the primary A–D contrast therefore compares *system-prompt text* only (which was delivered intact, see §2).16Recommended fix. Replace "delivered via files" everywhere with: "the gate/review feedback mechanism was **not delivered** in any session of the frozen study (verified: all 15 gate-feedback and all 12 resume sessions received a one-word prompt); the frozen comparison therefore tests the protocol text, not the feedback loop". Rewrite the T1 paragraph in `transfer.tex` to say the three continuation sessions were vacuous and that the unchanged resubmission is an artefact of the defect, not a judgement by the executor. Rewrite `recovery.tex:8`. If the paper wants any claim about controller-enforced repair, a rerun of the C/D gate-failure cases with the corrected launcher is required (the four T1 C/D runs and RR0015 would suffice for a minimal demonstration).1718### MAJOR1920**M1 — The delivered tool inventory is not the one documented, and `--bare` did not yield an empty plugin/agent set.**21Evidence. All 497 `system/init` events in `runs/*/*/raw_stream.jsonl` report `tools = ["Bash","Edit","Read"]`, `agents = [claude, Explore, general-purpose, Plan, statusline-setup]`, `plugins = [cc-plugin-agents-md, cc-plugin-plugin-authoring]`, `mcp_servers = []`, `permissionMode = default`, `apiKeySource = ANTHROPIC_API_KEY`, version 2.1.314. The argv (all 501 `sessions.command_json`) passes `--tools Bash,Read,Write,Edit,Glob,Grep --allowedTools Bash Read Write Edit Glob Grep`. Tool-use census: only Bash (4 356), Read (69) and Edit (12) were ever called — Write/Glob/Grep never, consistent with their absence. Uniform across arms, so not a confound, but `reports/isolation_audit.md` ("Tools | `--tools Bash,Read,Write,Edit,Glob,Grep` … the stream's init event lists the tool set"; "`--bare`: no hooks, plugins, MCP"), `implementation.tex:27` ("fixed tool inventory (Bash, Read, Write, Edit, Glob, Grep)") and `scientific_contract.md` row A are not supported by the init events.22Fix. Report the delivered inventory (Bash, Edit, Read) and say that Write/Glob/Grep were requested but not exposed by client 2.1.314 (cause unknown — worth a probe); replace "no plugins" with "no user plugins, MCP or hooks; two built-in plugins and five built-in agent definitions were listed by the client, and no `Agent` tool was exposed".2324**M2 — The behavioural audit over-counts and the paper under-reports; one genuine sibling-directory listing is not mentioned.**25Evidence. `isolation.scan_events_for_escapes` (isolation.py:70–86) compares path tokens against the absolute workspace path but does not normalise `~/` or `$HOME`; consequently `~/wp29/runs/<same run>/out/results.json` is flagged as "outside". Aggregate of `behaviour_audit.json`: confirmatory D = 169 "outside" refs in 58 runs, pilot D 23, recovery D 18, transfer D 8, almost all of this form (reviewer sessions use `~`); RC0275 (A) is an awk regex `!~/^[0-9.]+$/`; RC0274 is `out/../../../venv/bin/python`. `limitations.tex:16` nevertheless states "zero outside-workspace references in every audited run", which contradicts the audit files. The scanner also cannot see relative escapes (`cd ..`, `$HOME`); my independent grep of every tool_use input found no `curl`/`wget`/`pip`/`http(s)://`, no nested `claude`, no `ssh`/`scp`/`rsync`, no `env`/`printenv`, no `oracle`, no `wp29-oracles`, no other run id, and `pgar_gate` only in C/D (0 in A/B/L). **One genuine escape**: RC0094 attempt 1 (arm D executor, line 58) ran `ls /Users/simon-pierreboucher/wp29/runs/ | grep -v RC | head` and received the names of sibling run directories (RD0002, RD0003, RP0002, RP0006, RP0010, RP0014 and their `.ctl` dirs); no sibling content was read. RA0008 (L) read the client's own tool-result spill file under `~/.claude/projects/...` (benign, but proves that directory is readable).26Fix. Normalise `~` and `$HOME` to the home path before comparison, add `..`-resolution relative to the Bash cwd, rescan, regenerate the audit column; change the limitations sentence to "one directory listing of the parent of all workspaces (RC0094, aborted attempt), no reads of sibling content, no network commands, no nested sessions".2728**M3 — Two reviewer `revise` verdicts occurred; both were lost; the review-repair loop was never exercised.**29Evidence. `reviews`: 87 accept, 1 revise (RC0094 attempt 1, which then crashed on the A-004 ledger bug), 1 `null` (RC0159). RC0159's reviewer wrote a `revise` verdict with a major issue, but its Bash cwd had drifted into `out/` (the client's Bash tool keeps cwd between calls), so `cat > out/review/review.json` landed at `<ws>/out/out/review/review.json` (present locally at `runs/confirmatory/RC0159/out/out/review/review.json`, hash recorded in `artifact_manifest.json` as `out/review/review.json`); the controller's fixed-path fetch (`scheduler.py:151`) returned `null` and the run was closed as accept-equivalent. `sessions` contains **zero** `review_repair` rows. RC0009 shows the same `out/out` drift. `implementation.tex:35` ("If the verdict is revise, the executor session is resumed once with the review") describes code that never ran.30Fix. Count RC0159 as a `revise` verdict in every review statistic (2/89, not 1/89); state that the repair loop had zero executions; harden the fetch (`find <ws> -name review.json`, or parse the reviewer's final text as fallback) and reset cwd per call via `cd <ws> &&` in the preamble instructions.3132**M4 — The event ledger silently drops the events of a re-attempt; `first_turn_input_tokens` is NULL for both RC0094 attempt-2 sessions.**33Evidence. `events.event_id = f"{run_id}:{seq:05d}"` (events.py:14), `seq` restarts at 0 on each attempt (scheduler.py:112), `add_event` uses `INSERT OR IGNORE` (state.py:180). For RC0094, `events` holds only sessions 45da… (72 rows) and af15… (35 rows) of the aborted attempt; the 95 events of the evaluated attempt (989b…, 99a3…) exist only in `raw_stream.jsonl` (202 lines, both attempts appended) and in `sessions/*.jsonl`. Any analysis built on `events` (tool-use counts, timelines) for RC0094 describes the aborted attempt.34Fix. Key `event_id` by `(run_id, attempt, seq)` or by `(session_id, seq)`; document in the data description that RC0094's `events` rows belong to attempt 1. Minor related point: `rsync_from` of `out/` has no `--delete` (scheduler.py:161), so stale attempt-1 artefacts could survive locally — not observed for RC0094 (attempt 2 wrote every file), but a latent hazard.3536**M5 — Harness code changed during the confirmatory campaign and every run carries a `-dirty` commit, so the exact executed code is not recoverable from git; the freeze manifest predates two fixes.**37Evidence. `runs.manifest_json.code_commit` over the 288 confirmatory runs: 449400e-dirty (40), 142b688-dirty (77), 502400b-dirty (171); pilot ran on 76de75e/55b92f9; transfer on 502400b/4f59ed5/6df43c3. Amendment A-003 says the freeze manifest records SHA-256 of launcher/scheduler, yet A-004 (scheduler) and A-006 (launcher) changed them afterwards. Before A-004 any continuation would have aborted the run (`UNIQUE constraint failed: sessions.session_id`); only RC0094 was hit because the controller gate passed first time in all 144 confirmatory C/D runs (`gate_receipts`: 72 C + 73 D controller receipts, all `passed=1`), so the gate-cycle path was not exercised in the main benchmark at all.38Fix. State in the paper that the main benchmark never triggered a gate cycle; list the three code states with their run counts; add a per-run hash of `src/pgar/*.py` to the manifest (the `-dirty` flag alone is not provenance); update the freeze manifest or add an explicit "post-freeze code changes" table.3940### MINOR4142**m1 — Screenshot captions.** S002 is stamped 03:34:33.673 UTC with the caption "while it runs on M3U96b", but RD0002's executor session ended 03:33:52.873, its reviewer 03:34:30.522, and the run reached `evaluated` at 03:34:32.321 — the capture is 1.4 s after the run was closed (the Terminal was still tailing the finished file). S007 (RC0009, 4 s before session end), S008 and S004 (7 runs in `running` at both timestamps, verified from `run_transitions`) are consistent. S005/S006/S008 carry `redaction_description = "cropped: top 300 px removed"` while `redacted_sha256 == original_sha256` and the files are byte-identical (1312×822 / 2624×1644): the retained "original" is already the cropped region capture, so the manifest cannot evidence an uncropped source. All 9 `original_sha256` values match the files on disk. Fix: reword S002 as "immediately after completion"; set S005/S006/S008 redaction to "none (region capture)".4344**m2 — Killed sessions are charged 0 in `sessions`.** For the 12 interrupt runs the initial row has `total_cost_usd = session_cost_usd = 0.0` because the list-price fallback (`scheduler.py:221`) only triggers on `timed_out`. The envelope is nonetheless not under-charged: the resumed session's `total_cost_usd` is cumulative (RR0002: 0.375 reported ≈ 0.319 killed-session usage + 0.076 own; RR0010: 0.196 ≈ 0.127 + 0.045), as it is for gate cycles (RT0009: 0.912 → 0.935 → 0.951 → 0.965). Fix: charge the kill-time estimate to the killed row and subtract it from the resume, or document the convention.4546**m3 — Amendment timestamps are hand-rounded and one is mislabelled.** A-002 is stamped 2026-10-30T04:40:00Z and labelled `before_confirmatory_results`, but its commit (55b92f9) is 03:57Z and the confirmatory assignment started 04:02Z; A-001/A-004/A-005/A-006 are at :00/:40/:40/:20. Fix: use the commit timestamps.4748**m4 — Sibling exposure is persistent, not just concurrent, and includes other arms' instructions.** On M3U96a: `~/wp29/runs/` holds 199 entries (≈ 99 workspaces + their `.ctl` dirs, none cleaned), `~/.claude/projects/` holds 98 `wp29-runs-*` transcript directories, all owned by the same user; each `.ctl` dir contains that run's `system_append.md` (arm text) and `reviewer_system.md`. The audit's wording "sibling outputs of *other* runs" understates it: a later executor can read any earlier run's deliverables, transcript and arm text (none did, per M2). Fix: say so; delete workspaces/transcripts after collection in future runs.4950**m5 — "Offline nodes" / "no network access".** `scientific_contract.md` row A says "offline nodes" and both the common preamble and `reviewer.md` tell executors "There is no network access". The nodes have Internet access (the API calls require it); only the web *tools* were withheld and `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1` set. The grep in M2 found zero network commands, so the claim holds behaviourally, not mechanically. Fix: "no web tools; network not blocked; absence of network commands audited".5152**m6 — The API key is in the executor's process environment.** `remote_shell_line` sources `~/wp29/.env` with `set -a` (launcher.py:46), so every Bash tool subprocess inherits `ANTHROPIC_API_KEY`; an executor running `env` would have written it into the stream. None did (0 `env`/`printenv` calls; `grep -r sk-ant- runs reports paper release results` → only the pattern in `release/validate_release.py`). `.env` is mode 600 on the node. Fix: mention the inheritance as a residual; consider `env ANTHROPIC_API_KEY=… claude …` with `unset` in the preamble is not possible, so at minimum keep the release grep.5354**m7 — Nested-session guard is narrower than stated.** `PGAR_DEPTH` (launcher.py:51) only stops the Python controller; the `claude` binary is on the node and callable from Bash with the inherited key. No executor did so (0 hits). `scientific_contract.md` invariant 7 ("executors cannot launch controllers") is true; "nothing goes deeper" (launcher docstring) is not enforced. Fix: wording.5556**m8 — Dead code / cosmetics.** scheduler.py:201–202 (no-op query), 280 (`… if False else None`); `--permission-prompts none` is passed while init reports `permissionMode: default` (allowedTools pre-approval does the work — fine, but say so).5758**m9 — Post-fix verification runs are not comparable on first-turn size.** RD0005–RD0008 show `first_turn_input_tokens` 2273/3752 vs 2238/3717 for the frozen runs (+35 tokens = the full prompt), confirming the fix but also that the frozen study and the verification set differ in delivered context; results.tex Table "Post-fix verification" should carry that caveat (it says "not pooled", which is right).5960---6162## 2. Claims verified as supported (so the reader can weigh the above)6364- **Launcher flags identical across arms.** Normalising `sessions.command_json.argv` over all 501 sessions (replacing cap, session id and the per-run `.ctl` path) yields exactly one argv per (split, role, kind); arms differ only in the content of `system_append.md`. Flags present: `--bare -p … --output-format stream-json --verbose --model claude-opus-5-5 --tools … --allowedTools … --permission-prompts none --max-budget-usd <cap> --max-turns 80|150 --append-system-prompt-file … --no-chrome --disable-slash-commands --session-id|--resume <sid>`; no `--no-session-persistence` (hence transcripts persist, m4).65- **Same treatment text within an arm.** `protocol_hash` is unique per arm across every split (A 1d6fe7…, B bc05a0…, C f8afd6…, D d1a7be…, L c25619…); it equals SHA-256 of each run's stored `system_append.md` (0 mismatches over 384 runs) and equals what the current `protocol/arms/common_preamble.md + arm_X.md` regenerate. `first_turn_input_tokens` is constant within arm across all splits (A 2238, B 3717, C 4111, D 4319, L 3190; reviewer 2224).66- **Workspace verification.** 384 `prepare.json` manifests, 0 hash mismatches against the task `bundle.json`; `verified_identical` enforced before launch (scheduler.py:86).67- **No oracle material in bundles or on nodes.** No forbidden names in `tasks/public_bundles` (`F4-conf-04/data/reference_list.json` is the bibliography the task asks the executor to verify, not an answer); no `oracle`/`REFERENCE`/`planted`/`trap` strings; on M3U96a no `~/wp29-oracles`, no repository copy, no oracle-like file under `~/wp29`; laptop port 22 unreachable from the node (`nc -z` → unreachable, re-checked); `.env` mode 600; gate `pgar_gate.py` mode 444.68- **Reviewer sessions.** 89 reviewer sessions all used `reviewer_system.md` (argv) and the one-word prompt "Review"; none referenced `.claude/projects` or `transcript` (the 15 "transcript" string hits are "transcription error" in task text).69- **Budget enforcement.** For all 501 sessions `cap_usd == min(max, max − cumulative session cost of the attempt)` (0 exceptions); no attempt exceeds 4.00 (benchmark) or 12.00 (transfer); max observed per run 1.244 (confirmatory), 1.604 (transfer). Reviewer and continuation caps equal the remaining envelope (e.g. RC0094 reviewer cap 3.2497 = 4.0 − 0.7503).70- **Durable execution.** Invariant 1 enforced in `state.transition` and tested; R3 (`recovery_r3_duplicate.json`): RR0013 submitted 3×, 2 `duplicate_ignored`, 1 session launched; RC0094 attempt 1 cost (0.998 USD) is retained in `sessions` and in `reports/experiment_accounting.md` ("Sessions cost (all attempts)" 127.42 vs envelope 126.42); run_transitions show the legal `error → prepared` path.71- **Clocks.** DB `started_utc` precedes the first raw-stream event by 1.7–3.4 s (SSH + client start-up), `ended_utc` trails the last event by ≤ 0.3 s, `duration_ms` equals DB wall time (12-session random sample).72- **Evaluators do not execute executor code on the oracle host** (no `subprocess` in any `evaluate.py`; the reproducibility re-run happens on the node in a temp copy, `runtime_gates.py:185–193`).73- **Tests.** `pytest -q tests` → 10 passed (incl. the new quoting test).7475## 3. Claims in `reports/isolation_audit.md` and `paper/sections/implementation.tex` NOT supported by the data76771. isolation_audit "Tools" row and implementation.tex:27 — tool inventory Bash/Read/Write/Edit/Glob/Grep (delivered: Bash/Edit/Read). [M1]782. isolation_audit "Instruction loading" row — "no … plugins" (two built-in plugins and five agent definitions listed in every init event). [M1]793. implementation.tex:27 (A-006 sentence) — "continuation feedback … reached the executor only through files … we treat the feedback mechanism as delivered via files". [C1]804. implementation.tex:31 — "a failing receipt is returned to the executor as a resumed turn, at most three times" (as delivered: the word "The"). [C1]815. implementation.tex:35 — "If the verdict is revise, the executor session is resumed once with the review" (never executed; both revise verdicts lost). [M3]826. implementation.tex:23 / isolation_audit "Reviewer" row — audit "reports the counts per run": the counts are wrong by construction (tilde paths) and the paper's derived statement (limitations.tex:16 "zero outside-workspace references in every audited run") contradicts the audit files. [M2]837. implementation.tex:37 — "Eleven tables record … events" as a faithful record: a re-attempt's events are dropped. [M4]848. isolation_audit "Nested invocation" row — "executors cannot launch controllers" is fine; the implicit "no nested sessions" is unenforced. [m7]859. isolation_audit "Residual risks 1" — understates persistence of sibling workspaces/transcripts and the readability of other runs' `.ctl` instruction files. [m4]8610. (contract) "offline nodes"; (preamble/reviewer text) "There is no network access". [m5]87Also outside the two documents but load-bearing: transfer.tex:18 and recovery.tex:8 (C1); amendments A-002 timestamp (m3).8889## 4. Overall verdict9091The isolation claims that matter most for validity — that every arm received byte-identical launcher flags and a single, hashed treatment text; that executors never had the oracles, the repository, SSH to the laptop, web tools or any observed network/nested-session behaviour; that workspaces were verified before launch; that budgets were enforced as the remaining envelope and never exceeded; and that costs, duplicates and re-attempts are honestly accounted — are supported by the code, the database and the raw streams, and I could not find a channel by which reference answers reached an executor. What is **not** supported is the paper's account of the *delivered* execution mechanism: the tool inventory was Bash/Edit/Read rather than six tools, `--bare` still listed built-in plugins and agents, the behavioural audit numbers are wrong in both directions, and — most seriously — because of the unquoted prompt expansion (A-006) no continuation session in the frozen study ever received its gate feedback, review feedback or interruption notice; the paper's new wording that feedback "reached the executor through files" is contradicted by the transcripts, and the transfer and recovery sections narrate executor behaviour that did not occur. The frozen A–D comparison therefore remains a clean test of protocol *text* delivered via the system prompt, but any sentence that attributes an outcome to controller-enforced feedback, review-repair or the resumption instruction must be removed or supported by a rerun with the corrected launcher.92