SPB Git

spb/wp29_uqo

Public
5.1 KB · 25 lines markdown
1# Isolation audit — WP2923## Boundaries (what actually separates executors from hidden material)4| Boundary | Mechanism | Verification |5|---|---|---|6| Physical/host | Executors run on cluster nodes (M3U96a, M3U96b, M4M64a, M4M64b); oracles, evaluators, this repository and the series exist only on the laptop (and `~/wp29-oracles/` on the laptop) | nodes hold no copy (no rsync of oracles ever; `tests/isolation/test_bundle_hygiene.py` checks every public bundle for forbidden material); node → laptop SSH port 22 unreachable (`nc -z` from M3U96a, 2026-10-29) |7| Workspace | one fresh directory per run containing exactly the public bundle | `isolation.prepare_workspace` compares the remote SHA-256 manifest with the local bundle manifest before launch; mismatch ⇒ infrastructure error (this actually fired twice in the first smoke attempt because of an unexpanded `~`, and the runs were recorded as `error` with no inference) |8| Instruction loading | `--bare`: no CLAUDE.md discovery, no hooks, plugins, MCP, auto-memory, skills; instructions injected only via `--append-system-prompt-file`; identical flags across arms | sentinel probes: laptop normal mode loaded 49,077 tokens incl. the user's cluster CLAUDE.md; bare mode 144 tokens and no sentinel; on each node bare mode saw neither the project sentinel nor the node's `~/.claude/CLAUDE.md` (`reports/sentinel_probes_nodes.jsonl`, 175–177 input tokens) |9| Tools | requested `--tools Bash,Read,Write,Edit,Glob,Grep`; **delivered** (init event of every session): `Bash`, `Edit`, `Read` only; no web tools, no MCP, no sub-agent tool; two built-in plugins (`cc-plugin-agents-md`, `cc-plugin-plugin-authoring`) still load under `--bare`; identical across arms and sessions (systems review, 497 init events) | behaviour audit scans every tool_use input (path regex, `~` and `$HOME` normalised after the review found false positives): 4 of 384 runs referenced a path outside the workspace (RC0094 attempt 1 listed sibling run names; RA0008 read its own client transcript; RC0275 and RP0023 referenced `~`/home in shell idioms); 0 network commands, 0 reads of other runs' files, 0 nested `claude` invocations |10| Prompt delivery | intended user prompt per role in `protocol/arms/` | **defect A-006**: the remote shell line expanded the prompt file unquoted, so only its first word reached the client in every frozen session ('Begin', 'Review', 'The', 'Your'); the continuation sessions did not read the feedback files either (1–2 turns, asked for the missing message); treatment texts (system prompt) unaffected; fixed and verified post hoc (full prompt in RD0005–RD0008, RT0025, RT0026, RR0016–RR0017) |11| Credentials | `~/wp29/.env` (mode 600) on nodes, sourced by the launching shell; never written to logs or the repository; `--bare` never reads keychain/OAuth | grep of `runs/` for the key prefix is part of the release validation |12| Reviewer (arm D) | separate session id, separate system prompt, sees `TASK.md` + `out/` | residual leak path: the executor transcript exists under `~/.claude/projects` on the node and could be read with Bash; no reviewer session touched `.claude/projects` (systems review); one reviewer (RC0159) wrote its verdict to `out/out/review/review.json` after a `cd` drift — recovered from the collected artefact (A-008); review-repair never executed in the frozen benchmark |13| Nested invocation | no Agent tool; `PGAR_DEPTH` guard; executors cannot launch controllers | code review + probe |1415## Contamination of the task suite16Tasks were authored by Claude Code subagents (fresh contexts, no access to executor transcripts) under a frozen interface; the builder (this session) wrote the smoke task. Executor sessions are fresh (`--bare`, new session id, new workspace) and cannot access sibling runs, previous attempts or oracles. Pretraining familiarity with textbook methods is expected and reported per task (`contamination_risk` in each task card). Reference answers were accessible to: the authoring subagent (designer) and the evaluator host. No executor ever had access.1718## Residual risks191. Same user account on the nodes: an executor could read other runs' workspaces under `~/wp29/runs/` (sibling outputs of *other* runs) via Bash. Behaviour audit flags any such path; none occurred in the smoke. A stricter per-run OS identity was not implemented (documented limitation).202. The reviewer can in principle read the executor transcript (see above).213. The gate program is world-readable on the node; arms A/B were not told about it and never called it (0 invocations in 384 runs), which the audit records for every run.224. The nodes are not offline: they have Internet access; executors lacked web tools and issued no network command (audit), but the common preamble's sentence "There is no network access" was inaccurate.235. The harness changed after the freeze (A-004 scheduler, A-006 launcher); every run manifest records its code commit; no task, oracle, treatment text or analysis interface changed.246. Sibling exposure persists on the nodes after the study (workspaces, `.ctl` directories with arm texts, client transcript directories) under the shared user account.25
Files · main
Menu

Theme