SPB Git

spb/wp29_uqo

Public
3.5 KB · 20 lines markdown
1# Treatment definitions (frozen for the pilot; re-frozen before the confirmatory stage)23All arms share: the same `TASK.md` and bundle; the same client (Claude Code 2.1.314, `--bare`), model (`claude-opus-5-5`, CLI default effort), tool inventory (requested Bash, Read, Write, Edit, Glob, Grep; delivered Bash, Edit, Read — bare mode; no web, no sub-agents), permission policy (`--permission-prompts none`, listed tools pre-approved), offline executor node with the same pinned Python environment, the same resource envelope (4.00 USD, 1,500 s wall, 80 turns per session), the same user prompt (`protocol/arms/task_prompt.md`) and the same common preamble (`protocol/arms/common_preamble.md`: environment facts, output contract, integrity rules). The only difference is the text appended after the common preamble in `--append-system-prompt-file`:45| Arm | Appended text | Words (preamble + arm) | Controller behaviour after the session |6|---|---|---|---|7| A — Open instruction | `arm_A.md` (one sentence) | 285 | collect, post-hoc contract check (not fed back), evaluate |8| B — Structured protocol | `arm_B.md` (phases P0–P5, stopping rules, recovery rules, phase ledger, claims ledger) | 944 | same as A |9| C — Protocol + executable gates | `arm_C.md` = B + mandatory gate program (`~/wp29/gates/pgar_gate.py`) | 1,106 | run the pristine gate; if it fails, resume the session with the gate output (≤ 3 cycles, within the envelope); evaluate |10| D — Protocol + gates + separate review | `arm_D.md` = C + announcement of a separate review | 1,200 | as C; then a **separate** reviewer session (fresh session id, `reviewer.md` system prompt, sees only `TASK.md` + `out/`) writes `out/review/review.json`; if `revise`, the executor is resumed once with the review; gate again; evaluate |1112Measured initial context (first assistant turn, smoke run, tokens): A 2,238; B 3,717; C 4,111; D 4,319 (`sessions.first_turn_input_tokens`). These include the client's own system prompt and the task prompt; the difference between arms is the protocol text only.1314Content-equivalence audit: the arm texts add **no** scientific information (no data facts, no method hints beyond "compute the simplest valid baseline", which is already a required output in every `TASK.md` that defines a baseline key). The common preamble is byte-identical across arms (`protocol_hash` per run = SHA-256 of preamble+arm text; `effective_context_hash` adds TASK.md and the user prompt). A length-matched unstructured control (`arm_A_long.md`, same word count as B, generic research advice without phases, ledgers or gates) is a Phase 08 ablation if the budget allows.1516What the gate cannot do: it contains no reference values and cannot judge scientific correctness; it checks the output contract, internal consistency and reproducibility. What the reviewer cannot do: it has no reference answers and no access to the executor's transcript (it is told not to look for it; the transcript lives outside the workspace under `~/.claude/projects` on the node — a residual, documented leak path that was not exercised in the smoke run: the reviewer's tool uses referenced only workspace paths).171819**Post-freeze note (2026-10-30, A-006):** in every frozen session the user prompt was truncated to its first word by a launcher quoting defect; the arm texts (system prompt) were delivered intact. The controller gate loop (C/D) and the review-repair stage (D) therefore did not act through prompt text in any frozen run; see the paper's Section 5 and amendments A-004/A-006/A-008.20
Files · main
Menu

Theme