# WP29 — Final review record

Assembled 2026-10-30T06:09:39+00:00. Reviewers are Claude Code sub-sessions (claude-fable-5-1) of the same client family as the builder; no human review had occurred at build time. The builder's responses are in reports/final_review.md.

# Adversarial systems and isolation review — WP29 (PGAR)

Reviewer identity. This review was produced by a Claude Code sub-session (model claude-fable-5-1, same client family as the builder session), launched as a fresh context with no stake in the result. It is not an independent human reviewer, and it shares the vendor, client and account of the builder; treat it as an internal adversarial check, not as external validation. No file was modified except the creation of this report; no Claude Code session and no inference was run; pgar.cli run was not invoked. Everything below was obtained from read-only commands on the laptop, read-only SQL on runs/state.sqlite, the unit tests, and one read-only SSH inspection of M3U96a.

State of the repository at review time (2026-10-30 ≈ 06:00–06:40 UTC). The builder was still active: src/pgar/launcher.py, paper/sections/implementation.tex, protocol/amendments.csv (A-006) and the development split (RD0005–RD0008) changed on disk while this review ran; commit 452964e landed during the review. Findings refer to the state observed; where the builder's concurrent fix is relevant it is noted. DB totals at review time: 384 runs (288 confirmatory, 24 pilot, 24 ablation, 24+1 transfer, 15 recovery, 8 development), 501 sessions, ≈ 172 USD of list-price inference.

# 1. Findings, ranked by severity

# CRITICAL

C1 — No continuation session ever received its prompt; the gate-feedback loop of arms C/D was never delivered, and three sections of the paper describe it as if it had been. Evidence. launcher.py (pre-fix, line 23/43) placed $(cat prompt_file) unquoted on the remote shell line; the node's shell word-split it and the client took only the first word as the prompt. Transcript census over all runs/*/*/sessions/*.jsonl: first user turn = 'Begin' × 379, 'Review' × 89, 'The' × 15 (every gate-feedback cycle), 'Your' × 12 (every resume-after-fault); the full prompt appears only in the 5 post-fix development sessions. What the 15 gate-feedback sessions did (raw streams of RR0015, RT0009, RT0010, RT0014, RT0015): 1–2 turns, 0–1 trivial tool calls (ls), cost 0.011–0.034 USD, and every one answered in the style of "Your message came through as just 'The'… could you resend it?" None read out/state/gate_receipts/ or out/review/. The builder discovered the same defect concurrently (amendment A-006, commit 452964e, unit test tests/unit/test_launcher_quoting.py, launcher now double-quotes the expansion), but the remediation text is itself unsupported: A-006 and implementation.tex:27 now state that "continuation feedback … reached the executor only through files the controller had written into the workspace … we treat the feedback mechanism as delivered via files". The transcripts show it reached no executor by any channel. Consequences. (i) implementation.tex:31 ("a failing receipt is returned to the executor as a resumed turn, at most three times") was never realised. (ii) transfer.tex:18 — "in arms C and D the controller returned it to the executor three times. The executors did not fabricate a chronology …: they re-examined their ledgers, explained in their reports why the dates were correct, and resubmitted unchanged" — is false as a description of those sessions; the four T1 C/D runs (RT0009/10/14/15) spent three one-turn sessions asking for the missing message, and any explanation of the dates in their reports was written in the initial session. (iii) recovery.tex:8 — "resumed once with a neutral prompt stating that the interruption was an infrastructure event" — the resume prompt was the word "Your"; 2 of 12 executors (RR0003, RR0012) explicitly said so; the other 10 inferred from workspace state and continued. (iv) limitations.tex:14 "the gate and the reviewer could only act through files" — they did not act at all through continuations. (v) The design claim that arms C/D include a controller-enforced repair loop is, for this dataset, a claim about code, not about delivered treatment; the primary A–D contrast therefore compares system-prompt text only (which was delivered intact, see §2). Recommended fix. Replace "delivered via files" everywhere with: "the gate/review feedback mechanism was not delivered in any session of the frozen study (verified: all 15 gate-feedback and all 12 resume sessions received a one-word prompt); the frozen comparison therefore tests the protocol text, not the feedback loop". Rewrite the T1 paragraph in transfer.tex to say the three continuation sessions were vacuous and that the unchanged resubmission is an artefact of the defect, not a judgement by the executor. Rewrite recovery.tex:8. If the paper wants any claim about controller-enforced repair, a rerun of the C/D gate-failure cases with the corrected launcher is required (the four T1 C/D runs and RR0015 would suffice for a minimal demonstration).

# MAJOR

M1 — The delivered tool inventory is not the one documented, and --bare did not yield an empty plugin/agent set. Evidence. All 497 system/init events in runs/*/*/raw_stream.jsonl report tools = ["Bash","Edit","Read"], agents = [claude, Explore, general-purpose, Plan, statusline-setup], plugins = [cc-plugin-agents-md, cc-plugin-plugin-authoring], mcp_servers = [], permissionMode = default, apiKeySource = ANTHROPIC_API_KEY, version 2.1.314. The argv (all 501 sessions.command_json) passes --tools Bash,Read,Write,Edit,Glob,Grep --allowedTools Bash Read Write Edit Glob Grep. Tool-use census: only Bash (4 356), Read (69) and Edit (12) were ever called — Write/Glob/Grep never, consistent with their absence. Uniform across arms, so not a confound, but reports/isolation_audit.md ("Tools | --tools Bash,Read,Write,Edit,Glob,Grep … the stream's init event lists the tool set"; "--bare: no hooks, plugins, MCP"), implementation.tex:27 ("fixed tool inventory (Bash, Read, Write, Edit, Glob, Grep)") and scientific_contract.md row A are not supported by the init events. Fix. Report the delivered inventory (Bash, Edit, Read) and say that Write/Glob/Grep were requested but not exposed by client 2.1.314 (cause unknown — worth a probe); replace "no plugins" with "no user plugins, MCP or hooks; two built-in plugins and five built-in agent definitions were listed by the client, and no Agent tool was exposed".

M2 — The behavioural audit over-counts and the paper under-reports; one genuine sibling-directory listing is not mentioned. Evidence. isolation.scan_events_for_escapes (isolation.py:70–86) compares path tokens against the absolute workspace path but does not normalise ~/ or $HOME; consequently ~/wp29/runs/<same run>/out/results.json is flagged as "outside". Aggregate of behaviour_audit.json: confirmatory D = 169 "outside" refs in 58 runs, pilot D 23, recovery D 18, transfer D 8, almost all of this form (reviewer sessions use ~); RC0275 (A) is an awk regex !~/^[0-9.]+$/; RC0274 is out/../../../venv/bin/python. limitations.tex:16 nevertheless states "zero outside-workspace references in every audited run", which contradicts the audit files. The scanner also cannot see relative escapes (cd .., $HOME); my independent grep of every tool_use input found no curl/wget/pip/http(s)://, no nested claude, no ssh/scp/rsync, no env/printenv, no oracle, no wp29-oracles, no other run id, and pgar_gate only in C/D (0 in A/B/L). One genuine escape: RC0094 attempt 1 (arm D executor, line 58) ran ls /Users/simon-pierreboucher/wp29/runs/ | grep -v RC | head and received the names of sibling run directories (RD0002, RD0003, RP0002, RP0006, RP0010, RP0014 and their .ctl dirs); no sibling content was read. RA0008 (L) read the client's own tool-result spill file under ~/.claude/projects/... (benign, but proves that directory is readable). Fix. Normalise ~ and $HOME to the home path before comparison, add ..-resolution relative to the Bash cwd, rescan, regenerate the audit column; change the limitations sentence to "one directory listing of the parent of all workspaces (RC0094, aborted attempt), no reads of sibling content, no network commands, no nested sessions".

M3 — Two reviewer revise verdicts occurred; both were lost; the review-repair loop was never exercised. Evidence. reviews: 87 accept, 1 revise (RC0094 attempt 1, which then crashed on the A-004 ledger bug), 1 null (RC0159). RC0159's reviewer wrote a revise verdict with a major issue, but its Bash cwd had drifted into out/ (the client's Bash tool keeps cwd between calls), so cat > out/review/review.json landed at <ws>/out/out/review/review.json (present locally at runs/confirmatory/RC0159/out/out/review/review.json, hash recorded in artifact_manifest.json as out/review/review.json); the controller's fixed-path fetch (scheduler.py:151) returned null and the run was closed as accept-equivalent. sessions contains zero review_repair rows. RC0009 shows the same out/out drift. implementation.tex:35 ("If the verdict is revise, the executor session is resumed once with the review") describes code that never ran. Fix. Count RC0159 as a revise verdict in every review statistic (2/89, not 1/89); state that the repair loop had zero executions; harden the fetch (find <ws> -name review.json, or parse the reviewer's final text as fallback) and reset cwd per call via cd <ws> && in the preamble instructions.

M4 — The event ledger silently drops the events of a re-attempt; first_turn_input_tokens is NULL for both RC0094 attempt-2 sessions. Evidence. events.event_id = f"{run_id}:{seq:05d}" (events.py:14), seq restarts at 0 on each attempt (scheduler.py:112), add_event uses INSERT OR IGNORE (state.py:180). For RC0094, events holds only sessions 45da… (72 rows) and af15… (35 rows) of the aborted attempt; the 95 events of the evaluated attempt (989b…, 99a3…) exist only in raw_stream.jsonl (202 lines, both attempts appended) and in sessions/*.jsonl. Any analysis built on events (tool-use counts, timelines) for RC0094 describes the aborted attempt. Fix. Key event_id by (run_id, attempt, seq) or by (session_id, seq); document in the data description that RC0094's events rows belong to attempt 1. Minor related point: rsync_from of out/ has no --delete (scheduler.py:161), so stale attempt-1 artefacts could survive locally — not observed for RC0094 (attempt 2 wrote every file), but a latent hazard.

M5 — Harness code changed during the confirmatory campaign and every run carries a -dirty commit, so the exact executed code is not recoverable from git; the freeze manifest predates two fixes. Evidence. runs.manifest_json.code_commit over the 288 confirmatory runs: 449400e-dirty (40), 142b688-dirty (77), 502400b-dirty (171); pilot ran on 76de75e/55b92f9; transfer on 502400b/4f59ed5/6df43c3. Amendment A-003 says the freeze manifest records SHA-256 of launcher/scheduler, yet A-004 (scheduler) and A-006 (launcher) changed them afterwards. Before A-004 any continuation would have aborted the run (UNIQUE constraint failed: sessions.session_id); only RC0094 was hit because the controller gate passed first time in all 144 confirmatory C/D runs (gate_receipts: 72 C + 73 D controller receipts, all passed=1), so the gate-cycle path was not exercised in the main benchmark at all. Fix. State in the paper that the main benchmark never triggered a gate cycle; list the three code states with their run counts; add a per-run hash of src/pgar/*.py to the manifest (the -dirty flag alone is not provenance); update the freeze manifest or add an explicit "post-freeze code changes" table.

# MINOR

m1 — Screenshot captions. S002 is stamped 03:34:33.673 UTC with the caption "while it runs on M3U96b", but RD0002's executor session ended 03:33:52.873, its reviewer 03:34:30.522, and the run reached evaluated at 03:34:32.321 — the capture is 1.4 s after the run was closed (the Terminal was still tailing the finished file). S007 (RC0009, 4 s before session end), S008 and S004 (7 runs in running at both timestamps, verified from run_transitions) are consistent. S005/S006/S008 carry redaction_description = "cropped: top 300 px removed" while redacted_sha256 == original_sha256 and the files are byte-identical (1312×822 / 2624×1644): the retained "original" is already the cropped region capture, so the manifest cannot evidence an uncropped source. All 9 original_sha256 values match the files on disk. Fix: reword S002 as "immediately after completion"; set S005/S006/S008 redaction to "none (region capture)".

m2 — Killed sessions are charged 0 in sessions. For the 12 interrupt runs the initial row has total_cost_usd = session_cost_usd = 0.0 because the list-price fallback (scheduler.py:221) only triggers on timed_out. The envelope is nonetheless not under-charged: the resumed session's total_cost_usd is cumulative (RR0002: 0.375 reported ≈ 0.319 killed-session usage + 0.076 own; RR0010: 0.196 ≈ 0.127 + 0.045), as it is for gate cycles (RT0009: 0.912 → 0.935 → 0.951 → 0.965). Fix: charge the kill-time estimate to the killed row and subtract it from the resume, or document the convention.

m3 — Amendment timestamps are hand-rounded and one is mislabelled. A-002 is stamped 2026-10-30T04:40:00Z and labelled before_confirmatory_results, but its commit (55b92f9) is 03:57Z and the confirmatory assignment started 04:02Z; A-001/A-004/A-005/A-006 are at :00/:40/:40/:20. Fix: use the commit timestamps.

m4 — Sibling exposure is persistent, not just concurrent, and includes other arms' instructions. On M3U96a: ~/wp29/runs/ holds 199 entries (≈ 99 workspaces + their .ctl dirs, none cleaned), ~/.claude/projects/ holds 98 wp29-runs-* transcript directories, all owned by the same user; each .ctl dir contains that run's system_append.md (arm text) and reviewer_system.md. The audit's wording "sibling outputs of other runs" understates it: a later executor can read any earlier run's deliverables, transcript and arm text (none did, per M2). Fix: say so; delete workspaces/transcripts after collection in future runs.

m5 — "Offline nodes" / "no network access". scientific_contract.md row A says "offline nodes" and both the common preamble and reviewer.md tell executors "There is no network access". The nodes have Internet access (the API calls require it); only the web tools were withheld and CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 set. The grep in M2 found zero network commands, so the claim holds behaviourally, not mechanically. Fix: "no web tools; network not blocked; absence of network commands audited".

m6 — The API key is in the executor's process environment. remote_shell_line sources ~/wp29/.env with set -a (launcher.py:46), so every Bash tool subprocess inherits ANTHROPIC_API_KEY; an executor running env would have written it into the stream. None did (0 env/printenv calls; grep -r sk-ant- runs reports paper release results → only the pattern in release/validate_release.py). .env is mode 600 on the node. Fix: mention the inheritance as a residual; consider env ANTHROPIC_API_KEY=… claude … with unset in the preamble is not possible, so at minimum keep the release grep.

m7 — Nested-session guard is narrower than stated. PGAR_DEPTH (launcher.py:51) only stops the Python controller; the claude binary is on the node and callable from Bash with the inherited key. No executor did so (0 hits). scientific_contract.md invariant 7 ("executors cannot launch controllers") is true; "nothing goes deeper" (launcher docstring) is not enforced. Fix: wording.

m8 — Dead code / cosmetics. scheduler.py:201–202 (no-op query), 280 (… if False else None); --permission-prompts none is passed while init reports permissionMode: default (allowedTools pre-approval does the work — fine, but say so).

m9 — Post-fix verification runs are not comparable on first-turn size. RD0005–RD0008 show first_turn_input_tokens 2273/3752 vs 2238/3717 for the frozen runs (+35 tokens = the full prompt), confirming the fix but also that the frozen study and the verification set differ in delivered context; results.tex Table "Post-fix verification" should carry that caveat (it says "not pooled", which is right).

# 2. Claims verified as supported (so the reader can weigh the above)

• Launcher flags identical across arms. Normalising sessions.command_json.argv over all 501 sessions (replacing cap, session id and the per-run .ctl path) yields exactly one argv per (split, role, kind); arms differ only in the content of system_append.md. Flags present: --bare -p … --output-format stream-json --verbose --model claude-opus-5-5 --tools … --allowedTools … --permission-prompts none --max-budget-usd <cap> --max-turns 80|150 --append-system-prompt-file … --no-chrome --disable-slash-commands --session-id|--resume <sid>; no --no-session-persistence (hence transcripts persist, m4).
• Same treatment text within an arm. protocol_hash is unique per arm across every split (A 1d6fe7…, B bc05a0…, C f8afd6…, D d1a7be…, L c25619…); it equals SHA-256 of each run's stored system_append.md (0 mismatches over 384 runs) and equals what the current protocol/arms/common_preamble.md + arm_X.md regenerate. first_turn_input_tokens is constant within arm across all splits (A 2238, B 3717, C 4111, D 4319, L 3190; reviewer 2224).
• Workspace verification. 384 prepare.json manifests, 0 hash mismatches against the task bundle.json; verified_identical enforced before launch (scheduler.py:86).
• No oracle material in bundles or on nodes. No forbidden names in tasks/public_bundles (F4-conf-04/data/reference_list.json is the bibliography the task asks the executor to verify, not an answer); no oracle/REFERENCE/planted/trap strings; on M3U96a no ~/wp29-oracles, no repository copy, no oracle-like file under ~/wp29; laptop port 22 unreachable from the node (nc -z → unreachable, re-checked); .env mode 600; gate pgar_gate.py mode 444.
• Reviewer sessions. 89 reviewer sessions all used reviewer_system.md (argv) and the one-word prompt "Review"; none referenced .claude/projects or transcript (the 15 "transcript" string hits are "transcription error" in task text).
• Budget enforcement. For all 501 sessions cap_usd == min(max, max − cumulative session cost of the attempt) (0 exceptions); no attempt exceeds 4.00 (benchmark) or 12.00 (transfer); max observed per run 1.244 (confirmatory), 1.604 (transfer). Reviewer and continuation caps equal the remaining envelope (e.g. RC0094 reviewer cap 3.2497 = 4.0 − 0.7503).
• Durable execution. Invariant 1 enforced in state.transition and tested; R3 (recovery_r3_duplicate.json): RR0013 submitted 3×, 2 duplicate_ignored, 1 session launched; RC0094 attempt 1 cost (0.998 USD) is retained in sessions and in reports/experiment_accounting.md ("Sessions cost (all attempts)" 127.42 vs envelope 126.42); run_transitions show the legal error → prepared path.
• Clocks. DB started_utc precedes the first raw-stream event by 1.7–3.4 s (SSH + client start-up), ended_utc trails the last event by ≤ 0.3 s, duration_ms equals DB wall time (12-session random sample).
• Evaluators do not execute executor code on the oracle host (no subprocess in any evaluate.py; the reproducibility re-run happens on the node in a temp copy, runtime_gates.py:185–193).
• Tests. pytest -q tests → 10 passed (incl. the new quoting test).

# 3. Claims in reports/isolation_audit.md and paper/sections/implementation.tex NOT supported by the data

1. isolation_audit "Tools" row and implementation.tex:27 — tool inventory Bash/Read/Write/Edit/Glob/Grep (delivered: Bash/Edit/Read). [M1]
2. isolation_audit "Instruction loading" row — "no … plugins" (two built-in plugins and five agent definitions listed in every init event). [M1]
3. implementation.tex:27 (A-006 sentence) — "continuation feedback … reached the executor only through files … we treat the feedback mechanism as delivered via files". [C1]
4. implementation.tex:31 — "a failing receipt is returned to the executor as a resumed turn, at most three times" (as delivered: the word "The"). [C1]
5. implementation.tex:35 — "If the verdict is revise, the executor session is resumed once with the review" (never executed; both revise verdicts lost). [M3]
6. implementation.tex:23 / isolation_audit "Reviewer" row — audit "reports the counts per run": the counts are wrong by construction (tilde paths) and the paper's derived statement (limitations.tex:16 "zero outside-workspace references in every audited run") contradicts the audit files. [M2]
7. implementation.tex:37 — "Eleven tables record … events" as a faithful record: a re-attempt's events are dropped. [M4]
8. isolation_audit "Nested invocation" row — "executors cannot launch controllers" is fine; the implicit "no nested sessions" is unenforced. [m7]
9. isolation_audit "Residual risks 1" — understates persistence of sibling workspaces/transcripts and the readability of other runs' .ctl instruction files. [m4]
10. (contract) "offline nodes"; (preamble/reviewer text) "There is no network access". [m5] Also outside the two documents but load-bearing: transfer.tex:18 and recovery.tex:8 (C1); amendments A-002 timestamp (m3).

# 4. Overall verdict

The isolation claims that matter most for validity — that every arm received byte-identical launcher flags and a single, hashed treatment text; that executors never had the oracles, the repository, SSH to the laptop, web tools or any observed network/nested-session behaviour; that workspaces were verified before launch; that budgets were enforced as the remaining envelope and never exceeded; and that costs, duplicates and re-attempts are honestly accounted — are supported by the code, the database and the raw streams, and I could not find a channel by which reference answers reached an executor. What is not supported is the paper's account of the delivered execution mechanism: the tool inventory was Bash/Edit/Read rather than six tools, --bare still listed built-in plugins and agents, the behavioural audit numbers are wrong in both directions, and — most seriously — because of the unquoted prompt expansion (A-006) no continuation session in the frozen study ever received its gate feedback, review feedback or interruption notice; the paper's new wording that feedback "reached the executor through files" is contradicted by the transcripts, and the transfer and recovery sections narrate executor behaviour that did not occur. The frozen A–D comparison therefore remains a clean test of protocol text delivered via the system prompt, but any sentence that attributes an outcome to controller-enforced feedback, review-repair or the resumption instruction must be removed or supported by a rerun with the corrected launcher.

# Adversarial review — statistics and evaluator validity (WP29, confirmatory stage)

Reviewer: a fresh Claude Code sub-session (Claude, same client family as the builder; model string claude-fable-5-1). Same-family dependence applies: this reviewer may share systematic blind spots with the builder and with the executors. Nothing in the repository was modified; no language model was called; all checks are the reviewer's own pandas/numpy code, direct queries of runs/state.sqlite, re-execution of the hidden oracles on the stored deliverables and on their seeded artefacts, and reading of the oracle files in ~/wp29-oracles/.

Note on concurrency: the analysis pipeline was rebuilt while this review ran (results/analysis_data/runs.csv went from 379 to 384 rows; build_summary.json records SHA bac93f… while the file on disk is edd666…; paper/sections/results.tex and protocol/amendments.csv gained A-006 mid-review). The 288 confirmatory rows were re-verified unchanged after the rebuild; every number below refers to them.

# 1. Recomputed headline numbers (reviewer's own code, from runs.csv)

The arithmetic of the primary analysis is correct and reproducible. The findings below concern what the numbers measure.

# 2. Findings, severity-ranked

# CRITICAL

C1. Every one of the 25 confirmatory failures is an evaluator artefact, not an executor error; the paper attributes two of them to executors and the A-005 sensitivity corrects only two of the four defects.

Evidence (all 288 receipts inspected; failing cells only):

Consequences: (i) under a rubric that also waives the rounding and the regex artefacts, all four arms are at 1.000 (72/72 each); the A-005 sensitivity (A 0.986, B 1.000, C 1.000, D 0.986, Δ_DA = 0 [−0.042, +0.042]) still carries two artefacts, one per arm, which is why A and D sit below B and C. (ii) The only arm-specific violation in the study (D's 4 vs 3) is the regex artefact; the "Violation" rows of tab_contrasts_confirmatory (D−A = +0.014, D−C = +0.014) measure a regex. (iii) The paragraph "The remaining failures are isolated: …" must be rewritten; the claim that the pilot "did not exercise these rules" now covers four distinct rule failures (feasibility semantics, generator sanity, rounding tolerance, negation regex).

C1a — is partial the right label for F1-conf-01? tasks/TASK_AUTHORING_SPEC.md §4 defines feasibility only as the enumeration complete / partial / infeasible with feasibility_reason "why, if not complete" — no semantics. TASK.md says "Months for which g_t cannot be computed are reported as missing and counted" and the oracle's own reference has n_months_missing_growth = 1, i.e. the designer knew one estimand was undefined and still expected complete; the oracle's own validation/correct_out/analysis.py line 41 writes "feasibility": "complete". Twelve independent executor sessions, under all four arms, read the same text and all chose partial (one of 309 monthly estimands undefined; the summary estimands computed on 308). Both readings are defensible because the spec never says whether partial refers to the deliverables or to the estimands. Verdict: the frozen criterion is too strict under an undefined label, and the correct fix is in the authoring spec, not in A-005's run-content-dependent waiver (n_months_missing_growth == 1 or "2025-10" in reason), which is a post hoc rule written after reading the deliverables.

# MAJOR

M1. The review-and-repair mechanism of arm D never ran in the confirmatory stage, and results.tex says the opposite. sessions for the 288 runs contain only initial (289) and review (73) kinds; there is no review_repair and no gate_cycle_* session. The only revise verdict (RC0094, attempt 1, 04:33 UTC, one major issue) aborted on the ledger defect before the repair session was launched; attempt 2's reviewer returned accept with 3 minor issues (reviews id 80). Sentences that are therefore unsupported: "the one revision cycle that completed (RC0094, second attempt) ended in acceptance and a scientifically successful run"; "the single review-repair cycle of the benchmark … resolved [via files]" (A-006 paragraph); "asked for a revision \nReviewsRevise{} time(s)" counts the aborted attempt. In addition, RC0159 (F1-conf-03, arm D): the reviewer found a major inconsistency (report.md "other coefficients" do not match results.json: cash_ratio −0.002 vs −0.186, log_sales sign flipped) but never wrote out/review/review.json; scheduler.py line 151–153 treats a missing review as "not revise", the run proceeded to evaluation as a success, and reviews.verdict is NULL with n_issues = 0. So: 73 reviews = 71 accept + 1 revise (aborted attempt) + 1 unparsable (major issue silently dropped). H3 (claim validity higher in D at a cost) is not evaluable from these data, and the paper should say that the review step produced commentary only.

M2. Preregistration adherence — deviations and gaps (none changes the estimate, all must be disclosed).

1. Y definition: common.py implements scientific_success ∧ status = evaluated ∧ terminal_reason ∈ {executor_finished*, result_*, max_turns_reached}; the "no human intervention" conjunct is not implemented (0 interventions were logged, so no numerical effect, but the endpoint code does not read the ledger). max_turns_reached is counted as within the envelope although A-001 defines the envelope as 4 USD / 1,500 s / 80 turns; no run hit it (0 effect), but the rule contradicts §7 "runs that … exhaust the budget … count as Y = 0" if turns are part of the budget.
2. RC0094 rerun: §7 and A-004 promise an "attempt-numbered policy" with "the original attempt reported". In the store the rerun was executed in place under the same run_id: run_transitions shows error → prepared → running → completed → evaluated, the manifest's started_at_utc is the rerun's, and spent_usd = 0.897 excludes the aborted attempt's 0.999 USD. The aborted attempt survives only as two sessions rows and in experiment_accounting.md ("Sessions cost (all attempts)" 127.42 vs 126.42). Mean cost of arm D in tab_arms/tab_cost is therefore the final-attempt mean (0.640) not the all-attempts mean (0.654); state which.
3. A-005 provenance: the amendment cites "preregistration §14.4 'uncertain rubric labels'"; the preregistration has §1–§10 and no such clause — the citation is to the mandate §14.4. The sensitivity analysis is legitimate under the mandate but is not prespecified in the frozen document; results.tex calls it "the prespecified rubric-label sensitivity analysis" — change to "the mandate-required, post-results sensitivity analysis (A-005, after_results)".
4. Sidedness: H1 is directional (p_D − p_A > 0); permutation_p is two-sided (|mean|); the preregistration is silent. One-sided p = 0.50 (exact). Report both or fix the prereg text.
5. Bootstrap/permutation parameters: B = 5,000 percentile, 20,000 sign-flips, task clusters, Holm over {B−A, C−B, D−C} — all as preregistered. Note that with only 3 non-zero task differences the Monte-Carlo permutation is a 20,000-draw approximation of an 8-point distribution; the exact answer (1.00) coincides.
6. Missingness: no missing evaluator output, no timeout, no budget exhaustion — the rules were not exercised.

M3. A-005 loosened a frozen tolerance without need or justification. rubric_sensitivity.py accepts the 1,188-row naive CI within 0.006 + 0.004 instead of the frozen 0.006. The executors' 1,188-row CI bounds differ from the independently recomputed reference (sensitivity_rubric_F6-conf-04_reference_1188.json) by 5.3e−7, so the frozen 0.006 suffices; revert to the frozen tolerances so that the sensitivity analysis changes only the rule under test.

M4. analysis/robustness.py row "run-level unweighted mean difference" is broken (n_tasks = 0, NaN estimate in confirmatory_robustness.csv): assigning task_id = run_id leaves no task common to both arms in paired_task_diff. The intended anti-pattern comparison (treating runs as independent) must be computed as a difference of run means with a run-level bootstrap, or dropped.

M5. The claims endpoint (claims_analysis.py) is implemented as documented but is entirely driven by the same artefacts. All 12 runs with "unsupported supported claims" are the 11 F6-conf-04 1,188-row runs plus RC0063 (rounding). The arm gradient in confirmatory_claims_by_arm.csv (unsupported per run A 0.139 → D 0.056) is the by-arm distribution of those 12 runs (A 3, B 3, C 3, D 2) and must not be read as claim-validity evidence; not_assessable_per_run (A 0.96 → D 1.67) shows that B/C/D write more claims about keys no oracle checks. The evaluation plan's definition ("claims with status supported whose results key fails its criterion") is followed; keys covered = keys appearing in any criterion's observed dict.

M6. Generated numbers and tables are internally inconsistent across builds. paper/tables/numbers.tex had nTransferEvaluated 11, nAblationEvaluated 16, costAllTotal 157.63 while runs.csv had 16/24 and the accounting report 165.58; tab_accounting.tex reports confirmatory "Infra. error 0" while numbers.tex has nMainInfraErrors 1 and the accounting report "Infra attempts 1"; tab_accounting development row (8 assigned / 5 completed) disagrees with runs.csv (4, then 8); tab_postfix.tex contains nan rows; build_summary.json flags duplicate (task, arm, repeat) cells in development (4), recovery (2) and transfer (1). Nothing confirmatory is affected, but the paper must be built from one atomic rebuild after all splits are final, and the duplicates explained (post-fix verification runs share cells with earlier runs?).

# MINOR

• m1. results.tex: "or a replication script that did not regenerate the results byte for byte" — the 16 A/B contract failures are claims_json (A 5, B 5), section:Results (A 2), exclusion_ledger (B 4); zero reproducibility failures in any arm (reproducible_rate = 1.0). Delete the clause.
• m2. "excludes differences larger than about six percentage points in either direction" — the interval is [−2.8, +5.6] pp; say "larger than 5.6 pp in favour of D or 2.8 pp in favour of A".
• m3. Tables print "−0.000" for D−C and D−B (−2.3e−18); round to 0.
• m4. interventions table in state.sqlite is empty while reports/interventions.csv has I-001; the exports do not read it.
• m5. Reviewer-issue severities over the 72 parsed reviews: 197 minor, 1 major (the aborted RC0094 attempt); plus RC0159's unparsed major. "almost all minor" is correct; add the RC0159 case and the 2.7 mean includes a 0 for the unparsed review.
• m6. The shared F6 evaluator's NEGATION regex is fragile in both directions: any negation word anywhere in a claim ("not only … but also an effect") suppresses the flag; conclusion codes with underscores trigger it. Replace with a check on the conclusion key / CI sign rather than prose.
• m7. F4-conf-01 has 31 criteria of which 22 loc_* locator checks never fail on the seeded set or on any executor; q_correctness for F4 is therefore dominated by locator bookkeeping (a wrong value fails 1 of 31). Report correctness per task family with this caveat or weight criteria.
• m8. F3 oracles use rel 1e−6 on RMSE/MAE "because values have one decimal"; this is fragile to any rounding (RC0063) — the tolerance should be derived from the output's required precision, which TASK.md does not state.
• m9. common.py sets Y = 0 for terminal_reason containing "budget" or equal to wall_clock_limit after the status == evaluated filter; fine, but n_timeout/n_budget_exhausted in the arm table count terminal reasons only, not gate_failed_budget_exhausted continuation outcomes (none occurred).
• m10. effective_context_hash has 24 distinct values per arm (one per task) and protocol_hash one per arm — consistent with "task text read from the workspace"; treatment integrity verified.

# 3. Per-task evaluator audit (24 confirmatory tasks, 12 runs each)

Method: for each task, all 12 receipts (criteria observed/expected/tolerance, violations) were tabulated; each oracle was re-run by the reviewer on its validation/correct_out and every wrong_out (official validate.py: 24/24 PASS re-confirmed; reviewer's own harness: every wrong artefact rejected, criterion that catches it recorded); tolerance margins max |obs − ref| / tol over the 12 runs were computed for every numeric key. "Non-discriminating" lists criteria that fail on no seeded wrong artefact and on no executor run.

Near-ceiling: permissive oracles or easy tasks? Six spot-checks (F1-conf-02, F2-conf-01, F3-conf-03, F4-conf-01, F5-conf-02, F6-conf-02): in each, at least one wrong artefact fails on a scientific criterion (not only provenance), and the executor's numbers coincide with REFERENCE.md to the printed precision (e.g. F1-conf-02 balanced change 3.4928 = 3.4928; F3-conf-03 ridge RMSE 0.56079893 = 0.56079893; F4-conf-01 Table 6/7/2 values exact; F5-conf-02 analytical bias −0.11333 exact, MC bias within 0.9 MC-SE). Across all 24 tasks the median of max |obs − ref| / tol is 0.000 and exceeds 0.5 only where tolerances are MC-noise-sized (F2-conf-03) or rounding-sized (F3). The ceiling is therefore not a product of permissive tolerances: executors reproduced the references essentially exactly. Whether the tasks are "easy" is a separate question the design cannot answer (no arm or model failed on substance).

# 4. Sentences in paper/sections/results.tex that the numbers do not support

1. "one arm-A run of F3-conf-02 reported naive forecasts and a model-error summary inconsistent with its own forecast ledger (four criteria failed, no violation)" → the run rounded its summary statistics to four decimals; the ledger is consistent; the failure is a tolerance artefact (C1).
2. "one arm-D run of F6-conf-01 marked an effect claim as supported on the genuine-null weekday task (an invalid-inference violation)" → the claim states no_detectable_effect; the violation is a regex artefact (C1).
3. "The remaining failures are isolated" / "\nMainViolRuns{} runs with a violation in total, twelve of them that label" → all 13 violations and all 25 failures are evaluator artefacts; say so and give the fully corrected rates (1.000 in every arm).
4. "the prespecified rubric-label sensitivity analysis (amendment A-005)" → not in the preregistration; after-results, mandate-required (M2.3). Also extend it to the rounding and regex artefacts or state why not.
5. "the one revision cycle that completed (RC0094, second attempt) ended in acceptance" and "the single review-repair cycle of the benchmark … resolved" → no review-repair session exists; attempt 2 was accepted at first review (M1).
6. "\nReviewsRevise{} of \nReviews{} reviews returned revise" → 1 of 73, but that review belongs to the aborted attempt; one further review (RC0159) carried a major issue and was lost because no review.json was written; the harness treated it as acceptance.
7. "Reviewers … asked for a revision \nReviewsRevise{} time(s)" → zero effective revision requests reached an executor.
8. "or a replication script that did not regenerate the results byte for byte" → no reproducibility failure occurred (m1).
9. "excludes differences larger than about six percentage points in either direction" → [−2.8, +5.6] pp (m2).
10. "with the first attempt retained in the ledger and its cost charged to the accounting" → retained as session rows only; the run record and tab_cost use the final attempt (M2.2); state both figures.
11. Any use of the "Violation" contrast rows (D−A, D−C = +0.014) and of confirmatory_claims_by_arm as mechanism evidence (H2, H3) → artefact-driven (C1, M5).
12. "Both are evaluator defects of the kind the pilot was meant to catch and did not" → four defects, two of them generic (rounding tolerance, negation regex) that a pilot with rounded outputs or underscore conclusion codes would have caught.

# 5. Overall verdict

The statistical pipeline does what the preregistration says: task-weighted rates, paired task-level differences, task-cluster percentile bootstrap (B = 5,000), sign-flip permutation (20,000), Holm over the three mechanism contrasts, balanced 24 × 4 × 3 design with no exclusion and the RC0094 rerun counted once; every headline cell was reproduced independently to the last digit, the negative controls hold, and the seeded-artefact validation of all 24 oracles passes under re-execution. The null result (Δ̂_DA = +0.014 [−0.028, +0.056], p = 1.00) is correct and robust. What is not supportable is the paper's account of where the 25 failures come from: all of them, including the two the draft attributes to executors and the single arm-specific violation, are rubric artefacts (an undefined feasibility label, a generator producing impossible values, a rel 1e−6 tolerance applied to rounded output, and a negation regex defeated by an underscore). Under a corrected rubric every arm is at 72/72, so the honest statement is a complete ceiling with zero discriminating power, and the mechanism evidence for H2/H3 (violation and claim contrasts) is noise generated by the evaluators. Separately, the arm-D review-and-repair loop never executed in the confirmatory stage (no review_repair session; the only revise was in the aborted attempt; one major reviewer finding was lost to a missing file), so claims about what the reviewer "resolved" must be withdrawn and H3 declared not evaluable. Required before any manuscript build: rewrite the failure paragraph and the review paragraph as above, extend or re-scope A-005, fix the robustness script, regenerate all tables and numbers.tex from a single atomic build, and disclose the prereg deviations in M2. With those changes the paper's central claim — no detectable protocol effect at this task difficulty with this model, at 1.7–2.6× the cost — stands.

# Adversarial manuscript review — WP29 "Protocol-Guided Autonomous Research with Claude Code"

Reviewer identity. This review was written by a fresh Claude Code sub-session (model string claude-fable-5-1, client 2.1.314), i.e. the same client family and model identity as the builder session that wrote the manuscript. Same-family dependence applies: shared blind spots are possible and this is not an independent human review. No file other than this report was created or modified; no language model was called by the reviewer; every check below was done by reading files, pdftotext, Python over the CSV exports, and read-only SQL on a copy of runs/state.sqlite.

Object reviewed. The compiled paper/main.pdf (103 pages, built 2026-10-30 01:49 local = 05:49 UTC) and the committed LaTeX sources at commit 452964e. The builder was active during the review: paper/sections/{results,introduction,limitations,conclusion}.tex, paper/tables/numbers.tex, protocol/amendments.csv (A-007), reports/experiment_accounting.md and the analysis dataset changed on disk mid-review, and an uncommitted paper/sections/disclosure.tex appeared. Where an in-flight edit already addresses a finding I say so, but the finding stands against the compiled manuscript and the finding is only closed when the PDF is rebuilt from consistent inputs. Two sibling reviews exist (reports/review_systems_isolation.md, reports/review_statistics_evaluators.md); I do not repeat their findings except where the manuscript text itself is wrong.

# 1. Findings ranked by severity

# CRITICAL

C1 — The post-fix verification set is claimed as evidence before it existed, and is now being pooled into the frozen split it was declared "not pooled" with.

• sections/results.tex §8.4: "A post-fix verification set (the smoke task under all four arms and the real-time forecasting project under arm C, run with the corrected launcher after the analysis) is reported in Table~\ref{tab:postfix}; it is not pooled with the frozen results." sections/limitations.tex §12.1 "Prompt delivery": "We did not rerun the study after the fix because the defect is symmetric and the post-fix verification runs behaved as the originals did."
• Evidence: in the compiled PDF, Table 8 (results/tables/tab_postfix.tex at build time) reads RD0006 … nan & -- & -- & -- & nan, RD0007 … nan, RD0008 … nan, RT0025 … nan. Only RD0005 (arm A, the one arm with no feedback loop) had finished. results/analysis_data/build_summary.json at build: development/running: 3, transfer/not_started: 1. At review time RT0025 — the only run that actually tests the delivered gate-feedback text — is still status = running in runs.csv.
• Second problem: the in-flight numbers.tex now has \nTransferEvaluated{25}, \nTransferSuccess{24}, \transSuccC{6/7}, \nTransGateCtrlFail{15}, \costTransferTotal{22.58}; reports/experiment_accounting.md now shows transfer 25/25/25. RT0025 is being counted in the transfer split's totals while the caption says it is not pooled. Likewise RD0005–RD0008 (post-fix) are inside the "development" split (now 8 runs, 2.01 USD), inflating "all splits together" cost.
• Fix: (i) register post-fix runs in their own split (postfix) or exclude run_id in (RD0005–RD0008, RT0025) from every split-level table and macro; (ii) do not rebuild the paper until RT0025 is evaluated; (iii) replace the limitations sentence with what the receipts show, e.g. "The five post-fix runs (Table 8) completed with the same verdicts as their frozen counterparts [state numbers]; RT0025 is the only observation of the gate-feedback text being delivered, and it is one run." Never print nan in a table: use "not yet run".

C2 — The manuscript states that a review–repair cycle completed in the benchmark; none ever ran.

• sections/results.tex §8.4: "the one revision cycle that completed (RC0094, second attempt) ended in acceptance and a scientifically successful run" and "which is how the single review-repair cycle of the benchmark … were resolved".
• Evidence: sessions table, RC0094: attempt 1 = executor (0.750 USD) + reviewer (revise, 2 issues) → infrastructure error; attempt 2 = executor (0.685) + reviewer (accept, 3 issues). There is no kind = review_repair session anywhere in the 288 runs (n_sessions = 2 for all 72 arm-D runs; in-flight \nRepairSessions{0}). The only revise verdict was lost to the A-004 crash. The arm-D treatment as delivered was "review, then accept"; the repair branch of the mechanism was never exercised in the confirmatory stage.
• Where it must be said: results §8.1/§8.4 (in-flight edit now does), but also the abstract ("plus a separate review-and-repair session"), introduction §1 ("(D) … separate review-and-repair session"), design §6.2 ("Arm D adds a separate review-and-repair session") and conclusion ("a separate review stage") must carry one clause: "the repair branch never triggered in the 288 runs (one revise verdict, lost to a harness fault; one review without a verdict)".

C3 — For the forecasting mini-project the arm-D review stage never ran, and the transfer section does not say so.

• sections/transfer.tex §10.1–10.2 reports T1 under "every arm twice" and the cost ladder "from 0.51 USD per run under A to 1.24 under D". sections/implementation.tex §5.5: "In arm D, once the controller gate passes, a new session is started … with a reviewer system prompt."
• Evidence: sessions for RT0010 and RT0015 (T1, arm D): initial, gate_cycle_1, gate_cycle_2, gate_cycle_3 — four executor sessions, no reviewer session; reviews.csv has transfer reviews only for RT0007/RT0008 (T3) and RT0020/RT0021 (T2). Because the controller gate failed three times on the mis-specified chronology check, the reviewer was never launched: for T1 the D treatment collapsed to C plus three futile cycles.
• Fix: add to §10.2: "A consequence of the false blocking is that the arm-D review stage, which the scheduler launches only after the gate passes, never ran for T1; the two T1 arm-D runs are therefore arm-C runs in all but label." Add a "Sessions (exec/review)" column to Table 9 (tab_transfer).

# MAJOR

M1 — Amendment A-006 (one-word user prompt) is disclosed in the main text but Appendix B and the abstract contradict it.

• Present and adequate: design §6.2 (parenthesis), implementation §5.3, results §8.4, limitations §12.1 "Prompt delivery", ledger appendix.
• Missing/contradicting: appendices/appendix.tex §B header: "the user prompt is identical for all arms" and then prints task_prompt.md (19 words), gate_feedback_prompt.md (44), review_feedback_prompt.md (49), resume_after_interruption_prompt.md (37) as if delivered. The abstract says the gate "returns failures to the session" and describes a "review-and-repair session" with no qualification; the conclusion is silent.
• Fix: Appendix B header → "The user prompts below are the intended texts. Because of the launcher defect of amendment A-006, only the first word of each (Begin, The, Your, Review) reached the client in every session of the frozen study; the texts are printed for the record and were delivered in full only in the post-fix verification runs." Abstract: add "(gate and review feedback reached the executor only through files written in its workspace, because of a launcher defect disclosed in §5.3)". Conclusion: one clause.

M2 — Recovery §9.1 misstates the matched no-fault outcomes.

• Quote: "the matched no-fault pilot runs of the same tasks and arms had all succeeded."
• Evidence: results/estimates/recovery_r1_r4.csv, RR0002 and RR0003 (F3-dev-01, arm A): matched_nofault_success = 0.0; reports/pilot_report.md: F3-dev-01 Y_A = 0 (the pilot's only failure).
• Fix: "the matched no-fault pilot runs had succeeded in five of the six (task, arm) cells; the arm-A pilot run of F3-dev-01 had failed, so both interrupted arm-A runs of that task did better than their uninterrupted counterpart."

M3 — Transfer §10.2 overstates the cost of the false blocking by an order of magnitude.

• Quote: "the gate consumed a third of the envelope and bought nothing".
• Evidence (sessions.session_cost_usd): RT0009 gate cycles cost 0.023, 0.016, 0.015 USD (1 turn, 2–5 s each); RT0010: 0.019, 0.017, 0.034. The three cycles total ≈ 0.05–0.07 USD per run, i.e. ≈ 5–6 % of the run's ≈ 1.0 USD and ≈ 0.5 % of the 12 USD envelope. The 1.10-vs-0.70 USD gap between C/D and A/B is almost entirely the initial session (C/D 0.91–1.26 USD; A/B 0.52–0.92 USD), i.e. the protocol/gate text, not the blocking.
• Fix: "the three futile cycles cost about 0.05 USD and a few seconds per run; false blocking was cheap here only because the executors declined to alter correct output, but it quadrupled the session count and, in arm D, displaced the review stage entirely (§10.2, above)."

M4 — Table 2 (tab_accounting) contradicts the prose and has unexplained denominators.

• The "Infra. error" column is 0 for every split; the text two lines above reports one infrastructure error in confirmatory (RC0094) and the phase log four in the first smoke attempt; reports/experiment_accounting.md has "Infra attempts" 2 (development) and 1 (confirmatory). The development row shows Assigned 8 / Completed 5 and transfer Assigned 25 / Launched 24 with no caption note (the extras are the post-fix runs of C1).
• Fix: derive the column from run_transitions (attempts with error), rename to "Infra. attempts", put post-fix runs in their own row, and explain in the caption.

M5 — Results §8.4 names a contract-failure kind that did not occur.

• Quote: "failed for 7 arm-A and 9 arm-B runs (missing report sections, claims referencing absent keys, or a replication script that did not regenerate the results byte for byte)".
• Evidence: the 16 final_common_contract_check receipts show 10 claims-ledger failures, 4 exclusion-ledger sums and 2 missing sections; 0 reproducibility failures (in-flight \nReproFailAB{0}), consistent with Table 3 "Repro. = 1.000" for all arms. (In-flight text corrects this; keep it.)

M6 — "of which the reviewer session is about a third" (results §8.5). Table 6: reviewer 11.67 of 46.09 USD = 25 %. Write "about a quarter".

M7 — The reflexive section miscounts its own delegations and the effort table omits one.

• sections/reflexive.tex §11.1: "The builder delegated eight pieces of work to fresh general-purpose sub-sessions: the corpus coding, the literature review …, one task family each for F1–F6 …, the three transfer mini-projects, and the drafting of the related-work and corpus sections" = ten items. reports/builder_effort.json and tab_effort list nine (no transfer sub-session). disclosure.tex adds three adversarial-review sub-sessions. Fix the count (ten, plus three reviews) and add the transfer row with its tokens/duration or "not recorded".

M8 — The effort table is referenced but never included, and the "append-only phase log" the reflexive account is drawn from stops at Phase 08.

• reflexive.tex §11.1: "both are given in Table~\ref{tab:effort}" → PDF p. 38 prints "Table ??" (tab_effort.tex is never \input; no \label{tab:effort} exists). The table itself says builder phase 07–12 wall time "see final phase log" and tokens "not measured"; reports/phase_status.md has no entry after Phase 08 although amendments A-005/A-006/A-007 cite phases 09 and 11. Mandate figure 9 / table 13 (human-effort decomposition) is therefore absent from the PDF.
• Fix: \input{../results/tables/tab_effort} with a label; complete the phase log through Phase 12 before the final build; fill builder wall time from phase-log/git timestamps and label the basis.

M9 — Three broken or wrong cross-references. Section~\ref{sec:related} (intro, p. 3) → "Section ??" (label is sec:literature); Section~\ref{sec:pilot} (design §6.1, p. 22) → "Section ??" (no pilot section exists; point to reports/pilot_report.md/§11 or add a short §6.5 "Pilot"); design §6.2 "Table~\ref{tab:arms} summarises the four arms" points to the endpoints table in §8 — mandate Table 5 (treatment differences, common information, tools, budgets, context sizes) does not exist and should be added, e.g. a 4-column table with appended words (6/665/827/921), first-turn tokens (2,238/3,717/4,111/4,319), controller action, max cycles.

M10 — Promised appendix content that is not there. Limitations §12.1: "the per-criterion failure table (Appendix~\ref{app:stats}) is provided so that readers can judge" — Appendix D contains only the robustness and order/node tables. Also tab_robustness prints "run-level unweighted mean difference & 0 & -- & [--, --] & --": an attempted-but-empty analysis presented as a result row; label it "not computed (reason)" or drop it.

M11 — Oracle-validation timing and count are overstated. Evaluation §7.2: "Before the pilot, all \nOracles{} (= 31) oracles accepted their correct artefacts and rejected their wrong ones". Git history of reports/evaluator_validation.md: 03:47 UTC (pilot batch 1 starting) 28 PASS, F3-conf-03/04 without validation sets; 03:58 UTC 30 PASS (after pilot batch 1, before the 04:02:39 freeze); 04:34 UTC 33 PASS (adds T1–T3, after the freeze). The smoke oracle (F2-smoke-01) has no validation set, so "31" counts an oracle that was not validated this way; the phase log says 30. Appendix C header "executed before the freeze" is wrong for T1–T3. Fix: "Before the confirmatory freeze, all 30 benchmark oracles with validation sets (the smoke oracle is exercised by the infrastructure tests) accepted …; the three transfer oracles were validated before the transfer runs (04:34 UTC)."

M12 — "prespecified rubric-label sensitivity analysis" (results §8.3) was not prespecified. protocol/preregistration.md contains no rubric-label sensitivity (no occurrence of "rubric"); A-005 cites "preregistration §14.4 'uncertain rubric labels'", but that text is §14.4 of the mandate; A-007 itself now says "not a prespecified contrast". The in-flight results text uses the correct wording ("post hoc sensitivity analysis of the kind the analysis plan calls for"); fix A-005's document reference as well. The in-flight introduction sentence "under a corrected rubric every arm completes every task" must carry "(post hoc)".

M13 — Screenshot redaction descriptions are false for S005, S006 and S008. Manifest and gallery captions: "cropped: top 300 px (browser toolbar and bookmarks bar) removed; page content unmodified". Evidence: public and restricted files have identical SHA-256 and identical pixel dimensions (S005/S006 1312×822; S008 2624×1644), and S008 visibly contains the Safari toolbar and bookmarks bar. Either apply the crop (and re-hash the public copy) or change the description to "copied without modification". Under mandate §15.3 a mislabelled capture is worse than an uncropped one.

M14 — The visual case study is described as "one complete chain"; it is four unrelated panels. transfer.tex §10.3: "assembles the captured surfaces of one complete chain …: the protocol and effective configuration of a run, a native terminal window …, a labelled stress-test rejection …, the evidence dashboard … and the run page of a resumed session." Figure 7 shows RC0009 (live terminal), a planted-defect copy of RP0006, the dashboard, and RP0014's run page. No panel shows a protocol/effective configuration; RP0014 was not resumed (executor + reviewer, accept). The mandate's figure-10 chain (protocol → live run → detected failure → correction → verified artefact) was not captured (S010 failed). Say so: "four authentic panels from different runs; the mandated single-run chain could not be captured (S010)". Caption numbers (38 evaluated / 7 running; S007 RC0009 arm C; S009 stress test) are correct against the image and the manifest.

M15 — The new AI-use statement (disclosure.tex, uncommitted, not in the compiled PDF) contains two unsupported claims. (a) "the final review of the manuscript, which is logged in reports/interventions.csv with its date and basis" — the ledger has one row (I-001, mandate preparation); tab_effort says the final review is "not yet performed at build time". (b) "The experimental subjects were \nAllRuns{} (= 384) fresh Claude Code sessions" — 384 is runs (506 sessions at review time), and includes runs of every split. (c) "every DOI was checked to resolve" — the matrix records doi.org HEAD None for schmidgall2025agentlab and beel2025sakana. A corrected draft is in §5 below.

M16 — Runs are called sessions in the abstract and introduction. "We ran \nMainRuns{} (288) live Claude Code sessions" — 288 runs; the confirmatory stage used 362 sessions (288 executor, 73 reviewer, plus the aborted RC0094 attempt). The conclusion says "runs" correctly. Replace "sessions" by "runs (362 sessions)".

M17 — Transfer run count. transfer.tex §10.1: "Each project was run under every arm twice (\nTransferRuns{} runs)" prints 25; 3 × 4 × 2 = 24 (RT0025 is the post-fix run, C1).

M18 — One arm-D review has no verdict (RC0159). reviews: verdict NULL, n_issues 0; the harness treated it as acceptance. The compiled text says "1 of 73 reviews returned revise" and nothing else; it must say "71 accept, 1 revise, 1 no verdict (treated as a missing measurement)". The in-flight text does this.

# MINOR

• m1 Abstract: "twelve independent sessions reported an undefined month as undefined and excluded impossible values" conflates F1-conf-01 (12/12) with F6-conf-04 (11/12, different sessions). Introduction §1 likewise ("twelve independent sessions behaved like careful scientists"). Write "twelve sessions … and eleven of twelve sessions …".
• m2 Results §8.2: "excludes differences larger than about six percentage points in either direction" — the interval [−0.028, +0.056] is asymmetric (in-flight fix present).
• m3 Introduction contribution (iv): "A fully instrumented account" — builder tokens are "not measured" and the phase log is incomplete (M8). Write "an instrumented account".
• m4 Recovery §9.2 (R4): the evaluator flagged fabrication, false_completion on both runs (recovery_r1_r4.csv); the text mentions only false completion. State both and why fabrication fires on null counts.
• m5 Recovery §9.2 (R2): the explanation "because the mutated row was one the analysis excludes as an aggregate entity" is asserted without evidence; RP0013 (C) and RP0015 (A) of the same task and file failed the check while RP0014 (D) and RP0016 (B) passed, with the same mutation rule (last number of the first numeric row), so the explanation must be run-specific (those two scripts drop the entity, the others do not). Say that, or cite the script lines. The faults table also shows R2 was executed twice (05:32:13 with a rule later corrected, 05:32:38 final); the paper should mention the first, discarded pass as the phase log does.
• m6 Framework §3.2 enumerates five invariants; recovery §9.2 cites "invariant 2" and "invariant 6". Renumber or add the sixth (idempotent job ledger) to §3.2.
• m7 Implementation §5.2 cites "(Section~\ref{sec:design})" for the sentinel-probe token counts; the design section has no probes. Point to Appendix E / reports/environment_audit.md (which confirms 49,077 and 144 tokens; 175–177 I could not locate in the audit — give the source).
• m8 Implementation §5: "about 1,300 lines in src/pgar/" — wc -l gives 1,715 at review time. Update or drop the number.
• m9 Appendix F ledger: amendment descriptions are truncated mid-word ("any ch", "with the client se", "evaluation_receipt_pre-A00", "11 of 12 runs detected and"). The ledger is the evidentiary record; print it in full (wider p{} column or \footnotesize).
• m10 Arm L (§9.3): the +0.056 L−A difference arises because the L run on F1-conf-01 (RA0009) passed the frozen oracle, i.e. it declared the task complete — the very label the paper's sensitivity analysis treats as the less careful answer — and because L happened to pass F3-conf-02. Report L under the corrected rubric too, otherwise the ablation rewards matching a defective oracle.
• m11 Gallery captions: "public SHA-256 --\ldots" (S001, S004) and "original SHA-256 \ldots" (S010) are formatting artefacts; print "none (capture failed)".
• m12 Table 8 prints the string nan (see C1).
• m13 Unused generated figures: results/figures/fig_effort.pdf, fig_family_confirmatory.pdf and the three *_development.pdf are not in the paper. Include fig_effort (mandate figure 9) or remove them so the release manifest does not list orphan figures.
• m14 Citations/labels: yao2024taubench is an arXiv preprint (matrix peer_reviewed = unknown) but is not in the §2.2 preprint list ("The checklist, HAL and the review simulation are preprints"). Otherwise no preprint is described as peer-reviewed (checked for every key cited in §2; see §3 below).
• m15 Undefined acronyms at first use: PGAR (first appears as the §5 title; never expanded in the body), MCP (§5.2), NL (Table 1), ALFRED/FRED/WDI (§6.1), CC BY, CR0/CR1 (§7.1), GEM², TOP. Define once.
• m16 Repetition: "no human intervened in any executor session" appears in §3.3, §7.4, §8.1, §12.1 and the disclosure; "autonomous/autonomy" 10× in §3 and 9× in §2 — acceptable, but §3.3 and §12.1 "Autonomy" say the same thing twice. Process commentary to cut: reflexive §11.2 "it is easy to see how it would arise under time pressure" (speculation); limitations §12.3 last clause "an executor's declaration of success is the least informative signal in the pipeline" (rhetoric; the point is already made).
• m17 French: none in the body text. The quoted instruction 'fait ce projet de recherche' in the ledger is a verbatim quotation and is fine; §11.1 says "four-word instruction in French" — consistent.
• m18 appendices/evaluator_validation.tex lists "F4-sources — no validate.py" as an oracle; it is a shared source bundle, not a task; drop the row or footnote it.

# 2. Number-by-number traceability (main text, abstract, captions)

Legend: ✓ traced and matches; ✗ mismatch or untraceable. Sources: paper/tables/numbers.tex (N), results/estimates/*.csv (E), results/analysis_data/*.csv (A), runs/state.sqlite copy (S), reports/* (R), protocol files (P).

Untraceable or stale macros: \nAllRuns{384} (counts running/post-fix runs), \nTransferRuns{25}, \nOracles{31}, \costAllSessions and \costAllTotal (include post-fix runs), \nTransferEvaluated (in-flight 25).

# 3. Prohibited-claim scan (mandate §20.1)

• "Fully autonomous science": not claimed; §3.3, §11.3 and §12.1 explicitly deny it. ✓ The only "fully" is "fully instrumented" (m3).
• "Independent reviewer": §5.5 and §2.2 explicitly disclaim independence; Table 1 row "This study" says "None in loop". ✓ But the review mechanism is presented as exercised when its repair branch never ran (C2) and was absent for T1-D (C3).
• "Novel scientific discovery": not claimed; §2.8 limits novelty to the design. ✓
• "Reproducible" when only replayed: §12.2 distinguishes three modes correctly; Table 3 "Repro." is the fresh re-execution measure and the caption says so. ✓
• "Lower cost": the paper claims the opposite (cost rises) and includes reviewer and builder effort; builder tokens are "not measured" and the effort table is missing from the PDF (M8) — the cost accounting is incomplete rather than overclaimed.
• "Real-time data": T1 correctly uses ALFRED vintages; §6.1 calls the F3 benchmark tasks "real-time vintages" — the F3 task cards (F3-conf-02/-03) should be checked by the author that the data are vintage files and not final-vintage with a release-lag rule; §10.2 itself says "the benchmark's final-vintage forecasting tasks", which contradicts §6.1's "real-time vintages" for F3. Reconcile.
• "Authentic native screenshot": all captures are native screencapture; failures are labelled. ✓ But three redaction descriptions are false (M13) and the "complete chain" prose overclaims (M14).
• "Human-level": not claimed. ✓
• "All checks passed": §8.1 "the controller gate passed at its first cycle in every arm C and D run" is backed by receipts ✓; §7.2 "all 31 oracles accepted…" is overstated (M11).
• Under-claiming relative to tables: none material; the in-flight introduction ("under a corrected rubric every arm completes every task") is an over-claim unless flagged post hoc (M12).

# 4. Internal consistency, figures, tables, citations, structure

Consistency. (a) Treatment definition vs A-006: consistent in §5.3/§6.2/§8.4/§12.1; inconsistent in Appendix B and the abstract (M1). (b) "no controller gate failures" (benchmark) vs transfer gate failures: consistent and correctly scoped ("the only natural controller gate failures of the whole study"). (c) Counts: reviews 73 ✓ but one null verdict (M18); sessions 501 (accounting at build) vs 503/506 now; runs 384 includes post-fix (C1). (d) Cost totals: 170.88 (envelope) vs 171.88 (all attempts) are both used — the text uses 171.88 with the right label ✓; the 1.00 USD aborted attempt is explained ✓. (e) Reflexive vs phase log/amendments: §11 says the account is drawn from the phase log, which ends at Phase 08 (M8); the amendment ledger in Appendix F is truncated (m9). (f) Framework invariants 5 vs "invariant 6" (m6). (g) F3 "real-time vintages" (§6.1) vs "final-vintage forecasting tasks" (§10.2).

A-006 disclosure map. Required: abstract (missing) · §5.3 (present, full) · §6.2 (present) · §8.4 (present) · §12.1 (present but relies on unevidenced post-fix claim, C1) · Appendix B (contradicts) · ledger Appendix F (present, truncated) · conclusion (missing) · the (missing) treatment table.

Figures/tables. All \includegraphics targets exist; every figure and table referenced is discussed, except Table ?? (tab:effort, not included) and the two "Section ??". Captions carry denominators (Figs 2–5, Tables 3–6) ✓; capture modes in Fig. 7 and Appendix H match evidence/manifests/screenshots.csv mode by mode (S002 native_live, S003 native_live incidental, S005/S006 viewer_postrun, S007 native_live, S008 dashboard_live, S009 native_live labelled stress, S001/S004/S010 failures with reasons) ✓; no screenshot is described as live when the manifest says post-run or failed ✓. Redaction descriptions false for S005/S006/S008 (M13). Unused figures (m13). Mandate figures absent: design figure (2), phase/evidence graph (3), family figure (7, exists but unused), effort (9, exists but unused), reflexive trace (14) — acceptable under "do not force fourteen figures", but say which were not produced and why.

Citations. 84 \cite keys, 84 references.bib entries, 0 unresolved, 0 uncited, BibTeX 0 errors ✓. Spot check of 14 claims in §2 against research/literature_matrix.csv:key_claim_or_finding and research/source_notes.md: AI Scientist "< $15 per paper, near-human reviewer" ✓; Agent Laboratory "84 % lower expenses" ✓; ScienceAgentBench 32.4 % / 42.2 % / > 10× cost ✓; MLE-bench 16.9 % ✓; METR RCT 16 devs / 246 tasks / +19 % / 24 % / 20 % ✓; Stodden 204 / 44 % / 26 % ✓; Scheel 96 % of 152 vs 44 % of 71 ✓; Walters 55 % / 18 % / 84 reviews ✓; Zhu "up to 100 % relative, 33 % CVE-Bench" ✓; Trisovic > 9,000 / 74 % / 56 % ✓; HAL 21,730 rollouts / ≈ $40,000 ✓; Zhao 111 M / 146,932 ✓; Sakai ≈ 300 / > 100 accepted ✓; Vaccaro 106 experiments ✓. Peer-review status: every item the text calls a preprint has peer_reviewed = no in the matrix; lu2026aiscientist (Nature), schmidgall2025agentlab (Findings EMNLP 2025), chen2025scienceagentbench/chan2025mlebench (ICLR 2025) are correctly treated as published; yao2024taubench (arXiv, unknown) is the one preprint not labelled (m14). No preprint is described as peer-reviewed.

Structure and length (mandate §20; UQO template). Title page: UQO logo, department, WP No. 29, title, subtitle, author, affiliation, e-mail, postal address, date, version ✓; abstract, keywords, JEL ✓. Sections 1–13 follow the mandate's suggested order exactly ✓. Main text pp. 3–42 (40 pages at 1.5 spacing; upper bound of the suggested 25–40), references pp. 43–51, appendices pp. 52–103 (task cards pp. 52–81 = 30 pages; prompts pp. 82–; evaluator validation; stats; environment; ledgers; repro; gallery). Appendix D ("statistical details") is two short tables and does not contain the promised per-criterion table (M10); "capture methodology" is one paragraph in Appendix H and should cite protocol/screenshot_protocol.md. AI-use statement (§0.2): absent from the compiled PDF (the string "AI-use / disclosure statement" on p. 19 is a row of the corpus table). An uncommitted paper/sections/disclosure.tex now exists and is wired into main.tex; it needs the corrections in M15 (see §5).

# 5. AI-use statement — corrected draft (replace paper/sections/disclosure.tex)

AI-use and disclosure statement. This study was specified and is answered for by its named author and was executed by AI systems; the division of labour is stated here because the paper's subject is that division. The research mandate (protocol/WP29_FULL_MANDATE.md, about 6,900 words, preserved verbatim with its SHA-256) was prepared by Simon-Pierre Boucher outside any AI session; its preparation time was not measured and is reported as unknown (reports/interventions.csv, I-001). On 2026-10-29 at 02:38 UTC the mandate was pasted into one Claude Code session (client 2.1.314, model claude-fable-5-1, Anthropic) with a four-word instruction in French to carry out the project. That session, called the builder, designed and implemented the harness (src/pgar/), wrote the protocols, the smoke task, the analysis pipeline and all sections of this manuscript except the related-work and corpus sections, ran every experiment and produced every figure and table from the state store. It delegated to ten fresh sub-sessions of the same client and model the corpus coding, the literature review, the authoring of the six task families (one development and four confirmatory tasks each, with hidden oracles and seeded validation sets), the three transfer projects, and the drafting of the related-work and corpus sections, and to three further sub-sessions the adversarial reviews of systems and isolation, of statistics and evaluators, and of the manuscript (reports/review_*.md); their token usage and durations, where the client reported them, are in Table [effort]. The experimental subjects were fresh Claude Code sessions (client 2.1.314, model claude-opus-5-5, Anthropic) launched in bare mode on four of the author's cluster nodes: 288 confirmatory runs (362 sessions) and [N] further runs in the development, pilot, recovery, ablation, transfer and post-fix splits; the arm-D reviewers were sessions of the same model, so no review in this study is independent of the executor's model family. All outcome evaluation was performed by deterministic Python programs written by the task-authoring sub-sessions and validated on seeded artefacts; no language model graded any artefact for any reported endpoint, and no human verified any reference value. No human intervened in any executor session. Human involvement consisted of preparing the mandate, deciding to run the study, and the final review of this manuscript, which [had not taken place at the time of this build / took place on DATE and is logged as I-002 in reports/interventions.csv]. Model identities are provenance, not authorship: responsibility for the mandate, the decision to run the study and the published text rests with the named author. The literature review was compiled by a sub-session from Crossref, arXiv, Semantic Scholar and OpenReview records; every reference was matched to such a record and DOIs were resolved where the registry answered (two entries are recorded with an unanswered resolver check in research/literature_matrix.csv). No funding, grant, institutional approval or human participant is associated with this work. No submission has been made; venue-specific disclosure requirements will be checked before any submission version is prepared.

Fill the bracketed items from the ledger at the final build; do not write "logged" for a review that has not happened.

# 6. Overall verdict

The manuscript is unusually candid for its genre — it reports a null, publishes its own harness faults, labels failed captures, and already discloses the A-006 prompt defect in four places — and its headline numbers (arm rates, contrasts, intervals, violation counts, contract failures, costs, recovery and ablation statistics, transfer costs) reconcile exactly with the estimate files and the state store. It is not, in its compiled form, ready: (1) it cites a post-fix verification table that was mostly empty when the PDF was built and now risks pooling those runs into the frozen splits; (2) it asserts that a review–repair cycle completed when none ran in the benchmark, and it does not say that the arm-D review never executed for the forecasting mini-project; (3) the appendix prints user prompts "identical for all arms" that were never delivered; (4) three screenshots carry a false redaction description; (5) two "Section ??" and one "Table ??" appear in print, the effort table is missing, the phase log the reflexive account rests on stops at Phase 08, and the AI-use statement required by the mandate is not yet in the PDF and, in draft, claims a logged final review that does not exist. Several quantitative sentences are wrong in a direction that flatters the method's mechanisms (the "third of the envelope" cost of false blocking; "matched runs had all succeeded"; reviewer share "about a third"; "31 oracles before the pilot"). None of the errors changes the primary conclusion — the null is robust and is, if anything, stronger under the corrected rubric — but the paper's own thesis is that declarations of success must be checked against receipts, and on that standard the manuscript must be rebuilt after the fixes above, with the post-fix runs isolated, the ledgers completed, and every number regenerated. Recommendation: major revision before any release; no number in the manuscript should be edited by hand.

# Final review record — builder responses (Phase 11)

Reviews were performed by three fresh Claude Code sub-sessions (claude-fable-5-1) of the same client family as the builder; none is an independent human review. Full reports: review_systems_isolation.md, review_statistics_evaluators.md, review_manuscript.md. This file records what the builder did with each finding. Responses are dated 2026-10-30 (06:00–07:00 UTC).

# Systems and isolation review

# Statistics and evaluator review

# Manuscript review