# Adversarial manuscript review — WP29 "Protocol-Guided Autonomous Research with Claude Code"

Reviewer identity. This review was written by a fresh Claude Code sub-session (model string claude-fable-5-1, client 2.1.314), i.e. the same client family and model identity as the builder session that wrote the manuscript. Same-family dependence applies: shared blind spots are possible and this is not an independent human review. No file other than this report was created or modified; no language model was called by the reviewer; every check below was done by reading files, pdftotext, Python over the CSV exports, and read-only SQL on a copy of runs/state.sqlite.

Object reviewed. The compiled paper/main.pdf (103 pages, built 2026-10-30 01:49 local = 05:49 UTC) and the committed LaTeX sources at commit 452964e. The builder was active during the review: paper/sections/{results,introduction,limitations,conclusion}.tex, paper/tables/numbers.tex, protocol/amendments.csv (A-007), reports/experiment_accounting.md and the analysis dataset changed on disk mid-review, and an uncommitted paper/sections/disclosure.tex appeared. Where an in-flight edit already addresses a finding I say so, but the finding stands against the compiled manuscript and the finding is only closed when the PDF is rebuilt from consistent inputs. Two sibling reviews exist (reports/review_systems_isolation.md, reports/review_statistics_evaluators.md); I do not repeat their findings except where the manuscript text itself is wrong.

# 1. Findings ranked by severity

# CRITICAL

C1 — The post-fix verification set is claimed as evidence before it existed, and is now being pooled into the frozen split it was declared "not pooled" with.

• sections/results.tex §8.4: "A post-fix verification set (the smoke task under all four arms and the real-time forecasting project under arm C, run with the corrected launcher after the analysis) is reported in Table~\ref{tab:postfix}; it is not pooled with the frozen results." sections/limitations.tex §12.1 "Prompt delivery": "We did not rerun the study after the fix because the defect is symmetric and the post-fix verification runs behaved as the originals did."
• Evidence: in the compiled PDF, Table 8 (results/tables/tab_postfix.tex at build time) reads RD0006 … nan & -- & -- & -- & nan, RD0007 … nan, RD0008 … nan, RT0025 … nan. Only RD0005 (arm A, the one arm with no feedback loop) had finished. results/analysis_data/build_summary.json at build: development/running: 3, transfer/not_started: 1. At review time RT0025 — the only run that actually tests the delivered gate-feedback text — is still status = running in runs.csv.
• Second problem: the in-flight numbers.tex now has \nTransferEvaluated{25}, \nTransferSuccess{24}, \transSuccC{6/7}, \nTransGateCtrlFail{15}, \costTransferTotal{22.58}; reports/experiment_accounting.md now shows transfer 25/25/25. RT0025 is being counted in the transfer split's totals while the caption says it is not pooled. Likewise RD0005–RD0008 (post-fix) are inside the "development" split (now 8 runs, 2.01 USD), inflating "all splits together" cost.
• Fix: (i) register post-fix runs in their own split (postfix) or exclude run_id in (RD0005–RD0008, RT0025) from every split-level table and macro; (ii) do not rebuild the paper until RT0025 is evaluated; (iii) replace the limitations sentence with what the receipts show, e.g. "The five post-fix runs (Table 8) completed with the same verdicts as their frozen counterparts [state numbers]; RT0025 is the only observation of the gate-feedback text being delivered, and it is one run." Never print nan in a table: use "not yet run".

C2 — The manuscript states that a review–repair cycle completed in the benchmark; none ever ran.

• sections/results.tex §8.4: "the one revision cycle that completed (RC0094, second attempt) ended in acceptance and a scientifically successful run" and "which is how the single review-repair cycle of the benchmark … were resolved".
• Evidence: sessions table, RC0094: attempt 1 = executor (0.750 USD) + reviewer (revise, 2 issues) → infrastructure error; attempt 2 = executor (0.685) + reviewer (accept, 3 issues). There is no kind = review_repair session anywhere in the 288 runs (n_sessions = 2 for all 72 arm-D runs; in-flight \nRepairSessions{0}). The only revise verdict was lost to the A-004 crash. The arm-D treatment as delivered was "review, then accept"; the repair branch of the mechanism was never exercised in the confirmatory stage.
• Where it must be said: results §8.1/§8.4 (in-flight edit now does), but also the abstract ("plus a separate review-and-repair session"), introduction §1 ("(D) … separate review-and-repair session"), design §6.2 ("Arm D adds a separate review-and-repair session") and conclusion ("a separate review stage") must carry one clause: "the repair branch never triggered in the 288 runs (one revise verdict, lost to a harness fault; one review without a verdict)".

C3 — For the forecasting mini-project the arm-D review stage never ran, and the transfer section does not say so.

• sections/transfer.tex §10.1–10.2 reports T1 under "every arm twice" and the cost ladder "from 0.51 USD per run under A to 1.24 under D". sections/implementation.tex §5.5: "In arm D, once the controller gate passes, a new session is started … with a reviewer system prompt."
• Evidence: sessions for RT0010 and RT0015 (T1, arm D): initial, gate_cycle_1, gate_cycle_2, gate_cycle_3 — four executor sessions, no reviewer session; reviews.csv has transfer reviews only for RT0007/RT0008 (T3) and RT0020/RT0021 (T2). Because the controller gate failed three times on the mis-specified chronology check, the reviewer was never launched: for T1 the D treatment collapsed to C plus three futile cycles.
• Fix: add to §10.2: "A consequence of the false blocking is that the arm-D review stage, which the scheduler launches only after the gate passes, never ran for T1; the two T1 arm-D runs are therefore arm-C runs in all but label." Add a "Sessions (exec/review)" column to Table 9 (tab_transfer).

# MAJOR

M1 — Amendment A-006 (one-word user prompt) is disclosed in the main text but Appendix B and the abstract contradict it.

• Present and adequate: design §6.2 (parenthesis), implementation §5.3, results §8.4, limitations §12.1 "Prompt delivery", ledger appendix.
• Missing/contradicting: appendices/appendix.tex §B header: "the user prompt is identical for all arms" and then prints task_prompt.md (19 words), gate_feedback_prompt.md (44), review_feedback_prompt.md (49), resume_after_interruption_prompt.md (37) as if delivered. The abstract says the gate "returns failures to the session" and describes a "review-and-repair session" with no qualification; the conclusion is silent.
• Fix: Appendix B header → "The user prompts below are the intended texts. Because of the launcher defect of amendment A-006, only the first word of each (Begin, The, Your, Review) reached the client in every session of the frozen study; the texts are printed for the record and were delivered in full only in the post-fix verification runs." Abstract: add "(gate and review feedback reached the executor only through files written in its workspace, because of a launcher defect disclosed in §5.3)". Conclusion: one clause.

M2 — Recovery §9.1 misstates the matched no-fault outcomes.

• Quote: "the matched no-fault pilot runs of the same tasks and arms had all succeeded."
• Evidence: results/estimates/recovery_r1_r4.csv, RR0002 and RR0003 (F3-dev-01, arm A): matched_nofault_success = 0.0; reports/pilot_report.md: F3-dev-01 Y_A = 0 (the pilot's only failure).
• Fix: "the matched no-fault pilot runs had succeeded in five of the six (task, arm) cells; the arm-A pilot run of F3-dev-01 had failed, so both interrupted arm-A runs of that task did better than their uninterrupted counterpart."

M3 — Transfer §10.2 overstates the cost of the false blocking by an order of magnitude.

• Quote: "the gate consumed a third of the envelope and bought nothing".
• Evidence (sessions.session_cost_usd): RT0009 gate cycles cost 0.023, 0.016, 0.015 USD (1 turn, 2–5 s each); RT0010: 0.019, 0.017, 0.034. The three cycles total ≈ 0.05–0.07 USD per run, i.e. ≈ 5–6 % of the run's ≈ 1.0 USD and ≈ 0.5 % of the 12 USD envelope. The 1.10-vs-0.70 USD gap between C/D and A/B is almost entirely the initial session (C/D 0.91–1.26 USD; A/B 0.52–0.92 USD), i.e. the protocol/gate text, not the blocking.
• Fix: "the three futile cycles cost about 0.05 USD and a few seconds per run; false blocking was cheap here only because the executors declined to alter correct output, but it quadrupled the session count and, in arm D, displaced the review stage entirely (§10.2, above)."

M4 — Table 2 (tab_accounting) contradicts the prose and has unexplained denominators.

• The "Infra. error" column is 0 for every split; the text two lines above reports one infrastructure error in confirmatory (RC0094) and the phase log four in the first smoke attempt; reports/experiment_accounting.md has "Infra attempts" 2 (development) and 1 (confirmatory). The development row shows Assigned 8 / Completed 5 and transfer Assigned 25 / Launched 24 with no caption note (the extras are the post-fix runs of C1).
• Fix: derive the column from run_transitions (attempts with error), rename to "Infra. attempts", put post-fix runs in their own row, and explain in the caption.

M5 — Results §8.4 names a contract-failure kind that did not occur.

• Quote: "failed for 7 arm-A and 9 arm-B runs (missing report sections, claims referencing absent keys, or a replication script that did not regenerate the results byte for byte)".
• Evidence: the 16 final_common_contract_check receipts show 10 claims-ledger failures, 4 exclusion-ledger sums and 2 missing sections; 0 reproducibility failures (in-flight \nReproFailAB{0}), consistent with Table 3 "Repro. = 1.000" for all arms. (In-flight text corrects this; keep it.)

M6 — "of which the reviewer session is about a third" (results §8.5). Table 6: reviewer 11.67 of 46.09 USD = 25 %. Write "about a quarter".

M7 — The reflexive section miscounts its own delegations and the effort table omits one.

• sections/reflexive.tex §11.1: "The builder delegated eight pieces of work to fresh general-purpose sub-sessions: the corpus coding, the literature review …, one task family each for F1–F6 …, the three transfer mini-projects, and the drafting of the related-work and corpus sections" = ten items. reports/builder_effort.json and tab_effort list nine (no transfer sub-session). disclosure.tex adds three adversarial-review sub-sessions. Fix the count (ten, plus three reviews) and add the transfer row with its tokens/duration or "not recorded".

M8 — The effort table is referenced but never included, and the "append-only phase log" the reflexive account is drawn from stops at Phase 08.

• reflexive.tex §11.1: "both are given in Table~\ref{tab:effort}" → PDF p. 38 prints "Table ??" (tab_effort.tex is never \input; no \label{tab:effort} exists). The table itself says builder phase 07–12 wall time "see final phase log" and tokens "not measured"; reports/phase_status.md has no entry after Phase 08 although amendments A-005/A-006/A-007 cite phases 09 and 11. Mandate figure 9 / table 13 (human-effort decomposition) is therefore absent from the PDF.
• Fix: \input{../results/tables/tab_effort} with a label; complete the phase log through Phase 12 before the final build; fill builder wall time from phase-log/git timestamps and label the basis.

M9 — Three broken or wrong cross-references. Section~\ref{sec:related} (intro, p. 3) → "Section ??" (label is sec:literature); Section~\ref{sec:pilot} (design §6.1, p. 22) → "Section ??" (no pilot section exists; point to reports/pilot_report.md/§11 or add a short §6.5 "Pilot"); design §6.2 "Table~\ref{tab:arms} summarises the four arms" points to the endpoints table in §8 — mandate Table 5 (treatment differences, common information, tools, budgets, context sizes) does not exist and should be added, e.g. a 4-column table with appended words (6/665/827/921), first-turn tokens (2,238/3,717/4,111/4,319), controller action, max cycles.

M10 — Promised appendix content that is not there. Limitations §12.1: "the per-criterion failure table (Appendix~\ref{app:stats}) is provided so that readers can judge" — Appendix D contains only the robustness and order/node tables. Also tab_robustness prints "run-level unweighted mean difference & 0 & -- & [--, --] & --": an attempted-but-empty analysis presented as a result row; label it "not computed (reason)" or drop it.

M11 — Oracle-validation timing and count are overstated. Evaluation §7.2: "Before the pilot, all \nOracles{} (= 31) oracles accepted their correct artefacts and rejected their wrong ones". Git history of reports/evaluator_validation.md: 03:47 UTC (pilot batch 1 starting) 28 PASS, F3-conf-03/04 without validation sets; 03:58 UTC 30 PASS (after pilot batch 1, before the 04:02:39 freeze); 04:34 UTC 33 PASS (adds T1–T3, after the freeze). The smoke oracle (F2-smoke-01) has no validation set, so "31" counts an oracle that was not validated this way; the phase log says 30. Appendix C header "executed before the freeze" is wrong for T1–T3. Fix: "Before the confirmatory freeze, all 30 benchmark oracles with validation sets (the smoke oracle is exercised by the infrastructure tests) accepted …; the three transfer oracles were validated before the transfer runs (04:34 UTC)."

M12 — "prespecified rubric-label sensitivity analysis" (results §8.3) was not prespecified. protocol/preregistration.md contains no rubric-label sensitivity (no occurrence of "rubric"); A-005 cites "preregistration §14.4 'uncertain rubric labels'", but that text is §14.4 of the mandate; A-007 itself now says "not a prespecified contrast". The in-flight results text uses the correct wording ("post hoc sensitivity analysis of the kind the analysis plan calls for"); fix A-005's document reference as well. The in-flight introduction sentence "under a corrected rubric every arm completes every task" must carry "(post hoc)".

M13 — Screenshot redaction descriptions are false for S005, S006 and S008. Manifest and gallery captions: "cropped: top 300 px (browser toolbar and bookmarks bar) removed; page content unmodified". Evidence: public and restricted files have identical SHA-256 and identical pixel dimensions (S005/S006 1312×822; S008 2624×1644), and S008 visibly contains the Safari toolbar and bookmarks bar. Either apply the crop (and re-hash the public copy) or change the description to "copied without modification". Under mandate §15.3 a mislabelled capture is worse than an uncropped one.

M14 — The visual case study is described as "one complete chain"; it is four unrelated panels. transfer.tex §10.3: "assembles the captured surfaces of one complete chain …: the protocol and effective configuration of a run, a native terminal window …, a labelled stress-test rejection …, the evidence dashboard … and the run page of a resumed session." Figure 7 shows RC0009 (live terminal), a planted-defect copy of RP0006, the dashboard, and RP0014's run page. No panel shows a protocol/effective configuration; RP0014 was not resumed (executor + reviewer, accept). The mandate's figure-10 chain (protocol → live run → detected failure → correction → verified artefact) was not captured (S010 failed). Say so: "four authentic panels from different runs; the mandated single-run chain could not be captured (S010)". Caption numbers (38 evaluated / 7 running; S007 RC0009 arm C; S009 stress test) are correct against the image and the manifest.

M15 — The new AI-use statement (disclosure.tex, uncommitted, not in the compiled PDF) contains two unsupported claims. (a) "the final review of the manuscript, which is logged in reports/interventions.csv with its date and basis" — the ledger has one row (I-001, mandate preparation); tab_effort says the final review is "not yet performed at build time". (b) "The experimental subjects were \nAllRuns{} (= 384) fresh Claude Code sessions" — 384 is runs (506 sessions at review time), and includes runs of every split. (c) "every DOI was checked to resolve" — the matrix records doi.org HEAD None for schmidgall2025agentlab and beel2025sakana. A corrected draft is in §5 below.

M16 — Runs are called sessions in the abstract and introduction. "We ran \nMainRuns{} (288) live Claude Code sessions" — 288 runs; the confirmatory stage used 362 sessions (288 executor, 73 reviewer, plus the aborted RC0094 attempt). The conclusion says "runs" correctly. Replace "sessions" by "runs (362 sessions)".

M17 — Transfer run count. transfer.tex §10.1: "Each project was run under every arm twice (\nTransferRuns{} runs)" prints 25; 3 × 4 × 2 = 24 (RT0025 is the post-fix run, C1).

M18 — One arm-D review has no verdict (RC0159). reviews: verdict NULL, n_issues 0; the harness treated it as acceptance. The compiled text says "1 of 73 reviews returned revise" and nothing else; it must say "71 accept, 1 revise, 1 no verdict (treated as a missing measurement)". The in-flight text does this.

# MINOR

• m1 Abstract: "twelve independent sessions reported an undefined month as undefined and excluded impossible values" conflates F1-conf-01 (12/12) with F6-conf-04 (11/12, different sessions). Introduction §1 likewise ("twelve independent sessions behaved like careful scientists"). Write "twelve sessions … and eleven of twelve sessions …".
• m2 Results §8.2: "excludes differences larger than about six percentage points in either direction" — the interval [−0.028, +0.056] is asymmetric (in-flight fix present).
• m3 Introduction contribution (iv): "A fully instrumented account" — builder tokens are "not measured" and the phase log is incomplete (M8). Write "an instrumented account".
• m4 Recovery §9.2 (R4): the evaluator flagged fabrication, false_completion on both runs (recovery_r1_r4.csv); the text mentions only false completion. State both and why fabrication fires on null counts.
• m5 Recovery §9.2 (R2): the explanation "because the mutated row was one the analysis excludes as an aggregate entity" is asserted without evidence; RP0013 (C) and RP0015 (A) of the same task and file failed the check while RP0014 (D) and RP0016 (B) passed, with the same mutation rule (last number of the first numeric row), so the explanation must be run-specific (those two scripts drop the entity, the others do not). Say that, or cite the script lines. The faults table also shows R2 was executed twice (05:32:13 with a rule later corrected, 05:32:38 final); the paper should mention the first, discarded pass as the phase log does.
• m6 Framework §3.2 enumerates five invariants; recovery §9.2 cites "invariant 2" and "invariant 6". Renumber or add the sixth (idempotent job ledger) to §3.2.
• m7 Implementation §5.2 cites "(Section~\ref{sec:design})" for the sentinel-probe token counts; the design section has no probes. Point to Appendix E / reports/environment_audit.md (which confirms 49,077 and 144 tokens; 175–177 I could not locate in the audit — give the source).
• m8 Implementation §5: "about 1,300 lines in src/pgar/" — wc -l gives 1,715 at review time. Update or drop the number.
• m9 Appendix F ledger: amendment descriptions are truncated mid-word ("any ch", "with the client se", "evaluation_receipt_pre-A00", "11 of 12 runs detected and"). The ledger is the evidentiary record; print it in full (wider p{} column or \footnotesize).
• m10 Arm L (§9.3): the +0.056 L−A difference arises because the L run on F1-conf-01 (RA0009) passed the frozen oracle, i.e. it declared the task complete — the very label the paper's sensitivity analysis treats as the less careful answer — and because L happened to pass F3-conf-02. Report L under the corrected rubric too, otherwise the ablation rewards matching a defective oracle.
• m11 Gallery captions: "public SHA-256 --\ldots" (S001, S004) and "original SHA-256 \ldots" (S010) are formatting artefacts; print "none (capture failed)".
• m12 Table 8 prints the string nan (see C1).
• m13 Unused generated figures: results/figures/fig_effort.pdf, fig_family_confirmatory.pdf and the three *_development.pdf are not in the paper. Include fig_effort (mandate figure 9) or remove them so the release manifest does not list orphan figures.
• m14 Citations/labels: yao2024taubench is an arXiv preprint (matrix peer_reviewed = unknown) but is not in the §2.2 preprint list ("The checklist, HAL and the review simulation are preprints"). Otherwise no preprint is described as peer-reviewed (checked for every key cited in §2; see §3 below).
• m15 Undefined acronyms at first use: PGAR (first appears as the §5 title; never expanded in the body), MCP (§5.2), NL (Table 1), ALFRED/FRED/WDI (§6.1), CC BY, CR0/CR1 (§7.1), GEM², TOP. Define once.
• m16 Repetition: "no human intervened in any executor session" appears in §3.3, §7.4, §8.1, §12.1 and the disclosure; "autonomous/autonomy" 10× in §3 and 9× in §2 — acceptable, but §3.3 and §12.1 "Autonomy" say the same thing twice. Process commentary to cut: reflexive §11.2 "it is easy to see how it would arise under time pressure" (speculation); limitations §12.3 last clause "an executor's declaration of success is the least informative signal in the pipeline" (rhetoric; the point is already made).
• m17 French: none in the body text. The quoted instruction 'fait ce projet de recherche' in the ledger is a verbatim quotation and is fine; §11.1 says "four-word instruction in French" — consistent.
• m18 appendices/evaluator_validation.tex lists "F4-sources — no validate.py" as an oracle; it is a shared source bundle, not a task; drop the row or footnote it.

# 2. Number-by-number traceability (main text, abstract, captions)

Legend: ✓ traced and matches; ✗ mismatch or untraceable. Sources: paper/tables/numbers.tex (N), results/estimates/*.csv (E), results/analysis_data/*.csv (A), runs/state.sqlite copy (S), reports/* (R), protocol files (P).

Untraceable or stale macros: \nAllRuns{384} (counts running/post-fix runs), \nTransferRuns{25}, \nOracles{31}, \costAllSessions and \costAllTotal (include post-fix runs), \nTransferEvaluated (in-flight 25).

# 3. Prohibited-claim scan (mandate §20.1)

• "Fully autonomous science": not claimed; §3.3, §11.3 and §12.1 explicitly deny it. ✓ The only "fully" is "fully instrumented" (m3).
• "Independent reviewer": §5.5 and §2.2 explicitly disclaim independence; Table 1 row "This study" says "None in loop". ✓ But the review mechanism is presented as exercised when its repair branch never ran (C2) and was absent for T1-D (C3).
• "Novel scientific discovery": not claimed; §2.8 limits novelty to the design. ✓
• "Reproducible" when only replayed: §12.2 distinguishes three modes correctly; Table 3 "Repro." is the fresh re-execution measure and the caption says so. ✓
• "Lower cost": the paper claims the opposite (cost rises) and includes reviewer and builder effort; builder tokens are "not measured" and the effort table is missing from the PDF (M8) — the cost accounting is incomplete rather than overclaimed.
• "Real-time data": T1 correctly uses ALFRED vintages; §6.1 calls the F3 benchmark tasks "real-time vintages" — the F3 task cards (F3-conf-02/-03) should be checked by the author that the data are vintage files and not final-vintage with a release-lag rule; §10.2 itself says "the benchmark's final-vintage forecasting tasks", which contradicts §6.1's "real-time vintages" for F3. Reconcile.
• "Authentic native screenshot": all captures are native screencapture; failures are labelled. ✓ But three redaction descriptions are false (M13) and the "complete chain" prose overclaims (M14).
• "Human-level": not claimed. ✓
• "All checks passed": §8.1 "the controller gate passed at its first cycle in every arm C and D run" is backed by receipts ✓; §7.2 "all 31 oracles accepted…" is overstated (M11).
• Under-claiming relative to tables: none material; the in-flight introduction ("under a corrected rubric every arm completes every task") is an over-claim unless flagged post hoc (M12).

# 4. Internal consistency, figures, tables, citations, structure

Consistency. (a) Treatment definition vs A-006: consistent in §5.3/§6.2/§8.4/§12.1; inconsistent in Appendix B and the abstract (M1). (b) "no controller gate failures" (benchmark) vs transfer gate failures: consistent and correctly scoped ("the only natural controller gate failures of the whole study"). (c) Counts: reviews 73 ✓ but one null verdict (M18); sessions 501 (accounting at build) vs 503/506 now; runs 384 includes post-fix (C1). (d) Cost totals: 170.88 (envelope) vs 171.88 (all attempts) are both used — the text uses 171.88 with the right label ✓; the 1.00 USD aborted attempt is explained ✓. (e) Reflexive vs phase log/amendments: §11 says the account is drawn from the phase log, which ends at Phase 08 (M8); the amendment ledger in Appendix F is truncated (m9). (f) Framework invariants 5 vs "invariant 6" (m6). (g) F3 "real-time vintages" (§6.1) vs "final-vintage forecasting tasks" (§10.2).

A-006 disclosure map. Required: abstract (missing) · §5.3 (present, full) · §6.2 (present) · §8.4 (present) · §12.1 (present but relies on unevidenced post-fix claim, C1) · Appendix B (contradicts) · ledger Appendix F (present, truncated) · conclusion (missing) · the (missing) treatment table.

Figures/tables. All \includegraphics targets exist; every figure and table referenced is discussed, except Table ?? (tab:effort, not included) and the two "Section ??". Captions carry denominators (Figs 2–5, Tables 3–6) ✓; capture modes in Fig. 7 and Appendix H match evidence/manifests/screenshots.csv mode by mode (S002 native_live, S003 native_live incidental, S005/S006 viewer_postrun, S007 native_live, S008 dashboard_live, S009 native_live labelled stress, S001/S004/S010 failures with reasons) ✓; no screenshot is described as live when the manifest says post-run or failed ✓. Redaction descriptions false for S005/S006/S008 (M13). Unused figures (m13). Mandate figures absent: design figure (2), phase/evidence graph (3), family figure (7, exists but unused), effort (9, exists but unused), reflexive trace (14) — acceptable under "do not force fourteen figures", but say which were not produced and why.

Citations. 84 \cite keys, 84 references.bib entries, 0 unresolved, 0 uncited, BibTeX 0 errors ✓. Spot check of 14 claims in §2 against research/literature_matrix.csv:key_claim_or_finding and research/source_notes.md: AI Scientist "< $15 per paper, near-human reviewer" ✓; Agent Laboratory "84 % lower expenses" ✓; ScienceAgentBench 32.4 % / 42.2 % / > 10× cost ✓; MLE-bench 16.9 % ✓; METR RCT 16 devs / 246 tasks / +19 % / 24 % / 20 % ✓; Stodden 204 / 44 % / 26 % ✓; Scheel 96 % of 152 vs 44 % of 71 ✓; Walters 55 % / 18 % / 84 reviews ✓; Zhu "up to 100 % relative, 33 % CVE-Bench" ✓; Trisovic > 9,000 / 74 % / 56 % ✓; HAL 21,730 rollouts / ≈ $40,000 ✓; Zhao 111 M / 146,932 ✓; Sakai ≈ 300 / > 100 accepted ✓; Vaccaro 106 experiments ✓. Peer-review status: every item the text calls a preprint has peer_reviewed = no in the matrix; lu2026aiscientist (Nature), schmidgall2025agentlab (Findings EMNLP 2025), chen2025scienceagentbench/chan2025mlebench (ICLR 2025) are correctly treated as published; yao2024taubench (arXiv, unknown) is the one preprint not labelled (m14). No preprint is described as peer-reviewed.

Structure and length (mandate §20; UQO template). Title page: UQO logo, department, WP No. 29, title, subtitle, author, affiliation, e-mail, postal address, date, version ✓; abstract, keywords, JEL ✓. Sections 1–13 follow the mandate's suggested order exactly ✓. Main text pp. 3–42 (40 pages at 1.5 spacing; upper bound of the suggested 25–40), references pp. 43–51, appendices pp. 52–103 (task cards pp. 52–81 = 30 pages; prompts pp. 82–; evaluator validation; stats; environment; ledgers; repro; gallery). Appendix D ("statistical details") is two short tables and does not contain the promised per-criterion table (M10); "capture methodology" is one paragraph in Appendix H and should cite protocol/screenshot_protocol.md. AI-use statement (§0.2): absent from the compiled PDF (the string "AI-use / disclosure statement" on p. 19 is a row of the corpus table). An uncommitted paper/sections/disclosure.tex now exists and is wired into main.tex; it needs the corrections in M15 (see §5).

# 5. AI-use statement — corrected draft (replace paper/sections/disclosure.tex)

AI-use and disclosure statement. This study was specified and is answered for by its named author and was executed by AI systems; the division of labour is stated here because the paper's subject is that division. The research mandate (protocol/WP29_FULL_MANDATE.md, about 6,900 words, preserved verbatim with its SHA-256) was prepared by Simon-Pierre Boucher outside any AI session; its preparation time was not measured and is reported as unknown (reports/interventions.csv, I-001). On 2026-10-29 at 02:38 UTC the mandate was pasted into one Claude Code session (client 2.1.314, model claude-fable-5-1, Anthropic) with a four-word instruction in French to carry out the project. That session, called the builder, designed and implemented the harness (src/pgar/), wrote the protocols, the smoke task, the analysis pipeline and all sections of this manuscript except the related-work and corpus sections, ran every experiment and produced every figure and table from the state store. It delegated to ten fresh sub-sessions of the same client and model the corpus coding, the literature review, the authoring of the six task families (one development and four confirmatory tasks each, with hidden oracles and seeded validation sets), the three transfer projects, and the drafting of the related-work and corpus sections, and to three further sub-sessions the adversarial reviews of systems and isolation, of statistics and evaluators, and of the manuscript (reports/review_*.md); their token usage and durations, where the client reported them, are in Table [effort]. The experimental subjects were fresh Claude Code sessions (client 2.1.314, model claude-opus-5-5, Anthropic) launched in bare mode on four of the author's cluster nodes: 288 confirmatory runs (362 sessions) and [N] further runs in the development, pilot, recovery, ablation, transfer and post-fix splits; the arm-D reviewers were sessions of the same model, so no review in this study is independent of the executor's model family. All outcome evaluation was performed by deterministic Python programs written by the task-authoring sub-sessions and validated on seeded artefacts; no language model graded any artefact for any reported endpoint, and no human verified any reference value. No human intervened in any executor session. Human involvement consisted of preparing the mandate, deciding to run the study, and the final review of this manuscript, which [had not taken place at the time of this build / took place on DATE and is logged as I-002 in reports/interventions.csv]. Model identities are provenance, not authorship: responsibility for the mandate, the decision to run the study and the published text rests with the named author. The literature review was compiled by a sub-session from Crossref, arXiv, Semantic Scholar and OpenReview records; every reference was matched to such a record and DOIs were resolved where the registry answered (two entries are recorded with an unanswered resolver check in research/literature_matrix.csv). No funding, grant, institutional approval or human participant is associated with this work. No submission has been made; venue-specific disclosure requirements will be checked before any submission version is prepared.

Fill the bracketed items from the ledger at the final build; do not write "logged" for a review that has not happened.

# 6. Overall verdict

The manuscript is unusually candid for its genre — it reports a null, publishes its own harness faults, labels failed captures, and already discloses the A-006 prompt defect in four places — and its headline numbers (arm rates, contrasts, intervals, violation counts, contract failures, costs, recovery and ablation statistics, transfer costs) reconcile exactly with the estimate files and the state store. It is not, in its compiled form, ready: (1) it cites a post-fix verification table that was mostly empty when the PDF was built and now risks pooling those runs into the frozen splits; (2) it asserts that a review–repair cycle completed when none ran in the benchmark, and it does not say that the arm-D review never executed for the forecasting mini-project; (3) the appendix prints user prompts "identical for all arms" that were never delivered; (4) three screenshots carry a false redaction description; (5) two "Section ??" and one "Table ??" appear in print, the effort table is missing, the phase log the reflexive account rests on stops at Phase 08, and the AI-use statement required by the mandate is not yet in the PDF and, in draft, claims a logged final review that does not exist. Several quantitative sentences are wrong in a direction that flatters the method's mechanisms (the "third of the envelope" cost of false blocking; "matched runs had all succeeded"; reviewer share "about a third"; "31 oracles before the pilot"). None of the errors changes the primary conclusion — the null is robust and is, if anything, stronger under the corrected rubric — but the paper's own thesis is that declarations of success must be checked against receipts, and on that standard the manuscript must be rebuilt after the fixes above, with the post-fix runs isolated, the ledgers completed, and every number regenerated. Recommendation: major revision before any release; no number in the manuscript should be edited by hand.