Final review record — builder responses (Phase 11)
Reviews were performed by three fresh Claude Code sub-sessions (claude-fable-5-1) of the same client family as the builder; none is an independent human review. Full reports: review_systems_isolation.md, review_statistics_evaluators.md, review_manuscript.md. This file records what the builder did with each finding. Responses are dated 2026-10-30 (06:00–07:00 UTC).
Systems and isolation review
| Finding | Response |
|---|---|
| C1 Continuation sessions never received their prompt (one word); remediation text claimed "delivered via files" | Accepted. The claim was wrong: transcripts show the resumed sessions asked for the missing message and stopped. The paper now states that the controller feedback loop, the review-repair stage and the resumption instruction were NOT delivered in any frozen run, that the frozen A–D contrast tests protocol text delivered through the system prompt, and reports seven post-fix verification runs (smoke ×4, T1 arms C and D, R1 ×2) with the corrected launcher (A-006). |
| M1 Delivered tools = Bash/Edit/Read; built-in plugins loaded under --bare | Accepted; implementation section, isolation audit, contract and treatment definitions corrected to the delivered inventory from the init events. |
M2 Behaviour audit miscounted (~ not normalised); "zero outside references" claim |
Accepted; scanner normalises ~/$HOME; audits recomputed for all 384 runs: 4 runs with outside references (listed), 0 network commands; limitations text corrected (A-008). |
M3 Both revise verdicts lost; RC0159 review written to out/out/ |
Accepted; RC0159 verdict recovered from the artefact and the reviews row corrected (A-008); collection now searches under out/; the paper states that review-repair never executed and that H3 is not evaluable from the benchmark. |
| M4 Re-attempt events dropped (event-id collision) | Accepted; RC0094 attempt-2 events re-ingested under attempt-scoped ids; ids are attempt-scoped going forward (A-008). |
M5 Harness changed mid-campaign; runs -dirty |
Accepted; disclosed in the implementation section and the isolation audit; commits per run are in the manifests; no task/oracle/treatment/analysis interface changed. |
| m1 S002 caption ("while it runs" but run had ended); cropped copies byte-identical | Accepted; S002 caption corrected to "immediately after the run ended"; crop re-implemented and verified (public hashes differ from originals). |
| m2 Interrupted sessions charged 0 in the sessions table | Accepted as a ledger limitation; envelope accounting is cumulative and correct; noted in the accounting report. |
| m3 Amendment timestamps hand-rounded | Accepted; the appendix states that git commits are the authoritative timestamps. |
| m4 Sibling exposure persists on nodes | Accepted; listed as a residual risk; workspaces are retained deliberately as evidence. |
| m5 "offline nodes" wording | Accepted; corrected to "no web tools; nodes have Internet; no network command observed". |
| m6–m9 | Noted; m8 dead code removed from the release; m9 post-fix runs are never pooled. |
Statistics and evaluator review
| Finding | Response |
|---|---|
| C1 All 25 confirmatory non-successes are evaluator artefacts (four defects: feasibility label, generator artefact, rounding tolerance, regex false positive); paper misattributed two to executors | Accepted. Sensitivity re-scoring extended to all four (A-007); the paper's failure subsection rewritten: under the corrected rubric every arm completes every task; the violation contrasts measure a regex. The frozen scoring is unchanged and reported first. |
| M1 Review-repair never executed; RC0159 verdict NULL | Accepted (see systems M3). |
| M2 Preregistration deviations (Y conjunct, max_turns, in-place rerun, "prespecified" wording, two-sided test) | Accepted: Y now includes the intervention conjunct and treats max_turns as outside the envelope (0 runs affected); RC0094's rerun is documented as in-place with both attempts' sessions retained and both cost figures reported; A-005 is no longer called prespecified; the permutation test is described as two-sided. |
| M3 A-005 loosened a frozen tolerance without need | Accepted; the loosening removed (A-007). |
| M4 robustness.py run-level row broken | Accepted; replaced by an unpaired run-level bootstrap labelled as an anti-pattern comparison. |
| M5 Claims endpoint gradient spurious | Accepted; the claims table is retained in the appendix with the statement that all flagged claims are the F6-conf-04/F3-conf-02 artefacts. |
| M6 Tables disagreed across builds | Accepted; one atomic rebuild (make analyze && make paper) precedes the release validation. |
| Minor items (CI wording, byte-for-byte claim, "about six points") | Accepted and corrected. |
Manuscript review
| Finding | Response |
|---|---|
| C1 Post-fix runs pooled into macros; post-fix claims before the runs existed | Accepted; post-fix runs (seeds 102/402/403/802) are excluded from every frozen macro, table and estimate and appear only in the post-fix table; the manuscript was rebuilt only after the last post-fix run (RT0026) was evaluated. |
| C2 Non-existent repair cycle asserted | Accepted; removed everywhere; abstract, introduction, design, results, limitations and conclusion now state that the repair branch never triggered and that the frozen comparison is one of protocol texts. |
| C3 T1 arm D: reviewer never ran | Accepted; disclosed in the transfer section. |
| M1 Appendix B prompt claim | Accepted; note added that only the first word was delivered in frozen runs. |
| M2 "matched no-fault runs all succeeded" | Accepted; corrected (five of six cells). |
| M3 "a third of the envelope" | Accepted; replaced by the measured per-cycle costs. |
| M4 Accounting table infra column and post-fix rows | Accepted; infrastructure attempts counted from the transition ledger; post-fix verification shown as its own row. |
| M5 byte-for-byte repro failures | Accepted; corrected to the measured composition (claims ledger, exclusion ledger, sections; 0 reproducibility failures). |
| M6 Reviewer share | Accepted; computed macro (\reviewerShareD{} %). |
| M7 Delegation count and effort table | Accepted; thirteen delegations; transfer and review sub-sessions added to the effort data; table now input. |
| M8–M10 Undefined references, missing treatments table, missing per-criterion table, phase log | Accepted; labels fixed, treatments table and pilot subsection added, per-criterion table generated into Appendix D, phase log completed through Phase 12. |
| M11 "all 31 oracles before the pilot" | Accepted; reworded to the git-history facts. |
| M12 "prespecified" sensitivity | Accepted; described as an after-results sensitivity analysis. |
| M13 False crop labels | Accepted; crop re-implemented and verified. |
| M14 "one complete chain" | Accepted; caption and prose corrected. |
| M15 Disclosure statement | Accepted; corrected (sessions vs runs; human review not yet performed). |
| M16–M18 | Accepted (runs vs sessions; transfer run count macro; RC0159 disclosed). |
| Minor items | Accepted where factual (preprint label, acronyms, "instrumented", arm L re-scoring note, timestamp note); unused figures retained in results/ but not referenced. |