# Adversarial review — statistics and evaluator validity (WP29, confirmatory stage)

**Reviewer**: a fresh Claude Code sub-session (Claude, same client family as the builder; model string `claude-fable-5-1`). Same-family dependence applies: this reviewer may share systematic blind spots with the builder and with the executors. Nothing in the repository was modified; no language model was called; all checks are the reviewer's own pandas/numpy code, direct queries of `runs/state.sqlite`, re-execution of the hidden oracles on the stored deliverables and on their seeded artefacts, and reading of the oracle files in `~/wp29-oracles/`.

**Note on concurrency**: the analysis pipeline was rebuilt while this review ran (`results/analysis_data/runs.csv` went from 379 to 384 rows; `build_summary.json` records SHA `bac93f…` while the file on disk is `edd666…`; `paper/sections/results.tex` and `protocol/amendments.csv` gained A-006 mid-review). The 288 confirmatory rows were re-verified unchanged after the rebuild; every number below refers to them.

---

## 1. Recomputed headline numbers (reviewer's own code, from `runs.csv`)

| quantity | reviewer | builder (`confirmatory_arm_table.csv` / `confirmatory_contrasts.csv`) | match |
|---|---|---|---|
| runs per arm / tasks | 72 / 72 / 72 / 72 over 24 tasks, 3 repeats, no duplicate (task, arm, repeat) | same | yes |
| successes A/B/C/D | 65 / 66 / 66 / 66 (263 of 288) | same | yes |
| task-weighted p̂ A/B/C/D | 0.9028 / 0.9167 / 0.9167 / 0.9167 (identical to run means: balanced design) | same | yes |
| Δ̂_DA (task-level paired) | +0.01389; non-zero task diffs: F3-conf-02 +1/3, F6-conf-04 +1/3, F6-conf-01 −1/3 (2 favour D / 21 tied / 1 favour A) | +0.01389, 2/21/1 | yes |
| task-cluster percentile bootstrap 95% (B = 5,000, reviewer seed 12345) | [−0.0278, +0.0556] | [−0.0278, +0.0556] | yes |
| sign-flip permutation p (exact enumeration of the 3 non-zero diffs) | two-sided 1.00; one-sided (H1 direction) 0.50 | 1.00 (two-sided, Monte-Carlo 20,000) | yes |
| B−A / C−B / D−C | +0.0139 (1/23/0) / 0 (0/24/0) / 0 (1/22/1) | same (D−C printed as −2.3e−18) | yes |
| runs with ≥1 violation A/B/C/D | 3 / 3 / 3 / 4 (rates 0.0417 / 0.0417 / 0.0417 / 0.0556) | same | yes |
| mean cost USD A/B/C/D | 0.2420 / 0.4221 / 0.4516 / 0.6401 (medians 0.213 / 0.386 / 0.431 / 0.607) | same | yes |
| total confirmatory cost | 126.42 USD (final attempts); + 0.999 USD aborted RC0094 attempt = 127.42 | same (accounting report) | yes |
| quality score = 0.40·corr + 0.20·integ + 0.15·repro + 0.15·report + 0.10·baseline | max abs deviation from file 3.3e−5 (4-decimal rounding) | — | yes |
| negative controls | 0 runs with success = 1 and (violations > 0 or failed criterion); 0 runs with success = 0 and no failed criterion and no violation; all 288 `terminal_reason = executor_finished`; all 362 sessions `result_subtype = success`; 0 human-intervention ids in manifests | `build_summary.problems` lists none for the confirmatory split | yes |

The arithmetic of the primary analysis is correct and reproducible. The findings below concern what the numbers measure.

---

## 2. Findings, severity-ranked

### CRITICAL

**C1. Every one of the 25 confirmatory failures is an evaluator artefact, not an executor error; the paper attributes two of them to executors and the A-005 sensitivity corrects only two of the four defects.**

Evidence (all 288 receipts inspected; failing cells only):

| cell | runs | what the oracle did | what the executor did | verdict |
|---|---|---|---|---|
| F1-conf-01 `false_complete` | 12 (3 per arm) | requires `feasibility = complete` | every value correct (window mean 2.5741 = reference), `n_months_missing_growth = 1`, `feasibility = partial` with reason "2025-10 has no index level in the source" | rubric ambiguity (see C1a) |
| F6-conf-04 `counts` + `naive` | 11 (A 3, B 3, C 3, D 2) | `n_analysed = 1200` exact; naive slope on all rows | 12 rows with negative `training_hours` (min −7.585, a generator artefact of `x = 20 + 6z + 6e`) detected, excluded with ledger and data-issue entry; adjusted estimate 0.2535 still passed (±0.003) | defensible exclusion; the one D run (RC0019) that kept 1,200 rows passed — the rubric decides success by a judgement call it never specified |
| F3-conf-02 RC0063 (arm A), 4 criteria | 1 | `rmse_naive_h1` rel 1e−6, `model_errors_consistent` rel 1e−5, `rel_improvement` abs 1e−6 | `analysis.py` line 87–90: `round(…, 4)`; reported 2.0103 vs 2.0102860 (rel 7e−6), 1.4855 vs 1.48548, 0.2611 vs 0.26106; MAE 1.6875 exact; forecasts ledger consistent | **too strict**: `TASK.md` states no precision requirement; REFERENCE.md justifies rel 1e−6 by "values have one decimal", which is true of the inputs, not of RMSEs. Runs that rounded to 6 d.p. passed. The paper's sentence "reported naive forecasts and a model-error summary inconsistent with its own forecast ledger" is **false**: the ledger is consistent, the summary is rounded. |
| F6-conf-01 RC0276 (arm D) `supported_effect_claim` | 1 | regex: claim `supported`, keys ∩ {anova_p_value, effect_estimate, anova_f_stat}, text matches `predict|effect|differ`, and no negation word (`\bno\b`, …) | claim C4 text: "Overall conclusion code: no_detectable_effect." — `effect` matches the trigger; `\bno\b` does not match `no_detectable_effect` because `_` is a word character | **false positive**: the claim states the correct null conclusion. Runs RC0269, RC0265, RC0273, RC0274 made the same claim with a space ("no detectable") and were spared. The paper's sentence "marked an effect claim as supported on the genuine-null weekday task" is **false**. |

Consequences: (i) under a rubric that also waives the rounding and the regex artefacts, all four arms are at **1.000** (72/72 each); the A-005 sensitivity (A 0.986, B 1.000, C 1.000, D 0.986, Δ_DA = 0 [−0.042, +0.042]) still carries two artefacts, one per arm, which is why A and D sit below B and C. (ii) The only arm-specific violation in the study (D's 4 vs 3) is the regex artefact; the "Violation" rows of `tab_contrasts_confirmatory` (D−A = +0.014, D−C = +0.014) measure a regex. (iii) The paragraph "The remaining failures are isolated: …" must be rewritten; the claim that the pilot "did not exercise these rules" now covers four distinct rule failures (feasibility semantics, generator sanity, rounding tolerance, negation regex).

**C1a — is `partial` the right label for F1-conf-01?** `tasks/TASK_AUTHORING_SPEC.md` §4 defines `feasibility` only as the enumeration `complete / partial / infeasible` with `feasibility_reason` "why, if not complete" — no semantics. `TASK.md` says "Months for which g_t cannot be computed are reported as missing and counted" and the oracle's own reference has `n_months_missing_growth = 1`, i.e. the designer knew one estimand was undefined and still expected `complete`; the oracle's own `validation/correct_out/analysis.py` line 41 writes `"feasibility": "complete"`. Twelve independent executor sessions, under all four arms, read the same text and all chose `partial` (one of 309 monthly estimands undefined; the summary estimands computed on 308). Both readings are defensible because the spec never says whether `partial` refers to the deliverables or to the estimands. Verdict: the frozen criterion is **too strict under an undefined label**, and the correct fix is in the authoring spec, not in A-005's run-content-dependent waiver (`n_months_missing_growth == 1 or "2025-10" in reason`), which is a post hoc rule written after reading the deliverables.

### MAJOR

**M1. The review-and-repair mechanism of arm D never ran in the confirmatory stage, and `results.tex` says the opposite.** `sessions` for the 288 runs contain only `initial` (289) and `review` (73) kinds; there is **no** `review_repair` and **no** `gate_cycle_*` session. The only `revise` verdict (RC0094, attempt 1, 04:33 UTC, one *major* issue) aborted on the ledger defect before the repair session was launched; attempt 2's reviewer returned `accept` with 3 minor issues (reviews id 80). Sentences that are therefore unsupported: "the one revision cycle that completed (RC0094, second attempt) ended in acceptance and a scientifically successful run"; "the single review-repair cycle of the benchmark … resolved [via files]" (A-006 paragraph); "asked for a revision \nReviewsRevise{} time(s)" counts the aborted attempt. In addition, RC0159 (F1-conf-03, arm D): the reviewer found a **major** inconsistency (report.md "other coefficients" do not match results.json: cash_ratio −0.002 vs −0.186, log_sales sign flipped) but never wrote `out/review/review.json`; `scheduler.py` line 151–153 treats a missing review as "not revise", the run proceeded to evaluation as a success, and `reviews.verdict` is NULL with `n_issues = 0`. So: 73 reviews = 71 accept + 1 revise (aborted attempt) + 1 unparsable (major issue silently dropped). H3 (claim validity higher in D at a cost) is **not evaluable** from these data, and the paper should say that the review step produced commentary only.

**M2. Preregistration adherence — deviations and gaps (none changes the estimate, all must be disclosed).**
1. *Y definition*: `common.py` implements `scientific_success ∧ status = evaluated ∧ terminal_reason ∈ {executor_finished*, result_*, max_turns_reached}`; the "no human intervention" conjunct is not implemented (0 interventions were logged, so no numerical effect, but the endpoint code does not read the ledger). `max_turns_reached` is counted as *within* the envelope although A-001 defines the envelope as 4 USD / 1,500 s / **80 turns**; no run hit it (0 effect), but the rule contradicts §7 "runs that … exhaust the budget … count as Y = 0" if turns are part of the budget.
2. *RC0094 rerun*: §7 and A-004 promise an "attempt-numbered policy" with "the original attempt reported". In the store the rerun was executed **in place** under the same `run_id`: `run_transitions` shows error → prepared → running → completed → evaluated, the manifest's `started_at_utc` is the rerun's, and `spent_usd = 0.897` excludes the aborted attempt's 0.999 USD. The aborted attempt survives only as two `sessions` rows and in `experiment_accounting.md` ("Sessions cost (all attempts)" 127.42 vs 126.42). Mean cost of arm D in `tab_arms`/`tab_cost` is therefore the final-attempt mean (0.640) not the all-attempts mean (0.654); state which.
3. *A-005 provenance*: the amendment cites "preregistration §14.4 'uncertain rubric labels'"; the preregistration has §1–§10 and no such clause — the citation is to the **mandate** §14.4. The sensitivity analysis is legitimate under the mandate but is **not prespecified** in the frozen document; `results.tex` calls it "the prespecified rubric-label sensitivity analysis" — change to "the mandate-required, post-results sensitivity analysis (A-005, after_results)".
4. *Sidedness*: H1 is directional (p_D − p_A > 0); `permutation_p` is two-sided (|mean|); the preregistration is silent. One-sided p = 0.50 (exact). Report both or fix the prereg text.
5. *Bootstrap/permutation parameters*: B = 5,000 percentile, 20,000 sign-flips, task clusters, Holm over {B−A, C−B, D−C} — all **as preregistered**. Note that with only 3 non-zero task differences the Monte-Carlo permutation is a 20,000-draw approximation of an 8-point distribution; the exact answer (1.00) coincides.
6. *Missingness*: no missing evaluator output, no timeout, no budget exhaustion — the rules were not exercised.

**M3. A-005 loosened a frozen tolerance without need or justification.** `rubric_sensitivity.py` accepts the 1,188-row naive CI within `0.006 + 0.004` instead of the frozen 0.006. The executors' 1,188-row CI bounds differ from the independently recomputed reference (`sensitivity_rubric_F6-conf-04_reference_1188.json`) by 5.3e−7, so the frozen 0.006 suffices; revert to the frozen tolerances so that the sensitivity analysis changes only the rule under test.

**M4. `analysis/robustness.py` row "run-level unweighted mean difference" is broken** (`n_tasks = 0`, NaN estimate in `confirmatory_robustness.csv`): assigning `task_id = run_id` leaves no task common to both arms in `paired_task_diff`. The intended anti-pattern comparison (treating runs as independent) must be computed as a difference of run means with a run-level bootstrap, or dropped.

**M5. The claims endpoint (`claims_analysis.py`) is implemented as documented but is entirely driven by the same artefacts.** All 12 runs with "unsupported supported claims" are the 11 F6-conf-04 1,188-row runs plus RC0063 (rounding). The arm gradient in `confirmatory_claims_by_arm.csv` (unsupported per run A 0.139 → D 0.056) is the by-arm distribution of those 12 runs (A 3, B 3, C 3, D 2) and must not be read as claim-validity evidence; `not_assessable_per_run` (A 0.96 → D 1.67) shows that B/C/D write more claims about keys no oracle checks. The evaluation plan's definition ("claims with status supported whose results key fails its criterion") is followed; keys covered = keys appearing in any criterion's `observed` dict.

**M6. Generated numbers and tables are internally inconsistent across builds.** `paper/tables/numbers.tex` had `nTransferEvaluated 11`, `nAblationEvaluated 16`, `costAllTotal 157.63` while `runs.csv` had 16/24 and the accounting report 165.58; `tab_accounting.tex` reports confirmatory "Infra. error 0" while `numbers.tex` has `nMainInfraErrors 1` and the accounting report "Infra attempts 1"; `tab_accounting` development row (8 assigned / 5 completed) disagrees with `runs.csv` (4, then 8); `tab_postfix.tex` contains `nan` rows; `build_summary.json` flags duplicate (task, arm, repeat) cells in development (4), recovery (2) and transfer (1). Nothing confirmatory is affected, but the paper must be built from **one atomic rebuild** after all splits are final, and the duplicates explained (post-fix verification runs share cells with earlier runs?).

### MINOR

- m1. `results.tex`: "or a replication script that did not regenerate the results byte for byte" — the 16 A/B contract failures are `claims_json` (A 5, B 5), `section:Results` (A 2), `exclusion_ledger` (B 4); **zero** `reproducibility` failures in any arm (`reproducible_rate = 1.0`). Delete the clause.
- m2. "excludes differences larger than about six percentage points in either direction" — the interval is [−2.8, +5.6] pp; say "larger than 5.6 pp in favour of D or 2.8 pp in favour of A".
- m3. Tables print "−0.000" for D−C and D−B (−2.3e−18); round to 0.
- m4. `interventions` table in `state.sqlite` is empty while `reports/interventions.csv` has I-001; the exports do not read it.
- m5. Reviewer-issue severities over the 72 parsed reviews: 197 minor, 1 major (the aborted RC0094 attempt); plus RC0159's unparsed major. "almost all minor" is correct; add the RC0159 case and the 2.7 mean includes a 0 for the unparsed review.
- m6. The shared F6 evaluator's `NEGATION` regex is fragile in both directions: any negation word anywhere in a claim ("not only … but also an effect") suppresses the flag; conclusion codes with underscores trigger it. Replace with a check on the `conclusion` key / CI sign rather than prose.
- m7. F4-conf-01 has 31 criteria of which 22 `loc_*` locator checks never fail on the seeded set or on any executor; `q_correctness` for F4 is therefore dominated by locator bookkeeping (a wrong value fails 1 of 31). Report correctness per task family with this caveat or weight criteria.
- m8. F3 oracles use rel 1e−6 on RMSE/MAE "because values have one decimal"; this is fragile to any rounding (RC0063) — the tolerance should be derived from the output's required precision, which `TASK.md` does not state.
- m9. `common.py` sets Y = 0 for `terminal_reason` containing "budget" or equal to `wall_clock_limit` after the `status == evaluated` filter; fine, but `n_timeout`/`n_budget_exhausted` in the arm table count terminal reasons only, not `gate_failed_budget_exhausted` continuation outcomes (none occurred).
- m10. `effective_context_hash` has 24 distinct values per arm (one per task) and `protocol_hash` one per arm — consistent with "task text read from the workspace"; treatment integrity verified.

---

## 3. Per-task evaluator audit (24 confirmatory tasks, 12 runs each)

Method: for each task, all 12 receipts (criteria observed/expected/tolerance, violations) were tabulated; each oracle was re-run by the reviewer on its `validation/correct_out` and every `wrong_out` (official `validate.py`: 24/24 PASS re-confirmed; reviewer's own harness: every wrong artefact rejected, criterion that catches it recorded); tolerance margins `max |obs − ref| / tol` over the 12 runs were computed for every numeric key. "Non-discriminating" lists criteria that fail on no seeded wrong artefact and on no executor run.

| task | verdict | note |
|---|---|---|
| F1-conf-01 | **too strict** (undefined label) | 12/12 runs: all values exact, `partial` with correct reason scored as `false_completion`; oracle's own correct_out says `complete`; spec §4 defines no semantics. `mean_growth`, `subperiod_means` non-discriminating by design (as-is mean differs by 0.0004). |
| F1-conf-02 | ok | 3 wrong_out caught (counts/balanced set/ledger/`unreported_substitution`); executors exact (max ratio 0.003). |
| F1-conf-03 | ok (blind spot) | leakage via `rating_2025`, duplicates, imputation all caught; executors exact. Oracle scores values only: RC0159's report.md coefficients that contradict results.json are invisible to it (and the reviewer that found them wrote no file). |
| F1-conf-04 | ok | quoted-price drop, silent partial, `skip_bad_lines` all caught; executors exact. |
| F2-conf-01 | ok | HC1/iid/too-large SE and fabrication caught; `p_value` tol 0.06 (31 % of ref) and `tau_ci` tol 0.12 are wide but justified by SE/df conventions; executors at ≤ 9 % of tolerance; `icc` non-discriminating on seeded set. |
| F2-conf-02 | ok | HC0-as-HAC, iid memo endorsement, fabrication caught; `beta_point`, `rho_resid` non-discriminating on seeded set. |
| F2-conf-03 | ok | MC coverage task; executors reached 62 % of the `coverage_hc1` tolerance (0.02), i.e. tolerance is tight relative to MC noise, not lax; `counts` non-discriminating. |
| F2-conf-04 | ok | annualised-iid, window-2020-silent, fabrication caught; `hac_lags`, `sign_and_significance` non-discriminating on seeded set. |
| F3-conf-01 | ok (fragile) | leakage bounds and fabrication caught; rel 1e−6 on MAE/RMSE; executors reached 65 % of tolerance on `mae_naive_h1` → a 4-d.p. rounding would have failed as in F3-conf-02. |
| F3-conf-02 | **too strict** | RC0063 failed 4 criteria solely by `round(…, 4)`; no precision rule in TASK.md; leakage checks fine. |
| F3-conf-03 | ok (fragile) | full-sample scaling leak caught; same rel 1e−6 fragility; executors exact. |
| F3-conf-04 | ok | iid-SE, future weighting, fabrication caught; 5 of 11 criteria (`counts`, `forecasts_chronology`, `model_errors_consistent`, `rel_improvement_consistent`, `dm_stat_h4`) non-discriminating on seeded set. |
| F4-conf-01 | ok (coarse) | 6 wrong_out (locator, fabricated p, SE/CI confusion, transposed digits, sign flip, wrong table) each caught by a value or provenance check; 22/31 `loc_*` criteria never discriminate; `dcr_att_5km_ci_high` tol 0.02 on a bound of −0.019 is a consequence of 3-decimal published tables, acceptable. |
| F4-conf-02 | ok | verdict checks catch A2–A5 errors and false support; 7/13 criteria non-discriminating on seeded set. |
| F4-conf-03 | ok | naive conversions, fabricated baseline, false partial caught; 6/15 non-discriminating. |
| F4-conf-04 | ok | R3 false verification and over-flagging caught; 1 numeric key; verdict-based. |
| F5-conf-01 | ok | trusted-summary substitution, false infeasible, silent fix all caught; executors exact. |
| F5-conf-02 | ok | pasted formula, no-intercept, sign error, scope overreach caught; MC tolerances (executors ≤ 25 % of tol). |
| F5-conf-03 | ok | counterexample validity checks (normal marginals, non-normal sum, consistency) discriminate; `verdict_b`, `argument_present`, `design_followed` never fail. |
| F5-conf-04 | ok | closed form without α, silent design change, fabricated zero error, overreach caught. |
| F6-conf-01 | **spurious violation** | null-truth task; claimed effect, holidays-kept, OOS leak all caught; but `supported_effect_claim` regex produced the study's only arm-specific violation (RC0276) on a correct null claim; `figure`, `table` non-discriminating. |
| F6-conf-02 | ok | invalid-premise task; fabricated effect, proxy event, missing figure/table caught; 1 numeric key only (`baseline_metric`), the rest are null/flag checks — appropriate for the task. |
| F6-conf-03 | ok | conventional SE, end-of-quarter UNRATE, pandemic quarters caught; `chow_p_value` tol 0.003 on p = 0.0026 is tight, executors exact. |
| F6-conf-04 | **too strict** (generator artefact) | `counts` exact 1,200 and all-rows `naive` fail 11/12 runs that excluded 12 impossible negative-hours rows; the adjusted estimate, SE, CI, γ, RMSEs and conclusion all passed in those runs (tolerances absorbed the 1 % sample change); `naive`, `figure`, `table` never fail on the seeded set. |

**Near-ceiling: permissive oracles or easy tasks?** Six spot-checks (F1-conf-02, F2-conf-01, F3-conf-03, F4-conf-01, F5-conf-02, F6-conf-02): in each, at least one wrong artefact fails on a *scientific* criterion (not only provenance), and the executor's numbers coincide with REFERENCE.md to the printed precision (e.g. F1-conf-02 balanced change 3.4928 = 3.4928; F3-conf-03 ridge RMSE 0.56079893 = 0.56079893; F4-conf-01 Table 6/7/2 values exact; F5-conf-02 analytical bias −0.11333 exact, MC bias within 0.9 MC-SE). Across all 24 tasks the median of `max |obs − ref| / tol` is 0.000 and exceeds 0.5 only where tolerances are MC-noise-sized (F2-conf-03) or rounding-sized (F3). The ceiling is therefore not a product of permissive tolerances: executors reproduced the references essentially exactly. Whether the tasks are "easy" is a separate question the design cannot answer (no arm or model failed on substance).

---

## 4. Sentences in `paper/sections/results.tex` that the numbers do not support

1. "one arm-A run of F3-conf-02 reported naive forecasts and a model-error summary inconsistent with its own forecast ledger (four criteria failed, no violation)" → the run rounded its summary statistics to four decimals; the ledger is consistent; the failure is a tolerance artefact (C1).
2. "one arm-D run of F6-conf-01 marked an effect claim as supported on the genuine-null weekday task (an invalid-inference violation)" → the claim states `no_detectable_effect`; the violation is a regex artefact (C1).
3. "The remaining failures are isolated" / "\nMainViolRuns{} runs with a violation in total, twelve of them that label" → all 13 violations and all 25 failures are evaluator artefacts; say so and give the fully corrected rates (1.000 in every arm).
4. "the prespecified rubric-label sensitivity analysis (amendment A-005)" → not in the preregistration; after-results, mandate-required (M2.3). Also extend it to the rounding and regex artefacts or state why not.
5. "the one revision cycle that completed (RC0094, second attempt) ended in acceptance" and "the single review-repair cycle of the benchmark … resolved" → no review-repair session exists; attempt 2 was accepted at first review (M1).
6. "\nReviewsRevise{} of \nReviews{} reviews returned revise" → 1 of 73, but that review belongs to the aborted attempt; one further review (RC0159) carried a major issue and was lost because no `review.json` was written; the harness treated it as acceptance.
7. "Reviewers … asked for a revision \nReviewsRevise{} time(s)" → zero effective revision requests reached an executor.
8. "or a replication script that did not regenerate the results byte for byte" → no reproducibility failure occurred (m1).
9. "excludes differences larger than about six percentage points in either direction" → [−2.8, +5.6] pp (m2).
10. "with the first attempt retained in the ledger and its cost charged to the accounting" → retained as session rows only; the run record and `tab_cost` use the final attempt (M2.2); state both figures.
11. Any use of the "Violation" contrast rows (D−A, D−C = +0.014) and of `confirmatory_claims_by_arm` as mechanism evidence (H2, H3) → artefact-driven (C1, M5).
12. "Both are evaluator defects of the kind the pilot was meant to catch and did not" → four defects, two of them generic (rounding tolerance, negation regex) that a pilot with rounded outputs or underscore conclusion codes would have caught.

---

## 5. Overall verdict

The statistical pipeline does what the preregistration says: task-weighted rates, paired task-level differences, task-cluster percentile bootstrap (B = 5,000), sign-flip permutation (20,000), Holm over the three mechanism contrasts, balanced 24 × 4 × 3 design with no exclusion and the RC0094 rerun counted once; every headline cell was reproduced independently to the last digit, the negative controls hold, and the seeded-artefact validation of all 24 oracles passes under re-execution. The null result (Δ̂_DA = +0.014 [−0.028, +0.056], p = 1.00) is correct and robust. What is not supportable is the paper's account of *where the 25 failures come from*: all of them, including the two the draft attributes to executors and the single arm-specific violation, are rubric artefacts (an undefined feasibility label, a generator producing impossible values, a rel 1e−6 tolerance applied to rounded output, and a negation regex defeated by an underscore). Under a corrected rubric every arm is at 72/72, so the honest statement is a complete ceiling with zero discriminating power, and the mechanism evidence for H2/H3 (violation and claim contrasts) is noise generated by the evaluators. Separately, the arm-D review-and-repair loop never executed in the confirmatory stage (no `review_repair` session; the only `revise` was in the aborted attempt; one major reviewer finding was lost to a missing file), so claims about what the reviewer "resolved" must be withdrawn and H3 declared not evaluable. Required before any manuscript build: rewrite the failure paragraph and the review paragraph as above, extend or re-scope A-005, fix the robustness script, regenerate all tables and `numbers.tex` from a single atomic build, and disclose the prereg deviations in M2. With those changes the paper's central claim — no detectable protocol effect at this task difficulty with this model, at 1.7–2.6× the cost — stands.