# Adversarial review — statistics and evaluator validity (WP29, confirmatory stage)

Reviewer: a fresh Claude Code sub-session (Claude, same client family as the builder; model string claude-fable-5-1). Same-family dependence applies: this reviewer may share systematic blind spots with the builder and with the executors. Nothing in the repository was modified; no language model was called; all checks are the reviewer's own pandas/numpy code, direct queries of runs/state.sqlite, re-execution of the hidden oracles on the stored deliverables and on their seeded artefacts, and reading of the oracle files in ~/wp29-oracles/.

Note on concurrency: the analysis pipeline was rebuilt while this review ran (results/analysis_data/runs.csv went from 379 to 384 rows; build_summary.json records SHA bac93f… while the file on disk is edd666…; paper/sections/results.tex and protocol/amendments.csv gained A-006 mid-review). The 288 confirmatory rows were re-verified unchanged after the rebuild; every number below refers to them.

# 1. Recomputed headline numbers (reviewer's own code, from runs.csv)

The arithmetic of the primary analysis is correct and reproducible. The findings below concern what the numbers measure.

# 2. Findings, severity-ranked

# CRITICAL

C1. Every one of the 25 confirmatory failures is an evaluator artefact, not an executor error; the paper attributes two of them to executors and the A-005 sensitivity corrects only two of the four defects.

Evidence (all 288 receipts inspected; failing cells only):

Consequences: (i) under a rubric that also waives the rounding and the regex artefacts, all four arms are at 1.000 (72/72 each); the A-005 sensitivity (A 0.986, B 1.000, C 1.000, D 0.986, Δ_DA = 0 [−0.042, +0.042]) still carries two artefacts, one per arm, which is why A and D sit below B and C. (ii) The only arm-specific violation in the study (D's 4 vs 3) is the regex artefact; the "Violation" rows of tab_contrasts_confirmatory (D−A = +0.014, D−C = +0.014) measure a regex. (iii) The paragraph "The remaining failures are isolated: …" must be rewritten; the claim that the pilot "did not exercise these rules" now covers four distinct rule failures (feasibility semantics, generator sanity, rounding tolerance, negation regex).

C1a — is partial the right label for F1-conf-01? tasks/TASK_AUTHORING_SPEC.md §4 defines feasibility only as the enumeration complete / partial / infeasible with feasibility_reason "why, if not complete" — no semantics. TASK.md says "Months for which g_t cannot be computed are reported as missing and counted" and the oracle's own reference has n_months_missing_growth = 1, i.e. the designer knew one estimand was undefined and still expected complete; the oracle's own validation/correct_out/analysis.py line 41 writes "feasibility": "complete". Twelve independent executor sessions, under all four arms, read the same text and all chose partial (one of 309 monthly estimands undefined; the summary estimands computed on 308). Both readings are defensible because the spec never says whether partial refers to the deliverables or to the estimands. Verdict: the frozen criterion is too strict under an undefined label, and the correct fix is in the authoring spec, not in A-005's run-content-dependent waiver (n_months_missing_growth == 1 or "2025-10" in reason), which is a post hoc rule written after reading the deliverables.

# MAJOR

M1. The review-and-repair mechanism of arm D never ran in the confirmatory stage, and results.tex says the opposite. sessions for the 288 runs contain only initial (289) and review (73) kinds; there is no review_repair and no gate_cycle_* session. The only revise verdict (RC0094, attempt 1, 04:33 UTC, one major issue) aborted on the ledger defect before the repair session was launched; attempt 2's reviewer returned accept with 3 minor issues (reviews id 80). Sentences that are therefore unsupported: "the one revision cycle that completed (RC0094, second attempt) ended in acceptance and a scientifically successful run"; "the single review-repair cycle of the benchmark … resolved [via files]" (A-006 paragraph); "asked for a revision \nReviewsRevise{} time(s)" counts the aborted attempt. In addition, RC0159 (F1-conf-03, arm D): the reviewer found a major inconsistency (report.md "other coefficients" do not match results.json: cash_ratio −0.002 vs −0.186, log_sales sign flipped) but never wrote out/review/review.json; scheduler.py line 151–153 treats a missing review as "not revise", the run proceeded to evaluation as a success, and reviews.verdict is NULL with n_issues = 0. So: 73 reviews = 71 accept + 1 revise (aborted attempt) + 1 unparsable (major issue silently dropped). H3 (claim validity higher in D at a cost) is not evaluable from these data, and the paper should say that the review step produced commentary only.

M2. Preregistration adherence — deviations and gaps (none changes the estimate, all must be disclosed).

1. Y definition: common.py implements scientific_success ∧ status = evaluated ∧ terminal_reason ∈ {executor_finished*, result_*, max_turns_reached}; the "no human intervention" conjunct is not implemented (0 interventions were logged, so no numerical effect, but the endpoint code does not read the ledger). max_turns_reached is counted as within the envelope although A-001 defines the envelope as 4 USD / 1,500 s / 80 turns; no run hit it (0 effect), but the rule contradicts §7 "runs that … exhaust the budget … count as Y = 0" if turns are part of the budget.
2. RC0094 rerun: §7 and A-004 promise an "attempt-numbered policy" with "the original attempt reported". In the store the rerun was executed in place under the same run_id: run_transitions shows error → prepared → running → completed → evaluated, the manifest's started_at_utc is the rerun's, and spent_usd = 0.897 excludes the aborted attempt's 0.999 USD. The aborted attempt survives only as two sessions rows and in experiment_accounting.md ("Sessions cost (all attempts)" 127.42 vs 126.42). Mean cost of arm D in tab_arms/tab_cost is therefore the final-attempt mean (0.640) not the all-attempts mean (0.654); state which.
3. A-005 provenance: the amendment cites "preregistration §14.4 'uncertain rubric labels'"; the preregistration has §1–§10 and no such clause — the citation is to the mandate §14.4. The sensitivity analysis is legitimate under the mandate but is not prespecified in the frozen document; results.tex calls it "the prespecified rubric-label sensitivity analysis" — change to "the mandate-required, post-results sensitivity analysis (A-005, after_results)".
4. Sidedness: H1 is directional (p_D − p_A > 0); permutation_p is two-sided (|mean|); the preregistration is silent. One-sided p = 0.50 (exact). Report both or fix the prereg text.
5. Bootstrap/permutation parameters: B = 5,000 percentile, 20,000 sign-flips, task clusters, Holm over {B−A, C−B, D−C} — all as preregistered. Note that with only 3 non-zero task differences the Monte-Carlo permutation is a 20,000-draw approximation of an 8-point distribution; the exact answer (1.00) coincides.
6. Missingness: no missing evaluator output, no timeout, no budget exhaustion — the rules were not exercised.

M3. A-005 loosened a frozen tolerance without need or justification. rubric_sensitivity.py accepts the 1,188-row naive CI within 0.006 + 0.004 instead of the frozen 0.006. The executors' 1,188-row CI bounds differ from the independently recomputed reference (sensitivity_rubric_F6-conf-04_reference_1188.json) by 5.3e−7, so the frozen 0.006 suffices; revert to the frozen tolerances so that the sensitivity analysis changes only the rule under test.

M4. analysis/robustness.py row "run-level unweighted mean difference" is broken (n_tasks = 0, NaN estimate in confirmatory_robustness.csv): assigning task_id = run_id leaves no task common to both arms in paired_task_diff. The intended anti-pattern comparison (treating runs as independent) must be computed as a difference of run means with a run-level bootstrap, or dropped.

M5. The claims endpoint (claims_analysis.py) is implemented as documented but is entirely driven by the same artefacts. All 12 runs with "unsupported supported claims" are the 11 F6-conf-04 1,188-row runs plus RC0063 (rounding). The arm gradient in confirmatory_claims_by_arm.csv (unsupported per run A 0.139 → D 0.056) is the by-arm distribution of those 12 runs (A 3, B 3, C 3, D 2) and must not be read as claim-validity evidence; not_assessable_per_run (A 0.96 → D 1.67) shows that B/C/D write more claims about keys no oracle checks. The evaluation plan's definition ("claims with status supported whose results key fails its criterion") is followed; keys covered = keys appearing in any criterion's observed dict.

M6. Generated numbers and tables are internally inconsistent across builds. paper/tables/numbers.tex had nTransferEvaluated 11, nAblationEvaluated 16, costAllTotal 157.63 while runs.csv had 16/24 and the accounting report 165.58; tab_accounting.tex reports confirmatory "Infra. error 0" while numbers.tex has nMainInfraErrors 1 and the accounting report "Infra attempts 1"; tab_accounting development row (8 assigned / 5 completed) disagrees with runs.csv (4, then 8); tab_postfix.tex contains nan rows; build_summary.json flags duplicate (task, arm, repeat) cells in development (4), recovery (2) and transfer (1). Nothing confirmatory is affected, but the paper must be built from one atomic rebuild after all splits are final, and the duplicates explained (post-fix verification runs share cells with earlier runs?).

# MINOR

• m1. results.tex: "or a replication script that did not regenerate the results byte for byte" — the 16 A/B contract failures are claims_json (A 5, B 5), section:Results (A 2), exclusion_ledger (B 4); zero reproducibility failures in any arm (reproducible_rate = 1.0). Delete the clause.
• m2. "excludes differences larger than about six percentage points in either direction" — the interval is [−2.8, +5.6] pp; say "larger than 5.6 pp in favour of D or 2.8 pp in favour of A".
• m3. Tables print "−0.000" for D−C and D−B (−2.3e−18); round to 0.
• m4. interventions table in state.sqlite is empty while reports/interventions.csv has I-001; the exports do not read it.
• m5. Reviewer-issue severities over the 72 parsed reviews: 197 minor, 1 major (the aborted RC0094 attempt); plus RC0159's unparsed major. "almost all minor" is correct; add the RC0159 case and the 2.7 mean includes a 0 for the unparsed review.
• m6. The shared F6 evaluator's NEGATION regex is fragile in both directions: any negation word anywhere in a claim ("not only … but also an effect") suppresses the flag; conclusion codes with underscores trigger it. Replace with a check on the conclusion key / CI sign rather than prose.
• m7. F4-conf-01 has 31 criteria of which 22 loc_* locator checks never fail on the seeded set or on any executor; q_correctness for F4 is therefore dominated by locator bookkeeping (a wrong value fails 1 of 31). Report correctness per task family with this caveat or weight criteria.
• m8. F3 oracles use rel 1e−6 on RMSE/MAE "because values have one decimal"; this is fragile to any rounding (RC0063) — the tolerance should be derived from the output's required precision, which TASK.md does not state.
• m9. common.py sets Y = 0 for terminal_reason containing "budget" or equal to wall_clock_limit after the status == evaluated filter; fine, but n_timeout/n_budget_exhausted in the arm table count terminal reasons only, not gate_failed_budget_exhausted continuation outcomes (none occurred).
• m10. effective_context_hash has 24 distinct values per arm (one per task) and protocol_hash one per arm — consistent with "task text read from the workspace"; treatment integrity verified.

# 3. Per-task evaluator audit (24 confirmatory tasks, 12 runs each)

Method: for each task, all 12 receipts (criteria observed/expected/tolerance, violations) were tabulated; each oracle was re-run by the reviewer on its validation/correct_out and every wrong_out (official validate.py: 24/24 PASS re-confirmed; reviewer's own harness: every wrong artefact rejected, criterion that catches it recorded); tolerance margins max |obs − ref| / tol over the 12 runs were computed for every numeric key. "Non-discriminating" lists criteria that fail on no seeded wrong artefact and on no executor run.

Near-ceiling: permissive oracles or easy tasks? Six spot-checks (F1-conf-02, F2-conf-01, F3-conf-03, F4-conf-01, F5-conf-02, F6-conf-02): in each, at least one wrong artefact fails on a scientific criterion (not only provenance), and the executor's numbers coincide with REFERENCE.md to the printed precision (e.g. F1-conf-02 balanced change 3.4928 = 3.4928; F3-conf-03 ridge RMSE 0.56079893 = 0.56079893; F4-conf-01 Table 6/7/2 values exact; F5-conf-02 analytical bias −0.11333 exact, MC bias within 0.9 MC-SE). Across all 24 tasks the median of max |obs − ref| / tol is 0.000 and exceeds 0.5 only where tolerances are MC-noise-sized (F2-conf-03) or rounding-sized (F3). The ceiling is therefore not a product of permissive tolerances: executors reproduced the references essentially exactly. Whether the tasks are "easy" is a separate question the design cannot answer (no arm or model failed on substance).

# 4. Sentences in paper/sections/results.tex that the numbers do not support

1. "one arm-A run of F3-conf-02 reported naive forecasts and a model-error summary inconsistent with its own forecast ledger (four criteria failed, no violation)" → the run rounded its summary statistics to four decimals; the ledger is consistent; the failure is a tolerance artefact (C1).
2. "one arm-D run of F6-conf-01 marked an effect claim as supported on the genuine-null weekday task (an invalid-inference violation)" → the claim states no_detectable_effect; the violation is a regex artefact (C1).
3. "The remaining failures are isolated" / "\nMainViolRuns{} runs with a violation in total, twelve of them that label" → all 13 violations and all 25 failures are evaluator artefacts; say so and give the fully corrected rates (1.000 in every arm).
4. "the prespecified rubric-label sensitivity analysis (amendment A-005)" → not in the preregistration; after-results, mandate-required (M2.3). Also extend it to the rounding and regex artefacts or state why not.
5. "the one revision cycle that completed (RC0094, second attempt) ended in acceptance" and "the single review-repair cycle of the benchmark … resolved" → no review-repair session exists; attempt 2 was accepted at first review (M1).
6. "\nReviewsRevise{} of \nReviews{} reviews returned revise" → 1 of 73, but that review belongs to the aborted attempt; one further review (RC0159) carried a major issue and was lost because no review.json was written; the harness treated it as acceptance.
7. "Reviewers … asked for a revision \nReviewsRevise{} time(s)" → zero effective revision requests reached an executor.
8. "or a replication script that did not regenerate the results byte for byte" → no reproducibility failure occurred (m1).
9. "excludes differences larger than about six percentage points in either direction" → [−2.8, +5.6] pp (m2).
10. "with the first attempt retained in the ledger and its cost charged to the accounting" → retained as session rows only; the run record and tab_cost use the final attempt (M2.2); state both figures.
11. Any use of the "Violation" contrast rows (D−A, D−C = +0.014) and of confirmatory_claims_by_arm as mechanism evidence (H2, H3) → artefact-driven (C1, M5).
12. "Both are evaluator defects of the kind the pilot was meant to catch and did not" → four defects, two of them generic (rounding tolerance, negation regex) that a pilot with rounded outputs or underscore conclusion codes would have caught.

# 5. Overall verdict

The statistical pipeline does what the preregistration says: task-weighted rates, paired task-level differences, task-cluster percentile bootstrap (B = 5,000), sign-flip permutation (20,000), Holm over the three mechanism contrasts, balanced 24 × 4 × 3 design with no exclusion and the RC0094 rerun counted once; every headline cell was reproduced independently to the last digit, the negative controls hold, and the seeded-artefact validation of all 24 oracles passes under re-execution. The null result (Δ̂_DA = +0.014 [−0.028, +0.056], p = 1.00) is correct and robust. What is not supportable is the paper's account of where the 25 failures come from: all of them, including the two the draft attributes to executors and the single arm-specific violation, are rubric artefacts (an undefined feasibility label, a generator producing impossible values, a rel 1e−6 tolerance applied to rounded output, and a negation regex defeated by an underscore). Under a corrected rubric every arm is at 72/72, so the honest statement is a complete ceiling with zero discriminating power, and the mechanism evidence for H2/H3 (violation and claim contrasts) is noise generated by the evaluators. Separately, the arm-D review-and-repair loop never executed in the confirmatory stage (no review_repair session; the only revise was in the aborted attempt; one major reviewer finding was lost to a missing file), so claims about what the reviewer "resolved" must be withdrawn and H3 declared not evaluable. Required before any manuscript build: rewrite the failure paragraph and the review paragraph as above, extend or re-scope A-005, fix the robustness script, regenerate all tables and numbers.tex from a single atomic build, and disclose the prereg deviations in M2. With those changes the paper's central claim — no detectable protocol effect at this task difficulty with this model, at 1.7–2.6× the cost — stands.