# WP29 — Limitations and the scope of claims actually supported

(Final numbers are in the manuscript and `results/estimates/`; this document states what each principal claim rests on and what it does not.)

## Claims supported by the evidence
1. **Null effect of protocol structure on verified completion for this agent and task suite.** 288 live confirmatory runs (24 tasks × 4 arms × 3 repeats; frozen design, seed 301); primary contrast D−A estimated with task-cluster bootstrap and permutation at the task level; interval excludes differences beyond ≈ ±6 percentage points. Evidence: `results/estimates/confirmatory_contrasts.csv`, `runs/state.sqlite`.
2. **Contract compliance improves under executable gates.** Post-hoc contract-check failures occurred only in arms A and B (counts in the manuscript). Evidence: `results/analysis_data/gate_receipts.csv`.
3. **Cost rises two- to three-fold with protocol, gates and review.** Client list-price estimates per run; reviewer sessions draw from the same envelope. Evidence: `results/analysis_data/sessions.csv`.
4. **Evaluator defects, not executor errors, account for every non-success of the frozen scoring.** Inspection of all 25 non-success runs plus an adversarial audit of all 24 oracles (four defects: feasibility label, generator artefact, rounding tolerance, regex false positive); the rubric-corrected sensitivity analysis (A-005/A-007) gives 100% completion in every arm and leaves the null unchanged. Corollary: the confirmatory stage has no discriminating power at the top.
5. **Arm D's review-repair loop never executed in the benchmark** (one `revise` in an aborted attempt; one review without a verdict file); H3 is not evaluable from the benchmark. Evidence: `results/estimates/sensitivity_rubric_*.csv`.
5. **Isolation and treatment delivery were verified, not assumed.** Sentinel probes (laptop and nodes), workspace-manifest verification before every launch, effective-context hashes per run, behavioural audit of every tool call (no outside-workspace references). Evidence: `reports/isolation_audit.md`, `runs/*/*/behaviour_audit.json`.

## Claims explicitly NOT made
- That protocols are useless for open-ended or long-horizon research (not tested: tasks are bounded, 10–25 min of agent work).
- That the result generalises to other models, client versions, or future provider behaviour (single model/version, one-week window).
- That the benchmark is externally independent (tasks, oracles and treatments authored within one organisation by sub-sessions of the same client family; no human verified reference values).
- That the evaluation was blinded or human-reviewed (deterministic programs; no human reviewer; cells "unavailable").
- That the study was autonomous (a human wrote the mandate and reviews the manuscript; builder effort is reported, preparation time unknown).
- That costs are billing figures (they are client estimates).
- That the reviewer in arm D is independent (same model family; fresh context only).

## Known defects and how they were handled
- Harness: evaluator workspace composition (A-002, before confirmatory runs), session-ledger key collision on resumed sessions (A-004, during the batch; one run re-executed under the attempt policy), tilde expansion and SQLite threading (smoke stage; recorded in the ledger).
- Oracles: F1-conf-01 feasibility rule, F6-conf-04 sample rule (A-005), F3-conf-02 rounding tolerance, F6-conf-01 regex (A-007) — after results; sensitivity analysis only.
- Launcher: unquoted prompt expansion delivered a one-word user prompt in every session (A-006, symmetric; post-fix verification unchanged).
- Captures: two Safari window-level captures produced blank page areas (recorded as failures in the screenshot manifest; replaced by composited region captures); one native capture matched the builder's own session window (retained, labelled).
- Design: F2-conf-04 and F6-conf-03 share data and the core regression (independent authoring by two sub-sessions); reported, both kept.

## Residual risks
Executors share a user account on nodes (sibling workspaces readable; audited, not prevented); the gate program is world-readable; the arm-D reviewer could in principle read the executor transcript (not observed).