# WP29 — Limitations and the scope of claims actually supported

(Final numbers are in the manuscript and results/estimates/; this document states what each principal claim rests on and what it does not.)

# Claims supported by the evidence

1. Null effect of protocol structure on verified completion for this agent and task suite. 288 live confirmatory runs (24 tasks × 4 arms × 3 repeats; frozen design, seed 301); primary contrast D−A estimated with task-cluster bootstrap and permutation at the task level; interval excludes differences beyond ≈ ±6 percentage points. Evidence: results/estimates/confirmatory_contrasts.csv, runs/state.sqlite.
2. Contract compliance improves under executable gates. Post-hoc contract-check failures occurred only in arms A and B (counts in the manuscript). Evidence: results/analysis_data/gate_receipts.csv.
3. Cost rises two- to three-fold with protocol, gates and review. Client list-price estimates per run; reviewer sessions draw from the same envelope. Evidence: results/analysis_data/sessions.csv.
4. Evaluator defects, not executor errors, account for every non-success of the frozen scoring. Inspection of all 25 non-success runs plus an adversarial audit of all 24 oracles (four defects: feasibility label, generator artefact, rounding tolerance, regex false positive); the rubric-corrected sensitivity analysis (A-005/A-007) gives 100% completion in every arm and leaves the null unchanged. Corollary: the confirmatory stage has no discriminating power at the top.
5. Arm D's review-repair loop never executed in the benchmark (one revise in an aborted attempt; one review without a verdict file); H3 is not evaluable from the benchmark. Evidence: results/estimates/sensitivity_rubric_*.csv.
6. Isolation and treatment delivery were verified, not assumed. Sentinel probes (laptop and nodes), workspace-manifest verification before every launch, effective-context hashes per run, behavioural audit of every tool call (no outside-workspace references). Evidence: reports/isolation_audit.md, runs/*/*/behaviour_audit.json.

# Claims explicitly NOT made

• That protocols are useless for open-ended or long-horizon research (not tested: tasks are bounded, 10–25 min of agent work).
• That the result generalises to other models, client versions, or future provider behaviour (single model/version, one-week window).
• That the benchmark is externally independent (tasks, oracles and treatments authored within one organisation by sub-sessions of the same client family; no human verified reference values).
• That the evaluation was blinded or human-reviewed (deterministic programs; no human reviewer; cells "unavailable").
• That the study was autonomous (a human wrote the mandate and reviews the manuscript; builder effort is reported, preparation time unknown).
• That costs are billing figures (they are client estimates).
• That the reviewer in arm D is independent (same model family; fresh context only).

# Known defects and how they were handled

• Harness: evaluator workspace composition (A-002, before confirmatory runs), session-ledger key collision on resumed sessions (A-004, during the batch; one run re-executed under the attempt policy), tilde expansion and SQLite threading (smoke stage; recorded in the ledger).
• Oracles: F1-conf-01 feasibility rule, F6-conf-04 sample rule (A-005), F3-conf-02 rounding tolerance, F6-conf-01 regex (A-007) — after results; sensitivity analysis only.
• Launcher: unquoted prompt expansion delivered a one-word user prompt in every session (A-006, symmetric; post-fix verification unchanged).
• Captures: two Safari window-level captures produced blank page areas (recorded as failures in the screenshot manifest; replaced by composited region captures); one native capture matched the builder's own session window (retained, labelled).
• Design: F2-conf-04 and F6-conf-03 share data and the core regression (independent authoring by two sub-sessions); reported, both kept.

# Residual risks

Executors share a user account on nodes (sibling workspaces readable; audited, not prevented); the gate program is world-readable; the arm-D reviewer could in principle read the executor transcript (not observed).