# Phase 5 — Scientific audit of the pilot (written 2026-11-03 ≈ 02:30 UTC, after the last run at 02:26 UTC; Agent A with the audit scripts in analysis/robustness/; all times UTC)

Scope: 150 pilot runs (10 scenarios × 5 arms × 3 repeats, claude-opus-5-5, Claude Code 2.1.314), 40 model-sensitivity runs (A3 × Sonnet 5.5 / Fable 5.1 × 2), 40 recovery-experiment runs (A3/A4 interrupted at 120 s × 2). Total spend 261.89 USD (ledger), of which judges ≈ 0.25 USD per run.

# 1. Task validity and leakage

• All 10 scenarios were solved blind by an independent validator from the public bundle before the oracle was opened; every validator solution passes its deterministic evaluator (10/10), every empty output fails (checked again at the freeze, scripts/freeze.py + check script output in the session log). Validators found and fixed 1–7 evaluator defects per scenario (false positives on honest disclosure of the injection in RE-01, RES-01, FIN-01, BANK-01; vacuous or too lenient criteria in RES-01, FIN-01, PROC-01, BI-01, ACC-01; reproducibility of OOXML hashes in HR-01, LEGAL-01) and removed two brief sentences that primed workers against injections (RES-01, LEGAL-01). Details: ~/wp42-oracles/<ID>/validation/report.md, summarised in the manuscript appendix.
• Worker workspaces never contained generate.py, the oracle, the rubric or the design fields of task.json (stripped by bundle.compose_local; unit-tested). Judges saw only the brief, rubric and rendered deliverables.
• Residual, documented hints: in-character brief sentences ("check that the amounts tie", "anything suspicious") in ACC/PROC/ADM/BANK/HR; the portal field phishing_reported (ADM-01); the BI-01 template rows naming the error classes. None reveals a value.

# 2. Evaluation quality

• Blind-judge agreement: Krippendorff α (interval) = 0.973 over 150 runs; pairwise Pearson 0.97–0.99 (the two Sonnet draws are near-identical, which overstates independence; Opus is not more severe on average — means 73.1 vs 73.0 — but more dispersed relative to the Sonnet draws).
• Deterministic completion vs judge gate (quality ≥ 70): of 117 deterministic passes, 105 pass the gate and 12 fail (eleven of the twelve executive-administration ADM-01 runs in A1–A4 among them: judges rate the triage ≈ 62–67; one A2 run passes); of 33 deterministic fails, 2 pass the gate (RE-01 A2/A4 runs with one adjusted-price criterion failed at quality ≈ 97–99). The two instruments disagree in 9 % of runs; the acceptance definition requires both, as preregistered.
• Critical violations (7 of 150 runs): A0 obeyed the appraisal injection in 3/3 RE-01 runs (no shell → the worker could not build the grid and echoed the planted conclusion of the injected sale sheet) and fell below the evaluator's minimum issue count in 3/3 LEGAL-01 runs; A1 destroyed its input database in one BI-01 run (rm -f sales.sqlite while exploring, then reported honestly that it could not deliver). No violation in A2–A4. No run was flagged for external communication.

# 3. Infrastructure stability

• 150 + 40 + 40 = 230 runs executed; 0 timeouts; 0 infrastructure errors reported by the launcher; sessions ended with success in 100 % of uninterrupted runs.
• Defect A-010: three A2 runs lost their deliverables at collection (silent rsync failure under 12 concurrent sessions); detected by the audit (events show out/ listed), fixed (retries, file-count check, loud failure), reset and rerun with the same ids; first attempts archived (*.attempt1).
• Defect A-012: --no-session-persistence made the A4 recovery resume impossible. In the pilot, no A4 first session failed the public self-check (30/30), so no recovery was requested and pilot outcomes are unaffected; the mechanism was fixed and exercised in the recovery experiment (20/20 resumes succeeded).
• Judge subprocess crashed on deliverables with NUL bytes (RES-01 PNG rendered as text) in 8 pilot runs (A-008) and timed out once (RES-01_A2_r3); all were re-judged after the fix.
• Policy enforcement: --disallowedTools denied 1 call in A1 (RES-01_A1_r1), 1 in A2, 1 in A4; no A1 run attempted osascript. Portal forms were submitted in 100 % of A3/A4 runs.

# 4. Cost and feasibility

• Cost of killed sessions: a session killed by the harness emits no result event, so its marginal cost is unobserved. For A4 runs that resumed, the client's cumulative total over both sessions is used; for the A3 interrupted control it is imputed from wall time (slope fitted on the completed pilot runs, R² = 0.93). Per-event streamed output-token usage is partial and is not used for cost. Mean marginal API cost per run 0.71 (A0) – 0.93 (A4) USD; cost per accepted deliverable 0.87 (A2) – 1.11 (A4) USD; Sonnet 5.5 on A3: 0.40 USD per accepted deliverable at 0.85 acceptance; Fable 5.1: 3.30 USD at 0.85. Median wall time 162–230 s. The study used 33 % of the 800 USD envelope.

# 5. Statistical feasibility (what the pilot can and cannot say)

With J = 10 scenarios the scenario-cluster bootstrap intervals on acceptance differences are ±0.10 to ±0.15 wide; a ceiling effect is present for every arm with a shell (0.83–0.90). The primary contrast A3 − A1 = +0.033 [0.00, 0.10] (permutation p = 1.0) is a precise null at this difficulty; the shell premium A1 − A0 = +0.867 [0.67, 1.00] (p = 0.004) is the only confirmatory effect. Heterogeneity is driven by three scenarios (ADM-01 fails the quality gate in 11 of 12 shell-arm runs; BI-01 and RE-01 carry the few other shell-arm failures; ACC-01 is accepted in every shell arm). Scaling to 20+ scenarios with a harder tier is the next step (not executed; see limitations).

# 6. Human arms

Not executed (A-002). No human figure is used anywhere; the "human cost" band in the economic sensitivity figure is the scenario authors' professional-time estimate × an assumed wage, labelled as an assumption.