1# Pilot report (development tasks, 1 repeat per arm)23Generated 2026-10-30T04:03:02+00:00 from `results/analysis_data/*.csv` by `analysis/pilot_report.py`.45Runs assigned 24, evaluated 24, tasks 6 (F1-dev-01, F2-dev-01, F3-dev-01, F4-dev-01, F5-dev-01, F6-dev-01). Model reported by the client: claude-opus-5-5; client 2.1.314 (Claude Code).67## Outcomes by arm89| arm | runs | Y (task-weighted) | success (any) | runs with violation | reproducible | quality mean | mean cost USD | median cost | max cost | mean wall s | max wall s | sessions/run |10|---|---|---|---|---|---|---|---|---|---|---|---|---|11| A | 6 | 0.83 | 0.83 | 0 | 1.00 | 0.990 | 0.221 | 0.204 | 0.353 | 70 | 114 | 1.0 |12| B | 6 | 1.00 | 1.00 | 0 | 1.00 | 1.000 | 0.369 | 0.334 | 0.489 | 111 | 154 | 1.0 |13| C | 6 | 1.00 | 1.00 | 0 | 1.00 | 1.000 | 0.406 | 0.412 | 0.506 | 117 | 151 | 1.0 |14| D | 6 | 1.00 | 1.00 | 0 | 1.00 | 1.000 | 0.579 | 0.565 | 0.840 | 162 | 229 | 2.0 |1516## Outcomes by task1718| task | Y A | Y B | Y C | Y D | cost A | cost B | cost C | cost D |19|---|---|---|---|---|---|---|---|---|20| F1-dev-01 | 1 | 1 | 1 | 1 | 0.19 | 0.30 | 0.40 | 0.46 |21| F2-dev-01 | 1 | 1 | 1 | 1 | 0.14 | 0.31 | 0.26 | 0.39 |22| F3-dev-01 | 0 | 1 | 1 | 1 | 0.27 | 0.49 | 0.50 | 0.68 |23| F4-dev-01 | 1 | 1 | 1 | 1 | 0.21 | 0.31 | 0.42 | 0.67 |24| F5-dev-01 | 1 | 1 | 1 | 1 | 0.16 | 0.36 | 0.35 | 0.43 |25| F6-dev-01 | 1 | 1 | 1 | 1 | 0.35 | 0.45 | 0.51 | 0.84 |2627## Sessions2829| | n | cost_mean | turns_mean | duration_s |30|:------------------------|----:|------------:|-------------:|-------------:|31| ('executor', 'initial') | 24 | 0.36 | 11.04 | 106.47 |32| ('reviewer', 'review') | 6 | 0.14 | 5.83 | 32.07 |3334## Gate receipts (controller-side)3536| origin | passed | n |37|:----------------------------|---------:|----:|38| controller | 1 | 12 |39| final_common_contract_check | 1 | 24 |4041## Reviews (arm D)4243| verdict | n | issues_mean |44|:----------|----:|--------------:|45| accept | 6 | 2.83333 |4647- Projected 24 tasks × 4 arms × 3 repeats: 288 runs ≈ 130 USD (1.15 × mean pilot cost 0.394), ≈ 1.1 h at 8 concurrent runs (mean pilot wall 115 s).48- Projected 24 × 4 × 2: 192 runs ≈ 87 USD (1.15 × mean pilot cost 0.394), ≈ 0.8 h at 8 concurrent runs (mean pilot wall 115 s).49- Projected 12 × 4 × 2: 96 runs ≈ 43 USD (1.15 × mean pilot cost 0.394), ≈ 0.4 h at 8 concurrent runs (mean pilot wall 115 s).5051Scale decision rule (preregistration §6): choose the largest configuration whose projected cost fits the remaining envelope after a 150 USD reserve and whose projected time is ≤ 12 h.52