# Pilot report (development tasks, 1 repeat per arm)
Generated 2026-10-30T04:03:02+00:00 from results/analysis_data/*.csv by analysis/pilot_report.py.
Runs assigned 24, evaluated 24, tasks 6 (F1-dev-01, F2-dev-01, F3-dev-01, F4-dev-01, F5-dev-01, F6-dev-01). Model reported by the client: claude-opus-5-5; client 2.1.314 (Claude Code).
# Outcomes by arm
| arm |
runs |
Y (task-weighted) |
success (any) |
runs with violation |
reproducible |
quality mean |
mean cost USD |
median cost |
max cost |
mean wall s |
max wall s |
sessions/run |
| A |
6 |
0.83 |
0.83 |
0 |
1.00 |
0.990 |
0.221 |
0.204 |
0.353 |
70 |
114 |
1.0 |
| B |
6 |
1.00 |
1.00 |
0 |
1.00 |
1.000 |
0.369 |
0.334 |
0.489 |
111 |
154 |
1.0 |
| C |
6 |
1.00 |
1.00 |
0 |
1.00 |
1.000 |
0.406 |
0.412 |
0.506 |
117 |
151 |
1.0 |
| D |
6 |
1.00 |
1.00 |
0 |
1.00 |
1.000 |
0.579 |
0.565 |
0.840 |
162 |
229 |
2.0 |
# Outcomes by task
| task |
Y A |
Y B |
Y C |
Y D |
cost A |
cost B |
cost C |
cost D |
| F1-dev-01 |
1 |
1 |
1 |
1 |
0.19 |
0.30 |
0.40 |
0.46 |
| F2-dev-01 |
1 |
1 |
1 |
1 |
0.14 |
0.31 |
0.26 |
0.39 |
| F3-dev-01 |
0 |
1 |
1 |
1 |
0.27 |
0.49 |
0.50 |
0.68 |
| F4-dev-01 |
1 |
1 |
1 |
1 |
0.21 |
0.31 |
0.42 |
0.67 |
| F5-dev-01 |
1 |
1 |
1 |
1 |
0.16 |
0.36 |
0.35 |
0.43 |
| F6-dev-01 |
1 |
1 |
1 |
1 |
0.35 |
0.45 |
0.51 |
0.84 |
# Sessions
|
n |
cost_mean |
turns_mean |
duration_s |
| ('executor', 'initial') |
24 |
0.36 |
11.04 |
106.47 |
| ('reviewer', 'review') |
6 |
0.14 |
5.83 |
32.07 |
# Gate receipts (controller-side)
| origin |
passed |
n |
| controller |
1 |
12 |
| final_common_contract_check |
1 |
24 |
# Reviews (arm D)
| verdict |
n |
issues_mean |
| accept |
6 |
2.83333 |
- Projected 24 tasks × 4 arms × 3 repeats: 288 runs ≈ 130 USD (1.15 × mean pilot cost 0.394), ≈ 1.1 h at 8 concurrent runs (mean pilot wall 115 s).
- Projected 24 × 4 × 2: 192 runs ≈ 87 USD (1.15 × mean pilot cost 0.394), ≈ 0.8 h at 8 concurrent runs (mean pilot wall 115 s).
- Projected 12 × 4 × 2: 96 runs ≈ 43 USD (1.15 × mean pilot cost 0.394), ≈ 0.4 h at 8 concurrent runs (mean pilot wall 115 s).
Scale decision rule (preregistration §6): choose the largest configuration whose projected cost fits the remaining envelope after a 150 USD reserve and whose projected time is ≤ 12 h.