# Pilot report (development tasks, 1 repeat per arm)
Generated 2026-10-30T03:57:27+00:00 from results/analysis_data/*.csv by analysis/pilot_report.py.
Runs assigned 20, evaluated 20, tasks 5 (F1-dev-01, F2-dev-01, F4-dev-01, F5-dev-01, F6-dev-01). Model reported by the client: claude-opus-5-5; client 2.1.314 (Claude Code).
# Outcomes by arm
| arm |
runs |
Y (task-weighted) |
success (any) |
runs with violation |
reproducible |
quality mean |
mean cost USD |
median cost |
max cost |
mean wall s |
max wall s |
sessions/run |
| A |
5 |
1.00 |
1.00 |
0 |
1.00 |
1.000 |
0.212 |
0.194 |
0.353 |
66 |
114 |
1.0 |
| B |
5 |
1.00 |
1.00 |
0 |
1.00 |
1.000 |
0.345 |
0.312 |
0.452 |
103 |
140 |
1.0 |
| C |
5 |
1.00 |
1.00 |
0 |
1.00 |
1.000 |
0.388 |
0.404 |
0.506 |
110 |
151 |
1.0 |
| D |
5 |
1.00 |
1.00 |
0 |
1.00 |
1.000 |
0.558 |
0.462 |
0.840 |
153 |
229 |
2.0 |
# Outcomes by task
| task |
Y A |
Y B |
Y C |
Y D |
cost A |
cost B |
cost C |
cost D |
| F1-dev-01 |
1 |
1 |
1 |
1 |
0.19 |
0.30 |
0.40 |
0.46 |
| F2-dev-01 |
1 |
1 |
1 |
1 |
0.14 |
0.31 |
0.26 |
0.39 |
| F4-dev-01 |
1 |
1 |
1 |
1 |
0.21 |
0.31 |
0.42 |
0.67 |
| F5-dev-01 |
1 |
1 |
1 |
1 |
0.16 |
0.36 |
0.35 |
0.43 |
| F6-dev-01 |
1 |
1 |
1 |
1 |
0.35 |
0.45 |
0.51 |
0.84 |
# Sessions
|
n |
cost_mean |
turns_mean |
duration_s |
| ('executor', 'initial') |
20 |
0.34 |
10.75 |
100.11 |
| ('reviewer', 'review') |
5 |
0.13 |
5.8 |
29.71 |
# Gate receipts (controller-side)
| origin |
passed |
n |
| controller |
1 |
10 |
| final_common_contract_check |
1 |
20 |
# Reviews (arm D)
| verdict |
n |
issues_mean |
| accept |
5 |
2.8 |
- Projected 24 tasks × 4 arms × 3 repeats: 288 runs ≈ 124 USD (1.15 × mean pilot cost 0.376), ≈ 1.1 h at 8 concurrent runs (mean pilot wall 108 s).
- Projected 24 × 4 × 2: 192 runs ≈ 83 USD (1.15 × mean pilot cost 0.376), ≈ 0.7 h at 8 concurrent runs (mean pilot wall 108 s).
- Projected 12 × 4 × 2: 96 runs ≈ 41 USD (1.15 × mean pilot cost 0.376), ≈ 0.4 h at 8 concurrent runs (mean pilot wall 108 s).
Scale decision rule (preregistration §6): choose the largest configuration whose projected cost fits the remaining envelope after a 150 USD reserve and whose projected time is ≤ 12 h.