# Pilot report (development tasks, 1 repeat per arm)

Generated 2026-10-30T04:03:02+00:00 from `results/analysis_data/*.csv` by `analysis/pilot_report.py`.

Runs assigned 24, evaluated 24, tasks 6 (F1-dev-01, F2-dev-01, F3-dev-01, F4-dev-01, F5-dev-01, F6-dev-01). Model reported by the client: claude-opus-5-5; client 2.1.314 (Claude Code).

## Outcomes by arm

| arm | runs | Y (task-weighted) | success (any) | runs with violation | reproducible | quality mean | mean cost USD | median cost | max cost | mean wall s | max wall s | sessions/run |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A | 6 | 0.83 | 0.83 | 0 | 1.00 | 0.990 | 0.221 | 0.204 | 0.353 | 70 | 114 | 1.0 |
| B | 6 | 1.00 | 1.00 | 0 | 1.00 | 1.000 | 0.369 | 0.334 | 0.489 | 111 | 154 | 1.0 |
| C | 6 | 1.00 | 1.00 | 0 | 1.00 | 1.000 | 0.406 | 0.412 | 0.506 | 117 | 151 | 1.0 |
| D | 6 | 1.00 | 1.00 | 0 | 1.00 | 1.000 | 0.579 | 0.565 | 0.840 | 162 | 229 | 2.0 |

## Outcomes by task

| task | Y A | Y B | Y C | Y D | cost A | cost B | cost C | cost D |
|---|---|---|---|---|---|---|---|---|
| F1-dev-01 | 1 | 1 | 1 | 1 | 0.19 | 0.30 | 0.40 | 0.46 |
| F2-dev-01 | 1 | 1 | 1 | 1 | 0.14 | 0.31 | 0.26 | 0.39 |
| F3-dev-01 | 0 | 1 | 1 | 1 | 0.27 | 0.49 | 0.50 | 0.68 |
| F4-dev-01 | 1 | 1 | 1 | 1 | 0.21 | 0.31 | 0.42 | 0.67 |
| F5-dev-01 | 1 | 1 | 1 | 1 | 0.16 | 0.36 | 0.35 | 0.43 |
| F6-dev-01 | 1 | 1 | 1 | 1 | 0.35 | 0.45 | 0.51 | 0.84 |

## Sessions

|                         |   n |   cost_mean |   turns_mean |   duration_s |
|:------------------------|----:|------------:|-------------:|-------------:|
| ('executor', 'initial') |  24 |        0.36 |        11.04 |       106.47 |
| ('reviewer', 'review')  |   6 |        0.14 |         5.83 |        32.07 |

## Gate receipts (controller-side)

| origin                      |   passed |   n |
|:----------------------------|---------:|----:|
| controller                  |        1 |  12 |
| final_common_contract_check |        1 |  24 |

## Reviews (arm D)

| verdict   |   n |   issues_mean |
|:----------|----:|--------------:|
| accept    |   6 |       2.83333 |

- Projected 24 tasks × 4 arms × 3 repeats: 288 runs ≈ 130 USD (1.15 × mean pilot cost 0.394), ≈ 1.1 h at 8 concurrent runs (mean pilot wall 115 s).
- Projected 24 × 4 × 2: 192 runs ≈ 87 USD (1.15 × mean pilot cost 0.394), ≈ 0.8 h at 8 concurrent runs (mean pilot wall 115 s).
- Projected 12 × 4 × 2: 96 runs ≈ 43 USD (1.15 × mean pilot cost 0.394), ≈ 0.4 h at 8 concurrent runs (mean pilot wall 115 s).

Scale decision rule (preregistration §6): choose the largest configuration whose projected cost fits the remaining envelope after a 150 USD reserve and whose projected time is ≤ 12 h.