# WP30 statistical analysis plan (registered with the protocol freeze)

- **Unit of analysis:** one (document, question, condition, model pair) trial. Documents are the independent clusters; all
  questions of a document are dependent observations.
- **Quality endpoints:** synthetic/exec: `correct` ∈ {0,1} (exec graded by hidden tests: all-pass = 1; pass rate reported as
  secondary); public QA: LongBench F1 (max over reference answers) with the LongBench normalization.
  `quality_operational` sets failed/invalid attempts to 0 so that failures cannot vanish; conditional quality over successes is
  reported separately.
- **Resource endpoints:** serialized payload bytes (measured), cache tensor bytes (measured), scenario latencies assembled from
  recorded spans: cold = build + compress/serialize + load + receiver; warm per-query = load + receiver; amortized(N) =
  build/N + warm; peak MLX memory (prefill and decode recorded separately).
- **Uncertainty:** document-level cluster bootstrap, 2 000 replicates, percentile 95 % intervals, for means and for paired
  differences on identical instances. Bootstrap two-sided p-values for the registered primary family, Holm-corrected (6 tests,
  `protocol/hypotheses.md`). Everything else is exploratory and labelled so.
- **Latency:** p50/p95 and means with sample counts; timing trials in the core run alone on the GPU (pilot timings excluded).
- **Failures:** operational success rate over all scheduled trials; categories success/invalid_output/timeout/oom/incompatible/
  implementation_error/unavailable/budget_exhausted; retries keep both attempts.
- **No test-set tuning:** prompts, budgets, selection hyper-parameters, and the router are fixed on dev before the core run.