1# WP30 statistical analysis plan (registered with the protocol freeze)23- **Unit of analysis:** one (document, question, condition, model pair) trial. Documents are the independent clusters; all4 questions of a document are dependent observations.5- **Quality endpoints:** synthetic/exec: `correct` ∈ {0,1} (exec graded by hidden tests: all-pass = 1; pass rate reported as6 secondary); public QA: LongBench F1 (max over reference answers) with the LongBench normalization.7 `quality_operational` sets failed/invalid attempts to 0 so that failures cannot vanish; conditional quality over successes is8 reported separately.9- **Resource endpoints:** serialized payload bytes (measured), cache tensor bytes (measured), scenario latencies assembled from10 recorded spans: cold = build + compress/serialize + load + receiver; warm per-query = load + receiver; amortized(N) =11 build/N + warm; peak MLX memory (prefill and decode recorded separately).12- **Uncertainty:** document-level cluster bootstrap, 2 000 replicates, percentile 95 % intervals, for means and for paired13 differences on identical instances. Bootstrap two-sided p-values for the registered primary family, Holm-corrected (6 tests,14 `protocol/hypotheses.md`). Everything else is exploratory and labelled so.15- **Latency:** p50/p95 and means with sample counts; timing trials in the core run alone on the GPU (pilot timings excluded).16- **Failures:** operational success rate over all scheduled trials; categories success/invalid_output/timeout/oom/incompatible/17 implementation_error/unavailable/budget_exhausted; retries keep both attempts.18- **No test-set tuning:** prompts, budgets, selection hyper-parameters, and the router are fixed on dev before the core run.19