# WP30 statistical analysis plan (registered with the protocol freeze)

• Unit of analysis: one (document, question, condition, model pair) trial. Documents are the independent clusters; all questions of a document are dependent observations.
• Quality endpoints: synthetic/exec: correct ∈ {0,1} (exec graded by hidden tests: all-pass = 1; pass rate reported as secondary); public QA: LongBench F1 (max over reference answers) with the LongBench normalization. quality_operational sets failed/invalid attempts to 0 so that failures cannot vanish; conditional quality over successes is reported separately.
• Resource endpoints: serialized payload bytes (measured), cache tensor bytes (measured), scenario latencies assembled from recorded spans: cold = build + compress/serialize + load + receiver; warm per-query = load + receiver; amortized(N) = build/N + warm; peak MLX memory (prefill and decode recorded separately).
• Uncertainty: document-level cluster bootstrap, 2 000 replicates, percentile 95 % intervals, for means and for paired differences on identical instances. Bootstrap two-sided p-values for the registered primary family, Holm-corrected (6 tests, protocol/hypotheses.md). Everything else is exploratory and labelled so.
• Latency: p50/p95 and means with sample counts; timing trials in the core run alone on the GPU (pilot timings excluded).
• Failures: operational success rate over all scheduled trials; categories success/invalid_output/timeout/oom/incompatible/ implementation_error/unavailable/budget_exhausted; retries keep both attempts.
• No test-set tuning: prompts, budgets, selection hyper-parameters, and the router are fixed on dev before the core run.