# Benchmark protocol (WP33 TITAN-MoE) — v0.1, 2026-11-01 (Gate A draft; frozen at Gate C before any optimization claim)

# 1. Models (exact revisions pinned in benchmarks/configs/models.yaml)

Tokenizers: the ones shipped in each repo. Decoding: greedy (temperature 0) for correctness and throughput; sampling only in quality checks with fixed seeds.

# 2. Baselines

• MLX-LM (python -m mlx_lm generate, same venv, same weights, same revision) — primary, exactly comparable precision.
• llama.cpp (llama-bench, llama-cli, Metal backend, commit a91eab6) on a GGUF of the same base checkpoint; the quantization scheme differs (Q4_K_M / Q4_0 vs MLX affine g64) → reported with a NON-COMPARABLE PRECISION flag and a quality check (perplexity on the same held-out text) next to the speed numbers.
• Colibri (0a5504a): OLMoE engine (int8 conversion, CPU; Vulkan/CUDA not on macOS; Metal backend only for GLM/Inkling/Kimi engines) → OLMoE on Colibri is CPU-only int8: reported as a different-precision, different-device reference, never as a same-conditions competitor. For the large tier (GLM-5.2 int4) Colibri is compared only if TITAN-MoE runs GLM-5.2 (not planned).
• Internal controls: TITAN reference backend with all experts resident (upper bound), no-prefetch demand loading, LRU, LFU, static placement, single-node.

# 3. Workloads (benchmarks/configs/workloads.yaml)

Prompt families (held-out split by family, never used to fit predictors): general chat (Alpaca-style instructions), code (HumanEval-like prompts), reasoning (GSM8K-style), long-context summarization (2k–8k tokens), and a "drift" family that switches domain mid-session to test changing expert popularity. Output lengths 64 / 256 / 1024. Batch 1 (latency) and 4 / 8 (throughput) where the backend supports batching. Cold start = process start with purged page cache; warm = second run.

# 4. Metrics (all logged per request as JSON lines in benchmarks/raw/<gate>/<exp>/<run>.jsonl)

prefill tok/s · decode tok/s · TTFT · inter-token latency p50/p95/p99 · total latency · expert hit rate (local RAM / remote RAM / SSD) · stall time (ms waiting for weights) · bytes read from SSD · bytes over network · peak RSS and mx.get_peak_memory() · GPU utilization (powermetrics --samplers gpu_power, sudo) · energy per token (powermetrics CPU+GPU+ANE package power × wall time / tokens) · correctness (argmax agreement and max |Δlogit| vs MLX-LM reference) · stability (variance across repeats).

# 5. Procedure

Fixed seeds; exact command recorded in each JSON; warm-up run discarded; benchmark order randomized per block (seeded); ≥ 5 repeats (bootstrap 95 % CI, 10 000 resamples); temperature/thermal pressure sampled every 1 s during runs (powermetrics --samplers thermal); hardware inventory hash attached; timers = time.perf_counter_ns on each node with a one-way clock-offset estimate from an NTP-style ping exchange before each distributed run; compilation/warm-up measured separately from steady state; different precisions never pooled in the same table without a precision column.

# 6. Ablation plan (factorial where feasible)

predictor ∈ {none, LRU, LFU, transition, learned} × prefetch depth ∈ {0, 1, 2 layers} × memory budget ∈ {25 %, 50 %, 75 %, 100 % of expert bytes} × placement ∈ {static-uniform, frequency, capacity-aware, topology-aware} × replication ∈ {0, 1 hot set} × transport ∈ {TCP LAN, (Thunderbolt/JACCL if cabled)} × kernels ∈ {stock MLX, TITAN Metal} × batch ∈ {1, 4, 8}. Interactions reported, including losses.

# 7. Statistical audit (Gate G)

Pre-registered primary endpoints: H1 decode tok/s at 50 % budget (predictor vs no-prefetch); H2 decode tok/s 2–4 nodes (topology-aware vs static); H3 tok/s vs budget curve AUC; H4 end-to-end decode tok/s (TITAN kernels vs stock); H5 hypervolume of the (latency, throughput, energy) Pareto set. Effects reported with CIs; p-values secondary.