# Benchmark protocol (WP33 TITAN-MoE) — v0.1, 2026-11-01 (Gate A draft; frozen at Gate C before any optimization claim)

## 1. Models (exact revisions pinned in `benchmarks/configs/models.yaml`)
| tier | model | HF repo / revision | licence | layers | experts | top-k | shared | active / total params | quantization |
|---|---|---|---|---|---|---|---|---|---|
| small | OLMoE-1B-7B-0125-Instruct | `mlx-community/OLMoE-1B-7B-0125-Instruct-4bit` @ `e2384419` | Apache-2.0 | 16 | 64 | 8 | none | 1.3B / 6.9B | affine 4-bit, g64 (MLX) |
| medium (Gate C+, needs download authorization ≤ 20 GB → granted by default rule in CLAUDE.md: 17.2 GB) | Qwen3-30B-A3B-Instruct-2507 | `mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit` @ `e9675aa3` | Apache-2.0 | 48 | 128 | 8 | none | 3.3B / 30.5B | affine 4-bit, g64 |
| large (Gate F, requires explicit authorization: 62 GB) | gpt-oss-120b | `mlx-community/gpt-oss-120b-MXFP4-Q4` @ `bce781be` | Apache-2.0 | 36 | 128 | 4 | none | 5.1B / 117B | MXFP4 experts |

Tokenizers: the ones shipped in each repo. Decoding: greedy (temperature 0) for correctness and throughput; sampling only in
quality checks with fixed seeds.

## 2. Baselines
- **MLX-LM** (`python -m mlx_lm generate`, same venv, same weights, same revision) — primary, exactly comparable precision.
- **llama.cpp** (`llama-bench`, `llama-cli`, Metal backend, commit `a91eab6`) on a GGUF of the *same* base checkpoint; the
  quantization scheme differs (Q4_K_M / Q4_0 vs MLX affine g64) → reported with a NON-COMPARABLE PRECISION flag and a quality
  check (perplexity on the same held-out text) next to the speed numbers.
- **Colibri** (`0a5504a`): OLMoE engine (int8 conversion, CPU; Vulkan/CUDA not on macOS; Metal backend only for GLM/Inkling/Kimi
  engines) → OLMoE on Colibri is CPU-only int8: reported as a different-precision, different-device reference, never as a
  same-conditions competitor. For the large tier (GLM-5.2 int4) Colibri is compared only if TITAN-MoE runs GLM-5.2 (not planned).
- Internal controls: TITAN reference backend with all experts resident (upper bound), no-prefetch demand loading, LRU, LFU,
  static placement, single-node.

## 3. Workloads (`benchmarks/configs/workloads.yaml`)
Prompt families (held-out split by family, never used to fit predictors): general chat (Alpaca-style instructions), code
(HumanEval-like prompts), reasoning (GSM8K-style), long-context summarization (2k–8k tokens), and a "drift" family that
switches domain mid-session to test changing expert popularity. Output lengths 64 / 256 / 1024. Batch 1 (latency) and 4 / 8
(throughput) where the backend supports batching. Cold start = process start with purged page cache; warm = second run.

## 4. Metrics (all logged per request as JSON lines in `benchmarks/raw/<gate>/<exp>/<run>.jsonl`)
prefill tok/s · decode tok/s · TTFT · inter-token latency p50/p95/p99 · total latency · expert hit rate (local RAM / remote RAM / SSD)
· stall time (ms waiting for weights) · bytes read from SSD · bytes over network · peak RSS and `mx.get_peak_memory()` ·
GPU utilization (`powermetrics --samplers gpu_power`, sudo) · energy per token (powermetrics CPU+GPU+ANE package power × wall time
/ tokens) · correctness (argmax agreement and max |Δlogit| vs MLX-LM reference) · stability (variance across repeats).

## 5. Procedure
Fixed seeds; exact command recorded in each JSON; warm-up run discarded; benchmark order randomized per block (seeded);
≥ 5 repeats (bootstrap 95 % CI, 10 000 resamples); temperature/thermal pressure sampled every 1 s during runs
(`powermetrics --samplers thermal`); hardware inventory hash attached; timers = `time.perf_counter_ns` on each node with a
one-way clock-offset estimate from an NTP-style ping exchange before each distributed run; compilation/warm-up measured
separately from steady state; different precisions never pooled in the same table without a precision column.

## 6. Ablation plan (factorial where feasible)
predictor ∈ {none, LRU, LFU, transition, learned} × prefetch depth ∈ {0, 1, 2 layers} × memory budget ∈ {25 %, 50 %, 75 %, 100 %
of expert bytes} × placement ∈ {static-uniform, frequency, capacity-aware, topology-aware} × replication ∈ {0, 1 hot set} ×
transport ∈ {TCP LAN, (Thunderbolt/JACCL if cabled)} × kernels ∈ {stock MLX, TITAN Metal} × batch ∈ {1, 4, 8}.
Interactions reported, including losses.

## 7. Statistical audit (Gate G)
Pre-registered primary endpoints: H1 decode tok/s at 50 % budget (predictor vs no-prefetch); H2 decode tok/s 2–4 nodes
(topology-aware vs static); H3 tok/s vs budget curve AUC; H4 end-to-end decode tok/s (TITAN kernels vs stock); H5 hypervolume of
the (latency, throughput, energy) Pareto set. Effects reported with CIs; p-values secondary.