Related work and novelty matrix (Gate A, 2026-11-01)
Sources: research/prior_art.csv (59 works, each verified on its arXiv / publisher / GitHub page on 2026-11-01 by a
web-research pass; evidence URL per row) and a direct code/doc audit of the four reference repositories shallow-cloned in
third_party/ (hashes below). Venue fields left empty in the CSV were not verifiable (ACM DL blocked fetches); they are
cited as arXiv preprints until re-verified at Gate G.
| repo | commit | date | what we read |
|---|---|---|---|
| JustVugg/colibri | 0a5504a1dc64f86fcd0fe7db298b0448c730e7e6 |
2026-10-31 | README (How it works, Benchmarks), docs/cluster.md, docs/CACHE_ROUTE.md, docs/metal.md, docs/METAL-M5MAX-PERF-REPORT.md, docs/routing-telemetry.md (titles) |
| ml-explore/mlx | bd63d3803008edda03ce74080955d9879e2ef1eb (version.h 0.32.5) |
2026-11-01 | mlx/distributed/{ring,mpi,jaccl,nccl}, docs/src/usage/distributed.rst (JACCL = RDMA over Thunderbolt 5, macOS ≥ 26.2, rdma_ctl enable), ops gather_qmm/gather_mm |
| ml-explore/mlx-lm | 736ab0d8c1d0edc79c3d798044fe6904e2fa23be |
2026-10-30 | models/switch_layers.py (SwitchGLU → gather_qmm, sort when ≥ 64 rows), models/olmoe.py, qwen3_moe.py (PipelineMixin), generate.py (wired limit, async eval), 50+ MoE architectures |
| ggml-org/llama.cpp | a91eab67d8389d979d29fdf6633b2edd0718eb4b |
2026-10-31 | common/arg.cpp (--cpu-moe, --n-cpu-moe, --override-tensor, --rpc), tools/rpc, ggml-metal backend |
1. What is already known (and therefore not claimed)
Expert offloading and SSD streaming. Per-expert offloading with an LRU cache was established by Eliseev & Mazur (2023) and
MoE-Infinity (Xue et al. 2024); SSD-tier streaming is demonstrated in Colibri (C engine: per-layer LRU + learned pinned hot set
.coli_usage, one pread per expert, O_DIRECT, loader pool, Metal backend opt-in, 2.24 tok/s GLM-5.2 int4 on an M5 Max 128 GB),
SSD-LLaMA (Liang et al. 2026) and FlashMoE (Kim et al. 2026, learned replacement beating LRU/LFU). Colibri explicitly states the
design principle we adopt: placement only decides speed, never which experts run.
Expert prediction. Pre-gated MoE (Hwang et al., ISCA 2024) and Read-ME (Cai et al. 2024) change the architecture to route one
layer ahead; Fate (Fang et al. 2025), ProMoE, ExpertFlow (He et al. 2026), LayerScope, DuoServe, SpecPrefetch and SeqMoE
(2025–2026) learn cross-layer predictors on GPUs/NPUs; Colibri's PILOT prefetches the next layer from the router lookahead
(GLM-5.2: 71.6 % predictable one layer ahead, per README). Liang et al. (ICLR 2026) quantify routing locality and warn that
not all models suit offloading. Cache-aware routing (Skliar et al. 2025, arXiv 2412.00099; Colibri CACHE_ROUTE) changes the
expert choice and is therefore excluded from TITAN-MoE by construction (correctness constraint).
Distributed inference on Macs. MLX ships ring (TCP/Thunderbolt), MPI and JACCL (RDMA over Thunderbolt 5, macOS ≥ 26.2, Mac Studio); Apple reports 816B-parameter MoE inference across 4 × M5 Ultra (tensor/pipeline sharding of whole layers). exo, prima.cpp, Petals and llama.cpp RPC shard layers or tensors, not experts. Colibri's cluster mode delegates routed experts to remote "expert workers" over persistent TCP with a deterministic weighted hash (weights ∝ measured disk speed), batch-union requests, q8 activations (not token-exact) and timeouts; it has no topology or latency model and no replication.
Apple Silicon characterization. powermetrics-based energy per token for dense models (GreenBench, Silicon Showdown), MoE on M-series with all experts resident (NPUMoE/ANE, BaseRT Metal, Alfarizy et al.). Silicon Showdown reports MoE on Apple Silicon is latency-bound rather than bandwidth-bound when resident — a finding our Gate B baseline is consistent with (OLMoE 4-bit: 494 tok/s decode on M5 Max with 432 MiB of expert weights touched per token, i.e. ≈ 0.2 TB/s effective).
2. Novelty matrix (hypotheses vs prior art)
| TITAN-MoE element | closest prior art | already shown there? | what remains open (our claim, to be tested) |
|---|---|---|---|
| H1 causal cross-layer/cross-token prefetch on Apple Silicon with an SSD tier | Colibri PILOT (1 layer ahead), Fate, SeqMoE, FlashMoE (GPU/x86) | prediction works on GPUs; Colibri has a single heuristic | head-to-head of no-prefetch / LRU / LFU / transition / learned predictors under identical Metal kernels and SSD, net of predictor overhead, on held-out prompt families |
| H2 topology-aware expert placement on heterogeneous Macs | ExFlow (GPU affinity), OrderMoE (edge servers), Colibri cluster (hash by disk speed) | datacenter GPUs only; Colibri has no topology model | cost-model placement using measured per-node compute, RAM, SSD, link latency/bandwidth; comparison with static/uniform/frequency/capacity placement |
| H3 remote RAM vs local SSD as the second tier | MLX JACCL, exo, llama.cpp RPC (whole shards) ; SSD-LLaMA, Klotski (local SSD) | never compared as expert cache tiers | measured tier model (10 GbE / Thunderbolt vs NVMe) and a coordinated 3-tier policy under fixed budgets |
| H4 fused Metal kernels for quantized grouped MoE | MLX gather_qmm, llama.cpp mul_mat_id, BaseRT, Colibri Metal |
strong stock kernels exist | gains only if profiling shows a bottleneck; end-to-end (not kernel-only) comparison with explicit tolerances |
| H5 latency × throughput × energy Pareto with adaptive orchestration | GreenBench, Silicon Showdown (dense, single node) | no energy for offloaded/distributed MoE on Macs | powermetrics-based J/token across budgets, nodes and policies |
Honest expectations recorded before experiments: on a 48–96 GB node a small MoE is fully resident and no cache policy can beat the all-resident baseline (H1/H3 only matter when the expert set exceeds the budget); distribution over Ethernet adds ≥ 100 µs per layer of RTT, so a 16-layer model would lose ≥ 1.6 ms per token versus 2 ms per token local decode — H2 can only win at scales where a single node cannot hold the model or where SSD streaming is slower than the link. These are the regimes the protocol targets (medium/large tiers, memory budgets < 100 %).
3. Baselines and comparability (from the audit)
- mlx-lm: same weights, same kernels → exact comparison (Gate B shows bit-identical logits).
- llama.cpp Metal: different quantization (K-quants) → speed reported with a precision flag + quality check.
- Colibri on macOS: OLMoE engine is CPU int8 (Metal backend covers GLM/Inkling/Kimi engines only) → reference only.