SPB Git

spb/wp33_uqo

Public MIT
1.5 KB · 16 lines latex
1\newcommand{\WPabstract}{%2Sparse mixture-of-experts (MoE) language models touch only a few experts per token, so a model whose weights exceed a3machine's memory can still be served if the right experts are in the right place at the right time. This working paper4reports the design, implementation and measurement of \titan{}, an MLX-native inference runtime for Apple Silicon that5manages expert residency with bounded per-layer slot tables, loads experts on demand from local NVMe, prefetches them with6causal predictors, and (in its distributed mode) executes them on other Macs of a heterogeneous cluster connected by7gigabit Ethernet. Every design choice is validated against mlx-lm on identical weights: with all experts resident the runtime is8bit-identical in logits and within one percent of mlx-lm's decode throughput; with bounded residency the router's choices and9the logits are unchanged by construction. We state five hypotheses (causal prefetch, topology-aware placement, hierarchical10caching, fused Metal kernels, and adaptive orchestration on the latency--throughput--energy frontier) and test each one with a11pre-registered protocol on a live inventory of a 21-node Mac cluster. This version (0.1) is a \emph{protocol and preliminary}12working paper: it contains the verified hardware inventory, the prior-art audit, the single-node runtime and its correctness13evidence, and the first controlled single-node experiments; distributed and energy experiments are reported only where their14raw logs exist, and pending experiments are explicitly labelled as such.%15}16
Files · f419985eab99e178b5c98c31ba0a179ada784682
Menu

Theme