SPB Git

spb/wp33_uqo

Public MIT
2.1 KB · 21 lines latex
1\newcommand{\WPabstract}{%2Sparse mixture-of-experts (MoE) language models touch only a few experts per token, so a model whose weights exceed a3machine's memory can still be served if the right experts are in the right place at the right time. This working paper4reports the design, implementation and measurement of \titan{}, an MLX-native inference runtime for Apple Silicon that5manages expert residency with bounded per-layer slot tables, loads experts on demand from local NVMe, prefetches them with6causal predictors, and (in its distributed mode) executes them on other Macs of a heterogeneous cluster connected by7gigabit Ethernet. Every design choice is validated against mlx-lm on identical weights: with all experts resident the runtime is8bit-identical in logits and within one percent of mlx-lm's decode throughput; with bounded residency the router's choices and9the logits are unchanged by construction. We state five hypotheses (causal prefetch, topology-aware placement, hierarchical10caching, fused Metal kernels, and adaptive orchestration on the latency--throughput--energy frontier) and test each one with a11pre-registered protocol on a live inventory of a 21-node Mac cluster. This version (0.1) is a \emph{preliminary} working paper whose12results are mostly negative and fully attributed: on the small model, the best causal predictor (a per-layer logistic probe,13recall@8 of 0.76, within 0.16 of the oracle hit rate) only ties with demand loading because installing a prefetched expert costs14as much graph time as the read it saves (H1 not supported); kernel fusion is immaterial because decode at batch size one is15launch-bound at 28\,\% of memory bandwidth (H4 not supported); and executing half the experts on a second Mac Studio over the16gigabit Ethernet that actually connects the cluster costs 2.5~ms per layer, more than a cold-NVMe miss, so the remote tier is a17capacity mechanism rather than a latency one (H2/H3 in this regime). The storage layout, by contrast, matters: an18expert-contiguous record file raises cold-NVMe decoding from 8 to 36~tokens/s. Energy measurements and the adaptive scheduler19are pending and labelled as such.%20}21
Files · 2d987b30cb427aa933cb485dd908495f7ea82b47
Menu

Theme