SPB Git

spb/wp33_uqo

Public MIT
3.7 KB · 42 lines latex
1\section{Related Work}2\label{sec:related}34The prior-art table in the reproducibility appendix (and \code{research/prior\_art.csv} in the repository) lists 59 works5verified on their arXiv, publisher or repository pages on 2026-11-01; here we summarise what they establish and where6\titan{} differs.78\paragraph{Expert offloading and SSD streaming.} \citet{eliseev2023fast} introduced per-expert offloading with an LRU9cache for Mixtral on consumer GPUs; MoE-Infinity \citep{xue2024moeinfinity} built sparsity-aware expert caches from10activation traces; Fiddler \citep{kamahori2025fiddler}, HybriMoE \citep{zhong2025hybrimoe} and DAOP \citep{zhang2025daop}11execute cold experts on the CPU rather than moving them; MoE-Lightning \citep{cao2025moelightning}, fMoE12\citep{yu2025finemoe}, MoE-Gen \citep{xu2025moegen} and Klotski \citep{fang2025klotski} optimise throughput under memory13constraints with pipelined offloading. SSD-tier streaming has been demonstrated at trillion-parameter scale on x8614\citep{liang2026ssdllama}; FlashMoE \citep{kim2026flashmoe} learns a cache-replacement policy that beats LRU and LFU; colibri15\citep{colibri2026} combines a per-layer LRU, a pinned hot set learned across sessions and a one-layer router lookahead in a16pure C engine. The principle that placement must not change the model's semantics is stated explicitly by colibri and adopted17here.1819\paragraph{Expert prediction.} Pre-gated MoE \citep{hwang2024pregated} and Read-ME \citep{cai2024readme} modify the20architecture so that layer $l$'s router is computed from layer $l-1$'s representation; Fate \citep{fang2025fate}, ProMoE21\citep{song2024promoe}, ExpertFlow \citep{he2026expertflow,shen2025expertflow}, LayerScope \citep{yu2026layerscope},22DuoServe \citep{zhang2025duoserve}, SpecPrefetch \citep{kong2026specprefetch} and SeqMoE \citep{wang2026seqmoe} learn23cross-layer or multi-step predictors on GPUs and NPUs; \citet{liang2026notall} show that routing locality varies across24models and that a cache of about twice the active expert count is often enough. \titan{}'s predictors are deliberately25simple (transition counts at depth $d$, a linear probe on router log-probabilities, a previous-token rank predictor) so26that their cost can be measured against their benefit; the novelty we test is not a predictor but the \emph{condition},27derived in Section~\ref{sec:formulation}, under which any predictor can pay on a given tier of a given machine.2829\paragraph{Distributed inference on consumer hardware.} Petals \citep{borzunov2023petals}, exo \citep{exo2025}, prima.cpp30\citep{li2025primacpp} and llama.cpp's RPC backend shard layers or tensors across devices; MLX's distributed backends31(ring, MPI, JACCL) and Apple's own 4$\times$M5~Ultra results \citep{apple2026mlxdistributed} shard whole layers over a32Thunderbolt mesh. Expert-level placement across nodes exists for datacenter GPUs (ExFlow \citep{yao2024exflow}, Lina33\citep{li2023lina}, DeepEP \citep{deepep2025}) and, in a simple form, in colibri's cluster mode, where experts are hashed to34workers with weights proportional to disk speed. None of these models node heterogeneity, link latency or replication; H235targets that gap on the only topology our cluster offers, gigabit Ethernet.3637\paragraph{Energy and characterisation on Apple Silicon.} GreenBench \citep{kannan2026greenbench} and Silicon Showdown38\citep{javat2026siliconshowdown} measure joules per token with \code{powermetrics} for dense models; \citet{alfarizy2026doesmoe}39find that MoE energy on edge devices tracks total rather than active parameters; NPUMoE \citep{benazir2026npumoe} runs MoE40on the Neural Engine with all experts resident. No prior work reports an energy--throughput frontier for \emph{offloaded} or41\emph{distributed} MoE on Macs, which H5 addresses.42
Files · 9b97c26f010ceb73ef82c28ebf35c3792fd17460
Menu

Theme