1key,authors,year,title,venue,url,category,hardware,precision,offload_tiers,prediction_method,distributed,metal_or_apple,limitations_or_notes,evidence_url_seen2eliseev2023fast,Artyom Eliseev; Denis Mazur,2023,Fast Inference of Mixture-of-Experts Language Models with Offloading,arXiv technical report (no venue),https://arxiv.org/abs/2312.17238,expert offloading,"consumer desktop GPU, free-tier Colab T4",mixed quantization (HQQ; attention and experts quantized separately),"GPU <- CPU RAM (per-expert, per-layer)",LRU expert cache across adjacent tokens; speculative expert prefetch listed as future work in repo,no,no,"Mixtral-8x7B only; CUDA; repo github.com/dvmazur/mixtral-offloading (MIT, ~2.3k stars)",https://arxiv.org/abs/2312.17238 ; https://github.com/dvmazur/mixtral-offloading3xue2024moeinfinity,Leyang Xue; Yao Fu; Zhan Lu; Luo Mai; Mahesh Marina,2024,MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache,(not stated on arXiv page; v3 Mar 2025),https://arxiv.org/abs/2401.14361,expert offloading,"personal machine, single GPU, batch size 1",not stated,GPU <- host memory,sparsity-aware expert cache built from traced expert-activation traces (representative traces guide replacement + prefetch),no,no,venue empty (unverified),https://arxiv.org/abs/2401.143614hwang2024pregated,Ranggi Hwang; Jianyu Wei; Shijie Cao; Changho Hwang; Xiaohu Tang; Ting Cao; Mao Yang,2024,Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference,ISCA 2024 (51st IEEE/ACM ISCA; DOI 10.1109/ISCA59077.2024.00078 per UNIST record + MSR camera-ready PDF),https://arxiv.org/abs/2308.12066,predictive routing / offloading,single GPU + CPU memory,not stated,GPU <- CPU memory,"pre-gating: layer l gate computed from layer l-1 activations (model architecture change, requires fine-tuning)",no,no,requires modifying the MoE architecture; DOI seen only in secondary record,https://arxiv.org/abs/2308.12066 ; https://www.microsoft.com/en-us/research/uploads/prodnew/2024/05/isca24_pregated_moe_camera_ready.pdf5yi2025edgemoe,Rongjie Yi; Liwei Guo; Shiyun Wei; Ao Zhou; Shangguang Wang; Mengwei Xu,2025,EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices,"IEEE Transactions on Mobile Computing 24(8):7059-7073, DOI 10.1109/TMC.2025.3546466 (arXiv v1 2023)",https://arxiv.org/abs/2308.14352,expert offloading (mobile),mobile/edge devices,per-expert bitwidth adaptation,device memory <- external storage (flash),"expert preloading: predicts likely experts, compute-I/O pipeline",no,no,mobile SoCs not Apple,https://arxiv.org/abs/2308.14352 ; https://api.openalex.org/works/doi:10.1109/TMC.2025.35464666zhong2024adapmoe,Shuzhang Zhong; Ling Liang; Yuan Wang; Runsheng Wang; Ru Huang; Meng Li,2024,AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference,"ICCAD 2024 (inferred from related DOI 10.1145/3676536.3676741; ACM page returned 403, unverified)",https://arxiv.org/abs/2408.10284,expert offloading / adaptive gating,edge GPU,not stated,GPU <- host,sensitivity-based dynamic number of activated experts + prefetch + cache management,no,no,venue inferred from DOI prefix only,https://arxiv.org/abs/2408.102847tang2024hobbit,Peng Tang; Jiacheng Liu; Xiaofeng Hou; Yifei Pu; Jing Wang; Pheng-Ann Heng; Chao Li; Minyi Guo,2024,HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference,(not stated; venue unverified),https://arxiv.org/abs/2411.01433,expert offloading,memory-limited edge devices; built on llama.cpp,mixed precision (low-precision substitutes for cache-miss experts),device <- host/storage via llama.cpp,"token-level dynamic loading, layer-level adaptive prefetch, sequence-level multidimensional cache",no,indirectly (llama.cpp codebase) — no Apple results stated,up to 9.93x decode speedup claimed,https://arxiv.org/abs/2411.014338kamahori2025fiddler,Keisuke Kamahori; Tian Tang; Yile Gu; Kan Zhu; Baris Kasikci,2025,Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models,ICLR 2025,https://arxiv.org/abs/2402.07033,CPU-GPU hybrid execution,single consumer GPU + CPU,not stated,GPU + CPU (compute experts on CPU instead of moving weights),none (executes misses on CPU),no,no,x86 CPU + NVIDIA,https://arxiv.org/abs/2402.070339cao2025moelightning,Shiyi Cao; Shu Liu; Tyler Griggs; Peter Schafhalter; Xiaoxuan Liu; Ying Sheng; Joseph E. Gonzalez; Matei Zaharia; Ion Stoica,2025,MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs,ASPLOS 2025 (per Zaharia publication list; ACM DOI not obtained),https://arxiv.org/abs/2411.11217,throughput-oriented offloading,T4 16GB and 2-4 T4,not stated,"GPU <- CPU <- disk I/O (CGOPipe, paged weights)",Hierarchical Roofline performance model (no routing prediction),multi-GPU single host,no,"batch-throughput, not latency",https://arxiv.org/abs/2411.1121710du2024sida,Zhixu Du; Shiyu Li; Yuhao Wu; Xiangyu Jiang; Jingwei Sun; Qilin Zheng; Yongkai Wu; Ang Li; Hai Li; Yiran Chen,2024,SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models,MLSys 2024,https://arxiv.org/abs/2310.18859,predictive offloading,GPU + main memory,not stated,GPU <- main memory,offline-trained hash/predictor of expert activation from input data,no,no,,https://arxiv.org/abs/2310.1885911song2024promoe,Xiaoniu Song; Zihang Zhong; Rong Chen; Haibo Chen,2024,ProMoE: Fast MoE-based LLM Serving using Proactive Caching,(not stated; v3 Sep 2025),https://arxiv.org/abs/2410.22134,predictive prefetching,edge device with limited GPU memory,not stated,GPU <- host memory,"learned predictor using intermediate results to prefetch future-layer experts (proactive, off critical path)",no,no,,https://arxiv.org/abs/2410.2213412he2026expertflow,Xin He; Shunkang Zhang; Kaijie Tang; Shaohuai Shi; Yuxin Wang; Zihao Zeng; Zhenheng Tang; Xiaowen Chu; Haiyan Yin; Ivor W. Tsang; Yew Soon Ong,2026,ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling,DAC 2026 (arXiv Comments),https://arxiv.org/abs/2410.17954,predictive caching,single GPU,not stated,GPU <- CPU memory,transformer-based routing-path predictor for all layers in one pass + token scheduler,no,no,"arXiv v1 2024, v2 Apr 2026",https://arxiv.org/abs/2410.1795413yu2025finemoe,Hanfei Yu; Xingqi Cui; Hong Zhang; Hao Wang; Hao Wang,2025,Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading,(not stated),https://arxiv.org/abs/2502.05370,fine-grained offloading (fMoE/FineMoE),"6-GPU testbed, HF Transformers",not stated,GPU <- CPU,fine-grained expert-selection patterns + semantic hints from prompts,multi-GPU host,no,"author list shows ""Hao Wang"" twice on arXiv",https://arxiv.org/abs/2502.0537014zhang2025daop,Yujie Zhang; Shivam Aggarwal; Tulika Mitra,2025,DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference,DATE 2025 (arXiv Comments),https://arxiv.org/abs/2501.10375,CPU-GPU hybrid + prediction,CPU + GPU,not stated,GPU + CPU,per-sequence activation pattern assignment; predicted experts pre-computed on CPU; graceful degradation,no,no,,https://arxiv.org/abs/2501.1037515xu2025moegen,Tairan Xu; Leyang Xue; Zhan Lu; Adrian Jackson; Luo Mai,2025,MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching,(not stated),https://arxiv.org/abs/2503.09716,throughput offloading,single GPU + host memory,not stated,GPU <- host,none (module-based batching),no,no,"throughput, not single-user latency",https://arxiv.org/abs/2503.0971616fang2025klotski,Zhiyuan Fang; Yuegui Huang; Zicong Hong; Yufeng Lyu; Wuhui Chen; Yue Yu; Fan Yu; Zibin Zheng,2025,Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline,(not stated),https://arxiv.org/abs/2502.06888,offloading pipeline,GPU + CPU + disk,not stated,GPU <- CPU <- disk,correlation-aware expert prefetcher,no,no,,https://arxiv.org/abs/2502.0688817tairin2025emoe,Suraiya Tairin; Shohaib Mahmud; Haiying Shen; Anand Iyer,2025,eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference,(not stated),https://arxiv.org/abs/2503.06823,predictive loading,GPU,not stated,not stated on abstract,recurring routing-pattern predictor run every few prompts; task-aware skip,no,no,,https://arxiv.org/abs/2503.0682318suo2025coserve,Jiashun Suo; Xiaojian Liao; Limin Xiao; Li Ruan; Jinquan Wang; Xiao Su; Zhisheng Huo,2025,CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory,ASPLOS 2025 (arXiv journal-ref),https://arxiv.org/abs/2503.02354,CoE offloading,heterogeneous CPU+GPU,not stated,memory <-> storage,expert-dependency-aware scheduler + offline profiler for processor/device allocation,no,no,"Collaboration-of-Experts, not classic token-routed MoE",https://arxiv.org/abs/2503.0235419skliar2025cacheaware,Andrii Skliar; Ties van Rozendaal; Romain Lepert; Todor Boinovski; Mart van Baalen; Markus Nagel; Paul Whatmough; Babak Ehteshami Bejnordi,2025,Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference,TMLR (June 2025),https://arxiv.org/abs/2412.00099,cache-aware routing,"mobile device, DRAM-limited",not stated,DRAM <- storage,training-free cache-aware routing that biases router toward cached experts (~2x speedup),no,no,Qualcomm; no Apple,https://arxiv.org/abs/2412.0009920alizadeh2024flash,Keivan Alizadeh; Iman Mirzadeh; Dmitry Belenko; Karen Khatamifard; Minsik Cho; Carlo C Del Mundo; Mohammad Rastegari; Mehrdad Farajtabar,2024,LLM in a flash: Efficient Large Language Model Inference with Limited Memory,ACL 2024,https://arxiv.org/abs/2312.11514,"flash/SSD streaming (dense, Apple)",Apple M1 Max + consumer GPU (per paper; abstract says flash memory),not stated,DRAM <- flash,windowing (recent-neuron reuse) + row-column bundling; flash cost model,no,yes (Apple authors; M1 Max evaluation in paper body — not on abstract),"dense FFN sparsity, not MoE experts",https://arxiv.org/abs/2312.1151421song2024powerinfer,Yixin Song; Zeyu Mi; Haotong Xie; Haibo Chen,2024,PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU,SOSP 2024,https://arxiv.org/abs/2312.12456,hot/cold neuron locality,consumer GPU + CPU,not stated,GPU (hot neurons) + CPU (cold),adaptive activation predictors,no,no,"dense ReLU-sparsity, context for locality arguments",https://arxiv.org/abs/2312.1245622huang2023towards,Haiyang Huang; Newsha Ardalani; Anna Sun; Liu Ke; Hsien-Hsin S. Lee; Anjali Sridhar; Shruti Bhosale; Carole-Jean Wu; Benjamin Lee,2023,Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference,(not stated),https://arxiv.org/abs/2303.06182,characterization / expert buffering,GPU + CPU,not stated,GPU (frequent experts) <- CPU memory,dynamic gating + expert buffering by frequency,no,no,Meta; early locality characterization,https://arxiv.org/abs/2303.0618223cai2024readme,Ruisi Cai; Yeonju Ro; Geon-Woo Kim; Peihao Wang; Babak Ehteshami Bejnordi; Aditya Akella; Zhangyang Wang,2024,Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design,NeurIPS 2024,https://arxiv.org/abs/2410.19123,pre-gating router / lookahead,GPU,not stated,not stated,"decoupled pre-gating router computed ahead -> lookahead scheduling, expert-aware batching/caching",no,no,requires refactorized model,https://arxiv.org/abs/2410.1912324yao2024exflow,Jinghan Yao; Quentin Anthony; Aamir Shafi; Hari Subramoni; Dhabaleswar K. Panda,2024,Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference,(not stated),https://arxiv.org/abs/2401.08383,expert-parallel placement (ExFlow),multi-GPU,not stated,none,conditional inter-layer expert affinity -> integer-programming expert-to-GPU placement,yes (expert parallel),no,"placement, not offloading",https://arxiv.org/abs/2401.0838325fang2025fate,Zhiyuan Fang; Zicong Hong; Yuegui Huang; Yufeng Lyu; Wuhui Chen; Yue Yu; Fan Yu; Zibin Zheng,2025,Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate,(not stated),https://arxiv.org/abs/2502.12224,cross-layer expert prediction,edge device,quantization for cache/I/O,device <- host/storage,adjacent-layer gate inputs predict next-layer experts (97% prefetch acc.) + shallow-favoring cache,no,no,,https://arxiv.org/abs/2502.1222426liang2026notall,Jingcong Liang; Siyuan Wang; Miren Tian; Yitong Li; Duyu Tang; Zhongyu Wei,2026,Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models,ICLR 2026 (iclr.cc poster page; arXiv Comments empty),https://arxiv.org/abs/2505.16056,routing locality study,n/a (20 MoE LLMs analysed),n/a,n/a,metrics SRP/SCH; finds cache ~2x active experts; domain-specialized experts raise consistency,no,no,analysis only,https://arxiv.org/abs/2505.16056 ; https://iclr.cc/virtual/2026/poster/1001173227zhong2025hybrimoe,Shuzhang Zhong; Yanfan Sun; Ling Liang; Runsheng Wang; Ru Huang; Meng Li,2025,HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference,DAC 2025 (arXiv Comments),https://arxiv.org/abs/2504.05897,CPU-GPU hybrid,CPU + GPU,not stated,GPU <- host,impact-driven prefetch + score-based cache,no,no,,https://arxiv.org/abs/2504.0589728huang2026cpugpu,En-Ming Huang; Li-Shang Lin; Chun-Yi Lee,2026,Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems,ASP-DAC 2026 (arXiv Comments),https://arxiv.org/abs/2512.16473,CPU-GPU hybrid,consumer CPU + GPU,not stated,GPU cache + CPU execution of misses,frequency cache; argues prefetch cannot hide transfer,no,no,,https://arxiv.org/abs/2512.1647329zhao2026mobile,Yushu Zhao; Yubin Qin; Yang Wang; Xiaolong Yang; Huiming Han; Shaojun Wei; Yang Hu; Shouyi Yin,2026,MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts,ASP-DAC 2026 (arXiv Comments),https://arxiv.org/abs/2510.12357,offloading + expert substitution,consumer GPU,not stated,GPU <- CPU,"small ""little"" experts for unimportant tokens + prefetch/fallback",no,no,,https://arxiv.org/abs/2510.1235730yu2026layerscope,Enda Yu; Dezun Dong; Zhaoning Zhang; Zhe Bai; Weiling Yang; Haojie Wang; Dongsheng Li; Yongwei Wu; Xiangke Liao,2026,LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers,ICS 2026 (arXiv Comments),https://arxiv.org/abs/2509.23638,cross-layer prediction,"legacy GPU servers, PCIe",not stated,GPU <- CPU,learnable layer-aware predictor + cross-layer prefetch scheduling,no,no,"arXiv title earlier ""PreScope""",https://arxiv.org/abs/2509.2363831shen2025expertflow,Zixu Shen; Kexin Chu; Yifan Zhang; Dawei Xiang; Runxin Wu; Wei Zhang,2025,ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference,(not stated),https://arxiv.org/abs/2510.26730,adaptive prefetch horizon,memory-constrained GPU,not stated,GPU <- host,pregating + intermediate states; runtime-adaptive prefetch distance; cache-aware routing,no,no,name collision with he2026expertflow,https://arxiv.org/abs/2510.2673032zhang2025duoserve,Yuning Zhang; Grant Pinkert; Nan Yang; Yanli Li; Dong Yuan,2025,DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance,(not stated),https://arxiv.org/abs/2509.07379,prefill/decode-phase prefetch,CUDA GPU,not stated,GPU <- host,offline-trained layer-level predictor for decode; two-stream prefill pipeline,no,no,,https://arxiv.org/abs/2509.0737933kim2026flashmoe,Byeongju Kim; Jungwan Lee; Donghyeon Han; Hoi-Jun Yoo; Sangyeob Kim,2026,FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices,(not stated),https://arxiv.org/abs/2601.17063,SSD streaming + learned cache,user-grade desktop,not stated,RAM <- SSD,learned replacement (recency+frequency) beats LRU/LFU by up to 51% hit rate,no,no,"closest ""LRU vs learned"" comparison found",https://arxiv.org/abs/2601.1706334pan2026trimoe,Yudong Pan; Yintao He; Tianhua Han; Lian Liu; Shixin Zhao; Zhirong Chen; Mengdi Wang; Cangyuan Li; Yinhe Han; Ying Wang,2026,TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading,DAC 2026 (arXiv Comments),https://arxiv.org/abs/2603.01058,heterogeneous offloading,GPU + AMX CPU + DIMM NDP,not stated,GPU / CPU / near-memory,hot/warm/cold expert classification + prediction-driven rebalancing,no,no,server-class,https://arxiv.org/abs/2603.0105835kong2026specprefetch,Jinwei Kong; Runqi Meng; Fanyi Wang; Wentao Qiu; Haotian Hu; Yongjian Zhou; Zhenhua Ge,2026,SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models,(not stated),https://arxiv.org/abs/2607.24787,cross-layer prediction (mobile),Snapdragon 8 Elite,not stated,accelerator <- host/storage,small shared adapter predicts next-layer experts; window-aware transfer scheduler,no,no,,https://arxiv.org/abs/2607.2478736wang2026seqmoe,Zihan Wang; Yuqi Wang; Lei Gong; Cheng Tang; Wenqi Lou; Teng Wang; Chao Wang; Xuehai Zhou,2026,SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading,(not stated),https://arxiv.org/abs/2609.12978,multi-step prediction,device with ~45% experts resident,not stated,device <- host,seq2seq model forecasting experts multiple steps and layers ahead; probabilistic Belady eviction,no,no,,https://arxiv.org/abs/2609.1297837liang2026ssdllama,Fangzhou Liang; Yibin Shen; Jianmin Hu; Jiayang Xu; Hanchi Gao; Minxian Xu; Zili Meng,2026,SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC,(not stated),https://arxiv.org/abs/2609.18110,SSD streaming,RTX 5090 + <=32 GB RAM,quantized (unspecified) + lossless compression,VRAM <- RAM <- SSD (3 tiers),not stated; argues prefetch alone insufficient,no,no,,https://arxiv.org/abs/2609.1811038yang2026moecore,Ke Yang; Yongji Gao; Xushi Li; Kui Luo; Sicheng Zhang; Tianming Zhou; Keyi Liu; Shufang Lu; Aoxuan Chen; Jie Meng; Jingchun Gao; Dan Li; Xinkai You; Zhixiang Xia; Yan Shi; Yang Liu; Yanjia Zeng; Liangjun Feng,2026,MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference,(not stated),https://arxiv.org/abs/2610.01950,cross-layer prefetch + residency,NPU with 84 GB cap,not stated,device <- host,cross-layer prefetcher (1 expert/layer default) + routing-history-aware per-layer cache,no,no,"""Dan Li"" listed twice on arXiv",https://arxiv.org/abs/2610.0195039yuan2026ordermoe,Xin Yuan; Ning Li; Quan Chen; Wenchao Xu; Song Guo,2026,OrderMoE: An expert similarity driven distributed edge MoE inference,(not stated),https://arxiv.org/abs/2607.17154,distributed expert placement,"edge servers, bandwidth-limited",not stated,none,expert-similarity grouping -> placement; runtime local-substitute selection,yes (multi-server expert placement),no,no device heterogeneity model,https://arxiv.org/abs/2607.1715440benazir2026npumoe,Afsara Benazir; Felix Xiaozhu Lin,2026,Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs,(not stated),https://arxiv.org/abs/2604.18788,Apple Silicon MoE runtime (ANE),Apple M2 / M2 Ultra (Neural Engine),not stated,unified memory (all experts resident),offline calibration of expert capacity/popularity; load-aware graph residency,no,"yes (ANE + CPU/GPU fallback, energy mJ/token)",assumes all experts fit in unified memory — no SSD tier,https://arxiv.org/abs/2604.1878841alfarizy2026doesmoe,Alfarizy Alfarizy; Hung Truong Thanh Nguyen; Rene Richard; Roozbeh Razavi-Far; Hung Cao,2026,Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study,submitted to FAIEMA 2026 (arXiv Comments),https://arxiv.org/abs/2606.21428,Apple Silicon MoE energy characterization,Apple M2 Pro laptop; Jetson Orin Nano,llama.cpp GGUF,none,none,no,"yes (M2 Pro, llama.cpp)",OLMoE only; energy tracks total params,https://arxiv.org/abs/2606.2142842kannan2026greenbench,Rajeswari Kannan; Raj Firke; Shreya Bengle; Srushti Deshmukh,2026,GreenBench: Benchmarking Energy Efficiency and Carbon Footprint of Open-Source LLM Inference on Apple Silicon,IEEE ICCUBEA 2026 (arXiv Comments),https://arxiv.org/abs/2608.28667,Apple Silicon energy (powermetrics),Apple M4 Pro 48 GB,Ollama defaults,none,none,no,yes (powermetrics),"dense 3-9B only, no MoE",https://arxiv.org/abs/2608.2866743javat2026siliconshowdown,Abdurrahman Javat; Allan Kazakov,2026,"Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference",(not stated),https://arxiv.org/abs/2605.00519,Apple vs NVIDIA tokens/joule,"M3 Ultra, M4 Pro vs Blackwell","NVFP4 vs BF16, quantized",CPU offload penalties measured,none,no,yes (powermetrics per search snippet; MoE latency-bound finding),powermetrics not on abstract page,https://arxiv.org/abs/2605.0051944rajesh2025productiongrade,Varun Rajesh; Om Jodhpurkar; Pooja Anbuselvan; Mantinder Singh; Ashok Jallepali; Shantanu Godbole; Pradeep Kumar Sharma; Hritvik Shrivastava,2025,"Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS",(not stated),https://arxiv.org/abs/2511.05502,Apple Silicon runtime characterization,Mac Studio M2 Ultra 192 GB,various,none,none,no,"yes (MLX, llama.cpp Metal)",no energy measurement; dense Qwen-2.5,https://arxiv.org/abs/2511.0550245rathnayaka2026basert,Prabod Rathnayaka; Fabian Waschkowski; Lukas Wesemann,2026,BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal,(not stated),https://arxiv.org/abs/2607.00501,Metal kernel runtime,"Apple M3, M4 Pro",not stated,unified memory,none,no,yes (native Metal kernel fusion; MoE prefill gains),no offloading,https://arxiv.org/abs/2607.0050146wilkins2024hybrid,Grant Wilkins; Srinivasan Keshav; Richard Mortier,2024,Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads,(not stated),https://arxiv.org/abs/2407.00010,energy scheduling,heterogeneous processors,not stated,none,cost-based scheduler,cluster-level,Apple Silicon/powermetrics not confirmed on abstract page (search snippet only),7.5% energy reduction,https://arxiv.org/abs/2407.0001047borzunov2023petals,Alexander Borzunov; Dmitry Baranchuk; Tim Dettmers; Maksim Riabinin; Younes Belkada; Artem Chumachenko; Pavel Samygin; Colin Raffel,2023,Petals: Collaborative Inference and Fine-tuning of Large Models,"ACL 2023 System Demonstrations, pp. 558-568, DOI 10.18653/v1/2023.acl-demo.54",https://aclanthology.org/2023.acl-demo.54/,distributed inference (volunteer),consumer GPUs over Internet,8-bit (per paper),none,none,yes (pipeline over WAN),no,dense BLOOM-176B,https://aclanthology.org/2023.acl-demo.54/ ; https://arxiv.org/abs/2209.0118848li2025primacpp,Zonghang Li; Tao Li; Wenjiao Feng; Rongxing Xiao; Jianshu She; Hong Huang; Mohsen Guizani; Hongfang Yu; Qirong Ho; Wei Xiang; Xue Liu,2025,Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters,"(not stated; ""38 pages"")",https://arxiv.org/abs/2504.08791,heterogeneous home cluster,"mixed CPUs/GPUs, Wi-Fi, slow disks",llama.cpp quants,disk I/O overlapped via pipelined-ring,Halda scheduler (per-device CPU/GPU split under memory limits),yes (pipelined ring),not stated on abstract,dense models; no expert-level placement,https://arxiv.org/abs/2504.0879149exo2025,exo labs (exo-explore),2025,exo: Run frontier AI locally (GitHub repository),GitHub,https://github.com/exo-explore/exo,distributed Apple cluster,"Apple Silicon Macs (M4 Pro mini, M4 Max Studio/MBP, M3 Ultra Studio)",MLX quants,none,none,"yes (topology-aware auto-parallel, tensor + pipeline, RDMA over TB5)","yes (MLX backend; macOS 15.1+, RDMA needs macOS 26.2)","no expert-level offload; RDMA requires full mesh and identical macOS builds; ~48k stars, Apache-2.0",https://github.com/exo-explore/exo50llamacpp2026,Georgi Gerganov and ggml-org contributors,2026,llama.cpp: LLM inference in C/C++ (GitHub repository; RPC backend),GitHub,https://github.com/ggml-org/llama.cpp,Metal kernels + RPC distributed,"Apple Silicon (Metal), CUDA, CPU",GGUF quant families,none (full model split across devices),none,yes (RPC: memory-proportional tensor split; RDMA over TB5 on macOS 26.2+),yes (Metal first-class),"RPC ""proof-of-concept"", insecure; no MoE expert placement; MIT, ~133k stars",https://github.com/ggml-org/llama.cpp ; https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md51hannun2023mlx,Awni Hannun; Jagrit Digani; Angelos Katharopoulos; Ronan Collobert,2023,MLX: Efficient and flexible machine learning on Apple silicon,GitHub software (repo citation block),https://github.com/ml-explore/mlx,Apple array framework,Apple Silicon unified memory,various,none,none,"yes (MPI / ring over Thunderbolt / JACCL RDMA backends, docs v0.32.4)",yes,"JACCL requires TB5 + macOS 26.2, rdma_ctl enable in Recovery, fully connected mesh only",https://github.com/ml-explore/mlx ; https://ml-explore.github.io/mlx/build/html/usage/distributed.html52apple2026mlxdistributed,Apple Machine Learning Research (no authors listed),2026,Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio,"Apple ML Research blog, 2026-10-15",https://machinelearning.apple.com/research/mlx-distributed-inference,official Apple distributed MoE result,Mac Studio M3 Ultra / M5 Ultra 512 GB; 2 and 4 M5 Ultra TB5 mesh,4-bit NVFP4 (Muse Spark Open ~493 GB),none (all in unified memory),none,yes (pipeline + tensor parallel via JACCL),yes,"Muse Spark Open 816B/42B-active: 15.7 (2 PP) / 25.0 (2 TP) / 36.7 tok/s (4 TP); MoE handled by TP, no expert parallel; no energy figures",https://machinelearning.apple.com/research/mlx-distributed-inference53apple2026wwdc233,Apple (WWDC26 session 233),2026,Explore distributed inference and training with MLX,WWDC 2026 session 233,https://developer.apple.com/videos/play/wwdc2026/233/,official RDMA/JACCL source,Mac clusters over Thunderbolt 5,n/a,none,none,"yes (mesh/ring topologies, tensor + pipeline parallel)",yes (RDMA over TB5 from macOS 26.2; JACCL open-source),"video session, no numbers extracted",https://developer.apple.com/videos/play/wwdc2026/233/54geerling2025rdma,Jeff Geerling,2025,1.5 TB of VRAM on Mac Studio - RDMA over Thunderbolt 5,"blog post, 2025-12-18",https://jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5,independent Mac cluster measurement,"4x M3 Ultra Mac Studio (2x512 GB + 2x256 GB), TB5 full mesh",MLX/llama.cpp quants,none,none,yes (exo 1.0 vs llama.cpp RPC),yes (macOS 26.2),"TB5 ~50-60 Gbps real; RDMA latency ~300us -> <50us; Qwen3-235B ~32 tok/s, Kimi K2 ~30 tok/s on 4 nodes; idle <10 W per Studio but no energy/token",https://jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-555rajbhandari2022deepspeedmoe,Samyam Rajbhandari; Conglong Li; Zhewei Yao; Minjia Zhang; Reza Yazdani Aminabadi; Ammar Ahmad Awan; Jeff Rasley; Yuxiong He,2022,DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale,ICML 2022 (arXiv Comments),https://arxiv.org/abs/2201.05596,expert-parallel serving (context),multi-GPU datacenter,FP16/INT8 (per paper),none,none,yes (expert + tensor + data parallel),no,,https://arxiv.org/abs/2201.0559656deepep2025,Chenggang Zhao et al. (DeepSeek-AI),2025,DeepEP: an efficient expert-parallel communication library (GitHub repository),GitHub (README BibTeX key deepep2025),https://github.com/deepseek-ai/DeepEP,expert-parallel all-to-all kernels,"NVIDIA Hopper+, NVLink + RDMA",FP8 dispatch,none,none,"yes (intra/inter-node all-to-all, low-latency decode kernels)",no,Linux/CUDA only; MIT,https://github.com/deepseek-ai/DeepEP57li2023lina,Jiamin Li; Yimin Jiang; Yibo Zhu; Cong Wang; Hong Xu,2023,Accelerating Distributed MoE Training and Inference with Lina,USENIX ATC 2023 (pp. 945-959),https://www.usenix.org/conference/atc23/presentation/li-jiamin,expert-parallel serving,A100 + 100 Gbps InfiniBand,not stated,none,expert-popularity-aware resource scheduling for inference,yes,no,,https://www.usenix.org/conference/atc23/presentation/li-jiamin ; https://arxiv.org/abs/2210.1722358hwang2023tutel,Changho Hwang; Wei Cui; Yifan Xiong; Ziyue Yang; Ze Liu; Han Hu; Zilong Wang; Rafael Salas; Jithin Jose; Prabhat Ram; Joe Chau; Peng Cheng; Fan Yang; Mao Yang; Yongqiang Xiong,2023,Tutel: Adaptive Mixture-of-Experts at Scale,(MLSys 2023 believed; NOT shown on arXiv page — unverified),https://arxiv.org/abs/2206.03382,adaptive parallelism (context),multi-GPU,not stated,none,none,yes,no,venue unverified,https://arxiv.org/abs/2206.0338259gale2023megablocks,Trevor Gale; Deepak Narayanan; Cliff Young; Matei Zaharia,2023,MegaBlocks: Efficient Sparse Training with Mixture-of-Experts,(MLSys 2023 believed; NOT shown on arXiv page — unverified),https://arxiv.org/abs/2211.15841,block-sparse MoE kernels (context),GPU,not stated,none,none,no,no,training-focused; venue unverified,https://arxiv.org/abs/2211.1584160colibri2026,Vincenzo Fornaro (GitHub: JustVugg),2026,"colibri: Tiny engine, immense model — local inference engine streaming routed MoE experts from disk (GitHub repository, v2.1.0)","GitHub, Apache-2.0, ~45k stars, 5,150 commits",https://github.com/JustVugg/colibri,SSD expert streaming engine (C),x86 benchmark machines (e.g. Ryzen AI Max+ 395 128 GB); Apple Silicon via Metal backend (no Mac benchmarks listed),"GGUF IQ1_S..Q8_0, F16, BF16, MXFP4; int4/int8 containers; fp4 routed experts (DeepSeek V4.1 Flash)",RAM (pinned hot set + per-layer LRU) <- disk; GPU via Metal/CUDA/Vulkan,"router-lookahead thread prefetches next layer (""71.6% predictable one layer ahead"" for GLM-5.2); pinned hot set learned from user's conversations (.coli_usage)","yes (cluster mode: TCP, coordinator + disk-backed expert workers, experts hashed by expert ID with disk-speed-weighted shares; no topology awareness)",yes (COLI_METAL=1; macOS builds),"Terms ""PILOT"" and ""CACHE_ROUTE"" NOT found in README, docs site, or cluster.md — do not cite them as Colibri features without checking source; no paper, no BibTeX, no citation; cluster benchmark only prefill 618s->454s on 2 SATA workers",https://github.com/JustVugg/colibri ; https://raw.githubusercontent.com/JustVugg/colibri/main/README.md ; https://github.com/JustVugg/colibri/blob/main/docs/cluster.md ; https://justvugg.github.io/colibri61