SPB Git
spboucher.ai

spb/wp33_uqo

Public MIT

Paper skeleton v0.1 (protocol + preliminary), README, models config, stacked-layout registry (Qwen3-30B), generalized tests

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 1, 2026) parent a301b4c Browse files

34 changed files +1,901 −13

34 changed files
Load all 34 files Unified Split
added LICENSE +9 −0
@@ -0,0 +1,9 @@
1 +MIT License
2 +
3 +Copyright (c) 2026 Simon-Pierre Boucher
4 +
5 +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 +
7 +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
8 +
9 +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
added README.md +93 −0
@@ -0,0 +1,93 @@
1 +# TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon
2 +
3 +**UQO Working Paper No. 33** — *A Measured Systems Study on a Heterogeneous Mac Cluster*
4 +
5 +![Paper](https://img.shields.io/badge/paper-PDF-blue) ![Python](https://img.shields.io/badge/python-3.12-blue) ![MLX](https://img.shields.io/badge/MLX-0.32.4-black) ![Status](https://img.shields.io/badge/status-v0.1%20protocol%20%2B%20preliminary-orange) ![UQO](https://img.shields.io/badge/UQO-DSA-green) ![Author](https://img.shields.io/badge/author-S.--P.%20Boucher-lightgrey)
6 +
7 +## TL;DR
8 +- An MLX-native MoE inference runtime with bounded per-layer expert residency (slot tables), a `pread` NVMe tier, LRU/LFU eviction,
9 + causal prefetch predictors and an asynchronous prefetcher; the router's choices and the logits are those of mlx-lm by construction.
10 +- Correctness: **bit-identical logits** (max |Δ| = 0.0) and token-identical greedy decoding vs mlx-lm on OLMoE-1B-7B-0125 4-bit, at 64/32/16/8 slots.
11 +- All-resident control: **489 tok/s** vs mlx-lm 494 tok/s on an M5 Max laptop (−1 %). Bounded residency with demand loading from the page cache: 51–73 tok/s at 16 slots.
12 +- Honest first result on H1: one-layer-ahead prefetch raises the hit rate (0.48→0.63) but **lowers** throughput; the paper derives the lead-time condition d·τ_layer ≥ λ_tier and tests deeper horizons.
13 +- Live cluster inventory: 21 reachable Macs (M1–M6, 8–96 GB), **1 GbE everywhere**, no active Thunderbolt link → distributed gates use TCP; RDMA/JACCL blocked pending cabling.
14 +
15 +## Status
16 +Version 0.1 (2026-11-01), **protocol + preliminary** working paper; not submitted. Gates A–B complete, Gate C in progress, D–G pending
17 +(see `research/LOG.md`). Nothing in the paper is reported without a raw log under `benchmarks/raw/`.
18 +
19 +## What this paper does
20 +Designs, implements and measures TITAN-MoE against five pre-stated hypotheses (causal prefetch, topology-aware placement,
21 +hierarchical caching, Metal kernels, adaptive orchestration on the latency–throughput–energy frontier), with an exact-semantics
22 +constraint: placement decides speed, never which experts run.
23 +
24 +## Headline results (version 0.1, single node, OLMoE-1B-7B-0125-Instruct-4bit, M5 Max 48 GB)
25 +| Configuration | decode tok/s | hit rate | source |
26 +|---|---|---|---|
27 +| mlx-lm (reference) | 491–496 | — | `benchmarks/raw/gateB/capacity_sweep_laptop/sweep.jsonl` |
28 +| TITAN, all experts resident (64 slots) | 489 | 1.00 | same |
29 +| TITAN, 32 slots, LRU, demand loading (page cache) | 56–80 | 0.73–0.81 | same (before lazy-install fix) |
30 +| TITAN, 16 slots, LRU, demand loading (page cache) | 51–73 | 0.48–0.67 | `benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl` |
31 +| TITAN, 16 slots + transition prefetch (d = 1, m = 8) | 35–56 | 0.63–0.73 | same |
32 +
33 +## Repository layout
34 +```
35 +CLAUDE.md README.md LICENSE docs/MANDATE.md
36 +titan/ registry.py store.py slots.py moe.py prefetch.py predictors.py runtime.py
37 +metal/ (Gate D) transport/ (Gate E) adapters/ (formats beyond OLMoE/Qwen3-MoE)
38 +tests/ test_registry_store.py test_numerical.py
39 +benchmarks/ configs/ (workloads, models) scripts/ (bench_decode, collect_traces, simulate_cache, ssd_bandwidth) raw/ (immutable) processed/
40 +research/ LOG.md DECISIONS.md RELATED_WORK.md prior_art.csv protocol.md
41 +paper/ main.tex sections/ references.bib (UQO template; `cd paper && make`)
42 +artifacts/ environment.json (sanitized live inventory) manifest.json
43 +scripts/ cluster_setup.sh make_environment.py
44 +third_party/ shallow clones of colibri / mlx / mlx-lm / llama.cpp (git-ignored; hashes in research/prior_art.csv)
45 +```
46 +
47 +## Quick start
48 +```bash
49 +uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements-lock.txt
50 +# model (3.9 GB, Apache-2.0, pinned revision) goes to the Hugging Face cache
51 +.venv/bin/python -c "from huggingface_hub import snapshot_download as s; s('mlx-community/OLMoE-1B-7B-0125-Instruct-4bit', revision='e2384419')"
52 +.venv/bin/python -m pytest tests -q # correctness vs mlx-lm (writes evidence to benchmarks/raw/tests/)
53 +M=$(ls -d ~/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/*)
54 +.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend mlx_lm --out /tmp/b.jsonl
55 +.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend titan --capacity 16 --predictor transition2 --traces benchmarks/raw/gateC/traces_olmoe_v2_laptop --out /tmp/b.jsonl
56 +cd paper && make # PDF
57 +```
58 +Local-only mode: everything above runs on one Mac; the distributed components are only enabled with an explicit host list.
59 +
60 +## Data
61 +No external datasets. Workloads are synthetic prompts written for this study (`benchmarks/configs/workloads*.yaml`); routing
62 +traces derived from them are in `benchmarks/raw/gateC/traces_*`. Models: OLMoE-1B-7B-0125-Instruct (Apache-2.0),
63 +Qwen3-30B-A3B-Instruct-2507 (Apache-2.0), both as mlx-community 4-bit conversions with pinned revisions.
64 +
65 +## Methodology
66 +See `research/protocol.md` (frozen at Gate C) and `paper/sections/formulation.tex`.
67 +
68 +## Figures & tables inventory
69 +Generated by `scripts/make_figures.py` / `scripts/make_tables.py` from `benchmarks/raw/` (Gate G).
70 +
71 +## Limitations
72 +Small model only in v0.1; distributed, Metal and energy experiments pending; 1 GbE topology; see paper Section "Limitations".
73 +
74 +## Citation
75 +```bibtex
76 +@techreport{boucher2026titanmoe,
77 + title = {TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon},
78 + author = {Boucher, Simon-Pierre},
79 + institution = {Université du Québec en Outaouais, Département des sciences administratives},
80 + type = {UQO Working Paper},
81 + number = {33},
82 + year = {2026}
83 +}
84 +```
85 +
86 +## Author & contact
87 +Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca
88 +
89 +## Notes
90 +- The mandate's repository name `wp33_titan_moe` is kept as the spbgit repository name (`wp33_uqo.git`); the folder follows the
91 + Recherche-UQO series layout (`ia/titan-moe`). Internal structure follows the mandate (D-001).
92 +- Written with Claude Code (Claude Fable 5.1) acting as autonomous research engineer under `docs/MANDATE.md`; AI-assistance
93 + disclosure is included in the paper's reproducibility appendix.
added artifacts/environment.json +650 −0
@@ -0,0 +1,650 @@
1 +{
2 + "generated": "2026-11-01T01:37:38",
3 + "note": "Live inventory of 2026-11-01; aliases are cluster names, no addresses or serials. Unreachable aliases are listed but unused.",
4 + "laptop": {
5 + "role": "development laptop (not on the cluster LAN)",
6 + "chip": "Apple M5 Max",
7 + "ram_gb": 48,
8 + "macos": "27.0",
9 + "gpu_cores": 40,
10 + "cpu_cores": 18,
11 + "ssd_random_3p4MB_GBps_cold": 8.5,
12 + "ssd_seq_GBps_cold": 11.5,
13 + "ssd_source": "benchmarks/raw/gateA/ssd/laptop-M5Max.json"
14 + },
15 + "cluster_nodes": {
16 + "M1M32": {
17 + "status": "reachable",
18 + "chip": "Apple M1 Max",
19 + "ram_gb": 32,
20 + "macos": "26.5.1",
21 + "gpu_cores": 24,
22 + "disk_free_gb": 208,
23 + "system_python": "3.9.6",
24 + "mlx_preinstalled": false,
25 + "load_avg_at_probe": "{ 1,52 1,31 1,21 }",
26 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
27 + "homebrew_pythons": [],
28 + "uv": true,
29 + "xcode": "/Applications/Xcode.app/Contents/Developer",
30 + "metal_compiler": "Apple metal version 32023.883 (metalfe-32023.883)",
31 + "rdma_ctl_present": true,
32 + "iperf3": false
33 + },
34 + "M2M32": {
35 + "status": "unreachable (SSH timeout 2026-11-01)"
36 + },
37 + "M2M32b": {
38 + "status": "unreachable (SSH timeout 2026-11-01)"
39 + },
40 + "M2M32c": {
41 + "status": "reachable",
42 + "chip": "Apple M2 Max",
43 + "ram_gb": 32,
44 + "macos": "26.5.1",
45 + "gpu_cores": 30,
46 + "disk_free_gb": 417,
47 + "system_python": "3.9.6",
48 + "mlx_preinstalled": false,
49 + "load_avg_at_probe": "{ 1,66 1,22 1,12 }",
50 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
51 + "homebrew_pythons": [
52 + "/opt/homebrew/bin/python3.13",
53 + "/opt/homebrew/bin/python3.14"
54 + ],
55 + "uv": true,
56 + "xcode": "/Library/Developer/CommandLineTools",
57 + "metal_compiler": null,
58 + "rdma_ctl_present": true,
59 + "iperf3": false
60 + },
61 + "M2U64": {
62 + "status": "reachable",
63 + "chip": "Apple M2 Ultra",
64 + "ram_gb": 64,
65 + "macos": "26.5",
66 + "gpu_cores": 60,
67 + "disk_free_gb": 308,
68 + "system_python": "3.9.6",
69 + "mlx_preinstalled": false,
70 + "load_avg_at_probe": "{ 1,04 1,13 1,14 }",
71 + "en0_media": "autoselect (1000baseT <full-duplex>)",
72 + "homebrew_pythons": [
73 + "/opt/homebrew/bin/python3.14"
74 + ],
75 + "uv": true,
76 + "xcode": "/Applications/Xcode.app/Contents/Developer",
77 + "metal_compiler": null,
78 + "rdma_ctl_present": true,
79 + "iperf3": false
80 + },
81 + "M3BA16": {
82 + "status": "reachable",
83 + "chip": "Apple M3",
84 + "ram_gb": 16,
85 + "macos": "26.5.1",
86 + "gpu_cores": 10,
87 + "disk_free_gb": 409,
88 + "system_python": "3.9.6",
89 + "mlx_preinstalled": false,
90 + "load_avg_at_probe": "{ 1,27 1,26 1,27 }",
91 + "en0_media": "autoselect",
92 + "homebrew_pythons": [],
93 + "uv": false,
94 + "xcode": "/Library/Developer/CommandLineTools",
95 + "metal_compiler": null,
96 + "rdma_ctl_present": true,
97 + "iperf3": false
98 + },
99 + "M3BA24": {
100 + "status": "reachable",
101 + "chip": "Apple M3",
102 + "ram_gb": 24,
103 + "macos": "26.5.1",
104 + "gpu_cores": 10,
105 + "disk_free_gb": 420,
106 + "system_python": "3.9.6",
107 + "mlx_preinstalled": false,
108 + "load_avg_at_probe": "{ 1,31 1,25 1,23 }",
109 + "en0_media": "autoselect",
110 + "homebrew_pythons": [],
111 + "uv": false,
112 + "xcode": "/Library/Developer/CommandLineTools",
113 + "metal_compiler": null,
114 + "rdma_ctl_present": true,
115 + "iperf3": false
116 + },
117 + "M3U96a": {
118 + "status": "reachable",
119 + "chip": "Apple M3 Ultra",
120 + "ram_gb": 96,
121 + "macos": "26.6",
122 + "gpu_cores": 60,
123 + "disk_free_gb": 512,
124 + "system_python": "3.14.5",
125 + "mlx_preinstalled": true,
126 + "load_avg_at_probe": "{ 1,43 1,63 1,64 }",
127 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
128 + "homebrew_pythons": [
129 + "/opt/homebrew/bin/python3.14"
130 + ],
131 + "uv": true,
132 + "xcode": "/Applications/Xcode.app/Contents/Developer",
133 + "metal_compiler": "Apple metal version 32023.883 (metalfe-32023.883)",
134 + "rdma_ctl_present": true,
135 + "iperf3": true
136 + },
137 + "M3U96b": {
138 + "status": "reachable",
139 + "chip": "Apple M3 Ultra",
140 + "ram_gb": 96,
141 + "macos": "26.4",
142 + "gpu_cores": 60,
143 + "disk_free_gb": 432,
144 + "system_python": "3.14.4",
145 + "mlx_preinstalled": true,
146 + "load_avg_at_probe": "{ 1,93 1,92 1,93 }",
147 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
148 + "homebrew_pythons": [
149 + "/opt/homebrew/bin/python3.14"
150 + ],
151 + "uv": true,
152 + "xcode": "/Applications/Xcode.app/Contents/Developer",
153 + "metal_compiler": null,
154 + "rdma_ctl_present": true,
155 + "iperf3": true
156 + },
157 + "M4BP36": {
158 + "status": "reachable",
159 + "chip": "Apple M4 Max",
160 + "ram_gb": 36,
161 + "macos": "26.5.1",
162 + "gpu_cores": 32,
163 + "disk_free_gb": 843,
164 + "system_python": "3.9.6",
165 + "mlx_preinstalled": false,
166 + "load_avg_at_probe": "{ 1,08 1,13 1,16 }",
167 + "en0_media": "autoselect",
168 + "homebrew_pythons": [],
169 + "uv": true,
170 + "xcode": "/Library/Developer/CommandLineTools",
171 + "metal_compiler": null,
172 + "rdma_ctl_present": true,
173 + "iperf3": false
174 + },
175 + "M4BP48": {
176 + "status": "unreachable (SSH timeout 2026-11-01)"
177 + },
178 + "M4M36": {
179 + "status": "unreachable (SSH timeout 2026-11-01)"
180 + },
181 + "M4M64a": {
182 + "status": "reachable",
183 + "chip": "Apple M4 Max",
184 + "ram_gb": 64,
185 + "macos": "26.5.1",
186 + "gpu_cores": 40,
187 + "disk_free_gb": 730,
188 + "system_python": "3.14.4",
189 + "mlx_preinstalled": true,
190 + "load_avg_at_probe": "{ 1,88 2,01 1,78 }",
191 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
192 + "homebrew_pythons": [
193 + "/opt/homebrew/bin/python3.14"
194 + ],
195 + "uv": true,
196 + "xcode": "/Applications/Xcode.app/Contents/Developer",
197 + "metal_compiler": null,
198 + "rdma_ctl_present": true,
199 + "iperf3": true
200 + },
201 + "M4M64b": {
202 + "status": "reachable",
203 + "chip": "Apple M4 Max",
204 + "ram_gb": 64,
205 + "macos": "26.5.1",
206 + "gpu_cores": 40,
207 + "disk_free_gb": 83,
208 + "system_python": "3.14.4",
209 + "mlx_preinstalled": true,
210 + "load_avg_at_probe": "{ 1,39 1,40 1,41 }",
211 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
212 + "homebrew_pythons": [
213 + "/opt/homebrew/bin/python3.13",
214 + "/opt/homebrew/bin/python3.14"
215 + ],
216 + "uv": false,
217 + "xcode": "/Applications/Xcode.app/Contents/Developer",
218 + "metal_compiler": null,
219 + "rdma_ctl_present": true,
220 + "iperf3": true
221 + },
222 + "M4M64c": {
223 + "status": "reachable",
224 + "chip": "Apple M4 Max",
225 + "ram_gb": 64,
226 + "macos": "26.4",
227 + "gpu_cores": 40,
228 + "disk_free_gb": 321,
229 + "system_python": "3.9.6",
230 + "mlx_preinstalled": false,
231 + "load_avg_at_probe": "{ 1,63 1,73 1,75 }",
232 + "en0_media": "autoselect (1000baseT <full-duplex>)",
233 + "homebrew_pythons": [
234 + "/opt/homebrew/bin/python3.13",
235 + "/opt/homebrew/bin/python3.14"
236 + ],
237 + "uv": true,
238 + "xcode": "/Library/Developer/CommandLineTools",
239 + "metal_compiler": null,
240 + "rdma_ctl_present": true,
241 + "iperf3": false
242 + },
243 + "M5M36": {
244 + "status": "reachable",
245 + "chip": "Apple M5 Max",
246 + "ram_gb": 36,
247 + "macos": "27.0",
248 + "gpu_cores": 32,
249 + "disk_free_gb": 355,
250 + "system_python": "3.9.6",
251 + "mlx_preinstalled": false,
252 + "load_avg_at_probe": "{ 2,52 2,53 2,41 }",
253 + "en0_media": "autoselect (1000baseT <full-duplex>)",
254 + "homebrew_pythons": [
255 + "/opt/homebrew/bin/python3.13",
256 + "/opt/homebrew/bin/python3.14"
257 + ],
258 + "uv": true,
259 + "xcode": "/Library/Developer/CommandLineTools",
260 + "metal_compiler": null,
261 + "rdma_ctl_present": true,
262 + "iperf3": false
263 + },
264 + "m2m16": {
265 + "status": "unreachable (SSH timeout 2026-11-01)"
266 + },
267 + "m2m8a": {
268 + "status": "reachable",
269 + "chip": "Apple M2",
270 + "ram_gb": 8,
271 + "macos": "26.5.1",
272 + "gpu_cores": 10,
273 + "disk_free_gb": 424,
274 + "system_python": "3.9.6",
275 + "mlx_preinstalled": false,
276 + "load_avg_at_probe": "{ 1,69 1,63 1,57 }",
277 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
278 + "homebrew_pythons": [],
279 + "uv": false,
280 + "xcode": "/Library/Developer/CommandLineTools",
281 + "metal_compiler": null,
282 + "rdma_ctl_present": true,
283 + "iperf3": false
284 + },
285 + "m2m8b": {
286 + "status": "reachable",
287 + "chip": "Apple M2",
288 + "ram_gb": 8,
289 + "macos": "26.5.1",
290 + "gpu_cores": 10,
291 + "disk_free_gb": 195,
292 + "system_python": null,
293 + "mlx_preinstalled": false,
294 + "load_avg_at_probe": "{ 2,12 1,54 1,45 }",
295 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
296 + "homebrew_pythons": [],
297 + "uv": false,
298 + "xcode": "",
299 + "metal_compiler": null,
300 + "rdma_ctl_present": true,
301 + "iperf3": false
302 + },
303 + "m4ma": {
304 + "status": "unreachable (SSH timeout 2026-11-01)"
305 + },
306 + "m4mb": {
307 + "status": "unreachable (SSH timeout 2026-11-01)"
308 + },
309 + "m4mc": {
310 + "status": "unreachable (SSH timeout 2026-11-01)"
311 + },
312 + "m4md": {
313 + "status": "reachable",
314 + "chip": "Apple M4",
315 + "ram_gb": 16,
316 + "macos": "26.6.2",
317 + "gpu_cores": 10,
318 + "disk_free_gb": 195,
319 + "system_python": "3.9.6",
320 + "mlx_preinstalled": false,
321 + "load_avg_at_probe": "{ 1,25 1,34 1,37 }",
322 + "en0_media": "autoselect (1000baseT <full-duplex>)",
323 + "homebrew_pythons": [
324 + "/opt/homebrew/bin/python3.13",
325 + "/opt/homebrew/bin/python3.14"
326 + ],
327 + "uv": true,
328 + "xcode": "/Library/Developer/CommandLineTools",
329 + "metal_compiler": null,
330 + "rdma_ctl_present": true,
331 + "iperf3": false
332 + },
333 + "m4ml": {
334 + "status": "reachable",
335 + "chip": "Apple M4",
336 + "ram_gb": 16,
337 + "macos": "26.7",
338 + "gpu_cores": 10,
339 + "disk_free_gb": 178,
340 + "system_python": "3.9.6",
341 + "mlx_preinstalled": false,
342 + "load_avg_at_probe": "{ 1,47 1,45 1,45 }",
343 + "en0_media": "autoselect (1000baseT <full-duplex>)",
344 + "homebrew_pythons": [
345 + "/opt/homebrew/bin/python3.13",
346 + "/opt/homebrew/bin/python3.14"
347 + ],
348 + "uv": true,
349 + "xcode": "/Library/Developer/CommandLineTools",
350 + "metal_compiler": null,
351 + "rdma_ctl_present": true,
352 + "iperf3": false
353 + },
354 + "m4mm": {
355 + "status": "reachable",
356 + "chip": "Apple M4 Pro",
357 + "ram_gb": 48,
358 + "macos": "26.5.1",
359 + "gpu_cores": 16,
360 + "disk_free_gb": 375,
361 + "system_python": "3.9.6",
362 + "mlx_preinstalled": false,
363 + "load_avg_at_probe": "{ 1,26 1,38 1,39 }",
364 + "en0_media": "autoselect (1000baseT <full-duplex>)",
365 + "homebrew_pythons": [
366 + "/opt/homebrew/bin/python3.13",
367 + "/opt/homebrew/bin/python3.14"
368 + ],
369 + "uv": true,
370 + "xcode": "/Library/Developer/CommandLineTools",
371 + "metal_compiler": null,
372 + "rdma_ctl_present": true,
373 + "iperf3": false
374 + },
375 + "m4mn": {
376 + "status": "reachable",
377 + "chip": "Apple M4 Pro",
378 + "ram_gb": 48,
379 + "macos": "26.5.1",
380 + "gpu_cores": 16,
381 + "disk_free_gb": 367,
382 + "system_python": "3.9.6",
383 + "mlx_preinstalled": false,
384 + "load_avg_at_probe": "{ 1,11 1,27 1,35 }",
385 + "en0_media": "autoselect (1000baseT <full-duplex>)",
386 + "homebrew_pythons": [
387 + "/opt/homebrew/bin/python3.13",
388 + "/opt/homebrew/bin/python3.14"
389 + ],
390 + "uv": true,
391 + "xcode": "/Library/Developer/CommandLineTools",
392 + "metal_compiler": null,
393 + "rdma_ctl_present": true,
394 + "iperf3": false
395 + },
396 + "m5ma": {
397 + "status": "reachable",
398 + "chip": "Apple M5 Pro",
399 + "ram_gb": 24,
400 + "macos": "27.0",
401 + "gpu_cores": 16,
402 + "disk_free_gb": 382,
403 + "system_python": "3.9.6",
404 + "mlx_preinstalled": false,
405 + "load_avg_at_probe": "{ 2,66 2,58 2,53 }",
406 + "en0_media": "autoselect (1000baseT <full-duplex>)",
407 + "homebrew_pythons": [
408 + "/opt/homebrew/bin/python3.13",
409 + "/opt/homebrew/bin/python3.14"
410 + ],
411 + "uv": true,
412 + "xcode": "/Library/Developer/CommandLineTools",
413 + "metal_compiler": null,
414 + "rdma_ctl_present": true,
415 + "iperf3": false
416 + },
417 + "m6ma": {
418 + "status": "reachable",
419 + "chip": "Apple M6",
420 + "ram_gb": 24,
421 + "macos": "27.0",
422 + "gpu_cores": 12,
423 + "disk_free_gb": 396,
424 + "system_python": "3.9.6",
425 + "mlx_preinstalled": false,
426 + "load_avg_at_probe": "{ 2,45 2,45 2,42 }",
427 + "en0_media": "autoselect (1000baseT <full-duplex>)",
428 + "homebrew_pythons": [
429 + "/opt/homebrew/bin/python3.13",
430 + "/opt/homebrew/bin/python3.14"
431 + ],
432 + "uv": true,
433 + "xcode": "/Library/Developer/CommandLineTools",
434 + "metal_compiler": null,
435 + "rdma_ctl_present": true,
436 + "iperf3": false
437 + },
438 + "m6mb": {
439 + "status": "reachable",
440 + "chip": "Apple M6",
441 + "ram_gb": 24,
442 + "macos": "27.0",
443 + "gpu_cores": 12,
444 + "disk_free_gb": 394,
445 + "system_python": "3.9.6",
446 + "mlx_preinstalled": false,
447 + "load_avg_at_probe": "{ 1,99 2,24 2,27 }",
448 + "en0_media": "autoselect (1000baseT <full-duplex>)",
449 + "homebrew_pythons": [
450 + "/opt/homebrew/bin/python3.13",
451 + "/opt/homebrew/bin/python3.14"
452 + ],
453 + "uv": true,
454 + "xcode": "/Library/Developer/CommandLineTools",
455 + "metal_compiler": null,
456 + "rdma_ctl_present": true,
457 + "iperf3": false
458 + }
459 + },
460 + "network": {
461 + "lan": "1 GbE switch (all nodes negotiate 1000baseT on en0)",
462 + "thunderbolt": "no active Thunderbolt link between any two nodes at probe time (RDMA/JACCL blocked pending cabling)",
463 + "rtt_ms": [
464 + {
465 + "from": "M3U96a",
466 + "to": "lan-host-85",
467 + "min": 0.596,
468 + "avg": 0.732,
469 + "max": 1.0
470 + },
471 + {
472 + "from": "M3U96a",
473 + "to": "lan-host-61",
474 + "min": 0.367,
475 + "avg": 0.572,
476 + "max": 1.179
477 + },
478 + {
479 + "from": "M3U96a",
480 + "to": "lan-host-70",
481 + "min": 0.529,
482 + "avg": 0.634,
483 + "max": 0.865
484 + },
485 + {
486 + "from": "M3U96a",
487 + "to": "lan-host-77",
488 + "min": 0.357,
489 + "avg": 0.53,
490 + "max": 0.741
491 + },
492 + {
493 + "from": "M3U96a",
494 + "to": "lan-host-103",
495 + "min": 0.623,
496 + "avg": 0.742,
497 + "max": 1.216
498 + },
499 + {
500 + "from": "M3U96a",
501 + "to": "lan-host-104",
502 + "min": 0.583,
503 + "avg": 0.789,
504 + "max": 1.446
505 + },
506 + {
507 + "from": "M3U96a",
508 + "to": "lan-host-92",
509 + "min": 0.582,
510 + "avg": 0.771,
511 + "max": 1.318
512 + },
513 + {
514 + "from": "M3U96a",
515 + "to": "lan-host-93",
516 + "min": 0.6,
517 + "avg": 1.021,
518 + "max": 1.187
519 + },
520 + {
521 + "from": "M3U96a",
522 + "to": "lan-host-105",
523 + "min": 0.53,
524 + "avg": 0.741,
525 + "max": 0.905
526 + },
527 + {
528 + "from": "M3U96a",
529 + "to": "lan-host-106",
530 + "min": 0.571,
531 + "avg": 0.766,
532 + "max": 1.275
533 + },
534 + {
535 + "from": "M3U96a",
536 + "to": "lan-host-112",
537 + "min": 0.593,
538 + "avg": 0.786,
539 + "max": 1.431
540 + },
541 + {
542 + "from": "M3U96a",
543 + "to": "lan-host-72",
544 + "min": 0.393,
545 + "avg": 0.577,
546 + "max": 0.872
547 + },
548 + {
549 + "from": "M3U96a",
550 + "to": "lan-host-76",
551 + "min": 0.538,
552 + "avg": 0.762,
553 + "max": 1.287
554 + },
555 + {
556 + "from": "M3U96a",
557 + "to": "lan-host-62",
558 + "min": 0.468,
559 + "avg": 0.694,
560 + "max": 1.189
561 + },
562 + {
563 + "from": "M3U96a",
564 + "to": "lan-host-63",
565 + "min": 0.54,
566 + "avg": 0.657,
567 + "max": 1.317
568 + },
569 + {
570 + "from": "M3U96a",
571 + "to": "lan-host-66",
572 + "min": 0.455,
573 + "avg": 0.618,
574 + "max": 1.019
575 + },
576 + {
577 + "from": "M3U96a",
578 + "to": "lan-host-68",
579 + "min": 0.487,
580 + "avg": 0.667,
581 + "max": 1.29
582 + },
583 + {
584 + "from": "M4M64a",
585 + "to": "M3U96b",
586 + "min": 0.726,
587 + "avg": 0.869,
588 + "max": 1.328
589 + },
590 + {
591 + "from": "M4M64a",
592 + "to": "M2U64",
593 + "min": 0.598,
594 + "avg": 0.69,
595 + "max": 0.862
596 + }
597 + ],
598 + "thunderbolt_active_links": {
599 + "M3U96a": false,
600 + "M3U96b": false,
601 + "M4M64a": false,
602 + "M2U64": false,
603 + "M4M64c": false,
604 + "M5M36": false
605 + },
606 + "iperf3_gbits": [
607 + {
608 + "from": "M3U96a",
609 + "to": "M3U96b",
610 + "sent": 0.945,
611 + "recv": 0.94
612 + },
613 + {
614 + "from": "M3U96a",
615 + "to": "M4M64a",
616 + "sent": 0.947,
617 + "recv": 0.941
618 + },
619 + {
620 + "from": "M3U96a",
621 + "to": "M2U64",
622 + "sent": 0.945,
623 + "recv": 0.94
624 + },
625 + {
626 + "from": "M3U96a",
627 + "to": "M4M64c",
628 + "sent": 0.946,
629 + "recv": 0.94
630 + },
631 + {
632 + "from": "M3U96a",
633 + "to": "M3U96b",
634 + "sent": 0.945,
635 + "recv": 0.94
636 + }
637 + ]
638 + },
639 + "software": {
640 + "mlx_venv": "0.32.4",
641 + "mlx_lm_venv": "0.32.2",
642 + "python_venv": "3.12",
643 + "third_party_commits": {
644 + "colibri": "0a5504a1dc64f86fcd0fe7db298b0448c730e7e6",
645 + "mlx": "bd63d3803008edda03ce74080955d9879e2ef1eb",
646 + "mlx-lm": "736ab0d8c1d0edc79c3d798044fe6904e2fa23be",
647 + "llama.cpp": "a91eab67d8389d979d29fdf6633b2edd0718eb4b"
648 + }
649 + }
650 +}
\ No newline at end of file
added benchmarks/configs/models.yaml +27 −0
@@ -0,0 +1,27 @@
1 +models:
2 + olmoe-1b-7b-0125-4bit:
3 + repo: mlx-community/OLMoE-1B-7B-0125-Instruct-4bit
4 + revision: e23844197887b031e7ddddbb0b8959c5a6853a7b
5 + license: apache-2.0
6 + base: allenai/OLMoE-1B-7B-0125-Instruct
7 + layers: 16
8 + experts: 64
9 + top_k: 8
10 + shared_experts: 0
11 + params_active_total: "1.3B / 6.9B"
12 + quantization: {bits: 4, group_size: 64, mode: affine}
13 + bytes_per_expert: 3538944
14 + tier: small
15 + qwen3-30b-a3b-2507-4bit:
16 + repo: mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit
17 + revision: e9675aa3ca5f900ccef55267914466d55ab325fa
18 + license: apache-2.0
19 + base: Qwen/Qwen3-30B-A3B-Instruct-2507
20 + layers: 48
21 + experts: 128
22 + top_k: 8
23 + shared_experts: 0
24 + params_active_total: "3.3B / 30.5B"
25 + quantization: {bits: 4, group_size: 64, mode: affine}
26 + tier: medium
27 + downloaded_on: [M3U96a]
added benchmarks/raw/tests/greedy_olmoe_cap16_20261101T014415.json +14 −0
@@ -0,0 +1,14 @@
1 +[
2 + {
3 + "prompt": "The capital of Canada is",
4 + "ref": "Ottawa.\n\nOttawa is the capital of Canada, and it is located in the province of Ontario. It is situated on the south bank of the Ottawa",
5 + "titan": "Ottawa.\n\nOttawa is the capital of Canada, and it is located in the province of Ontario. It is situated on the south bank of the Ottawa",
6 + "equal": true
7 + },
8 + {
9 + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
10 + "ref": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n",
11 + "titan": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n",
12 + "equal": true
13 + }
14 +]
\ No newline at end of file
added benchmarks/raw/tests/numerical_olmoe_cap16_2026-11-01T014410.json +50 −0
@@ -0,0 +1,50 @@
1 +{
2 + "capacity": 16,
3 + "prompts": [
4 + {
5 + "prompt": "The capital of Canada is",
6 + "tokens": 5,
7 + "max_abs_dlogit": 0.0,
8 + "argmax_equal": true,
9 + "hit_rate": 0.2580101180438449,
10 + "misses": 440,
11 + "stall_s": 0.08842262497637421,
12 + "layers_routed": 46
13 + },
14 + {
15 + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
16 + "tokens": 21,
17 + "max_abs_dlogit": 0.0,
18 + "argmax_equal": true,
19 + "hit_rate": 0.42512908777969016,
20 + "misses": 1002,
21 + "stall_s": 0.19870067050214857,
22 + "layers_routed": 127
23 + },
24 + {
25 + "prompt": "Q: If a train travels 60 km in 45 minute",
26 + "tokens": 25,
27 + "max_abs_dlogit": 0.0,
28 + "argmax_equal": true,
29 + "hit_rate": 0.41904344010530936,
30 + "misses": 1324,
31 + "stall_s": 0.2609664188930765,
32 + "layers_routed": 161
33 + },
34 + {
35 + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
36 + "tokens": 26,
37 + "max_abs_dlogit": 0.0,
38 + "argmax_equal": true,
39 + "hit_rate": 0.45278969957081544,
40 + "misses": 1275,
41 + "stall_s": 0.2480922951363027,
42 + "layers_routed": 164
43 + }
44 + ],
45 + "tol": 0.001,
46 + "ts": "2026-11-01T01:44:10",
47 + "max_abs_dlogit_worst": 0.0,
48 + "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b",
49 + "model_type": "olmoe"
50 +}
\ No newline at end of file
added benchmarks/raw/tests/numerical_olmoe_cap32_2026-11-01T014409.json +50 −0
@@ -0,0 +1,50 @@
1 +{
2 + "capacity": 32,
3 + "prompts": [
4 + {
5 + "prompt": "The capital of Canada is",
6 + "tokens": 5,
7 + "max_abs_dlogit": 0.0,
8 + "argmax_equal": true,
9 + "hit_rate": 0.0,
10 + "misses": 430,
11 + "stall_s": 0.08661957993172109,
12 + "layers_routed": 16
13 + },
14 + {
15 + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
16 + "tokens": 21,
17 + "max_abs_dlogit": 0.0,
18 + "argmax_equal": true,
19 + "hit_rate": 0.6060606060606061,
20 + "misses": 429,
21 + "stall_s": 0.085544079891406,
22 + "layers_routed": 40
23 + },
24 + {
25 + "prompt": "Q: If a train travels 60 km in 45 minute",
26 + "tokens": 25,
27 + "max_abs_dlogit": 0.0,
28 + "argmax_equal": true,
29 + "hit_rate": 0.6090458488228006,
30 + "misses": 631,
31 + "stall_s": 0.12506033421959728,
32 + "layers_routed": 58
33 + },
34 + {
35 + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
36 + "tokens": 26,
37 + "max_abs_dlogit": 0.0,
38 + "argmax_equal": true,
39 + "hit_rate": 0.6224188790560472,
40 + "misses": 512,
41 + "stall_s": 0.10236366488970816,
42 + "layers_routed": 49
43 + }
44 + ],
45 + "tol": 0.001,
46 + "ts": "2026-11-01T01:44:09",
47 + "max_abs_dlogit_worst": 0.0,
48 + "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b",
49 + "model_type": "olmoe"
50 +}
\ No newline at end of file
added benchmarks/raw/tests/numerical_olmoe_cap64_2026-11-01T014408.json +50 −0
@@ -0,0 +1,50 @@
1 +{
2 + "capacity": 64,
3 + "prompts": [
4 + {
5 + "prompt": "The capital of Canada is",
6 + "tokens": 5,
7 + "max_abs_dlogit": 0.0,
8 + "argmax_equal": true,
9 + "hit_rate": 0.0,
10 + "misses": 430,
11 + "stall_s": 0.08624804276041687,
12 + "layers_routed": 16
13 + },
14 + {
15 + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
16 + "tokens": 21,
17 + "max_abs_dlogit": 0.0,
18 + "argmax_equal": true,
19 + "hit_rate": 0.4637904468412943,
20 + "misses": 348,
21 + "stall_s": 0.0698862480930984,
22 + "layers_routed": 16
23 + },
24 + {
25 + "prompt": "Q: If a train travels 60 km in 45 minute",
26 + "tokens": 25,
27 + "max_abs_dlogit": 0.0,
28 + "argmax_equal": true,
29 + "hit_rate": 0.823076923076923,
30 + "misses": 138,
31 + "stall_s": 0.028100415016524494,
32 + "layers_routed": 16
33 + },
34 + {
35 + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
36 + "tokens": 26,
37 + "max_abs_dlogit": 0.0,
38 + "argmax_equal": true,
39 + "hit_rate": 0.9234828496042217,
40 + "misses": 58,
41 + "stall_s": 0.012272500898689032,
42 + "layers_routed": 16
43 + }
44 + ],
45 + "tol": 0.001,
46 + "ts": "2026-11-01T01:44:08",
47 + "max_abs_dlogit_worst": 0.0,
48 + "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b",
49 + "model_type": "olmoe"
50 +}
\ No newline at end of file
added benchmarks/raw/tests/numerical_olmoe_cap8_2026-11-01T014411.json +50 −0
@@ -0,0 +1,50 @@
1 +{
2 + "capacity": 8,
3 + "prompts": [
4 + {
5 + "prompt": "The capital of Canada is",
6 + "tokens": 5,
7 + "max_abs_dlogit": 0.0,
8 + "argmax_equal": true,
9 + "hit_rate": 0.1625,
10 + "misses": 536,
11 + "stall_s": 0.1006133749615401,
12 + "layers_routed": 80
13 + },
14 + {
15 + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
16 + "tokens": 21,
17 + "max_abs_dlogit": 0.0,
18 + "argmax_equal": true,
19 + "hit_rate": 0.4425981873111782,
20 + "misses": 1476,
21 + "stall_s": 0.2634192098630592,
22 + "layers_routed": 331
23 + },
24 + {
25 + "prompt": "Q: If a train travels 60 km in 45 minute",
26 + "tokens": 25,
27 + "max_abs_dlogit": 0.0,
28 + "argmax_equal": true,
29 + "hit_rate": 0.375,
30 + "misses": 1995,
31 + "stall_s": 0.3573505834210664,
32 + "layers_routed": 399
33 + },
34 + {
35 + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
36 + "tokens": 26,
37 + "max_abs_dlogit": 0.0,
38 + "argmax_equal": true,
39 + "hit_rate": 0.3878048780487805,
40 + "misses": 2008,
41 + "stall_s": 0.3646473821718246,
42 + "layers_routed": 410
43 + }
44 + ],
45 + "tol": 0.001,
46 + "ts": "2026-11-01T01:44:11",
47 + "max_abs_dlogit_worst": 0.0,
48 + "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b",
49 + "model_type": "olmoe"
50 +}
\ No newline at end of file
added paper/main.tex +110 −0
@@ -0,0 +1,110 @@
1 +% ============================================================================
2 +% UQO Working Paper No. 33 — TITAN-MoE
3 +% ============================================================================
4 +\documentclass[12pt,letterpaper]{article}
5 +\usepackage[utf8]{inputenc}
6 +\usepackage[T1]{fontenc}
7 +\usepackage[english]{babel}
8 +\usepackage[letterpaper,top=1in,bottom=1in,left=1in,right=1in]{geometry}
9 +\usepackage{mathptmx}
10 +\usepackage{microtype}
11 +\usepackage{setspace}
12 +\onehalfspacing
13 +\usepackage{amsmath,amssymb,amsthm}
14 +\usepackage{booktabs}
15 +\usepackage{tabularx}
16 +\usepackage{multirow}
17 +\usepackage{threeparttable}
18 +\usepackage{siunitx}
19 +\sisetup{input-symbols = {()}, table-format = -1.3, parse-numbers = false}
20 +\usepackage{graphicx}
21 +\usepackage{subcaption}
22 +\usepackage{float}
23 +\usepackage[dvipsnames]{xcolor}
24 +\usepackage[colorlinks=true,linkcolor=NavyBlue,citecolor=NavyBlue,urlcolor=NavyBlue,breaklinks=true,
25 + pdftitle={TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon},
26 + pdfauthor={Simon-Pierre Boucher}]{hyperref}
27 +\usepackage[authoryear,round,semicolon]{natbib}
28 +\bibliographystyle{aer}
29 +\usepackage[font=small,labelfont=bf,labelsep=period,justification=justified,singlelinecheck=false]{caption}
30 +\usepackage{fancyhdr}
31 +\pagestyle{fancy}
32 +\fancyhf{}
33 +\renewcommand{\headrulewidth}{0pt}
34 +\fancyfoot[C]{\thepage}
35 +\usepackage{titlesec}
36 +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{}
37 +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{}
38 +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{}
39 +\usepackage[toc,page]{appendix}
40 +\usepackage{enumitem}
41 +\setlist{nosep,leftmargin=*}
42 +\usepackage{algorithm}
43 +\usepackage{algpseudocode}
44 +\usepackage{tikz}
45 +\usetikzlibrary{arrows.meta,positioning,shapes.geometric,fit,calc}
46 +\usepackage{array}
47 +\newcolumntype{L}[1]{>{\raggedright\arraybackslash}p{#1}}
48 +\newcolumntype{R}[1]{>{\raggedleft\arraybackslash}p{#1}}
49 +\graphicspath{{figures/}}
50 +
51 +% --- theorem environments (shared counter, numbered by section) ---
52 +\theoremstyle{plain}
53 +\newtheorem{theorem}{Theorem}[section]
54 +\newtheorem{proposition}[theorem]{Proposition}
55 +\newtheorem{lemma}[theorem]{Lemma}
56 +\newtheorem{corollary}[theorem]{Corollary}
57 +\theoremstyle{definition}
58 +\newtheorem{definition}[theorem]{Definition}
59 +\newtheorem{assumption}[theorem]{Assumption}
60 +\theoremstyle{remark}
61 +\newtheorem{remark}[theorem]{Remark}
62 +\newtheorem{observation}[theorem]{Observation}
63 +
64 +% --- macros ---
65 +\newcommand{\E}{\mathcal{E}}
66 +\newcommand{\Lset}{\mathcal{L}}
67 +\newcommand{\Nset}{\mathcal{N}}
68 +\newcommand{\R}{\mathbb{R}}
69 +\newcommand{\Ex}{\mathbb{E}}
70 +\newcommand{\ind}{\mathbb{1}}
71 +\newcommand{\toks}{\mathrm{tok/s}}
72 +\newcommand{\code}[1]{\texttt{#1}}
73 +\newcommand{\titan}{TITAN-MoE}
74 +\newcommand{\gatherqmm}{\code{gather\_qmm}}
75 +
76 +% ============================================================================
77 +% METADATA
78 +% ============================================================================
79 +\newcommand{\WPnumber}{33}
80 +\newcommand{\WPtitle}{TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon}
81 +\newcommand{\WPsubtitle}{A Measured Systems Study on a Heterogeneous Mac Cluster}
82 +\newcommand{\WPdate}{November 2026}
83 +\newcommand{\WPversion}{0.1}
84 +\input{sections/abstract}
85 +\newcommand{\WPkeywords}{mixture of experts, inference systems, Apple Silicon, expert offloading, predictive prefetching, distributed inference, energy efficiency}
86 +\newcommand{\WPjel}{C63, C88, L86, O33}
87 +\newcommand{\WPauthor}{Simon-Pierre Boucher}
88 +\newcommand{\WPaffiliation}{Département des sciences administratives\\Université du Québec en Outaouais}
89 +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca}
90 +\newcommand{\WPaddress}{Gatineau -- Pavillon Alexandre-Taché\\283, boulevard Alexandre-Taché\\Gatineau, Québec, Canada J9A 1L8}
91 +
92 +\begin{document}
93 +\input{sections/titlepage}
94 +\input{sections/introduction}
95 +\input{sections/background}
96 +\input{sections/related}
97 +\input{sections/formulation}
98 +\input{sections/architecture}
99 +\input{sections/algorithms}
100 +\input{sections/implementation}
101 +\input{sections/setup}
102 +\input{sections/results}
103 +\input{sections/ablations}
104 +\input{sections/robustness}
105 +\input{sections/limitations}
106 +\input{sections/conclusion}
107 +\newpage
108 +\bibliography{references}
109 +\input{sections/appendix}
110 +\end{document}
modified paper/references.bib +14 −0
@@ -394,3 +394,17 @@
394 394 year={2026},
395 395 note={Version 2.1.0, Apache-2.0. Accessed 2026-11-01. No paper or citation provided by the project}
396 396 }
397 +
398 +% --- added 2026-11-01, verified on arxiv.org/abs pages (title, first authors, date) ---
399 +@article{muennighoff2024olmoe,
400 + title = {{OLMoE}: Open Mixture-of-Experts Language Models},
401 + author = {Muennighoff, Niklas and Soldaini, Luca and Groeneveld, Dirk and Lo, Kyle and Morrison, Jacob and Min, Sewon and Shi, Weijia and Walsh, Pete and Tafjord, Oyvind and Lambert, Nathan and Gu, Yuling and Arora, Shane and Bhagia, Akshita and Schwenk, Dustin and Wadden, David and Wettig, Alexander and Hui, Binyuan and Dettmers, Tim and Kiela, Douwe and Farhadi, Ali and Smith, Noah A. and Koh, Pang Wei and Singh, Amanpreet and Hajishirzi, Hannaneh},
402 + journal = {arXiv preprint arXiv:2409.02060},
403 + year = {2024}
404 +}
405 +@article{yang2025qwen3,
406 + title = {Qwen3 Technical Report},
407 + author = {Yang, An and Li, Anfeng and Yang, Baosong and others},
408 + journal = {arXiv preprint arXiv:2505.09388},
409 + year = {2025}
410 +}
added paper/sections/ablations.tex +12 −0
@@ -0,0 +1,12 @@
1 +\section{Ablations}
2 +\label{sec:ablations}
3 +The ablation grid of the protocol (predictor $\times$ depth $\times$ budget $\times$ policy $\times$ placement $\times$
4 +transport $\times$ kernels $\times$ batch) is executed gate by gate. In this version only the single-node factors have logs:
5 +\begin{itemize}
6 + \item \textbf{Install path} (per-miss synchronisation vs lazy batched installs): $+80$ to $+90\,\%$ decode throughput at 16 slots
7 + (Section~\ref{sec:implementation}).
8 + \item \textbf{Fast path} (device-side slot map when a layer is complete): 203 $\to$ 489~\toks{} at 64 slots.
9 + \item \textbf{Eviction policy} (LRU vs LFU): within 0.01 hit rate at every capacity in simulation.
10 + \item \textbf{Predictor} (none / prev-rank / transition$_d$ / probe / oracle) and \textbf{depth}: Table~\ref{tab:sim}, \textsc{pending} until the v2
11 + simulation and its online confirmation complete.
12 +\end{itemize}
added paper/sections/abstract.tex +15 −0
@@ -0,0 +1,15 @@
1 +\newcommand{\WPabstract}{%
2 +Sparse mixture-of-experts (MoE) language models touch only a few experts per token, so a model whose weights exceed a
3 +machine's memory can still be served if the right experts are in the right place at the right time. This working paper
4 +reports the design, implementation and measurement of \titan{}, an MLX-native inference runtime for Apple Silicon that
5 +manages expert residency with bounded per-layer slot tables, loads experts on demand from local NVMe, prefetches them with
6 +causal predictors, and (in its distributed mode) executes them on other Macs of a heterogeneous cluster connected by
7 +gigabit Ethernet. Every design choice is validated against mlx-lm on identical weights: with all experts resident the runtime is
8 +bit-identical in logits and within one percent of mlx-lm's decode throughput; with bounded residency the router's choices and
9 +the logits are unchanged by construction. We state five hypotheses (causal prefetch, topology-aware placement, hierarchical
10 +caching, fused Metal kernels, and adaptive orchestration on the latency--throughput--energy frontier) and test each one with a
11 +pre-registered protocol on a live inventory of a 21-node Mac cluster. This version (0.1) is a \emph{protocol and preliminary}
12 +working paper: it contains the verified hardware inventory, the prior-art audit, the single-node runtime and its correctness
13 +evidence, and the first controlled single-node experiments; distributed and energy experiments are reported only where their
14 +raw logs exist, and pending experiments are explicitly labelled as such.%
15 +}
added paper/sections/algorithms.tex +40 −0
@@ -0,0 +1,40 @@
1 +\section{Algorithms}
2 +\label{sec:algorithms}
3 +
4 +\begin{algorithm}[t]
5 +\caption{Decode step of one MoE layer in \titan{} (single node)}
6 +\label{alg:layer}
7 +\begin{algorithmic}[1]
8 +\Require hidden rows $X$, router indices $I$ ($R \times K$), slot table $\mathcal{S}_l$, prefetcher $\mathcal{P}$, depth $d$
9 +\If{$\mathcal{S}_l$ holds every expert} \State \Return $\gatherqmm(X, \mathcal{S}_l, \mathrm{slotmap}[I])$ \Comment{no host sync}
10 +\EndIf
11 +\State $U \gets$ unique experts of $I$ (host) \Comment{the unavoidable synchronisation}
12 +\State $\mathcal{P}.\textsc{drain}(l, U)$ \Comment{install finished prefetches; wait for in-flight experts in $U$ (late)}
13 +\State $\hat S \gets \mathcal{P}.\textsc{predict}(l + d)$; $\mathcal{P}.\textsc{issue}(l + d, \hat S)$ \Comment{before this layer's compute}
14 +\State $M \gets U \setminus \mathrm{resident}(\mathcal{S}_l)$; read $M$ from the store as a batch; install into slots (lazy)
15 +\State $J \gets \mathrm{slot}(I)$
16 +\State \Return $\gatherqmm(X, \mathcal{S}_l, J)$ \Comment{same kernel, same bytes as mlx-lm}
17 +\end{algorithmic}
18 +\end{algorithm}
19 +
20 +\begin{algorithm}[t]
21 +\caption{Deterministic cost-aware placement (Gate F heuristic)}
22 +\label{alg:placement}
23 +\begin{algorithmic}[1]
24 +\Require expert popularity $\pi_e$ from fit traces, node costs $(c_n, M_n, \rho_{0n})$, expert size $b$
25 +\State sort experts by $\pi_e$ descending; $\mathrm{free}_n \gets M_n$
26 +\For{each expert $e$ in that order}
27 + \State $n^\star \gets \arg\min_{n:\ \mathrm{free}_n \ge b} \big(c_n + \ind[n \ne 0]\,\rho_{0n}\big) \cdot \pi_e$ \Comment{expected per-step cost}
28 + \State place $e$ on $n^\star$; $\mathrm{free}_{n^\star} \gets \mathrm{free}_{n^\star} - b$
29 +\EndFor
30 +\State replicate the $\rho$ most popular experts on node $0$ if $\mathrm{free}_0$ allows (hot set)
31 +\State experts placed nowhere fall back to node $0$'s slot table (SSD tier, demand loading)
32 +\end{algorithmic}
33 +\end{algorithm}
34 +
35 +Algorithm~\ref{alg:layer} is the per-layer path of Section~\ref{sec:architecture}; the prefetch for layer $l+d$ is issued
36 +\emph{before} the current layer's kernels are dispatched so that the reader threads overlap the GPU work of the next $d$ layers.
37 +Algorithm~\ref{alg:placement} is the first placement heuristic: greedy by popularity, cost-aware by node, with replication of a
38 +hot set on the coordinator. It runs in $\mathcal{O}(LE \log LE + LE N)$ time, negligible against inference, and will be the
39 +static baseline against which the adaptive scheduler (online re-estimation of $\pi$ and of the node costs) is compared.
40 +Predictor fitting is $\mathcal{O}(T L K^2)$ for transition counts and one logistic regression per layer for the probe.
added paper/sections/appendix.tex +21 −0
@@ -0,0 +1,21 @@
1 +\begin{appendices}
2 +\section{Reproducibility}
3 +\label{app:repro}
4 +\paragraph{Software.} Python 3.12 virtual environment with MLX 0.32.4 and mlx-lm 0.32.2 (\code{requirements-lock.txt});
5 +reference repositories audited at colibri \code{0a5504a} (2026-10-31), mlx \code{bd63d38} (2026-11-01), mlx-lm \code{736ab0d}
6 +(2026-10-30), llama.cpp \code{a91eab6} (2026-10-31).
7 +\paragraph{Commands.} Correctness: \code{.venv/bin/python -m pytest tests -q}. Capacity sweep:
8 +\code{benchmarks/scripts/bench\_decode.py --backend \{mlx\_lm,titan\} --capacity C --predictor P --traces T}. Traces:
9 +\code{benchmarks/scripts/collect\_traces.py}. Simulation: \code{benchmarks/scripts/simulate\_cache.py}. SSD tier:
10 +\code{benchmarks/scripts/ssd\_bandwidth.py} after \code{sudo purge}. Node preparation: \code{scripts/cluster\_setup.sh}.
11 +\paragraph{Raw data.} \code{benchmarks/raw/gateA/} (smoke baseline, SSD), \code{gateB/} (design microbenchmarks, capacity sweep,
12 +tests), \code{gateC/} (traces, online prefetch, miss-cost breakdown, Qwen3 baselines); processed simulations in
13 +\code{benchmarks/processed/}. Logs are append-only JSON lines.
14 +\paragraph{Prior-art table.} \code{research/prior\_art.csv} (59 rows, evidence URL per row) and
15 +\code{research/RELATED\_WORK.md}.
16 +\paragraph{AI-assistance disclosure.} The software, experiments and this manuscript were produced by Claude Code (Anthropic,
17 +model Claude Fable 5.1) acting as an autonomous research engineer under the author's written mandate
18 +(\code{docs/MANDATE.md}); the author set the research question, hypotheses, integrity rules and gates, and is responsible for
19 +the content. Every quantitative claim traces to a logged measurement; the agent's decisions are recorded in
20 +\code{research/DECISIONS.md} and its chronology in \code{research/LOG.md}.
21 +\end{appendices}
added paper/sections/architecture.tex +72 −0
@@ -0,0 +1,72 @@
1 +\section{Architecture}
2 +\label{sec:architecture}
3 +
4 +Figure~\ref{fig:arch} shows the components of \titan{}. The host model (embeddings, attention, norms, routers, KV cache) is
5 +mlx-lm's own module tree, loaded lazily so that the stacked expert tensors are never materialised; each layer's expert block
6 +is replaced by a \titan{} block that owns a \emph{slot table}. Everything else is orchestration around that table.
7 +
8 +\begin{figure}[t]
9 +\centering
10 +\begin{tikzpicture}[node distance=4mm and 6mm, every node/.style={font=\small},
11 + box/.style={draw, rounded corners=2pt, minimum height=7mm, minimum width=24mm, align=center, fill=gray!8},
12 + tier/.style={draw, dashed, rounded corners=2pt, minimum height=7mm, minimum width=24mm, align=center},
13 + arr/.style={-{Latex[length=2mm]}, thick}]
14 + \node[box] (reg) {Expert registry\\\scriptsize safetensors byte ranges};
15 + \node[box, right=of reg] (store) {Expert store\\\scriptsize \code{pread}, F\_NOCACHE};
16 + \node[box, right=of store] (slots) {Slot tables ($L$)\\\scriptsize capacity $C$, LRU/LFU};
17 + \node[box, right=of slots] (exec) {Execution block\\\scriptsize \gatherqmm{} on slots};
18 + \node[box, below=of store] (pred) {Predictors\\\scriptsize prev-rank, transition$_d$, probe};
19 + \node[box, right=of pred] (pf) {Prefetcher\\\scriptsize reader pool, drain/install};
20 + \node[box, right=of pf] (host) {Host model (mlx-lm)\\\scriptsize attention, router, KV};
21 + \node[tier, below=of pf] (net) {Transport (Gate E)\\\scriptsize TCP, authenticated};
22 + \node[tier, right=of net] (sched) {Scheduler (Gate F)\\\scriptsize cost model, placement};
23 + \draw[arr] (reg) -- (store);
24 + \draw[arr] (store) -- (slots);
25 + \draw[arr] (slots) -- (exec);
26 + \draw[arr] (host) -- node[above,font=\scriptsize]{routes} (exec);
27 + \draw[arr] (exec) -- node[left,font=\scriptsize]{routes, probs} (pf);
28 + \draw[arr] (pf) -- (pred);
29 + \draw[arr] (pf) -- node[left,font=\scriptsize]{install} (slots);
30 + \draw[arr] (pred) -- node[above,font=\scriptsize]{$\hat S$} (pf);
31 + \draw[arr, dashed] (net) -- (slots);
32 + \draw[arr, dashed] (sched) -- (net);
33 +\end{tikzpicture}
34 +\caption{Components of \titan{} on one node. Solid boxes are implemented and tested in version 0.1; dashed boxes are the
35 +distributed components whose status is given in Section~\ref{sec:limitations}.}
36 +\label{fig:arch}
37 +\end{figure}
38 +
39 +\paragraph{Expert registry.} The registry parses \code{config.json} and the safetensors headers of a checkpoint directory
40 +and records, for every (layer, expert) and every matrix (gate, up, down) and component (packed weight, scales, biases), the
41 +absolute byte offset and length in the file, together with the routing semantics (softmax-then-top-$k$, renormalisation flag,
42 +shared experts) and the quantization (bits, group size). It accepts only verified formats (OLMoE and Qwen3-MoE in this
43 +version) and refuses the others rather than guessing.
44 +
45 +\paragraph{Expert store (SSD tier).} An expert is read with one \code{pread} per tensor at the registry's offsets, wrapped
46 +into MLX arrays without further copies on the Python side; bytes, read count and read time are accounted. The file descriptor
47 +can be opened with \code{F\_NOCACHE} so that reads bypass the unified buffer cache, which is how the cold-NVMe tier is
48 +measured (after \code{purge}); without it the same code measures the page-cache tier.
49 +
50 +\paragraph{Slot tables.} Each layer owns $C$ slots in stacked MLX buffers of the exact shapes mlx-lm uses for its stacked
51 +experts, so that the same \gatherqmm{} kernel can run on them. A dictionary maps resident expert ids to slots; eviction is
52 +LRU by default (LFU available). Installing an expert is a slice assignment into the buffers, which MLX performs in place.
53 +Misses are read as a batch per layer and installed lazily so that the layer's kernel consumes the new buffers in the same
54 +graph: there is no per-expert synchronisation. The table records hits, misses, evictions, loads and stall time.
55 +
56 +\paragraph{Execution block.} Given the router's indices, the block materialises them on the host (the one synchronisation a
57 +demand-loading design cannot avoid), ensures residency, remaps expert ids to slot ids and runs \gatherqmm{} for the up, gate
58 +and down projections with mlx-lm's compiled SwiGLU. When a layer's table holds every expert (the all-resident control) a
59 +device-side slot map is used and no synchronisation occurs, which recovers mlx-lm's throughput. Prefill rows whose union of
60 +experts exceeds $C$ are chunked greedily by rows, which is exact because rows are independent.
61 +
62 +\paragraph{Predictors and prefetcher.} A predictor proposes up to $m$ experts for a layer from information available at the
63 +decision point: the previous token's routing and router probabilities (token horizon), or the routing of layer $l-d$ at the
64 +current token (layer horizon, depth $d$). The prefetcher submits reads to a small thread pool, installs completed fetches when
65 +the target layer is reached, waits for fetches that are needed but still in flight (counted as \emph{late}, with the residual
66 +wait charged as stall), and never changes which experts run.
67 +
68 +\paragraph{Hierarchical memory and distributed execution.} The remote tier does not move weights during decode (Section
69 +\ref{sec:prefetch-condition}): a remote node that holds an expert executes it on the coordinator's activations and returns the
70 +$d$-dimensional result, exactly as colibri's cluster mode and as expert parallelism do. The transport (Gate E) is a persistent
71 +authenticated TCP connection per worker with a per-layer batched request, timeouts and local fallback; placement (Gate F)
72 +follows the cost model of Section~\ref{sec:formulation} with measured $c_n$, $M_n$, $\rho_{0n}$ and $\lambda$ per node.
added paper/sections/background.tex +40 −0
@@ -0,0 +1,40 @@
1 +\section{Background}
2 +\label{sec:background}
3 +
4 +\subsection{Sparse MoE inference}
5 +Let a decoder have layers $l \in \Lset = \{1,\dots,L\}$. In an MoE layer the token representation $h \in \R^{d}$ is routed by
6 +$g = \operatorname{softmax}(W_r h) \in \R^{E}$; the top-$K$ entries of $g$ select experts $S(h) \subset \E=\{1,\dots,E\}$,
7 +$|S(h)| = K$, and the layer output is $\sum_{e \in S(h)} \tilde g_e\, f_e(h)$ where $f_e$ is a gated feed-forward network
8 +(SwiGLU in the models we use) and $\tilde g$ equals $g$ restricted to $S(h)$, renormalised in some architectures. OLMoE
9 +\citep{muennighoff2024olmoe} and Qwen3-MoE \citep{yang2025qwen3} follow this template without a shared expert; DeepSeek-style models add shared experts and
10 +grouped routing, which our registry records but this version does not execute.
11 +
12 +During prefill the $T$ prompt tokens are processed together and a layer typically needs most of its experts (with $T=20$ and
13 +$K=8$ the union of selected experts already approaches $E=64$ in OLMoE), so prefill is compute-bound and resembles dense
14 +inference. During decode one token at a time selects $K$ experts per layer; the $L K$ expert weight matrices must be read from
15 +wherever they reside. If every expert is resident in unified memory, the decode step is bound by memory bandwidth and kernel
16 +launch latency \citep{javat2026siliconshowdown}; if some are not, the step is bound by the latency of fetching the missing
17 +ones. The cache hit rate therefore translates almost directly into throughput, which is why expert caching and prediction
18 +dominate the literature of Section~\ref{sec:related}.
19 +
20 +\subsection{Apple Silicon as an inference platform}
21 +An Apple Silicon system-on-chip shares one physical memory between CPU, GPU and Neural Engine. For offloading this removes
22 +the PCIe copy but not the fetch: a weight on disk must still be read into memory, and a weight on another machine must cross a
23 +network. The MLX framework exposes lazy arrays evaluated as a graph; inserting an expert into a resident buffer is a slice
24 +assignment that MLX performs in place when the buffer is not otherwise referenced (we measure 0.3~ms for one OLMoE expert in
25 +Section~\ref{sec:implementation}). The \gatherqmm{} primitive multiplies a batch of rows by a batch of quantized matrices selected
26 +by index, which is exactly the operation an MoE layer needs; mlx-lm uses it for all its 50-odd MoE architectures.
27 +
28 +Three facts about our cluster matter for every later number. First, all nodes negotiate 1000BASE-T on their Ethernet port
29 +(the switch is gigabit even though the Mac Studios have 10~GbE ports); iperf3 measures 0.945~Gbit/s between any two Studios and
30 +ping 0.5--1.0~ms round trip. Second, no Thunderbolt link is active between any two nodes, so RDMA over Thunderbolt~5 and
31 +MLX's JACCL backend are unavailable until cables are installed. Third, the development laptop is not on the cluster LAN; all
32 +distributed experiments run between LAN nodes with a Mac Studio as coordinator.
33 +
34 +\subsection{Baselines}
35 +mlx-lm \citep{hannun2023mlx} is the exact baseline: same weights, same quantization, same kernels. llama.cpp
36 +\citep{llamacpp2026} runs the same base checkpoints through its Metal backend but with its own K-quant formats, so its
37 +speed is reported with a precision flag and a perplexity check rather than as a like-for-like competitor. colibri
38 +\citep{colibri2026} streams experts from disk in C; on macOS its OLMoE engine is CPU-only (its Metal backend covers other
39 +engines), so it serves as a reference for the design space rather than a same-hardware competitor; its cluster mode, which
40 +assigns experts to worker Macs by a deterministic weighted hash, is the closest existing analogue of our static placement.
added paper/sections/conclusion.tex +9 −0
@@ -0,0 +1,9 @@
1 +\section{Conclusion}
2 +\label{sec:conclusion}
3 +We have built an MLX-native MoE runtime whose bounded expert residency is semantically invisible (bit-identical logits to mlx-lm)
4 +and whose all-resident path costs nothing (489 vs 494~\toks{}), characterised the cluster it will run on as it actually is
5 +(gigabit Ethernet, no Thunderbolt), and obtained a first negative result that the formulation explains: one-layer-ahead
6 +prefetching cannot pay when the fetch latency exceeds the per-layer compute time. The next steps are fixed by the protocol:
7 +depth-$d$ and token-horizon prefetching against the cold-NVMe tier, the medium model on nodes that cannot hold it, the
8 +two-node TCP transport with static placement, the cost-aware placement of Algorithm~\ref{alg:placement}, profiling-driven Metal
9 +work, and the energy frontier. Each will be reported with its raw logs, including the cases where \titan{} loses.
added paper/sections/formulation.tex +99 −0
@@ -0,0 +1,99 @@
1 +\section{Mathematical Formulation}
2 +\label{sec:formulation}
3 +
4 +\subsection{Objects}
5 +Let $\E$ be the set of (layer, expert) pairs, $|\E| = LE$, each of size $b$ bytes (3.375~MiB for OLMoE at 4 bits; equal
6 +within an architecture). Let $\Nset = \{0,1,\dots,N\}$ be the nodes, node $0$ being the coordinator that runs attention, the
7 +router and the KV cache, and let tier $\mathcal{T}_n = \{\mathrm{ram}, \mathrm{ssd}\}$ be the storage tiers of node $n$.
8 +Tokens are steps $t = 1,2,\dots$; the router of layer $l$ at step $t$ selects $S_{t,l} \subset \E_l$, $|S_{t,l}| = K$.
9 +Decision variables are binary and indexed by step:
10 +\begin{align}
11 + x_{e,n,t} &\in \{0,1\} \quad \text{expert $e$ is resident in the memory of node $n$ at step $t$,} \\
12 + z_{e,n,t} &\in \{0,1\} \quad \text{expert $e$ is \emph{executed} on node $n$ at step $t$,} \\
13 + p_{e,n,t} &\in \{0,1\} \quad \text{a prefetch of $e$ into node $n$ is issued during step $t$.}
14 +\end{align}
15 +Every expert exists on the local NVMe of each node that holds a copy of the checkpoint (the SSD tier is unbounded for our
16 +purposes and costs only time), so residency refers to memory.
17 +
18 +\subsection{Constraints}
19 +\begin{align}
20 + \sum_{e \in \E} b\, x_{e,n,t} &\le M_n \qquad \forall n, t && \text{(memory budget of node $n$)} \label{eq:mem}\\
21 + \sum_{n \in \Nset} z_{e,n,t} &= \ind[e \in S_{t,l(e)}] \qquad \forall e, t && \text{(every selected expert executed exactly once)} \label{eq:exec}\\
22 + z_{e,n,t} &\le x_{e,n,t} + m_{e,n,t} \qquad \forall e, n, t && \text{(execution needs residency or a demand miss $m$)} \label{eq:res}\\
23 + x_{e,n,t+1} &\le x_{e,n,t} + p_{e,n,t} + m_{e,n,t} \qquad \forall e, n, t && \text{(residency only grows by prefetch or miss)} \label{eq:dyn}
24 +\end{align}
25 +where $m_{e,n,t} \in \{0,1\}$ is the demand-miss indicator, itself a consequence of the router and of $x$. Constraint
26 +\eqref{eq:exec} encodes the correctness invariant: the set of executed experts is the router's set, whatever the placement.
27 +
28 +\subsection{Costs}
29 +The per-step latency decomposes along the layer sequence, because layer $l$ cannot start before layer $l-1$'s output exists:
30 +\begin{equation}
31 + \Lambda_t \;=\; \sum_{l=1}^{L} \Big[ \underbrace{\tau^{\mathrm{attn}}_{0}}_{\text{attention, node 0}}
32 + + \max_{n \in \Nset}\big( \underbrace{c_n \cdot \textstyle\sum_{e \in S_{t,l}} z_{e,n,t}}_{\text{expert compute on $n$}}
33 + + \underbrace{\rho_{0n}\,\ind[\exists e\!:\! z_{e,n,t}=1, n\neq 0]}_{\text{activation round trip}} \big)
34 + + \underbrace{\sigma_{t,l}}_{\text{stall}} \Big],
35 + \label{eq:latency}
36 +\end{equation}
37 +with $c_n$ the measured per-expert compute time of node $n$, $\rho_{0n}$ the measured round-trip time of a
38 +$d$-dimensional activation and its result between node $0$ and $n$ (on our LAN, $0.5$--$1.0$~ms), and the stall
39 +\begin{equation}
40 + \sigma_{t,l} \;=\; \max\Big(0,\; \max_{e \in S_{t,l},\, m_{e,n,t} = 1} \big( \lambda_{\mathrm{tier}(e)} - \eta_{e,t} \big) \Big),
41 + \label{eq:stall}
42 +\end{equation}
43 +where $\lambda_{\mathrm{tier}}$ is the fetch latency of the tier the expert comes from ($\lambda_{\mathrm{pc}} = 0.21$~ms page
44 +cache, $\lambda_{\mathrm{ssd}} = 0.43$~ms cold NVMe, $\lambda_{\mathrm{lan}} \approx 28$~ms for 3.4~MB over 0.945~Gbit/s, all
45 +measured on our hardware) and $\eta_{e,t}$ is the lead time between the moment a prefetch of $e$ was issued and the moment
46 +$e$ is needed ($\eta = 0$ for a demand miss). Energy per step is $\Phi_t = \sum_n P_n \cdot \Lambda^{(n)}_t$ with $P_n$ the
47 +package power of node $n$ measured by \code{powermetrics} during the run, and communication cost $\Gamma_t = \sum_{n \ne 0}
48 +\beta_{0n}\,(\text{activation bytes} + \text{weight bytes moved})$.
49 +
50 +\subsection{Problem}
51 +\begin{equation}
52 + \min_{x,z,p}\; \Ex\Big[\sum_{t} \Lambda_t + \lambda\,\Phi_t + \mu\,\Gamma_t\Big] \quad \text{s.t. \eqref{eq:mem}--\eqref{eq:dyn}},
53 + \label{eq:problem}
54 +\end{equation}
55 +where the expectation is over the router's sequence $\{S_{t,l}\}$, which is revealed online: at step $t$, $S_{t,l}$ becomes
56 +known only after layer $l-1$'s output is computed. The problem is a stochastic online integer program; its offline relaxation
57 +with known $S$ is the paging problem with multiple caches and non-uniform fetch costs, for which Belady's furthest-in-future
58 +rule is optimal per cache and which our simulator's \emph{oracle} predictor approximates.
59 +
60 +\subsection{When can a prefetch pay?}
61 +\label{sec:prefetch-condition}
62 +Consider a predictor issuing, when layer $l-d$'s routing is known at step $t$, a set $\hat S_{t,l}$ of $m$ experts for layer
63 +$l$, with recall $r = \Ex|\hat S \cap S|/K$ and precision $q = \Ex|\hat S \cap S|/m$. With per-layer decode time $\tau_\ell$
64 +the lead time is $\eta = d\,\tau_\ell$. Per layer, the expected stall without prefetch is
65 +$\sigma_0 = \lceil K(1-h)/w \rceil\, \lambda$ for hit rate $h$ and $w$ parallel loaders, and with prefetch
66 +\begin{equation}
67 + \sigma_d \;\approx\; \lceil K(1-h)(1-r')/w \rceil\,\lambda \;+\; \lceil K(1-h)\, r'/w \rceil\,\max(0, \lambda - d\,\tau_\ell),
68 + \label{eq:prefetch-gain}
69 +\end{equation}
70 +where $r'$ is the recall among the experts that would have missed. The prefetch also adds $m(1-q)$ useless fetches whose
71 +bandwidth and install cost $\kappa$ each is paid on the critical path when the loader is shared with demand misses.
72 +A prefetch can only reduce latency if
73 +\begin{equation}
74 + r'\,\min(\lambda, d\,\tau_\ell) \;>\; m(1-q)\,\kappa\;/\;K(1-h),
75 + \label{eq:condition}
76 +\end{equation}
77 +and it can \emph{remove} the stall only if $d\,\tau_\ell \ge \lambda$. On the laptop, $\tau_\ell = 0.128$~ms (489~\toks{} over
78 +16 layers), so a page-cache fetch ($\lambda = 0.21$~ms) needs $d \ge 2$ and a cold-NVMe fetch ($\lambda = 0.43$~ms) needs
79 +$d \ge 4$; a LAN fetch ($\lambda \approx 28$~ms) can never be hidden within a token on this model, which is why weights are not
80 +moved over the LAN during decode in \titan{} and remote experts are instead executed where they reside. The one-layer online
81 +experiment of Section~\ref{sec:results-prefetch} violated \eqref{eq:condition} ($d = 1$, $q \approx 0.5$) and lost throughput, as
82 +predicted; the depth-$d$ experiments test the condition directly.
83 +
84 +\begin{proposition}[Demand loading is $\mathcal{O}(1)$-competitive per layer only when the table exceeds the per-step
85 +working set]
86 +\label{prop:capacity}
87 +With per-layer tables of capacity $C \ge K$, LRU eviction, and no prefetch, the number of demand misses at a decode step is at
88 +most $K$ and at least $K - |S_{t,l} \cap S_{t-1,l}| $ when $C = K$; for $C \ge |\bigcup_{s \le \Delta} S_{t-s,l}|$ the misses at
89 +step $t$ are exactly the experts not used in the last $\Delta$ steps at that layer.
90 +\end{proposition}
91 +\begin{proof}
92 +With $C = K$ the table after step $t-1$ holds exactly $S_{t-1,l}$, so every expert of $S_{t,l} \setminus S_{t-1,l}$ misses and
93 +no other does; misses are bounded by $K$ because $|S_{t,l}| = K$. For the second claim, LRU keeps the most recently used
94 +$C$ distinct experts; if $C$ is at least the number of distinct experts used in the last $\Delta$ steps, all of them are
95 +resident, so a miss at step $t$ is precisely an expert unused during those $\Delta$ steps.
96 +\end{proof}
97 +The proposition is elementary but useful: it says that the hit rate of LRU at capacity $C$ is the fraction of a step's
98 +experts that recur within the $C$-recent window of that layer, a quantity we measure directly from traces
99 +(Section~\ref{sec:results-locality}) and which, for OLMoE, is 0.57 at $C = 16$ and 0.80 at $C = 32$.
added paper/sections/implementation.tex +29 −0
@@ -0,0 +1,29 @@
1 +\section{Implementation}
2 +\label{sec:implementation}
3 +
4 +\titan{} is about 1\,100 lines of Python over MLX 0.32.4 and mlx-lm 0.32.2 (Python 3.12), organised as
5 +\code{titan/\{registry, store, slots, moe, prefetch, predictors, runtime\}.py}, with benchmark drivers in
6 +\code{benchmarks/scripts/} and tests in \code{tests/}. Three implementation findings determined the design.
7 +
8 +\paragraph{In-place slot updates.} Writing the nine tensors of one OLMoE expert (3.4~MB) into stacked buffers of 64 slots
9 +(216~MB per matrix) costs 0.305~ms, which shows that MLX reuses the buffer rather than copying it; a per-expert
10 +\code{quantized\_matmul} loop costs 0.43~ms per layer for one token against 0.26~ms for \gatherqmm{}, and restacking eight
11 +experts per call 0.50~ms. The slot table with \gatherqmm{} is therefore both the fastest and the most faithful option
12 +(\code{benchmarks/raw/gateB/design/}).
13 +
14 +\paragraph{No per-miss synchronisation.} The first version evaluated the table after every miss; removing that
15 +synchronisation and batching the reads of a layer's misses raised decode throughput at 16 slots from 28 to 51~\toks{}
16 +(general prompts) and from 41 to 73~\toks{} (code), with the stall halved
17 +(\code{benchmarks/raw/gateC/prefetch\_online\_laptop/}). The per-miss cost on the page-cache tier is now 0.21~ms: 0.13~ms
18 +of \code{pread}, 0.06~ms of array creation and the remainder in the lazy install.
19 +
20 +\paragraph{Activation parity.} An unfused float16 $\mathrm{up}\cdot\sigma(\mathrm{gate})\cdot\mathrm{gate}$ gave
21 +$\max|\Delta\mathrm{logit}| = 0.0195$ against mlx-lm; using mlx-lm's compiled SwiGLU gives $0.0$. Numerical parity in float16 is
22 +a property of the exact kernel sequence, not only of the weights, and the test suite pins it.
23 +
24 +\paragraph{Reproducibility.} Every benchmark writes JSON lines with the command, git commit, host, chip, memory, macOS,
25 +library versions, seed and timestamps; raw logs are append-only under \code{benchmarks/raw/} and tables and figures are
26 +generated from them by scripts. Model revisions are pinned by commit hash in \code{benchmarks/configs/models.yaml};
27 +weights are never stored in the repository. A node is prepared with \code{scripts/cluster\_setup.sh} (rsync of the code,
28 +pinned virtual environment, import check), and the sanitized inventory \code{artifacts/environment.json} is generated from
29 +raw probes that are kept out of version control.
added paper/sections/introduction.tex +68 −0
@@ -0,0 +1,68 @@
1 +\section{Introduction}
2 +\label{sec:intro}
3 +
4 +A sparse mixture-of-experts (MoE) transformer replaces the dense feed-forward block of each layer by a bank of $E$ expert
5 +networks and a router that sends every token to $K \ll E$ of them \citep{rajbhandari2022deepspeedmoe}. The consequence that
6 +matters for inference is arithmetic: a token activates $K/E$ of the expert parameters, so the \emph{working set} of a decode
7 +step is small even when the \emph{model} is enormous. OLMoE-1B-7B activates 1.3 of its 6.9 billion parameters; in its
8 +4-bit MLX conversion a decode step touches 128 experts of 3.4 MB each, 432 MiB out of 3.6 GB of expert weights
9 +(Section~\ref{sec:setup}). Engines such as colibri \citep{colibri2026} exploit exactly this to run models of hundreds of billions of
10 +parameters from an NVMe disk on a laptop; the price is that the disk, not the GPU, sets the speed.
11 +
12 +Apple Silicon is an unusual target for this problem. Its unified memory removes the host-to-device copy that dominates
13 +GPU offloading systems, its NVMe storage is fast (we measure 8.5~GB/s for random 3.4~MB reads on an M5~Max), and small
14 +clusters of Macs are common in laboratories that cannot or do not want to buy datacenter accelerators. The MLX framework
15 +\citep{hannun2023mlx} provides a quantized grouped matrix-multiplication kernel (\gatherqmm{}) that executes a batch of
16 +expert products in one dispatch, and, since macOS~26.2, RDMA over Thunderbolt~5 between Mac Studios
17 +\citep{apple2026mlxdistributed}. What does not exist is a runtime that decides, per expert and per token, \emph{where} an
18 +expert should reside (local memory, a peer's memory, or the local disk), \emph{when} to move it (on demand or ahead of the
19 +router), and \emph{which node} should execute it, using measured properties of the machines and links at hand.
20 +
21 +This working paper describes \titan{} (Topology-aware Intelligent exTert orchestrAtion for distributed MoE iNference), a
22 +runtime built to answer one question stated as five testable hypotheses (Section~\ref{sec:hypotheses}): can topology-aware
23 +placement, causal prefetching, hierarchical caching and specialised Metal kernels improve decode latency, prefill throughput,
24 +energy per token and feasible model capacity over strong existing baselines on Apple Silicon? We insist on three
25 +methodological commitments that shape the rest of the paper.
26 +
27 +\paragraph{Placement decides speed, never semantics.} The router's expert identities, gating weights and aggregation order
28 +are those of the reference implementation (mlx-lm). A resident, a prefetched and a remotely executed expert must produce the
29 +same bytes. Section~\ref{sec:results-correctness} shows that \titan{} with bounded residency reproduces mlx-lm's logits
30 +exactly ($\max|\Delta| = 0$ in float16) on the test prompts; cache-aware routing, which trades expert identity for hits
31 +\citep{skliar2025cacheaware}, is therefore out of scope by construction.
32 +
33 +\paragraph{Measure before optimising.} Every design decision in Section~\ref{sec:architecture} is preceded by a microbenchmark
34 +recorded in the repository, from the cost of writing one expert into a stacked buffer (0.3~ms) to the per-expert fetch
35 +latency of each tier (0.21~ms from the page cache, 0.43~ms from a cold NVMe, about 28~ms over the gigabit LAN that actually
36 +connects the cluster). The inventory of the cluster is read live and is reported as read, including the seven nodes that did
37 +not answer and the absence of any Thunderbolt link between the Mac Studios.
38 +
39 +\paragraph{Report losses.} Our first online experiment with one-layer-ahead prefetching \emph{raised} the expert hit rate and
40 +\emph{lowered} throughput (Section~\ref{sec:results-prefetch}); the paper keeps that result and uses it to derive the
41 +condition under which prefetching can pay on this hardware.
42 +
43 +\subsection{Hypotheses}
44 +\label{sec:hypotheses}
45 +\begin{description}[style=nextline]
46 + \item[H1 (causal prefetch)] Predicting upcoming expert demand from information available at decision time and fetching
47 + asynchronously reduces expert-loading stalls net of predictor overhead.
48 + \item[H2 (topology-aware placement)] Placing experts on heterogeneous nodes according to measured compute, memory and
49 + link costs outperforms uniform or static sharding.
50 + \item[H3 (hierarchical caching)] Coordinating local memory, remote memory and local NVMe under a fixed memory budget improves
51 + throughput over any single second tier.
52 + \item[H4 (Metal kernels)] Selected fused Metal kernels improve end-to-end inference over stock MLX kernels for matching
53 + shapes and precision.
54 + \item[H5 (adaptive orchestration)] Adapting placement and prefetching online improves the Pareto frontier of latency,
55 + throughput and energy, not only isolated microbenchmarks.
56 +\end{description}
57 +
58 +\subsection{Contributions and status of this version}
59 +Version~0.1 of this working paper contributes (i) a verified inventory and network characterisation of a 21-node Apple Silicon
60 +cluster and the audit of four reference code bases (colibri, MLX, mlx-lm, llama.cpp) with a prior-art matrix of 59 works;
61 +(ii) the \titan{} single-node runtime: an expert registry over safetensors byte ranges, a \code{pread}-based NVMe tier,
62 +per-layer slot tables with LRU/LFU eviction, an exact execution block over \gatherqmm{}, and an asynchronous prefetcher
63 +with four causal predictors; (iii) the correctness evidence (bit-identical logits, token-identical greedy decoding);
64 +(iv) a trace-driven simulator calibrated on measured constants and the first controlled single-node experiments on
65 +OLMoE-1B-7B; (v) a formal statement of the placement--prefetch problem (Section~\ref{sec:formulation}) and the deterministic
66 +cost-aware heuristic that the distributed gates will use. Distributed execution (Gates E--F of the protocol), Metal kernel work
67 +(Gate~D) and the energy Pareto study are implemented or planned as described in Section~\ref{sec:limitations} and reported only
68 +to the extent that raw logs exist.
added paper/sections/limitations.tex +19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/related.tex +41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/results.tex +66 −0
Large diff (66 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/robustness.tex +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added paper/sections/setup.tex +55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added requirements-lock.txt +50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added scripts/cluster_setup.sh +17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added scripts/make_environment.py +55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
modified tests/conftest.py +15 −5
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
modified tests/test_numerical.py +9 −5
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_registry_stacked.py +24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
modified tests/test_registry_store.py +6 −2
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
modified titan/registry.py +14 −1
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere