SPB Git
spboucher.ai

spb/wp33_uqo

Public MIT

Paper skeleton v0.1 (protocol + preliminary), README, models config, stacked-layout registry (Qwen3-30B), generalized tests

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 1, 2026) parent a301b4c Browse files

34 changed files +1,901 −13

34 changed files
Unified Split
added LICENSE +9 −0
@@ -0,0 +1,9 @@
1 +MIT License
2 +
3 +Copyright (c) 2026 Simon-Pierre Boucher
4 +
5 +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 +
7 +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
8 +
9 +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
added README.md +93 −0
@@ -0,0 +1,93 @@
1 +# TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon
2 +
3 +**UQO Working Paper No. 33** — *A Measured Systems Study on a Heterogeneous Mac Cluster*
4 +
5 +![Paper](https://img.shields.io/badge/paper-PDF-blue) ![Python](https://img.shields.io/badge/python-3.12-blue) ![MLX](https://img.shields.io/badge/MLX-0.32.4-black) ![Status](https://img.shields.io/badge/status-v0.1%20protocol%20%2B%20preliminary-orange) ![UQO](https://img.shields.io/badge/UQO-DSA-green) ![Author](https://img.shields.io/badge/author-S.--P.%20Boucher-lightgrey)
6 +
7 +## TL;DR
8 +- An MLX-native MoE inference runtime with bounded per-layer expert residency (slot tables), a `pread` NVMe tier, LRU/LFU eviction,
9 + causal prefetch predictors and an asynchronous prefetcher; the router's choices and the logits are those of mlx-lm by construction.
10 +- Correctness: **bit-identical logits** (max |Δ| = 0.0) and token-identical greedy decoding vs mlx-lm on OLMoE-1B-7B-0125 4-bit, at 64/32/16/8 slots.
11 +- All-resident control: **489 tok/s** vs mlx-lm 494 tok/s on an M5 Max laptop (−1 %). Bounded residency with demand loading from the page cache: 51–73 tok/s at 16 slots.
12 +- Honest first result on H1: one-layer-ahead prefetch raises the hit rate (0.48→0.63) but **lowers** throughput; the paper derives the lead-time condition d·τ_layer ≥ λ_tier and tests deeper horizons.
13 +- Live cluster inventory: 21 reachable Macs (M1–M6, 8–96 GB), **1 GbE everywhere**, no active Thunderbolt link → distributed gates use TCP; RDMA/JACCL blocked pending cabling.
14 +
15 +## Status
16 +Version 0.1 (2026-11-01), **protocol + preliminary** working paper; not submitted. Gates A–B complete, Gate C in progress, D–G pending
17 +(see `research/LOG.md`). Nothing in the paper is reported without a raw log under `benchmarks/raw/`.
18 +
19 +## What this paper does
20 +Designs, implements and measures TITAN-MoE against five pre-stated hypotheses (causal prefetch, topology-aware placement,
21 +hierarchical caching, Metal kernels, adaptive orchestration on the latency–throughput–energy frontier), with an exact-semantics
22 +constraint: placement decides speed, never which experts run.
23 +
24 +## Headline results (version 0.1, single node, OLMoE-1B-7B-0125-Instruct-4bit, M5 Max 48 GB)
25 +| Configuration | decode tok/s | hit rate | source |
26 +|---|---|---|---|
27 +| mlx-lm (reference) | 491–496 | — | `benchmarks/raw/gateB/capacity_sweep_laptop/sweep.jsonl` |
28 +| TITAN, all experts resident (64 slots) | 489 | 1.00 | same |
29 +| TITAN, 32 slots, LRU, demand loading (page cache) | 56–80 | 0.73–0.81 | same (before lazy-install fix) |
30 +| TITAN, 16 slots, LRU, demand loading (page cache) | 51–73 | 0.48–0.67 | `benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl` |
31 +| TITAN, 16 slots + transition prefetch (d = 1, m = 8) | 35–56 | 0.63–0.73 | same |
32 +
33 +## Repository layout
34 +```
35 +CLAUDE.md README.md LICENSE docs/MANDATE.md
36 +titan/ registry.py store.py slots.py moe.py prefetch.py predictors.py runtime.py
37 +metal/ (Gate D) transport/ (Gate E) adapters/ (formats beyond OLMoE/Qwen3-MoE)
38 +tests/ test_registry_store.py test_numerical.py
39 +benchmarks/ configs/ (workloads, models) scripts/ (bench_decode, collect_traces, simulate_cache, ssd_bandwidth) raw/ (immutable) processed/
40 +research/ LOG.md DECISIONS.md RELATED_WORK.md prior_art.csv protocol.md
41 +paper/ main.tex sections/ references.bib (UQO template; `cd paper && make`)
42 +artifacts/ environment.json (sanitized live inventory) manifest.json
43 +scripts/ cluster_setup.sh make_environment.py
44 +third_party/ shallow clones of colibri / mlx / mlx-lm / llama.cpp (git-ignored; hashes in research/prior_art.csv)
45 +```
46 +
47 +## Quick start
48 +```bash
49 +uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements-lock.txt
50 +# model (3.9 GB, Apache-2.0, pinned revision) goes to the Hugging Face cache
51 +.venv/bin/python -c "from huggingface_hub import snapshot_download as s; s('mlx-community/OLMoE-1B-7B-0125-Instruct-4bit', revision='e2384419')"
52 +.venv/bin/python -m pytest tests -q # correctness vs mlx-lm (writes evidence to benchmarks/raw/tests/)
53 +M=$(ls -d ~/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/*)
54 +.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend mlx_lm --out /tmp/b.jsonl
55 +.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend titan --capacity 16 --predictor transition2 --traces benchmarks/raw/gateC/traces_olmoe_v2_laptop --out /tmp/b.jsonl
56 +cd paper && make # PDF
57 +```
58 +Local-only mode: everything above runs on one Mac; the distributed components are only enabled with an explicit host list.
59 +
60 +## Data
61 +No external datasets. Workloads are synthetic prompts written for this study (`benchmarks/configs/workloads*.yaml`); routing
62 +traces derived from them are in `benchmarks/raw/gateC/traces_*`. Models: OLMoE-1B-7B-0125-Instruct (Apache-2.0),
63 +Qwen3-30B-A3B-Instruct-2507 (Apache-2.0), both as mlx-community 4-bit conversions with pinned revisions.
64 +
65 +## Methodology
66 +See `research/protocol.md` (frozen at Gate C) and `paper/sections/formulation.tex`.
67 +
68 +## Figures & tables inventory
69 +Generated by `scripts/make_figures.py` / `scripts/make_tables.py` from `benchmarks/raw/` (Gate G).
70 +
71 +## Limitations
72 +Small model only in v0.1; distributed, Metal and energy experiments pending; 1 GbE topology; see paper Section "Limitations".
73 +
74 +## Citation
75 +```bibtex
76 +@techreport{boucher2026titanmoe,
77 + title = {TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon},
78 + author = {Boucher, Simon-Pierre},
79 + institution = {Université du Québec en Outaouais, Département des sciences administratives},
80 + type = {UQO Working Paper},
81 + number = {33},
82 + year = {2026}
83 +}
84 +```
85 +
86 +## Author & contact
87 +Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca
88 +
89 +## Notes
90 +- The mandate's repository name `wp33_titan_moe` is kept as the spbgit repository name (`wp33_uqo.git`); the folder follows the
91 + Recherche-UQO series layout (`ia/titan-moe`). Internal structure follows the mandate (D-001).
92 +- Written with Claude Code (Claude Fable 5.1) acting as autonomous research engineer under `docs/MANDATE.md`; AI-assistance
93 + disclosure is included in the paper's reproducibility appendix.
added artifacts/environment.json +650 −0
@@ -0,0 +1,650 @@
1 +{
2 + "generated": "2026-11-01T01:37:38",
3 + "note": "Live inventory of 2026-11-01; aliases are cluster names, no addresses or serials. Unreachable aliases are listed but unused.",
4 + "laptop": {
5 + "role": "development laptop (not on the cluster LAN)",
6 + "chip": "Apple M5 Max",
7 + "ram_gb": 48,
8 + "macos": "27.0",
9 + "gpu_cores": 40,
10 + "cpu_cores": 18,
11 + "ssd_random_3p4MB_GBps_cold": 8.5,
12 + "ssd_seq_GBps_cold": 11.5,
13 + "ssd_source": "benchmarks/raw/gateA/ssd/laptop-M5Max.json"
14 + },
15 + "cluster_nodes": {
16 + "M1M32": {
17 + "status": "reachable",
18 + "chip": "Apple M1 Max",
19 + "ram_gb": 32,
20 + "macos": "26.5.1",
21 + "gpu_cores": 24,
22 + "disk_free_gb": 208,
23 + "system_python": "3.9.6",
24 + "mlx_preinstalled": false,
25 + "load_avg_at_probe": "{ 1,52 1,31 1,21 }",
26 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
27 + "homebrew_pythons": [],
28 + "uv": true,
29 + "xcode": "/Applications/Xcode.app/Contents/Developer",
30 + "metal_compiler": "Apple metal version 32023.883 (metalfe-32023.883)",
31 + "rdma_ctl_present": true,
32 + "iperf3": false
33 + },
34 + "M2M32": {
35 + "status": "unreachable (SSH timeout 2026-11-01)"
36 + },
37 + "M2M32b": {
38 + "status": "unreachable (SSH timeout 2026-11-01)"
39 + },
40 + "M2M32c": {
41 + "status": "reachable",
42 + "chip": "Apple M2 Max",
43 + "ram_gb": 32,
44 + "macos": "26.5.1",
45 + "gpu_cores": 30,
46 + "disk_free_gb": 417,
47 + "system_python": "3.9.6",
48 + "mlx_preinstalled": false,
49 + "load_avg_at_probe": "{ 1,66 1,22 1,12 }",
50 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
51 + "homebrew_pythons": [
52 + "/opt/homebrew/bin/python3.13",
53 + "/opt/homebrew/bin/python3.14"
54 + ],
55 + "uv": true,
56 + "xcode": "/Library/Developer/CommandLineTools",
57 + "metal_compiler": null,
58 + "rdma_ctl_present": true,
59 + "iperf3": false
60 + },
61 + "M2U64": {
62 + "status": "reachable",
63 + "chip": "Apple M2 Ultra",
64 + "ram_gb": 64,
65 + "macos": "26.5",
66 + "gpu_cores": 60,
67 + "disk_free_gb": 308,
68 + "system_python": "3.9.6",
69 + "mlx_preinstalled": false,
70 + "load_avg_at_probe": "{ 1,04 1,13 1,14 }",
71 + "en0_media": "autoselect (1000baseT <full-duplex>)",
72 + "homebrew_pythons": [
73 + "/opt/homebrew/bin/python3.14"
74 + ],
75 + "uv": true,
76 + "xcode": "/Applications/Xcode.app/Contents/Developer",
77 + "metal_compiler": null,
78 + "rdma_ctl_present": true,
79 + "iperf3": false
80 + },
81 + "M3BA16": {
82 + "status": "reachable",
83 + "chip": "Apple M3",
84 + "ram_gb": 16,
85 + "macos": "26.5.1",
86 + "gpu_cores": 10,
87 + "disk_free_gb": 409,
88 + "system_python": "3.9.6",
89 + "mlx_preinstalled": false,
90 + "load_avg_at_probe": "{ 1,27 1,26 1,27 }",
91 + "en0_media": "autoselect",
92 + "homebrew_pythons": [],
93 + "uv": false,
94 + "xcode": "/Library/Developer/CommandLineTools",
95 + "metal_compiler": null,
96 + "rdma_ctl_present": true,
97 + "iperf3": false
98 + },
99 + "M3BA24": {
100 + "status": "reachable",
101 + "chip": "Apple M3",
102 + "ram_gb": 24,
103 + "macos": "26.5.1",
104 + "gpu_cores": 10,
105 + "disk_free_gb": 420,
106 + "system_python": "3.9.6",
107 + "mlx_preinstalled": false,
108 + "load_avg_at_probe": "{ 1,31 1,25 1,23 }",
109 + "en0_media": "autoselect",
110 + "homebrew_pythons": [],
111 + "uv": false,
112 + "xcode": "/Library/Developer/CommandLineTools",
113 + "metal_compiler": null,
114 + "rdma_ctl_present": true,
115 + "iperf3": false
116 + },
117 + "M3U96a": {
118 + "status": "reachable",
119 + "chip": "Apple M3 Ultra",
120 + "ram_gb": 96,
121 + "macos": "26.6",
122 + "gpu_cores": 60,
123 + "disk_free_gb": 512,
124 + "system_python": "3.14.5",
125 + "mlx_preinstalled": true,
126 + "load_avg_at_probe": "{ 1,43 1,63 1,64 }",
127 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
128 + "homebrew_pythons": [
129 + "/opt/homebrew/bin/python3.14"
130 + ],
131 + "uv": true,
132 + "xcode": "/Applications/Xcode.app/Contents/Developer",
133 + "metal_compiler": "Apple metal version 32023.883 (metalfe-32023.883)",
134 + "rdma_ctl_present": true,
135 + "iperf3": true
136 + },
137 + "M3U96b": {
138 + "status": "reachable",
139 + "chip": "Apple M3 Ultra",
140 + "ram_gb": 96,
141 + "macos": "26.4",
142 + "gpu_cores": 60,
143 + "disk_free_gb": 432,
144 + "system_python": "3.14.4",
145 + "mlx_preinstalled": true,
146 + "load_avg_at_probe": "{ 1,93 1,92 1,93 }",
147 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
148 + "homebrew_pythons": [
149 + "/opt/homebrew/bin/python3.14"
150 + ],
151 + "uv": true,
152 + "xcode": "/Applications/Xcode.app/Contents/Developer",
153 + "metal_compiler": null,
154 + "rdma_ctl_present": true,
155 + "iperf3": true
156 + },
157 + "M4BP36": {
158 + "status": "reachable",
159 + "chip": "Apple M4 Max",
160 + "ram_gb": 36,
161 + "macos": "26.5.1",
162 + "gpu_cores": 32,
163 + "disk_free_gb": 843,
164 + "system_python": "3.9.6",
165 + "mlx_preinstalled": false,
166 + "load_avg_at_probe": "{ 1,08 1,13 1,16 }",
167 + "en0_media": "autoselect",
168 + "homebrew_pythons": [],
169 + "uv": true,
170 + "xcode": "/Library/Developer/CommandLineTools",
171 + "metal_compiler": null,
172 + "rdma_ctl_present": true,
173 + "iperf3": false
174 + },
175 + "M4BP48": {
176 + "status": "unreachable (SSH timeout 2026-11-01)"
177 + },
178 + "M4M36": {
179 + "status": "unreachable (SSH timeout 2026-11-01)"
180 + },
181 + "M4M64a": {
182 + "status": "reachable",
183 + "chip": "Apple M4 Max",
184 + "ram_gb": 64,
185 + "macos": "26.5.1",
186 + "gpu_cores": 40,
187 + "disk_free_gb": 730,
188 + "system_python": "3.14.4",
189 + "mlx_preinstalled": true,
190 + "load_avg_at_probe": "{ 1,88 2,01 1,78 }",
191 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
192 + "homebrew_pythons": [
193 + "/opt/homebrew/bin/python3.14"
194 + ],
195 + "uv": true,
196 + "xcode": "/Applications/Xcode.app/Contents/Developer",
197 + "metal_compiler": null,
198 + "rdma_ctl_present": true,
199 + "iperf3": true
200 + },
201 + "M4M64b": {
202 + "status": "reachable",
203 + "chip": "Apple M4 Max",
204 + "ram_gb": 64,
205 + "macos": "26.5.1",
206 + "gpu_cores": 40,
207 + "disk_free_gb": 83,
208 + "system_python": "3.14.4",
209 + "mlx_preinstalled": true,
210 + "load_avg_at_probe": "{ 1,39 1,40 1,41 }",
211 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
212 + "homebrew_pythons": [
213 + "/opt/homebrew/bin/python3.13",
214 + "/opt/homebrew/bin/python3.14"
215 + ],
216 + "uv": false,
217 + "xcode": "/Applications/Xcode.app/Contents/Developer",
218 + "metal_compiler": null,
219 + "rdma_ctl_present": true,
220 + "iperf3": true
221 + },
222 + "M4M64c": {
223 + "status": "reachable",
224 + "chip": "Apple M4 Max",
225 + "ram_gb": 64,
226 + "macos": "26.4",
227 + "gpu_cores": 40,
228 + "disk_free_gb": 321,
229 + "system_python": "3.9.6",
230 + "mlx_preinstalled": false,
231 + "load_avg_at_probe": "{ 1,63 1,73 1,75 }",
232 + "en0_media": "autoselect (1000baseT <full-duplex>)",
233 + "homebrew_pythons": [
234 + "/opt/homebrew/bin/python3.13",
235 + "/opt/homebrew/bin/python3.14"
236 + ],
237 + "uv": true,
238 + "xcode": "/Library/Developer/CommandLineTools",
239 + "metal_compiler": null,
240 + "rdma_ctl_present": true,
241 + "iperf3": false
242 + },
243 + "M5M36": {
244 + "status": "reachable",
245 + "chip": "Apple M5 Max",
246 + "ram_gb": 36,
247 + "macos": "27.0",
248 + "gpu_cores": 32,
249 + "disk_free_gb": 355,
250 + "system_python": "3.9.6",
251 + "mlx_preinstalled": false,
252 + "load_avg_at_probe": "{ 2,52 2,53 2,41 }",
253 + "en0_media": "autoselect (1000baseT <full-duplex>)",
254 + "homebrew_pythons": [
255 + "/opt/homebrew/bin/python3.13",
256 + "/opt/homebrew/bin/python3.14"
257 + ],
258 + "uv": true,
259 + "xcode": "/Library/Developer/CommandLineTools",
260 + "metal_compiler": null,
261 + "rdma_ctl_present": true,
262 + "iperf3": false
263 + },
264 + "m2m16": {
265 + "status": "unreachable (SSH timeout 2026-11-01)"
266 + },
267 + "m2m8a": {
268 + "status": "reachable",
269 + "chip": "Apple M2",
270 + "ram_gb": 8,
271 + "macos": "26.5.1",
272 + "gpu_cores": 10,
273 + "disk_free_gb": 424,
274 + "system_python": "3.9.6",
275 + "mlx_preinstalled": false,
276 + "load_avg_at_probe": "{ 1,69 1,63 1,57 }",
277 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
278 + "homebrew_pythons": [],
279 + "uv": false,
280 + "xcode": "/Library/Developer/CommandLineTools",
281 + "metal_compiler": null,
282 + "rdma_ctl_present": true,
283 + "iperf3": false
284 + },
285 + "m2m8b": {
286 + "status": "reachable",
287 + "chip": "Apple M2",
288 + "ram_gb": 8,
289 + "macos": "26.5.1",
290 + "gpu_cores": 10,
291 + "disk_free_gb": 195,
292 + "system_python": null,
293 + "mlx_preinstalled": false,
294 + "load_avg_at_probe": "{ 2,12 1,54 1,45 }",
295 + "en0_media": "autoselect (1000baseT <full-duplex,flow-control>)",
296 + "homebrew_pythons": [],
297 + "uv": false,
298 + "xcode": "",
299 + "metal_compiler": null,
300 + "rdma_ctl_present": true,
301 + "iperf3": false
302 + },
303 + "m4ma": {
304 + "status": "unreachable (SSH timeout 2026-11-01)"
305 + },
306 + "m4mb": {
307 + "status": "unreachable (SSH timeout 2026-11-01)"
308 + },
309 + "m4mc": {
310 + "status": "unreachable (SSH timeout 2026-11-01)"
311 + },
312 + "m4md": {
313 + "status": "reachable",
314 + "chip": "Apple M4",
315 + "ram_gb": 16,
316 + "macos": "26.6.2",
317 + "gpu_cores": 10,
318 + "disk_free_gb": 195,
319 + "system_python": "3.9.6",
320 + "mlx_preinstalled": false,
321 + "load_avg_at_probe": "{ 1,25 1,34 1,37 }",
322 + "en0_media": "autoselect (1000baseT <full-duplex>)",
323 + "homebrew_pythons": [
324 + "/opt/homebrew/bin/python3.13",
325 + "/opt/homebrew/bin/python3.14"
326 + ],
327 + "uv": true,
328 + "xcode": "/Library/Developer/CommandLineTools",
329 + "metal_compiler": null,
330 + "rdma_ctl_present": true,
331 + "iperf3": false
332 + },
333 + "m4ml": {
334 + "status": "reachable",
335 + "chip": "Apple M4",
336 + "ram_gb": 16,
337 + "macos": "26.7",
338 + "gpu_cores": 10,
339 + "disk_free_gb": 178,
340 + "system_python": "3.9.6",
341 + "mlx_preinstalled": false,
342 + "load_avg_at_probe": "{ 1,47 1,45 1,45 }",
343 + "en0_media": "autoselect (1000baseT <full-duplex>)",
344 + "homebrew_pythons": [
345 + "/opt/homebrew/bin/python3.13",
346 + "/opt/homebrew/bin/python3.14"
347 + ],
348 + "uv": true,
349 + "xcode": "/Library/Developer/CommandLineTools",
350 + "metal_compiler": null,
351 + "rdma_ctl_present": true,
352 + "iperf3": false
353 + },
354 + "m4mm": {
355 + "status": "reachable",
356 + "chip": "Apple M4 Pro",
357 + "ram_gb": 48,
358 + "macos": "26.5.1",
359 + "gpu_cores": 16,
360 + "disk_free_gb": 375,
361 + "system_python": "3.9.6",
362 + "mlx_preinstalled": false,
363 + "load_avg_at_probe": "{ 1,26 1,38 1,39 }",
364 + "en0_media": "autoselect (1000baseT <full-duplex>)",
365 + "homebrew_pythons": [
366 + "/opt/homebrew/bin/python3.13",
367 + "/opt/homebrew/bin/python3.14"
368 + ],
369 + "uv": true,
370 + "xcode": "/Library/Developer/CommandLineTools",
371 + "metal_compiler": null,
372 + "rdma_ctl_present": true,
373 + "iperf3": false
374 + },
375 + "m4mn": {
376 + "status": "reachable",
377 + "chip": "Apple M4 Pro",
378 + "ram_gb": 48,
379 + "macos": "26.5.1",
380 + "gpu_cores": 16,
381 + "disk_free_gb": 367,
382 + "system_python": "3.9.6",
383 + "mlx_preinstalled": false,
384 + "load_avg_at_probe": "{ 1,11 1,27 1,35 }",
385 + "en0_media": "autoselect (1000baseT <full-duplex>)",
386 + "homebrew_pythons": [
387 + "/opt/homebrew/bin/python3.13",
388 + "/opt/homebrew/bin/python3.14"
389 + ],
390 + "uv": true,
391 + "xcode": "/Library/Developer/CommandLineTools",
392 + "metal_compiler": null,
393 + "rdma_ctl_present": true,
394 + "iperf3": false
395 + },
396 + "m5ma": {
397 + "status": "reachable",
398 + "chip": "Apple M5 Pro",
399 + "ram_gb": 24,
400 + "macos": "27.0",
401 + "gpu_cores": 16,
402 + "disk_free_gb": 382,
403 + "system_python": "3.9.6",
404 + "mlx_preinstalled": false,
405 + "load_avg_at_probe": "{ 2,66 2,58 2,53 }",
406 + "en0_media": "autoselect (1000baseT <full-duplex>)",
407 + "homebrew_pythons": [
408 + "/opt/homebrew/bin/python3.13",
409 + "/opt/homebrew/bin/python3.14"
410 + ],
411 + "uv": true,
412 + "xcode": "/Library/Developer/CommandLineTools",
413 + "metal_compiler": null,
414 + "rdma_ctl_present": true,
415 + "iperf3": false
416 + },
417 + "m6ma": {
418 + "status": "reachable",
419 + "chip": "Apple M6",
420 + "ram_gb": 24,
421 + "macos": "27.0",
422 + "gpu_cores": 12,
423 + "disk_free_gb": 396,
424 + "system_python": "3.9.6",
425 + "mlx_preinstalled": false,
426 + "load_avg_at_probe": "{ 2,45 2,45 2,42 }",
427 + "en0_media": "autoselect (1000baseT <full-duplex>)",
428 + "homebrew_pythons": [
429 + "/opt/homebrew/bin/python3.13",
430 + "/opt/homebrew/bin/python3.14"
431 + ],
432 + "uv": true,
433 + "xcode": "/Library/Developer/CommandLineTools",
434 + "metal_compiler": null,
435 + "rdma_ctl_present": true,
436 + "iperf3": false
437 + },
438 + "m6mb": {
439 + "status": "reachable",
440 + "chip": "Apple M6",
441 + "ram_gb": 24,
442 + "macos": "27.0",
443 + "gpu_cores": 12,
444 + "disk_free_gb": 394,
445 + "system_python": "3.9.6",
446 + "mlx_preinstalled": false,
447 + "load_avg_at_probe": "{ 1,99 2,24 2,27 }",
448 + "en0_media": "autoselect (1000baseT <full-duplex>)",
449 + "homebrew_pythons": [
450 + "/opt/homebrew/bin/python3.13",
451 + "/opt/homebrew/bin/python3.14"
452 + ],
453 + "uv": true,
454 + "xcode": "/Library/Developer/CommandLineTools",
455 + "metal_compiler": null,
456 + "rdma_ctl_present": true,
457 + "iperf3": false
458 + }
459 + },
460 + "network": {
461 + "lan": "1 GbE switch (all nodes negotiate 1000baseT on en0)",
462 + "thunderbolt": "no active Thunderbolt link between any two nodes at probe time (RDMA/JACCL blocked pending cabling)",
463 + "rtt_ms": [
464 + {
465 + "from": "M3U96a",
466 + "to": "lan-host-85",
467 + "min": 0.596,
468 + "avg": 0.732,
469 + "max": 1.0
470 + },
471 + {
472 + "from": "M3U96a",
473 + "to": "lan-host-61",
474 + "min": 0.367,
475 + "avg": 0.572,
476 + "max": 1.179
477 + },
478 + {
479 + "from": "M3U96a",
480 + "to": "lan-host-70",
481 + "min": 0.529,
482 + "avg": 0.634,
483 + "max": 0.865
484 + },
485 + {
486 + "from": "M3U96a",
487 + "to": "lan-host-77",
488 + "min": 0.357,
489 + "avg": 0.53,
490 + "max": 0.741
491 + },
492 + {
493 + "from": "M3U96a",
494 + "to": "lan-host-103",
495 + "min": 0.623,
496 + "avg": 0.742,
497 + "max": 1.216
498 + },
499 + {
500 + "from": "M3U96a",
501 + "to": "lan-host-104",
502 + "min": 0.583,
503 + "avg": 0.789,
504 + "max": 1.446
505 + },
506 + {
507 + "from": "M3U96a",
508 + "to": "lan-host-92",
509 + "min": 0.582,
510 + "avg": 0.771,
511 + "max": 1.318
512 + },
513 + {
514 + "from": "M3U96a",
515 + "to": "lan-host-93",
516 + "min": 0.6,
517 + "avg": 1.021,
518 + "max": 1.187
519 + },
520 + {
521 + "from": "M3U96a",
522 + "to": "lan-host-105",
523 + "min": 0.53,
524 + "avg": 0.741,
525 + "max": 0.905
526 + },
527 + {
528 + "from": "M3U96a",
529 + "to": "lan-host-106",
530 + "min": 0.571,
531 + "avg": 0.766,
532 + "max": 1.275
533 + },
534 + {
535 + "from": "M3U96a",
536 + "to": "lan-host-112",
537 + "min": 0.593,
538 + "avg": 0.786,
539 + "max": 1.431
540 + },
541 + {
542 + "from": "M3U96a",
543 + "to": "lan-host-72",
544 + "min": 0.393,
545 + "avg": 0.577,
546 + "max": 0.872
547 + },
548 + {
549 + "from": "M3U96a",
550 + "to": "lan-host-76",
551 + "min": 0.538,
552 + "avg": 0.762,
553 + "max": 1.287
554 + },
555 + {
556 + "from": "M3U96a",
557 + "to": "lan-host-62",
558 + "min": 0.468,
559 + "avg": 0.694,
560 + "max": 1.189
561 + },
562 + {
563 + "from": "M3U96a",
564 + "to": "lan-host-63",
565 + "min": 0.54,
566 + "avg": 0.657,
567 + "max": 1.317
568 + },
569 + {
570 + "from": "M3U96a",
571 + "to": "lan-host-66",
572 + "min": 0.455,
573 + "avg": 0.618,
574 + "max": 1.019
575 + },
576 + {
577 + "from": "M3U96a",
578 + "to": "lan-host-68",
579 + "min": 0.487,
580 + "avg": 0.667,
581 + "max": 1.29
582 + },
583 + {
584 + "from": "M4M64a",
585 + "to": "M3U96b",
586 + "min": 0.726,
587 + "avg": 0.869,
588 + "max": 1.328
589 + },
590 + {
591 + "from": "M4M64a",
592 + "to": "M2U64",
593 + "min": 0.598,
594 + "avg": 0.69,
595 + "max": 0.862
596 + }
597 + ],
598 + "thunderbolt_active_links": {
599 + "M3U96a": false,
600 + "M3U96b": false,
601 + "M4M64a": false,
602 + "M2U64": false,
603 + "M4M64c": false,
604 + "M5M36": false
605 + },
606 + "iperf3_gbits": [
607 + {
608 + "from": "M3U96a",
609 + "to": "M3U96b",
610 + "sent": 0.945,
611 + "recv": 0.94
612 + },
613 + {
614 + "from": "M3U96a",
615 + "to": "M4M64a",
616 + "sent": 0.947,
617 + "recv": 0.941
618 + },
619 + {
620 + "from": "M3U96a",
621 + "to": "M2U64",
622 + "sent": 0.945,
623 + "recv": 0.94
624 + },
625 + {
626 + "from": "M3U96a",
627 + "to": "M4M64c",
628 + "sent": 0.946,
629 + "recv": 0.94
630 + },
631 + {
632 + "from": "M3U96a",
633 + "to": "M3U96b",
634 + "sent": 0.945,
635 + "recv": 0.94
636 + }
637 + ]
638 + },
639 + "software": {
640 + "mlx_venv": "0.32.4",
641 + "mlx_lm_venv": "0.32.2",
642 + "python_venv": "3.12",
643 + "third_party_commits": {
644 + "colibri": "0a5504a1dc64f86fcd0fe7db298b0448c730e7e6",
645 + "mlx": "bd63d3803008edda03ce74080955d9879e2ef1eb",
646 + "mlx-lm": "736ab0d8c1d0edc79c3d798044fe6904e2fa23be",
647 + "llama.cpp": "a91eab67d8389d979d29fdf6633b2edd0718eb4b"
648 + }
649 + }
650 +}
\ No newline at end of file
added benchmarks/configs/models.yaml +27 −0
@@ -0,0 +1,27 @@
1 +models:
2 + olmoe-1b-7b-0125-4bit:
3 + repo: mlx-community/OLMoE-1B-7B-0125-Instruct-4bit
4 + revision: e23844197887b031e7ddddbb0b8959c5a6853a7b
5 + license: apache-2.0
6 + base: allenai/OLMoE-1B-7B-0125-Instruct
7 + layers: 16
8 + experts: 64
9 + top_k: 8
10 + shared_experts: 0
11 + params_active_total: "1.3B / 6.9B"
12 + quantization: {bits: 4, group_size: 64, mode: affine}
13 + bytes_per_expert: 3538944
14 + tier: small
15 + qwen3-30b-a3b-2507-4bit:
16 + repo: mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit
17 + revision: e9675aa3ca5f900ccef55267914466d55ab325fa
18 + license: apache-2.0
19 + base: Qwen/Qwen3-30B-A3B-Instruct-2507
20 + layers: 48
21 + experts: 128
22 + top_k: 8
23 + shared_experts: 0
24 + params_active_total: "3.3B / 30.5B"
25 + quantization: {bits: 4, group_size: 64, mode: affine}
26 + tier: medium
27 + downloaded_on: [M3U96a]
added benchmarks/raw/tests/greedy_olmoe_cap16_20261101T014415.json +14 −0
@@ -0,0 +1,14 @@
1 +[
2 + {
3 + "prompt": "The capital of Canada is",
4 + "ref": "Ottawa.\n\nOttawa is the capital of Canada, and it is located in the province of Ontario. It is situated on the south bank of the Ottawa",
5 + "titan": "Ottawa.\n\nOttawa is the capital of Canada, and it is located in the province of Ontario. It is situated on the south bank of the Ottawa",
6 + "equal": true
7 + },
8 + {
9 + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
10 + "ref": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n",
11 + "titan": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n",
12 + "equal": true
13 + }
14 +]
\ No newline at end of file
added benchmarks/raw/tests/numerical_olmoe_cap16_2026-11-01T014410.json +50 −0
@@ -0,0 +1,50 @@
1 +{
2 + "capacity": 16,
3 + "prompts": [
4 + {
5 + "prompt": "The capital of Canada is",
6 + "tokens": 5,
7 + "max_abs_dlogit": 0.0,
8 + "argmax_equal": true,
9 + "hit_rate": 0.2580101180438449,
10 + "misses": 440,
11 + "stall_s": 0.08842262497637421,
12 + "layers_routed": 46
13 + },
14 + {
15 + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
16 + "tokens": 21,
17 + "max_abs_dlogit": 0.0,
18 + "argmax_equal": true,
19 + "hit_rate": 0.42512908777969016,
20 + "misses": 1002,
21 + "stall_s": 0.19870067050214857,
22 + "layers_routed": 127
23 + },
24 + {
25 + "prompt": "Q: If a train travels 60 km in 45 minute",
26 + "tokens": 25,
27 + "max_abs_dlogit": 0.0,
28 + "argmax_equal": true,
29 + "hit_rate": 0.41904344010530936,
30 + "misses": 1324,
31 + "stall_s": 0.2609664188930765,
32 + "layers_routed": 161
33 + },
34 + {
35 + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
36 + "tokens": 26,
37 + "max_abs_dlogit": 0.0,
38 + "argmax_equal": true,
39 + "hit_rate": 0.45278969957081544,
40 + "misses": 1275,
41 + "stall_s": 0.2480922951363027,
42 + "layers_routed": 164
43 + }
44 + ],
45 + "tol": 0.001,
46 + "ts": "2026-11-01T01:44:10",
47 + "max_abs_dlogit_worst": 0.0,
48 + "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b",
49 + "model_type": "olmoe"
50 +}
\ No newline at end of file
added benchmarks/raw/tests/numerical_olmoe_cap32_2026-11-01T014409.json +50 −0
@@ -0,0 +1,50 @@
1 +{
2 + "capacity": 32,
3 + "prompts": [
4 + {
5 + "prompt": "The capital of Canada is",
6 + "tokens": 5,
7 + "max_abs_dlogit": 0.0,
8 + "argmax_equal": true,
9 + "hit_rate": 0.0,
10 + "misses": 430,
11 + "stall_s": 0.08661957993172109,
12 + "layers_routed": 16
13 + },
14 + {
15 + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
16 + "tokens": 21,
17 + "max_abs_dlogit": 0.0,
18 + "argmax_equal": true,
19 + "hit_rate": 0.6060606060606061,
20 + "misses": 429,
21 + "stall_s": 0.085544079891406,
22 + "layers_routed": 40
23 + },
24 + {
25 + "prompt": "Q: If a train travels 60 km in 45 minute",
26 + "tokens": 25,
27 + "max_abs_dlogit": 0.0,
28 + "argmax_equal": true,
29 + "hit_rate": 0.6090458488228006,
30 + "misses": 631,
31 + "stall_s": 0.12506033421959728,
32 + "layers_routed": 58
33 + },
34 + {
35 + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
36 + "tokens": 26,
37 + "max_abs_dlogit": 0.0,
38 + "argmax_equal": true,
39 + "hit_rate": 0.6224188790560472,
40 + "misses": 512,
41 + "stall_s": 0.10236366488970816,
42 + "layers_routed": 49
43 + }
44 + ],
45 + "tol": 0.001,
46 + "ts": "2026-11-01T01:44:09",
47 + "max_abs_dlogit_worst": 0.0,
48 + "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b",
49 + "model_type": "olmoe"
50 +}
\ No newline at end of file
added benchmarks/raw/tests/numerical_olmoe_cap64_2026-11-01T014408.json +50 −0
@@ -0,0 +1,50 @@
1 +{
2 + "capacity": 64,
3 + "prompts": [
4 + {
5 + "prompt": "The capital of Canada is",
6 + "tokens": 5,
7 + "max_abs_dlogit": 0.0,
8 + "argmax_equal": true,
9 + "hit_rate": 0.0,
10 + "misses": 430,
11 + "stall_s": 0.08624804276041687,
12 + "layers_routed": 16
13 + },
14 + {
15 + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
16 + "tokens": 21,
17 + "max_abs_dlogit": 0.0,
18 + "argmax_equal": true,
19 + "hit_rate": 0.4637904468412943,
20 + "misses": 348,
21 + "stall_s": 0.0698862480930984,
22 + "layers_routed": 16
23 + },
24 + {
25 + "prompt": "Q: If a train travels 60 km in 45 minute",
26 + "tokens": 25,
27 + "max_abs_dlogit": 0.0,
28 + "argmax_equal": true,
29 + "hit_rate": 0.823076923076923,
30 + "misses": 138,
31 + "stall_s": 0.028100415016524494,
32 + "layers_routed": 16
33 + },
34 + {
35 + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
36 + "tokens": 26,
37 + "max_abs_dlogit": 0.0,
38 + "argmax_equal": true,
39 + "hit_rate": 0.9234828496042217,
40 + "misses": 58,
41 + "stall_s": 0.012272500898689032,
42 + "layers_routed": 16
43 + }
44 + ],
45 + "tol": 0.001,
46 + "ts": "2026-11-01T01:44:08",
47 + "max_abs_dlogit_worst": 0.0,
48 + "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b",
49 + "model_type": "olmoe"
50 +}
\ No newline at end of file
added benchmarks/raw/tests/numerical_olmoe_cap8_2026-11-01T014411.json +50 −0
@@ -0,0 +1,50 @@
1 +{
2 + "capacity": 8,
3 + "prompts": [
4 + {
5 + "prompt": "The capital of Canada is",
6 + "tokens": 5,
7 + "max_abs_dlogit": 0.0,
8 + "argmax_equal": true,
9 + "hit_rate": 0.1625,
10 + "misses": 536,
11 + "stall_s": 0.1006133749615401,
12 + "layers_routed": 80
13 + },
14 + {
15 + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
16 + "tokens": 21,
17 + "max_abs_dlogit": 0.0,
18 + "argmax_equal": true,
19 + "hit_rate": 0.4425981873111782,
20 + "misses": 1476,
21 + "stall_s": 0.2634192098630592,
22 + "layers_routed": 331
23 + },
24 + {
25 + "prompt": "Q: If a train travels 60 km in 45 minute",
26 + "tokens": 25,
27 + "max_abs_dlogit": 0.0,
28 + "argmax_equal": true,
29 + "hit_rate": 0.375,
30 + "misses": 1995,
31 + "stall_s": 0.3573505834210664,
32 + "layers_routed": 399
33 + },
34 + {
35 + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
36 + "tokens": 26,
37 + "max_abs_dlogit": 0.0,
38 + "argmax_equal": true,
39 + "hit_rate": 0.3878048780487805,
40 + "misses": 2008,
41 + "stall_s": 0.3646473821718246,
42 + "layers_routed": 410
43 + }
44 + ],
45 + "tol": 0.001,
46 + "ts": "2026-11-01T01:44:11",
47 + "max_abs_dlogit_worst": 0.0,
48 + "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b",
49 + "model_type": "olmoe"
50 +}
\ No newline at end of file
added paper/main.tex +110 −0
@@ -0,0 +1,110 @@
1 +% ============================================================================
2 +% UQO Working Paper No. 33 — TITAN-MoE
3 +% ============================================================================
4 +\documentclass[12pt,letterpaper]{article}
5 +\usepackage[utf8]{inputenc}
6 +\usepackage[T1]{fontenc}
7 +\usepackage[english]{babel}
8 +\usepackage[letterpaper,top=1in,bottom=1in,left=1in,right=1in]{geometry}
9 +\usepackage{mathptmx}
10 +\usepackage{microtype}
11 +\usepackage{setspace}
12 +\onehalfspacing
13 +\usepackage{amsmath,amssymb,amsthm}
14 +\usepackage{booktabs}
15 +\usepackage{tabularx}
16 +\usepackage{multirow}
17 +\usepackage{threeparttable}
18 +\usepackage{siunitx}
19 +\sisetup{input-symbols = {()}, table-format = -1.3, parse-numbers = false}
20 +\usepackage{graphicx}
21 +\usepackage{subcaption}
22 +\usepackage{float}
23 +\usepackage[dvipsnames]{xcolor}
24 +\usepackage[colorlinks=true,linkcolor=NavyBlue,citecolor=NavyBlue,urlcolor=NavyBlue,breaklinks=true,
25 + pdftitle={TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon},
26 + pdfauthor={Simon-Pierre Boucher}]{hyperref}
27 +\usepackage[authoryear,round,semicolon]{natbib}
28 +\bibliographystyle{aer}
29 +\usepackage[font=small,labelfont=bf,labelsep=period,justification=justified,singlelinecheck=false]{caption}
30 +\usepackage{fancyhdr}
31 +\pagestyle{fancy}
32 +\fancyhf{}
33 +\renewcommand{\headrulewidth}{0pt}
34 +\fancyfoot[C]{\thepage}
35 +\usepackage{titlesec}
36 +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{}
37 +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{}
38 +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{}
39 +\usepackage[toc,page]{appendix}
40 +\usepackage{enumitem}
41 +\setlist{nosep,leftmargin=*}
42 +\usepackage{algorithm}
43 +\usepackage{algpseudocode}
44 +\usepackage{tikz}
45 +\usetikzlibrary{arrows.meta,positioning,shapes.geometric,fit,calc}
46 +\usepackage{array}
47 +\newcolumntype{L}[1]{>{\raggedright\arraybackslash}p{#1}}
48 +\newcolumntype{R}[1]{>{\raggedleft\arraybackslash}p{#1}}
49 +\graphicspath{{figures/}}
50 +
51 +% --- theorem environments (shared counter, numbered by section) ---
52 +\theoremstyle{plain}
53 +\newtheorem{theorem}{Theorem}[section]
54 +\newtheorem{proposition}[theorem]{Proposition}
55 +\newtheorem{lemma}[theorem]{Lemma}
56 +\newtheorem{corollary}[theorem]{Corollary}
57 +\theoremstyle{definition}
58 +\newtheorem{definition}[theorem]{Definition}
59 +\newtheorem{assumption}[theorem]{Assumption}
60 +\theoremstyle{remark}
61 +\newtheorem{remark}[theorem]{Remark}
62 +\newtheorem{observation}[theorem]{Observation}
63 +
64 +% --- macros ---
65 +\newcommand{\E}{\mathcal{E}}
66 +\newcommand{\Lset}{\mathcal{L}}
67 +\newcommand{\Nset}{\mathcal{N}}
68 +\newcommand{\R}{\mathbb{R}}
69 +\newcommand{\Ex}{\mathbb{E}}
70 +\newcommand{\ind}{\mathbb{1}}
71 +\newcommand{\toks}{\mathrm{tok/s}}
72 +\newcommand{\code}[1]{\texttt{#1}}
73 +\newcommand{\titan}{TITAN-MoE}
74 +\newcommand{\gatherqmm}{\code{gather\_qmm}}
75 +
76 +% ============================================================================
77 +% METADATA
78 +% ============================================================================
79 +\newcommand{\WPnumber}{33}
80 +\newcommand{\WPtitle}{TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon}
81 +\newcommand{\WPsubtitle}{A Measured Systems Study on a Heterogeneous Mac Cluster}
82 +\newcommand{\WPdate}{November 2026}
83 +\newcommand{\WPversion}{0.1}
84 +\input{sections/abstract}
85 +\newcommand{\WPkeywords}{mixture of experts, inference systems, Apple Silicon, expert offloading, predictive prefetching, distributed inference, energy efficiency}
86 +\newcommand{\WPjel}{C63, C88, L86, O33}
87 +\newcommand{\WPauthor}{Simon-Pierre Boucher}
88 +\newcommand{\WPaffiliation}{Département des sciences administratives\\Université du Québec en Outaouais}
89 +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca}
90 +\newcommand{\WPaddress}{Gatineau -- Pavillon Alexandre-Taché\\283, boulevard Alexandre-Taché\\Gatineau, Québec, Canada J9A 1L8}
91 +
92 +\begin{document}
93 +\input{sections/titlepage}
94 +\input{sections/introduction}
95 +\input{sections/background}
96 +\input{sections/related}
97 +\input{sections/formulation}
98 +\input{sections/architecture}
99 +\input{sections/algorithms}
100 +\input{sections/implementation}
101 +\input{sections/setup}
102 +\input{sections/results}
103 +\input{sections/ablations}
104 +\input{sections/robustness}
105 +\input{sections/limitations}
106 +\input{sections/conclusion}
107 +\newpage
108 +\bibliography{references}
109 +\input{sections/appendix}
110 +\end{document}
modified paper/references.bib +14 −0
@@ -394,3 +394,17 @@
394 394 year={2026},
395 395 note={Version 2.1.0, Apache-2.0. Accessed 2026-11-01. No paper or citation provided by the project}
396 396 }
397 +
398 +% --- added 2026-11-01, verified on arxiv.org/abs pages (title, first authors, date) ---
399 +@article{muennighoff2024olmoe,
400 + title = {{OLMoE}: Open Mixture-of-Experts Language Models},
401 + author = {Muennighoff, Niklas and Soldaini, Luca and Groeneveld, Dirk and Lo, Kyle and Morrison, Jacob and Min, Sewon and Shi, Weijia and Walsh, Pete and Tafjord, Oyvind and Lambert, Nathan and Gu, Yuling and Arora, Shane and Bhagia, Akshita and Schwenk, Dustin and Wadden, David and Wettig, Alexander and Hui, Binyuan and Dettmers, Tim and Kiela, Douwe and Farhadi, Ali and Smith, Noah A. and Koh, Pang Wei and Singh, Amanpreet and Hajishirzi, Hannaneh},
402 + journal = {arXiv preprint arXiv:2409.02060},
403 + year = {2024}
404 +}
405 +@article{yang2025qwen3,
406 + title = {Qwen3 Technical Report},
407 + author = {Yang, An and Li, Anfeng and Yang, Baosong and others},
408 + journal = {arXiv preprint arXiv:2505.09388},
409 + year = {2025}
410 +}
added paper/sections/ablations.tex +12 −0
@@ -0,0 +1,12 @@
1 +\section{Ablations}
2 +\label{sec:ablations}
3 +The ablation grid of the protocol (predictor $\times$ depth $\times$ budget $\times$ policy $\times$ placement $\times$
4 +transport $\times$ kernels $\times$ batch) is executed gate by gate. In this version only the single-node factors have logs:
5 +\begin{itemize}
6 + \item \textbf{Install path} (per-miss synchronisation vs lazy batched installs): $+80$ to $+90\,\%$ decode throughput at 16 slots
7 + (Section~\ref{sec:implementation}).
8 + \item \textbf{Fast path} (device-side slot map when a layer is complete): 203 $\to$ 489~\toks{} at 64 slots.
9 + \item \textbf{Eviction policy} (LRU vs LFU): within 0.01 hit rate at every capacity in simulation.
10 + \item \textbf{Predictor} (none / prev-rank / transition$_d$ / probe / oracle) and \textbf{depth}: Table~\ref{tab:sim}, \textsc{pending} until the v2
11 + simulation and its online confirmation complete.
12 +\end{itemize}
added paper/sections/abstract.tex +15 −0
@@ -0,0 +1,15 @@
1 +\newcommand{\WPabstract}{%
2 +Sparse mixture-of-experts (MoE) language models touch only a few experts per token, so a model whose weights exceed a
3 +machine's memory can still be served if the right experts are in the right place at the right time. This working paper
4 +reports the design, implementation and measurement of \titan{}, an MLX-native inference runtime for Apple Silicon that
5 +manages expert residency with bounded per-layer slot tables, loads experts on demand from local NVMe, prefetches them with
6 +causal predictors, and (in its distributed mode) executes them on other Macs of a heterogeneous cluster connected by
7 +gigabit Ethernet. Every design choice is validated against mlx-lm on identical weights: with all experts resident the runtime is
8 +bit-identical in logits and within one percent of mlx-lm's decode throughput; with bounded residency the router's choices and
9 +the logits are unchanged by construction. We state five hypotheses (causal prefetch, topology-aware placement, hierarchical
10 +caching, fused Metal kernels, and adaptive orchestration on the latency--throughput--energy frontier) and test each one with a
11 +pre-registered protocol on a live inventory of a 21-node Mac cluster. This version (0.1) is a \emph{protocol and preliminary}
12 +working paper: it contains the verified hardware inventory, the prior-art audit, the single-node runtime and its correctness
13 +evidence, and the first controlled single-node experiments; distributed and energy experiments are reported only where their
14 +raw logs exist, and pending experiments are explicitly labelled as such.%
15 +}
added paper/sections/algorithms.tex +40 −0
@@ -0,0 +1,40 @@
1 +\section{Algorithms}
2 +\label{sec:algorithms}
3 +
4 +\begin{algorithm}[t]
5 +\caption{Decode step of one MoE layer in \titan{} (single node)}
6 +\label{alg:layer}
7 +\begin{algorithmic}[1]
8 +\Require hidden rows $X$, router indices $I$ ($R \times K$), slot table $\mathcal{S}_l$, prefetcher $\mathcal{P}$, depth $d$
9 +\If{$\mathcal{S}_l$ holds every expert} \State \Return $\gatherqmm(X, \mathcal{S}_l, \mathrm{slotmap}[I])$ \Comment{no host sync}
10 +\EndIf
11 +\State $U \gets$ unique experts of $I$ (host) \Comment{the unavoidable synchronisation}
12 +\State $\mathcal{P}.\textsc{drain}(l, U)$ \Comment{install finished prefetches; wait for in-flight experts in $U$ (late)}
13 +\State $\hat S \gets \mathcal{P}.\textsc{predict}(l + d)$; $\mathcal{P}.\textsc{issue}(l + d, \hat S)$ \Comment{before this layer's compute}
14 +\State $M \gets U \setminus \mathrm{resident}(\mathcal{S}_l)$; read $M$ from the store as a batch; install into slots (lazy)
15 +\State $J \gets \mathrm{slot}(I)$
16 +\State \Return $\gatherqmm(X, \mathcal{S}_l, J)$ \Comment{same kernel, same bytes as mlx-lm}
17 +\end{algorithmic}
18 +\end{algorithm}
19 +
20 +\begin{algorithm}[t]
21 +\caption{Deterministic cost-aware placement (Gate F heuristic)}
22 +\label{alg:placement}
23 +\begin{algorithmic}[1]
24 +\Require expert popularity $\pi_e$ from fit traces, node costs $(c_n, M_n, \rho_{0n})$, expert size $b$
25 +\State sort experts by $\pi_e$ descending; $\mathrm{free}_n \gets M_n$
26 +\For{each expert $e$ in that order}
27 + \State $n^\star \gets \arg\min_{n:\ \mathrm{free}_n \ge b} \big(c_n + \ind[n \ne 0]\,\rho_{0n}\big) \cdot \pi_e$ \Comment{expected per-step cost}
28 + \State place $e$ on $n^\star$; $\mathrm{free}_{n^\star} \gets \mathrm{free}_{n^\star} - b$
29 +\EndFor
30 +\State replicate the $\rho$ most popular experts on node $0$ if $\mathrm{free}_0$ allows (hot set)
31 +\State experts placed nowhere fall back to node $0$'s slot table (SSD tier, demand loading)
32 +\end{algorithmic}
33 +\end{algorithm}
34 +
35 +Algorithm~\ref{alg:layer} is the per-layer path of Section~\ref{sec:architecture}; the prefetch for layer $l+d$ is issued
36 +\emph{before} the current layer's kernels are dispatched so that the reader threads overlap the GPU work of the next $d$ layers.
37 +Algorithm~\ref{alg:placement} is the first placement heuristic: greedy by popularity, cost-aware by node, with replication of a
38 +hot set on the coordinator. It runs in $\mathcal{O}(LE \log LE + LE N)$ time, negligible against inference, and will be the
39 +static baseline against which the adaptive scheduler (online re-estimation of $\pi$ and of the node costs) is compared.
40 +Predictor fitting is $\mathcal{O}(T L K^2)$ for transition counts and one logistic regression per layer for the probe.
added paper/sections/appendix.tex +21 −0
@@ -0,0 +1,21 @@
1 +\begin{appendices}
2 +\section{Reproducibility}
3 +\label{app:repro}
4 +\paragraph{Software.} Python 3.12 virtual environment with MLX 0.32.4 and mlx-lm 0.32.2 (\code{requirements-lock.txt});
5 +reference repositories audited at colibri \code{0a5504a} (2026-10-31), mlx \code{bd63d38} (2026-11-01), mlx-lm \code{736ab0d}
6 +(2026-10-30), llama.cpp \code{a91eab6} (2026-10-31).
7 +\paragraph{Commands.} Correctness: \code{.venv/bin/python -m pytest tests -q}. Capacity sweep:
8 +\code{benchmarks/scripts/bench\_decode.py --backend \{mlx\_lm,titan\} --capacity C --predictor P --traces T}. Traces:
9 +\code{benchmarks/scripts/collect\_traces.py}. Simulation: \code{benchmarks/scripts/simulate\_cache.py}. SSD tier:
10 +\code{benchmarks/scripts/ssd\_bandwidth.py} after \code{sudo purge}. Node preparation: \code{scripts/cluster\_setup.sh}.
11 +\paragraph{Raw data.} \code{benchmarks/raw/gateA/} (smoke baseline, SSD), \code{gateB/} (design microbenchmarks, capacity sweep,
12 +tests), \code{gateC/} (traces, online prefetch, miss-cost breakdown, Qwen3 baselines); processed simulations in
13 +\code{benchmarks/processed/}. Logs are append-only JSON lines.
14 +\paragraph{Prior-art table.} \code{research/prior\_art.csv} (59 rows, evidence URL per row) and
15 +\code{research/RELATED\_WORK.md}.
16 +\paragraph{AI-assistance disclosure.} The software, experiments and this manuscript were produced by Claude Code (Anthropic,
17 +model Claude Fable 5.1) acting as an autonomous research engineer under the author's written mandate
18 +(\code{docs/MANDATE.md}); the author set the research question, hypotheses, integrity rules and gates, and is responsible for
19 +the content. Every quantitative claim traces to a logged measurement; the agent's decisions are recorded in
20 +\code{research/DECISIONS.md} and its chronology in \code{research/LOG.md}.
21 +\end{appendices}
added paper/sections/architecture.tex +72 −0
@@ -0,0 +1,72 @@
1 +\section{Architecture}
2 +\label{sec:architecture}
3 +
4 +Figure~\ref{fig:arch} shows the components of \titan{}. The host model (embeddings, attention, norms, routers, KV cache) is
5 +mlx-lm's own module tree, loaded lazily so that the stacked expert tensors are never materialised; each layer's expert block
6 +is replaced by a \titan{} block that owns a \emph{slot table}. Everything else is orchestration around that table.
7 +
8 +\begin{figure}[t]
9 +\centering
10 +\begin{tikzpicture}[node distance=4mm and 6mm, every node/.style={font=\small},
11 + box/.style={draw, rounded corners=2pt, minimum height=7mm, minimum width=24mm, align=center, fill=gray!8},
12 + tier/.style={draw, dashed, rounded corners=2pt, minimum height=7mm, minimum width=24mm, align=center},
13 + arr/.style={-{Latex[length=2mm]}, thick}]
14 + \node[box] (reg) {Expert registry\\\scriptsize safetensors byte ranges};
15 + \node[box, right=of reg] (store) {Expert store\\\scriptsize \code{pread}, F\_NOCACHE};
16 + \node[box, right=of store] (slots) {Slot tables ($L$)\\\scriptsize capacity $C$, LRU/LFU};
17 + \node[box, right=of slots] (exec) {Execution block\\\scriptsize \gatherqmm{} on slots};
18 + \node[box, below=of store] (pred) {Predictors\\\scriptsize prev-rank, transition$_d$, probe};
19 + \node[box, right=of pred] (pf) {Prefetcher\\\scriptsize reader pool, drain/install};
20 + \node[box, right=of pf] (host) {Host model (mlx-lm)\\\scriptsize attention, router, KV};
21 + \node[tier, below=of pf] (net) {Transport (Gate E)\\\scriptsize TCP, authenticated};
22 + \node[tier, right=of net] (sched) {Scheduler (Gate F)\\\scriptsize cost model, placement};
23 + \draw[arr] (reg) -- (store);
24 + \draw[arr] (store) -- (slots);
25 + \draw[arr] (slots) -- (exec);
26 + \draw[arr] (host) -- node[above,font=\scriptsize]{routes} (exec);
27 + \draw[arr] (exec) -- node[left,font=\scriptsize]{routes, probs} (pf);
28 + \draw[arr] (pf) -- (pred);
29 + \draw[arr] (pf) -- node[left,font=\scriptsize]{install} (slots);
30 + \draw[arr] (pred) -- node[above,font=\scriptsize]{$\hat S$} (pf);
31 + \draw[arr, dashed] (net) -- (slots);
32 + \draw[arr, dashed] (sched) -- (net);
33 +\end{tikzpicture}
34 +\caption{Components of \titan{} on one node. Solid boxes are implemented and tested in version 0.1; dashed boxes are the
35 +distributed components whose status is given in Section~\ref{sec:limitations}.}
36 +\label{fig:arch}
37 +\end{figure}
38 +
39 +\paragraph{Expert registry.} The registry parses \code{config.json} and the safetensors headers of a checkpoint directory
40 +and records, for every (layer, expert) and every matrix (gate, up, down) and component (packed weight, scales, biases), the
41 +absolute byte offset and length in the file, together with the routing semantics (softmax-then-top-$k$, renormalisation flag,
42 +shared experts) and the quantization (bits, group size). It accepts only verified formats (OLMoE and Qwen3-MoE in this
43 +version) and refuses the others rather than guessing.
44 +
45 +\paragraph{Expert store (SSD tier).} An expert is read with one \code{pread} per tensor at the registry's offsets, wrapped
46 +into MLX arrays without further copies on the Python side; bytes, read count and read time are accounted. The file descriptor
47 +can be opened with \code{F\_NOCACHE} so that reads bypass the unified buffer cache, which is how the cold-NVMe tier is
48 +measured (after \code{purge}); without it the same code measures the page-cache tier.
49 +
50 +\paragraph{Slot tables.} Each layer owns $C$ slots in stacked MLX buffers of the exact shapes mlx-lm uses for its stacked
51 +experts, so that the same \gatherqmm{} kernel can run on them. A dictionary maps resident expert ids to slots; eviction is
52 +LRU by default (LFU available). Installing an expert is a slice assignment into the buffers, which MLX performs in place.
53 +Misses are read as a batch per layer and installed lazily so that the layer's kernel consumes the new buffers in the same
54 +graph: there is no per-expert synchronisation. The table records hits, misses, evictions, loads and stall time.
55 +
56 +\paragraph{Execution block.} Given the router's indices, the block materialises them on the host (the one synchronisation a
57 +demand-loading design cannot avoid), ensures residency, remaps expert ids to slot ids and runs \gatherqmm{} for the up, gate
58 +and down projections with mlx-lm's compiled SwiGLU. When a layer's table holds every expert (the all-resident control) a
59 +device-side slot map is used and no synchronisation occurs, which recovers mlx-lm's throughput. Prefill rows whose union of
60 +experts exceeds $C$ are chunked greedily by rows, which is exact because rows are independent.
61 +
62 +\paragraph{Predictors and prefetcher.} A predictor proposes up to $m$ experts for a layer from information available at the
63 +decision point: the previous token's routing and router probabilities (token horizon), or the routing of layer $l-d$ at the
64 +current token (layer horizon, depth $d$). The prefetcher submits reads to a small thread pool, installs completed fetches when
65 +the target layer is reached, waits for fetches that are needed but still in flight (counted as \emph{late}, with the residual
66 +wait charged as stall), and never changes which experts run.
67 +
68 +\paragraph{Hierarchical memory and distributed execution.} The remote tier does not move weights during decode (Section
69 +\ref{sec:prefetch-condition}): a remote node that holds an expert executes it on the coordinator's activations and returns the
70 +$d$-dimensional result, exactly as colibri's cluster mode and as expert parallelism do. The transport (Gate E) is a persistent
71 +authenticated TCP connection per worker with a per-layer batched request, timeouts and local fallback; placement (Gate F)
72 +follows the cost model of Section~\ref{sec:formulation} with measured $c_n$, $M_n$, $\rho_{0n}$ and $\lambda$ per node.
added paper/sections/background.tex +40 −0
@@ -0,0 +1,40 @@
1 +\section{Background}
2 +\label{sec:background}
3 +
4 +\subsection{Sparse MoE inference}
5 +Let a decoder have layers $l \in \Lset = \{1,\dots,L\}$. In an MoE layer the token representation $h \in \R^{d}$ is routed by
6 +$g = \operatorname{softmax}(W_r h) \in \R^{E}$; the top-$K$ entries of $g$ select experts $S(h) \subset \E=\{1,\dots,E\}$,
7 +$|S(h)| = K$, and the layer output is $\sum_{e \in S(h)} \tilde g_e\, f_e(h)$ where $f_e$ is a gated feed-forward network
8 +(SwiGLU in the models we use) and $\tilde g$ equals $g$ restricted to $S(h)$, renormalised in some architectures. OLMoE
9 +\citep{muennighoff2024olmoe} and Qwen3-MoE \citep{yang2025qwen3} follow this template without a shared expert; DeepSeek-style models add shared experts and
10 +grouped routing, which our registry records but this version does not execute.
11 +
12 +During prefill the $T$ prompt tokens are processed together and a layer typically needs most of its experts (with $T=20$ and
13 +$K=8$ the union of selected experts already approaches $E=64$ in OLMoE), so prefill is compute-bound and resembles dense
14 +inference. During decode one token at a time selects $K$ experts per layer; the $L K$ expert weight matrices must be read from
15 +wherever they reside. If every expert is resident in unified memory, the decode step is bound by memory bandwidth and kernel
16 +launch latency \citep{javat2026siliconshowdown}; if some are not, the step is bound by the latency of fetching the missing
17 +ones. The cache hit rate therefore translates almost directly into throughput, which is why expert caching and prediction
18 +dominate the literature of Section~\ref{sec:related}.
19 +
20 +\subsection{Apple Silicon as an inference platform}
21 +An Apple Silicon system-on-chip shares one physical memory between CPU, GPU and Neural Engine. For offloading this removes
22 +the PCIe copy but not the fetch: a weight on disk must still be read into memory, and a weight on another machine must cross a
23 +network. The MLX framework exposes lazy arrays evaluated as a graph; inserting an expert into a resident buffer is a slice
24 +assignment that MLX performs in place when the buffer is not otherwise referenced (we measure 0.3~ms for one OLMoE expert in
25 +Section~\ref{sec:implementation}). The \gatherqmm{} primitive multiplies a batch of rows by a batch of quantized matrices selected
26 +by index, which is exactly the operation an MoE layer needs; mlx-lm uses it for all its 50-odd MoE architectures.
27 +
28 +Three facts about our cluster matter for every later number. First, all nodes negotiate 1000BASE-T on their Ethernet port
29 +(the switch is gigabit even though the Mac Studios have 10~GbE ports); iperf3 measures 0.945~Gbit/s between any two Studios and
30 +ping 0.5--1.0~ms round trip. Second, no Thunderbolt link is active between any two nodes, so RDMA over Thunderbolt~5 and
31 +MLX's JACCL backend are unavailable until cables are installed. Third, the development laptop is not on the cluster LAN; all
32 +distributed experiments run between LAN nodes with a Mac Studio as coordinator.
33 +
34 +\subsection{Baselines}
35 +mlx-lm \citep{hannun2023mlx} is the exact baseline: same weights, same quantization, same kernels. llama.cpp
36 +\citep{llamacpp2026} runs the same base checkpoints through its Metal backend but with its own K-quant formats, so its
37 +speed is reported with a precision flag and a perplexity check rather than as a like-for-like competitor. colibri
38 +\citep{colibri2026} streams experts from disk in C; on macOS its OLMoE engine is CPU-only (its Metal backend covers other
39 +engines), so it serves as a reference for the design space rather than a same-hardware competitor; its cluster mode, which
40 +assigns experts to worker Macs by a deterministic weighted hash, is the closest existing analogue of our static placement.
added paper/sections/conclusion.tex +9 −0
@@ -0,0 +1,9 @@
1 +\section{Conclusion}
2 +\label{sec:conclusion}
3 +We have built an MLX-native MoE runtime whose bounded expert residency is semantically invisible (bit-identical logits to mlx-lm)
4 +and whose all-resident path costs nothing (489 vs 494~\toks{}), characterised the cluster it will run on as it actually is
5 +(gigabit Ethernet, no Thunderbolt), and obtained a first negative result that the formulation explains: one-layer-ahead
6 +prefetching cannot pay when the fetch latency exceeds the per-layer compute time. The next steps are fixed by the protocol:
7 +depth-$d$ and token-horizon prefetching against the cold-NVMe tier, the medium model on nodes that cannot hold it, the
8 +two-node TCP transport with static placement, the cost-aware placement of Algorithm~\ref{alg:placement}, profiling-driven Metal
9 +work, and the energy frontier. Each will be reported with its raw logs, including the cases where \titan{} loses.
added paper/sections/formulation.tex +99 −0
@@ -0,0 +1,99 @@
1 +\section{Mathematical Formulation}
2 +\label{sec:formulation}
3 +
4 +\subsection{Objects}
5 +Let $\E$ be the set of (layer, expert) pairs, $|\E| = LE$, each of size $b$ bytes (3.375~MiB for OLMoE at 4 bits; equal
6 +within an architecture). Let $\Nset = \{0,1,\dots,N\}$ be the nodes, node $0$ being the coordinator that runs attention, the
7 +router and the KV cache, and let tier $\mathcal{T}_n = \{\mathrm{ram}, \mathrm{ssd}\}$ be the storage tiers of node $n$.
8 +Tokens are steps $t = 1,2,\dots$; the router of layer $l$ at step $t$ selects $S_{t,l} \subset \E_l$, $|S_{t,l}| = K$.
9 +Decision variables are binary and indexed by step:
10 +\begin{align}
11 + x_{e,n,t} &\in \{0,1\} \quad \text{expert $e$ is resident in the memory of node $n$ at step $t$,} \\
12 + z_{e,n,t} &\in \{0,1\} \quad \text{expert $e$ is \emph{executed} on node $n$ at step $t$,} \\
13 + p_{e,n,t} &\in \{0,1\} \quad \text{a prefetch of $e$ into node $n$ is issued during step $t$.}
14 +\end{align}
15 +Every expert exists on the local NVMe of each node that holds a copy of the checkpoint (the SSD tier is unbounded for our
16 +purposes and costs only time), so residency refers to memory.
17 +
18 +\subsection{Constraints}
19 +\begin{align}
20 + \sum_{e \in \E} b\, x_{e,n,t} &\le M_n \qquad \forall n, t && \text{(memory budget of node $n$)} \label{eq:mem}\\
21 + \sum_{n \in \Nset} z_{e,n,t} &= \ind[e \in S_{t,l(e)}] \qquad \forall e, t && \text{(every selected expert executed exactly once)} \label{eq:exec}\\
22 + z_{e,n,t} &\le x_{e,n,t} + m_{e,n,t} \qquad \forall e, n, t && \text{(execution needs residency or a demand miss $m$)} \label{eq:res}\\
23 + x_{e,n,t+1} &\le x_{e,n,t} + p_{e,n,t} + m_{e,n,t} \qquad \forall e, n, t && \text{(residency only grows by prefetch or miss)} \label{eq:dyn}
24 +\end{align}
25 +where $m_{e,n,t} \in \{0,1\}$ is the demand-miss indicator, itself a consequence of the router and of $x$. Constraint
26 +\eqref{eq:exec} encodes the correctness invariant: the set of executed experts is the router's set, whatever the placement.
27 +
28 +\subsection{Costs}
29 +The per-step latency decomposes along the layer sequence, because layer $l$ cannot start before layer $l-1$'s output exists:
30 +\begin{equation}
31 + \Lambda_t \;=\; \sum_{l=1}^{L} \Big[ \underbrace{\tau^{\mathrm{attn}}_{0}}_{\text{attention, node 0}}
32 + + \max_{n \in \Nset}\big( \underbrace{c_n \cdot \textstyle\sum_{e \in S_{t,l}} z_{e,n,t}}_{\text{expert compute on $n$}}
33 + + \underbrace{\rho_{0n}\,\ind[\exists e\!:\! z_{e,n,t}=1, n\neq 0]}_{\text{activation round trip}} \big)
34 + + \underbrace{\sigma_{t,l}}_{\text{stall}} \Big],
35 + \label{eq:latency}
36 +\end{equation}
37 +with $c_n$ the measured per-expert compute time of node $n$, $\rho_{0n}$ the measured round-trip time of a
38 +$d$-dimensional activation and its result between node $0$ and $n$ (on our LAN, $0.5$--$1.0$~ms), and the stall
39 +\begin{equation}
40 + \sigma_{t,l} \;=\; \max\Big(0,\; \max_{e \in S_{t,l},\, m_{e,n,t} = 1} \big( \lambda_{\mathrm{tier}(e)} - \eta_{e,t} \big) \Big),
41 + \label{eq:stall}
42 +\end{equation}
43 +where $\lambda_{\mathrm{tier}}$ is the fetch latency of the tier the expert comes from ($\lambda_{\mathrm{pc}} = 0.21$~ms page
44 +cache, $\lambda_{\mathrm{ssd}} = 0.43$~ms cold NVMe, $\lambda_{\mathrm{lan}} \approx 28$~ms for 3.4~MB over 0.945~Gbit/s, all
45 +measured on our hardware) and $\eta_{e,t}$ is the lead time between the moment a prefetch of $e$ was issued and the moment
46 +$e$ is needed ($\eta = 0$ for a demand miss). Energy per step is $\Phi_t = \sum_n P_n \cdot \Lambda^{(n)}_t$ with $P_n$ the
47 +package power of node $n$ measured by \code{powermetrics} during the run, and communication cost $\Gamma_t = \sum_{n \ne 0}
48 +\beta_{0n}\,(\text{activation bytes} + \text{weight bytes moved})$.
49 +
50 +\subsection{Problem}
51 +\begin{equation}
52 + \min_{x,z,p}\; \Ex\Big[\sum_{t} \Lambda_t + \lambda\,\Phi_t + \mu\,\Gamma_t\Big] \quad \text{s.t. \eqref{eq:mem}--\eqref{eq:dyn}},
53 + \label{eq:problem}
54 +\end{equation}
55 +where the expectation is over the router's sequence $\{S_{t,l}\}$, which is revealed online: at step $t$, $S_{t,l}$ becomes
56 +known only after layer $l-1$'s output is computed. The problem is a stochastic online integer program; its offline relaxation
57 +with known $S$ is the paging problem with multiple caches and non-uniform fetch costs, for which Belady's furthest-in-future
58 +rule is optimal per cache and which our simulator's \emph{oracle} predictor approximates.
59 +
60 +\subsection{When can a prefetch pay?}
61 +\label{sec:prefetch-condition}
62 +Consider a predictor issuing, when layer $l-d$'s routing is known at step $t$, a set $\hat S_{t,l}$ of $m$ experts for layer
63 +$l$, with recall $r = \Ex|\hat S \cap S|/K$ and precision $q = \Ex|\hat S \cap S|/m$. With per-layer decode time $\tau_\ell$
64 +the lead time is $\eta = d\,\tau_\ell$. Per layer, the expected stall without prefetch is
65 +$\sigma_0 = \lceil K(1-h)/w \rceil\, \lambda$ for hit rate $h$ and $w$ parallel loaders, and with prefetch
66 +\begin{equation}
67 + \sigma_d \;\approx\; \lceil K(1-h)(1-r')/w \rceil\,\lambda \;+\; \lceil K(1-h)\, r'/w \rceil\,\max(0, \lambda - d\,\tau_\ell),
68 + \label{eq:prefetch-gain}
69 +\end{equation}
70 +where $r'$ is the recall among the experts that would have missed. The prefetch also adds $m(1-q)$ useless fetches whose
71 +bandwidth and install cost $\kappa$ each is paid on the critical path when the loader is shared with demand misses.
72 +A prefetch can only reduce latency if
73 +\begin{equation}
74 + r'\,\min(\lambda, d\,\tau_\ell) \;>\; m(1-q)\,\kappa\;/\;K(1-h),
75 + \label{eq:condition}
76 +\end{equation}
77 +and it can \emph{remove} the stall only if $d\,\tau_\ell \ge \lambda$. On the laptop, $\tau_\ell = 0.128$~ms (489~\toks{} over
78 +16 layers), so a page-cache fetch ($\lambda = 0.21$~ms) needs $d \ge 2$ and a cold-NVMe fetch ($\lambda = 0.43$~ms) needs
79 +$d \ge 4$; a LAN fetch ($\lambda \approx 28$~ms) can never be hidden within a token on this model, which is why weights are not
80 +moved over the LAN during decode in \titan{} and remote experts are instead executed where they reside. The one-layer online
81 +experiment of Section~\ref{sec:results-prefetch} violated \eqref{eq:condition} ($d = 1$, $q \approx 0.5$) and lost throughput, as
82 +predicted; the depth-$d$ experiments test the condition directly.
83 +
84 +\begin{proposition}[Demand loading is $\mathcal{O}(1)$-competitive per layer only when the table exceeds the per-step
85 +working set]
86 +\label{prop:capacity}
87 +With per-layer tables of capacity $C \ge K$, LRU eviction, and no prefetch, the number of demand misses at a decode step is at
88 +most $K$ and at least $K - |S_{t,l} \cap S_{t-1,l}| $ when $C = K$; for $C \ge |\bigcup_{s \le \Delta} S_{t-s,l}|$ the misses at
89 +step $t$ are exactly the experts not used in the last $\Delta$ steps at that layer.
90 +\end{proposition}
91 +\begin{proof}
92 +With $C = K$ the table after step $t-1$ holds exactly $S_{t-1,l}$, so every expert of $S_{t,l} \setminus S_{t-1,l}$ misses and
93 +no other does; misses are bounded by $K$ because $|S_{t,l}| = K$. For the second claim, LRU keeps the most recently used
94 +$C$ distinct experts; if $C$ is at least the number of distinct experts used in the last $\Delta$ steps, all of them are
95 +resident, so a miss at step $t$ is precisely an expert unused during those $\Delta$ steps.
96 +\end{proof}
97 +The proposition is elementary but useful: it says that the hit rate of LRU at capacity $C$ is the fraction of a step's
98 +experts that recur within the $C$-recent window of that layer, a quantity we measure directly from traces
99 +(Section~\ref{sec:results-locality}) and which, for OLMoE, is 0.57 at $C = 16$ and 0.80 at $C = 32$.
added paper/sections/implementation.tex +29 −0
@@ -0,0 +1,29 @@
1 +\section{Implementation}
2 +\label{sec:implementation}
3 +
4 +\titan{} is about 1\,100 lines of Python over MLX 0.32.4 and mlx-lm 0.32.2 (Python 3.12), organised as
5 +\code{titan/\{registry, store, slots, moe, prefetch, predictors, runtime\}.py}, with benchmark drivers in
6 +\code{benchmarks/scripts/} and tests in \code{tests/}. Three implementation findings determined the design.
7 +
8 +\paragraph{In-place slot updates.} Writing the nine tensors of one OLMoE expert (3.4~MB) into stacked buffers of 64 slots
9 +(216~MB per matrix) costs 0.305~ms, which shows that MLX reuses the buffer rather than copying it; a per-expert
10 +\code{quantized\_matmul} loop costs 0.43~ms per layer for one token against 0.26~ms for \gatherqmm{}, and restacking eight
11 +experts per call 0.50~ms. The slot table with \gatherqmm{} is therefore both the fastest and the most faithful option
12 +(\code{benchmarks/raw/gateB/design/}).
13 +
14 +\paragraph{No per-miss synchronisation.} The first version evaluated the table after every miss; removing that
15 +synchronisation and batching the reads of a layer's misses raised decode throughput at 16 slots from 28 to 51~\toks{}
16 +(general prompts) and from 41 to 73~\toks{} (code), with the stall halved
17 +(\code{benchmarks/raw/gateC/prefetch\_online\_laptop/}). The per-miss cost on the page-cache tier is now 0.21~ms: 0.13~ms
18 +of \code{pread}, 0.06~ms of array creation and the remainder in the lazy install.
19 +
20 +\paragraph{Activation parity.} An unfused float16 $\mathrm{up}\cdot\sigma(\mathrm{gate})\cdot\mathrm{gate}$ gave
21 +$\max|\Delta\mathrm{logit}| = 0.0195$ against mlx-lm; using mlx-lm's compiled SwiGLU gives $0.0$. Numerical parity in float16 is
22 +a property of the exact kernel sequence, not only of the weights, and the test suite pins it.
23 +
24 +\paragraph{Reproducibility.} Every benchmark writes JSON lines with the command, git commit, host, chip, memory, macOS,
25 +library versions, seed and timestamps; raw logs are append-only under \code{benchmarks/raw/} and tables and figures are
26 +generated from them by scripts. Model revisions are pinned by commit hash in \code{benchmarks/configs/models.yaml};
27 +weights are never stored in the repository. A node is prepared with \code{scripts/cluster\_setup.sh} (rsync of the code,
28 +pinned virtual environment, import check), and the sanitized inventory \code{artifacts/environment.json} is generated from
29 +raw probes that are kept out of version control.
added paper/sections/introduction.tex +68 −0
@@ -0,0 +1,68 @@
1 +\section{Introduction}
2 +\label{sec:intro}
3 +
4 +A sparse mixture-of-experts (MoE) transformer replaces the dense feed-forward block of each layer by a bank of $E$ expert
5 +networks and a router that sends every token to $K \ll E$ of them \citep{rajbhandari2022deepspeedmoe}. The consequence that
6 +matters for inference is arithmetic: a token activates $K/E$ of the expert parameters, so the \emph{working set} of a decode
7 +step is small even when the \emph{model} is enormous. OLMoE-1B-7B activates 1.3 of its 6.9 billion parameters; in its
8 +4-bit MLX conversion a decode step touches 128 experts of 3.4 MB each, 432 MiB out of 3.6 GB of expert weights
9 +(Section~\ref{sec:setup}). Engines such as colibri \citep{colibri2026} exploit exactly this to run models of hundreds of billions of
10 +parameters from an NVMe disk on a laptop; the price is that the disk, not the GPU, sets the speed.
11 +
12 +Apple Silicon is an unusual target for this problem. Its unified memory removes the host-to-device copy that dominates
13 +GPU offloading systems, its NVMe storage is fast (we measure 8.5~GB/s for random 3.4~MB reads on an M5~Max), and small
14 +clusters of Macs are common in laboratories that cannot or do not want to buy datacenter accelerators. The MLX framework
15 +\citep{hannun2023mlx} provides a quantized grouped matrix-multiplication kernel (\gatherqmm{}) that executes a batch of
16 +expert products in one dispatch, and, since macOS~26.2, RDMA over Thunderbolt~5 between Mac Studios
17 +\citep{apple2026mlxdistributed}. What does not exist is a runtime that decides, per expert and per token, \emph{where} an
18 +expert should reside (local memory, a peer's memory, or the local disk), \emph{when} to move it (on demand or ahead of the
19 +router), and \emph{which node} should execute it, using measured properties of the machines and links at hand.
20 +
21 +This working paper describes \titan{} (Topology-aware Intelligent exTert orchestrAtion for distributed MoE iNference), a
22 +runtime built to answer one question stated as five testable hypotheses (Section~\ref{sec:hypotheses}): can topology-aware
23 +placement, causal prefetching, hierarchical caching and specialised Metal kernels improve decode latency, prefill throughput,
24 +energy per token and feasible model capacity over strong existing baselines on Apple Silicon? We insist on three
25 +methodological commitments that shape the rest of the paper.
26 +
27 +\paragraph{Placement decides speed, never semantics.} The router's expert identities, gating weights and aggregation order
28 +are those of the reference implementation (mlx-lm). A resident, a prefetched and a remotely executed expert must produce the
29 +same bytes. Section~\ref{sec:results-correctness} shows that \titan{} with bounded residency reproduces mlx-lm's logits
30 +exactly ($\max|\Delta| = 0$ in float16) on the test prompts; cache-aware routing, which trades expert identity for hits
31 +\citep{skliar2025cacheaware}, is therefore out of scope by construction.
32 +
33 +\paragraph{Measure before optimising.} Every design decision in Section~\ref{sec:architecture} is preceded by a microbenchmark
34 +recorded in the repository, from the cost of writing one expert into a stacked buffer (0.3~ms) to the per-expert fetch
35 +latency of each tier (0.21~ms from the page cache, 0.43~ms from a cold NVMe, about 28~ms over the gigabit LAN that actually
36 +connects the cluster). The inventory of the cluster is read live and is reported as read, including the seven nodes that did
37 +not answer and the absence of any Thunderbolt link between the Mac Studios.
38 +
39 +\paragraph{Report losses.} Our first online experiment with one-layer-ahead prefetching \emph{raised} the expert hit rate and
40 +\emph{lowered} throughput (Section~\ref{sec:results-prefetch}); the paper keeps that result and uses it to derive the
41 +condition under which prefetching can pay on this hardware.
42 +
43 +\subsection{Hypotheses}
44 +\label{sec:hypotheses}
45 +\begin{description}[style=nextline]
46 + \item[H1 (causal prefetch)] Predicting upcoming expert demand from information available at decision time and fetching
47 + asynchronously reduces expert-loading stalls net of predictor overhead.
48 + \item[H2 (topology-aware placement)] Placing experts on heterogeneous nodes according to measured compute, memory and
49 + link costs outperforms uniform or static sharding.
50 + \item[H3 (hierarchical caching)] Coordinating local memory, remote memory and local NVMe under a fixed memory budget improves
51 + throughput over any single second tier.
52 + \item[H4 (Metal kernels)] Selected fused Metal kernels improve end-to-end inference over stock MLX kernels for matching
53 + shapes and precision.
54 + \item[H5 (adaptive orchestration)] Adapting placement and prefetching online improves the Pareto frontier of latency,
55 + throughput and energy, not only isolated microbenchmarks.
56 +\end{description}
57 +
58 +\subsection{Contributions and status of this version}
59 +Version~0.1 of this working paper contributes (i) a verified inventory and network characterisation of a 21-node Apple Silicon
60 +cluster and the audit of four reference code bases (colibri, MLX, mlx-lm, llama.cpp) with a prior-art matrix of 59 works;
61 +(ii) the \titan{} single-node runtime: an expert registry over safetensors byte ranges, a \code{pread}-based NVMe tier,
62 +per-layer slot tables with LRU/LFU eviction, an exact execution block over \gatherqmm{}, and an asynchronous prefetcher
63 +with four causal predictors; (iii) the correctness evidence (bit-identical logits, token-identical greedy decoding);
64 +(iv) a trace-driven simulator calibrated on measured constants and the first controlled single-node experiments on
65 +OLMoE-1B-7B; (v) a formal statement of the placement--prefetch problem (Section~\ref{sec:formulation}) and the deterministic
66 +cost-aware heuristic that the distributed gates will use. Distributed execution (Gates E--F of the protocol), Metal kernel work
67 +(Gate~D) and the energy Pareto study are implemented or planned as described in Section~\ref{sec:limitations} and reported only
68 +to the extent that raw logs exist.
added paper/sections/limitations.tex +19 −0
@@ -0,0 +1,19 @@
1 +\section{Limitations}
2 +\label{sec:limitations}
3 +\begin{itemize}
4 + \item \textbf{Scope of evidence.} Version 0.1 reports single-node results on a small model whose experts fit in memory, so the
5 + regimes where H1--H3 can matter (expert set larger than the budget of a node) are only approached by artificially bounding the
6 + slot tables. The medium model on the 36--64~GB nodes is the first realistic regime.
7 + \item \textbf{Topology.} The cluster is connected by gigabit Ethernet with no Thunderbolt link; RDMA/JACCL results would require
8 + cabling the Studios (a physical change outside the agent's authority). A LAN fetch of one expert costs about 28~ms, which rules
9 + out moving weights during decode and confines the remote tier to remote \emph{execution}.
10 + \item \textbf{Synchronisation floor.} Any demand-loading design must read the router's choice on the host once per layer; on this
11 + model that costs 2.4$\times$ in throughput. Hiding it requires either complete tables (no benefit from caching) or speculative
12 + execution on predicted experts with verification, which changes nothing semantically only if the verification is exact --
13 + a Gate D question.
14 + \item \textbf{Predictors.} Only simple predictors are tested; the linear probe is under-trained with the small trace corpus.
15 + \item \textbf{Measurements.} Stall times in simulation use point estimates of tier latencies; online runs use the page-cache tier
16 + unless stated; energy is not yet measured.
17 + \item \textbf{Baselines.} llama.cpp and colibri comparisons are pending and will carry precision flags because their
18 + quantization formats differ from MLX's.
19 +\end{itemize}
added paper/sections/related.tex +41 −0
@@ -0,0 +1,41 @@
1 +\section{Related Work}
2 +\label{sec:related}
3 +
4 +The prior-art table in the reproducibility appendix (and \code{research/prior\_art.csv} in the repository) lists 59 works
5 +verified on their arXiv, publisher or repository pages on 2026-11-01; here we summarise what they establish and where
6 +\titan{} differs.
7 +
8 +\paragraph{Expert offloading and SSD streaming.} \citet{eliseev2023fast} introduced per-expert offloading with an LRU
9 +cache for Mixtral on consumer GPUs; MoE-Infinity \citep{xue2024moeinfinity} built sparsity-aware expert caches from
10 +activation traces; Fiddler \citep{kamahori2025fiddler}, HybriMoE \citep{zhong2025hybrimoe} and DAOP \citep{zhang2025daop}
11 +execute cold experts on the CPU rather than moving them; MoE-Lightning \citep{cao2025moelightning}, fMoE
12 +\citep{yu2025finemoe}, MoE-Gen \citep{xu2025moegen} and Klotski \citep{fang2025klotski} optimise throughput under memory
13 +constraints with pipelined offloading. SSD-tier streaming has been demonstrated at trillion-parameter scale on x86
14 +\citep{liang2026ssdllama}; FlashMoE \citep{kim2026flashmoe} learns a cache-replacement policy that beats LRU and LFU; colibri
15 +\citep{colibri2026} combines a per-layer LRU, a pinned hot set learned across sessions and a one-layer router lookahead in a
16 +pure C engine. The principle that placement must not change the model's semantics is stated explicitly by colibri and adopted
17 +here.
18 +
19 +\paragraph{Expert prediction.} Pre-gated MoE \citep{hwang2024pregated} and Read-ME \citep{cai2024readme} modify the
20 +architecture so that layer $l$'s router is computed from layer $l-1$'s representation; Fate \citep{fang2025fate}, ProMoE
21 +\citep{song2024promoe}, ExpertFlow \citep{he2026expertflow,shen2025expertflow}, LayerScope \citep{yu2026layerscope},
22 +DuoServe \citep{zhang2025duoserve}, SpecPrefetch \citep{kong2026specprefetch} and SeqMoE \citep{wang2026seqmoe} learn
23 +cross-layer or multi-step predictors on GPUs and NPUs; \citet{liang2026notall} show that routing locality varies across
24 +models and that a cache of about twice the active expert count is often enough. \titan{}'s predictors are deliberately
25 +simple (transition counts at depth $d$, a linear probe on router log-probabilities, a previous-token rank predictor) so
26 +that their cost can be measured against their benefit; the novelty we test is not a predictor but the \emph{condition},
27 +derived in Section~\ref{sec:formulation}, under which any predictor can pay on a given tier of a given machine.
28 +
29 +\paragraph{Distributed inference on consumer hardware.} Petals \citep{borzunov2023petals}, exo \citep{exo2025}, prima.cpp
30 +\citep{li2025primacpp} and llama.cpp's RPC backend shard layers or tensors across devices; MLX's distributed backends
31 +(ring, MPI, JACCL) and Apple's own 4$\times$M5~Ultra results \citep{apple2026mlxdistributed} shard whole layers over a
32 +Thunderbolt mesh. Expert-level placement across nodes exists for datacenter GPUs (ExFlow \citep{yao2024exflow}, Lina
33 +\citep{li2023lina}, DeepEP \citep{deepep2025}) and, in a simple form, in colibri's cluster mode, where experts are hashed to
34 +workers with weights proportional to disk speed. None of these models node heterogeneity, link latency or replication; H2
35 +targets that gap on the only topology our cluster offers, gigabit Ethernet.
36 +
37 +\paragraph{Energy and characterisation on Apple Silicon.} GreenBench \citep{kannan2026greenbench} and Silicon Showdown
38 +\citep{javat2026siliconshowdown} measure joules per token with \code{powermetrics} for dense models; \citet{alfarizy2026doesmoe}
39 +find that MoE energy on edge devices tracks total rather than active parameters; NPUMoE \citep{benazir2026npumoe} runs MoE
40 +on the Neural Engine with all experts resident. No prior work reports an energy--throughput frontier for \emph{offloaded} or
41 +\emph{distributed} MoE on Macs, which H5 addresses.
added paper/sections/results.tex +66 −0
@@ -0,0 +1,66 @@
1 +\section{Results}
2 +\label{sec:results}
3 +
4 +Every number below has a raw log in \code{benchmarks/raw/}; the path is given with each result. Results marked
5 +\textsc{pending} have a protocol but no log yet in this version.
6 +
7 +\subsection{Correctness}
8 +\label{sec:results-correctness}
9 +On four prompts (general, code, reasoning, French) \titan{} reproduces mlx-lm's logits with $\max|\Delta| = 0.0$ (float16) at
10 +capacities 64, 32, 16 and 8 slots per layer, with identical argmax at every position and identical greedy continuations of
11 +32 tokens at 16 slots (\code{benchmarks/raw/tests/numerical\_*.json}, \code{greedy\_*.json}, \code{pytest\_gateB\_*.txt},
12 +7 tests passed). The registry's byte ranges read with \code{pread} are byte-equal to \code{mx.load} for sampled experts of both the
13 +per-expert layout (OLMoE) and the stacked layout (Qwen3-30B-A3B, on M3U96a). Qwen3-30B-A3B logit parity:
14 +\textsc{pending} (test launched on M3U96a; result to be inserted from its log).
15 +
16 +\subsection{All-resident control and demand loading}
17 +\label{sec:results-capacity}
18 +Table~\ref{tab:capacity} summarises the laptop capacity sweep (64-token greedy generations, general and code prompts,
19 +page-cached checkpoint, \code{benchmarks/raw/gateB/capacity\_sweep\_laptop/sweep.jsonl} and
20 +\code{benchmarks/raw/gateC/prefetch\_online\_laptop/smoke\_lazyinstall.jsonl}).
21 +
22 +\begin{table}[t]
23 +\centering\small
24 +\caption{OLMoE-1B-7B 4-bit on the M5 Max laptop: decode throughput vs residency (2 repeats after warm-up; ranges over prompts).}
25 +\label{tab:capacity}
26 +\begin{tabular}{llrrr}
27 +\toprule
28 +Backend & Residency & decode \toks{} & hit rate & stall / 64 tokens \\
29 +\midrule
30 +mlx-lm & all & 491--496 & -- & -- \\
31 +\titan{} & 64 slots, preloaded & 489 & 1.00 & 0 \\
32 +\titan{} & 64 slots, not preloaded (sync path) & 201--203 & 1.00 & 0 \\
33 +\titan{} & 32 slots, LRU, demand (v1 installs) & 56--80 & 0.73--0.81 & 0.7--1.0 s \\
34 +\titan{} & 16 slots, LRU, demand (v1 installs) & 36--45 & 0.48--0.60 & 1.7--2.0 s \\
35 +\titan{} & 16 slots, LRU, demand (lazy installs) & 51--73 & 0.48--0.67 & 0.6--0.9 s \\
36 +\titan{} & 16 slots, LRU + transition prefetch ($d{=}1$, $m{=}8$) & 35--56 & 0.63--0.73 & 0.6--1.1 s \\
37 +\bottomrule
38 +\end{tabular}
39 +\end{table}
40 +
41 +Two observations. The all-resident control matches mlx-lm within one percent only when every layer's table is complete;
42 +otherwise each layer pays a host synchronisation, which alone costs 2.4$\times$ in throughput on this model and is the floor
43 +of any demand-loading design, including colibri's. And the demand-loading rows are the H1 \emph{control}: at 16 slots the
44 +model does 53 misses per token (Prop.~\ref{prop:capacity}), i.e.\ about 11~ms of fetch per 2~ms of compute.
45 +
46 +\subsection{Routing locality}
47 +\label{sec:results-locality}
48 +From the v1 traces (18 prompts, 3\,686 tokens), per-layer LRU hit rates are 0.57 at 16 slots, 0.80 at 32 and 0.93 at 48 (LFU
49 +within 0.01), nearly identical on fit and held-out prompts (\code{benchmarks/processed/gateC/sim\_olmoe\_laptop\_ssd\_v1.json});
50 +a one-layer-ahead transition predictor has recall@8 of 0.50 on held-out prompts; a linear probe on the previous layer's
51 +log-probabilities only 0.16 with these few traces. The v2 traces (65 prompts, 6 families) and the depth-$d$ predictors:
52 +\textsc{pending} (simulation running at the time of writing; Table~\ref{tab:sim} is generated from its JSON).
53 +
54 +\subsection{Prefetching (H1)}
55 +\label{sec:results-prefetch}
56 +With the one-layer horizon the online prefetcher raised the hit rate from 0.48 to 0.63 (general) and from 0.67 to 0.73
57 +(code) but reduced throughput from 51 to 35 and from 73 to 56~\toks{} (Table~\ref{tab:capacity}): the lead time
58 +$d\,\tau_\ell = 0.13$~ms is shorter than the 0.21~ms fetch, so predicted experts arrive late and are waited for, and the
59 +50\,\% of wrong predictions add traffic. This is exactly the regime excluded by condition~\eqref{eq:condition}.
60 +Depth-$d$ and token-horizon prefetching, and the cold-NVMe tier: \textsc{pending} (next experiments of Gate C).
61 +
62 +\subsection{Medium model (Qwen3-30B-A3B)}
63 +mlx-lm and \titan{} all-resident baselines on M3U96a (96~GB): \textsc{pending} (running).
64 +
65 +\subsection{Distributed execution (H2, H3), Metal kernels (H4), energy frontier (H5)}
66 +\textsc{pending}: Gates D--F have protocols (Section~\ref{sec:algorithms}, \code{research/protocol.md}) and no logs in this version.
added paper/sections/robustness.tex +8 −0
@@ -0,0 +1,8 @@
1 +\section{Robustness}
2 +\label{sec:robustness}
3 +Held-out prompt families give hit rates within 0.01 of the fit families at every capacity (v1 traces), and the transition
4 +predictor's recall drops from 0.53 (fit) to 0.50 (held-out), so the locality statistics are not an artefact of the fitting
5 +prompts. Throughput measurements on the laptop vary by less than 1\,\% across repeats for the all-resident configurations and by
6 +up to 40\,\% across prompt families for demand loading (code prompts route more repetitively than general prose, hence higher
7 +hit rates). Thermal state and competing processes are not yet controlled in the smoke tests reported here; the final sweeps
8 +sample \code{powermetrics} thermal pressure every second and discard runs under throttling.
added paper/sections/setup.tex +55 −0
@@ -0,0 +1,55 @@
1 +\section{Experimental Setup}
2 +\label{sec:setup}
3 +
4 +\subsection{Hardware (live inventory, 2026-11-01)}
5 +Table~\ref{tab:nodes} lists the machines whose specifications were read live (\code{sysctl}, \code{system\_profiler}) on
6 +the day of the experiments; the sanitized inventory is \code{artifacts/environment.json}. Eight aliases of the cluster's
7 +nominal 29 did not answer SSH within 15~s and are excluded rather than assumed. All nodes negotiate 1000BASE-T; iperf3 measures
8 +0.945--0.947~Gbit/s and ping 0.5--1.0~ms between the Studios; no Thunderbolt link is active between any two nodes.
9 +
10 +\begin{table}[t]
11 +\centering\small
12 +\caption{Machines used or available. GPU = GPU cores. Dev.\ laptop is not on the cluster LAN.}
13 +\label{tab:nodes}
14 +\begin{tabular}{llrrll}
15 +\toprule
16 +Alias & Chip & RAM (GB) & GPU & macOS & Role in this version \\
17 +\midrule
18 +laptop & Apple M5 Max & 48 & 40 & 27.0 & single-node experiments (Gates B--C); NVMe 8.5 GB/s random \\
19 +M3U96a & Apple M3 Ultra & 96 & 60 & 26.6 & coordinator; Qwen3-30B-A3B baselines \\
20 +M3U96b & Apple M3 Ultra & 96 & 60 & 26.4 & worker (Gate E) \\
21 +M2U64 & Apple M2 Ultra & 64 & 60 & 26.5 & worker (Gate E) \\
22 +M4M64a/b/c & Apple M4 Max & 64 & 40 & 26.4--26.5 & workers (Gate F) \\
23 +M5M36 & Apple M5 Max & 36 & 32 & 27.0 & heterogeneous worker (Gate F) \\
24 +m4mm, m4mn & Apple M4 Pro & 48 & 16 & 26.5 & heterogeneous workers (Gate F) \\
25 +others (12) & M1--M6, 8--32 GB & & & & inventoried, unused \\
26 +\bottomrule
27 +\end{tabular}
28 +\end{table}
29 +
30 +\subsection{Models}
31 +\begin{itemize}
32 + \item \textbf{Small}: OLMoE-1B-7B-0125-Instruct \citep{muennighoff2024olmoe}, mlx-community 4-bit conversion (affine, group 64),
33 + revision \code{e2384419}, Apache-2.0: 16 layers, 64 experts, top-8, no shared expert; 3.375~MiB per expert, 3.62~GB of experts,
34 + 0.27~GB dense; 128 experts (432~MiB) touched per decode token.
35 + \item \textbf{Medium}: Qwen3-30B-A3B-Instruct-2507 \citep{yang2025qwen3}, mlx-community 4-bit (routers 8-bit), revision \code{e9675aa3},
36 + Apache-2.0: 48 layers, 128 experts, top-8, renormalised gates; 2.53~MiB per expert, 16.3~GB of experts, 0.87~GB dense;
37 + 384 experts (0.97~GiB) per decode token. Stacked \code{switch\_mlp} layout (per-expert byte ranges are slices of the first axis).
38 + \item \textbf{Large} (gpt-oss-120b, 62~GB): not downloaded in this version (authorization pending).
39 +\end{itemize}
40 +
41 +\subsection{Workloads and metrics}
42 +Prompts are synthetic instructions written for this study in six families (general, code, reasoning, multilingual,
43 +long-context, domain drift), split into \emph{fit} prompts (used to fit predictors) and \emph{held-out} prompts (never used to
44 +fit anything); v1 has 18 prompts (Gate B), v2 has 65 (Gate C). Decoding is greedy. Metrics follow \code{research/protocol.md}:
45 +prefill and decode tok/s as reported by mlx-lm's generator plus wall-clock inter-token latencies (p50/p95/p99) measured in the
46 +driver, time to first token, expert hit rate, misses, stall time, bytes read, peak memory; energy (\code{powermetrics}) in
47 +the distributed and Pareto experiments. Every run records its command line, git commit and environment; order is shuffled with a
48 +fixed seed; the first run of each configuration is a discarded warm-up; repeats are $\ge 2$ in the smoke tests reported here
49 +and $\ge 5$ in the final sweeps.
50 +
51 +\subsection{Tiers and constants used by the simulator}
52 +Measured on the laptop: page-cache fetch of one OLMoE expert 0.21~ms (batched demand path), cold NVMe 0.43~ms (F\_NOCACHE after
53 +\code{purge}, single reader), 1~GbE transfer $\approx 28$~ms; per-layer decode compute $\tau_\ell = 0.128$~ms (489~\toks{}).
54 +The trace-driven simulator (\code{benchmarks/scripts/simulate\_cache.py}) is exact for hits and misses and uses these constants
55 +for stall estimates.
added requirements-lock.txt +50 −0
@@ -0,0 +1,50 @@
1 +annotated-doc==0.0.5
2 +anyio==4.15.1
3 +click==8.5.0
4 +contourpy==1.4.0
5 +cycler==0.12.1
6 +filelock==4.2.6
7 +fonttools==4.68.0
8 +fsspec==2026.9.0
9 +h11==0.16.0
10 +hf-xet==1.8.0
11 +httpcore2==2.16.0
12 +httpx2==2.16.0
13 +huggingface-hub==2.4.0
14 +idna==3.20
15 +iniconfig==2.3.1
16 +jinja2==3.1.6
17 +kiwisolver==1.5.1
18 +markdown-it-py==4.2.0
19 +markupsafe==3.0.4
20 +matplotlib==3.11.2
21 +mdurl==0.1.2
22 +mlx==0.32.4
23 +mlx-lm==0.32.2
24 +mlx-metal==0.32.4
25 +numpy==2.5.4
26 +packaging==26.3
27 +pandas==3.1.0
28 +pillow==13.0.0
29 +pluggy==1.7.0
30 +protobuf==7.36.2
31 +psutil==7.2.2
32 +pygments==2.21.0
33 +pyparsing==3.3.3
34 +pytest==9.2.0
35 +python-dateutil==3.0.0
36 +pyyaml==6.0.3
37 +regex==2026.9.29
38 +rich==15.0.0
39 +safetensors==0.8.0
40 +scipy==1.18.1
41 +sentencepiece==0.2.2
42 +shellingham==1.5.4
43 +six==1.17.0
44 +tokenizers==0.23.3
45 +tqdm==4.70.1
46 +transformers==5.22.1
47 +truststore==0.10.5
48 +typer==0.27.3
49 +typing-extensions==4.16.0
50 +tzdata==2026.6
added scripts/cluster_setup.sh +17 −0
@@ -0,0 +1,17 @@
1 +#!/bin/bash
2 +# Prepare a MacLustr node for WP33 experiments: rsync the repo (code only), create a Python 3.12 venv with pinned MLX, run the
3 +# unit tests that do not need a model. Usage: scripts/cluster_setup.sh <alias> [<alias>...]
4 +set -u
5 +SRC=~/Desktop/UQO/Recherche/Recherche-UQO/ia/titan-moe
6 +for n in "$@"; do
7 + (
8 + rsync -az --delete --exclude '.venv' --exclude 'third_party' --exclude 'benchmarks/raw' --exclude 'artifacts/private' --exclude '.git' \
9 + --exclude '__pycache__' --exclude 'paper' "$SRC/" "$n:~/wp33/titan-moe/" || { echo "$n: rsync FAILED"; exit 1; }
10 + ssh -o ConnectTimeout=15 "$n" 'export PATH=$HOME/.local/bin:/opt/homebrew/bin:$PATH; cd ~/wp33/titan-moe && \
11 + ( [ -x .venv/bin/python ] || uv venv --python 3.12 .venv -q ) && \
12 + uv pip install -q --python .venv/bin/python "mlx>=0.32,<0.33" "mlx-lm>=0.31" numpy pytest matplotlib pandas scipy huggingface_hub psutil pyyaml && \
13 + .venv/bin/python -c "import mlx.core as mx, mlx_lm, platform; print(\"'"$n"'\", platform.mac_ver()[0], \"mlx\", mx.__version__, \"mlx_lm\", mlx_lm.__version__, mx.default_device())"' \
14 + || echo "$n: venv FAILED"
15 + ) &
16 +done
17 +wait
added scripts/make_environment.py +55 −0
@@ -0,0 +1,55 @@
1 +#!/usr/bin/env python3
2 +"""Build the sanitized artifacts/environment.json from the raw probes kept in artifacts/private/ (not committed).
3 +Only hardware facts that were read live are included; no IP addresses, serial numbers or credentials."""
4 +import glob, json, os, re, subprocess, sys, time, platform
5 +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
6 +PRIV = os.path.join(ROOT, "artifacts", "private")
7 +
8 +def parse_kv(line):
9 + return dict(kv.split("=", 1) for kv in line.strip().split("|") if "=" in kv)
10 +
11 +nodes = {}
12 +for f in sorted(glob.glob(os.path.join(PRIV, "*.txt"))):
13 + name = os.path.basename(f).replace(".retry", "").replace(".txt", "")
14 + if name.startswith(("inventory", "linkprobe", "netprobe")):
15 + continue
16 + line = open(f).read().strip().split("\n")[-1]
17 + if "chip=" not in line:
18 + nodes.setdefault(name, {"status": "unreachable (SSH timeout 2026-11-01)"}); continue
19 + kv = parse_kv(line)
20 + nodes[name] = {"status": "reachable", "chip": kv.get("chip"), "ram_gb": int(kv["ram"]), "macos": kv.get("os"),
21 + "gpu_cores": int(kv["gpu"]) if kv.get("gpu", "").isdigit() else None, "disk_free_gb": int(kv["disk_free"].rstrip("G")) if kv.get("disk_free", "").rstrip("G").isdigit() else None,
22 + "system_python": kv.get("py"), "mlx_preinstalled": kv.get("mlx") not in (None, "none"), "load_avg_at_probe": kv.get("load")}
23 +lp = os.path.join(PRIV, "linkprobe-2026-11-01.txt")
24 +if os.path.exists(lp):
25 + for line in open(lp):
26 + kv = parse_kv(line); n = line.split("|")[0]
27 + if n in nodes and nodes[n].get("status") == "reachable":
28 + nodes[n].update({"en0_media": kv.get("media"), "homebrew_pythons": [p for p in kv.get("brewpy", "").split(",") if p], "uv": bool(kv.get("uv")),
29 + "xcode": kv.get("xcode"), "metal_compiler": kv.get("metal") or None, "rdma_ctl_present": bool(kv.get("rdma")), "iperf3": kv.get("iperf3") not in (None, "no", "")})
30 +net = {}
31 +for fn in ("netprobe-2026-11-01.txt", "netprobe2-2026-11-01.txt"):
32 + p = os.path.join(PRIV, fn)
33 + if not os.path.exists(p): continue
34 + for line in open(p):
35 + m = re.match(r"rtt (\S+)->(\S+) ([\d.]+)/([\d.]+)/([\d.]+)/([\d.]+) ms", line.strip())
36 + if m:
37 + net.setdefault("rtt_ms", []).append({"from": m.group(1), "to": "lan-host-" + m.group(2).split(".")[-1] if m.group(2)[0].isdigit() else m.group(2), "min": float(m.group(3)), "avg": float(m.group(4)), "max": float(m.group(5))})
38 + m = re.match(r"iperf3 (\S+)->(\S+) \(.*\): ([\d.]+) Gbit/s sent ([\d.]+) Gbit/s recv", line.strip())
39 + if m:
40 + net.setdefault("iperf3_gbits", []).append({"from": m.group(1), "to": m.group(2), "sent": float(m.group(3)), "recv": float(m.group(4))})
41 + if line.startswith("tb-ifaces"):
42 + n = line.split(":")[0].split()[1]
43 + net.setdefault("thunderbolt_active_links", {})[n] = ("status: active" in line and "en0" not in line.split("status: active")[0][-60:]) and False
44 +laptop = {"role": "development laptop (not on the cluster LAN)", "chip": subprocess.run(["sysctl", "-n", "machdep.cpu.brand_string"], capture_output=True, text=True).stdout.strip(),
45 + "ram_gb": int(subprocess.run(["sysctl", "-n", "hw.memsize"], capture_output=True, text=True).stdout) // 2**30, "macos": platform.mac_ver()[0],
46 + "gpu_cores": 40, "cpu_cores": 18, "ssd_random_3p4MB_GBps_cold": 8.50, "ssd_seq_GBps_cold": 11.5, "ssd_source": "benchmarks/raw/gateA/ssd/laptop-M5Max.json"}
47 +env = {"generated": time.strftime("%Y-%m-%dT%H:%M:%S"), "note": "Live inventory of 2026-11-01; aliases are cluster names, no addresses or serials. Unreachable aliases are listed but unused.",
48 + "laptop": laptop, "cluster_nodes": nodes, "network": {"lan": "1 GbE switch (all nodes negotiate 1000baseT on en0)", "thunderbolt": "no active Thunderbolt link between any two nodes at probe time (RDMA/JACCL blocked pending cabling)", **net},
49 + "software": {"mlx_venv": "0.32.4", "mlx_lm_venv": "0.32.2", "python_venv": "3.12", "third_party_commits": {"colibri": "0a5504a1dc64f86fcd0fe7db298b0448c730e7e6", "mlx": "bd63d3803008edda03ce74080955d9879e2ef1eb", "mlx-lm": "736ab0d8c1d0edc79c3d798044fe6904e2fa23be", "llama.cpp": "a91eab67d8389d979d29fdf6633b2edd0718eb4b"}}}
50 +out = os.path.join(ROOT, "artifacts", "environment.json")
51 +json.dump(env, open(out, "w"), indent=2)
52 +reach = [n for n, v in nodes.items() if v.get("status") == "reachable"]
53 +print(f"wrote {out}: {len(reach)} reachable nodes, {len(nodes) - len(reach)} unreachable")
54 +for n in sorted(reach, key=lambda x: -nodes[x]["ram_gb"]):
55 + v = nodes[n]; print(f" {n:8s} {v['chip']:16s} {v['ram_gb']:3d} GB GPU {v['gpu_cores']} macOS {v['macos']} free {v['disk_free_gb']} GB {v.get('en0_media','')}")
modified tests/conftest.py +15 −5
@@ -7,9 +7,19 @@ def _model_dir():
7 7 hits = glob.glob(os.path.expanduser("~/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/*"))
8 8 return hits[0] if hits else None
9 9
10 −@pytest.fixture(scope="session")
11 −def model_dir():
10 +def _model_dirs():
11 + env = os.environ.get("TITAN_MODEL_DIRS")
12 + if env:
13 + return [d for d in env.split(":") if d]
12 14 d = _model_dir()
13 − if d is None:
14 − pytest.skip("OLMoE-1B-7B-0125-Instruct-4bit not in the HF cache (set TITAN_MODEL_DIR)")
15 − return d
15 + return [d] if d else []
16 +
17 +def pytest_generate_tests(metafunc):
18 + if "model_dir" in metafunc.fixturenames:
19 + dirs = _model_dirs()
20 + metafunc.parametrize("model_dir", dirs or [None], ids=[os.path.basename(os.path.dirname(os.path.dirname(d))) if d else "none" for d in dirs] or ["none"], scope="module")
21 +
22 +@pytest.fixture(autouse=True)
23 +def _skip_if_no_model(request):
24 + if "model_dir" in request.fixturenames and request.getfixturevalue("model_dir") is None:
25 + pytest.skip("no MoE checkpoint in the HF cache (set TITAN_MODEL_DIRS)")
modified tests/test_numerical.py +9 −5
@@ -50,9 +50,11 @@ def ref(model_dir):
50 50 tok = load_tokenizer(Path(model_dir))
51 51 return m, tok
52 52
53 −@pytest.mark.parametrize("capacity", [64, 32, 16, 8])
54 −def test_logits_match_mlx_lm(model_dir, ref, capacity):
53 +@pytest.mark.parametrize("frac", [1.0, 0.5, 0.25, 0.125])
54 +def test_logits_match_mlx_lm(model_dir, ref, frac):
55 55 refm, tok = ref
56 + from titan.registry import load_spec
57 + capacity = max(8, int(load_spec(model_dir).num_experts * frac))
56 58 tm = TitanModel(model_dir, capacity=capacity)
57 59 trace = _route_trace(tm)
58 60 rec = {"capacity": capacity, "prompts": [], "tol": TOL, "ts": time.strftime("%Y-%m-%dT%H:%M:%S")}
@@ -75,19 +77,21 @@ def test_logits_match_mlx_lm(model_dir, ref, capacity):
75 77 assert len({l for l, _ in trace}) == tm.spec.num_layers
76 78 rec["max_abs_dlogit_worst"] = worst
77 79 os.makedirs("benchmarks/raw/tests", exist_ok=True)
78 − with open(f"benchmarks/raw/tests/numerical_cap{capacity}_{rec['ts'].replace(':','')}.json", "w") as f:
80 + rec["model"] = os.path.realpath(model_dir); rec["model_type"] = tm.spec.model_type
81 + with open(f"benchmarks/raw/tests/numerical_{tm.spec.model_type}_cap{capacity}_{rec['ts'].replace(':','')}.json", "w") as f:
79 82 json.dump(rec, f, indent=2)
80 83
81 84 def test_greedy_decode_identical(model_dir, ref):
82 85 """Greedy continuation (32 tokens) must be token-identical to mlx-lm with a bounded (16-slot) table."""
83 86 from mlx_lm import generate
84 87 refm, tok = ref
85 − tm = TitanModel(model_dir, capacity=16)
88 + from titan.registry import load_spec
89 + tm = TitanModel(model_dir, capacity=max(8, load_spec(model_dir).num_experts // 4))
86 90 outs = []
87 91 for p in PROMPTS[:2]:
88 92 a = generate(refm, tok, prompt=p, max_tokens=32, verbose=False)
89 93 b = generate(tm.model, tok, prompt=p, max_tokens=32, verbose=False)
90 94 outs.append({"prompt": p[:40], "ref": a, "titan": b, "equal": a == b})
91 95 assert a == b, (a, b)
92 − with open(f"benchmarks/raw/tests/greedy_cap16_{time.strftime('%Y%m%dT%H%M%S')}.json", "w") as f:
96 + with open(f"benchmarks/raw/tests/greedy_{tm.spec.model_type}_cap{tm.capacity}_{time.strftime('%Y%m%dT%H%M%S')}.json", "w") as f:
93 97 json.dump(outs, f, indent=2)
added tests/test_registry_stacked.py +24 −0
@@ -0,0 +1,24 @@
1 +"""Stacked-layout registry/store check (runs only where a stacked checkpoint such as Qwen3-30B-A3B-4bit is cached)."""
2 +import glob, os, pytest, mlx.core as mx
3 +from titan.registry import load_spec
4 +from titan.store import ExpertStore
5 +
6 +def _dir():
7 + hits = glob.glob(os.path.expanduser("~/.cache/huggingface/hub/models--mlx-community--Qwen3-30B-A3B-Instruct-2507-4bit/snapshots/*"))
8 + return hits[0] if hits else None
9 +
10 +@pytest.mark.skipif(_dir() is None, reason="Qwen3-30B-A3B-Instruct-2507-4bit not cached")
11 +def test_stacked_slices_match_mx_load():
12 + d = _dir(); s = load_spec(d)
13 + assert s.layout == "stacked" and s.num_layers == 48 and s.num_experts == 128 and s.top_k == 8
14 + st = ExpertStore(s)
15 + for (l, e) in [(0, 0), (23, 77), (47, 127)]:
16 + t = st.read_expert(l, e)
17 + f = s.experts[(l, e)].tensors["gate.weight"].file
18 + ref = mx.load(f)
19 + for name, arr in t.items():
20 + m, part = name.split(".")
21 + r = ref[f"model.layers.{l}.mlp.switch_mlp.{m}_proj.{part}"][e]
22 + assert arr.shape == r.shape and arr.dtype == r.dtype, (name, arr.shape, r.shape)
23 + assert bool(mx.array_equal(arr, r)), (l, e, name)
24 + print("bytes_per_expert", s.bytes_per_expert, "expert_bytes_total_GB", s.expert_bytes_total / 1e9)
modified tests/test_registry_store.py +6 −2
@@ -1,10 +1,12 @@
1 −import numpy as np, mlx.core as mx
1 +import numpy as np, mlx.core as mx, pytest
2 2 from titan.registry import load_spec
3 3 from titan.store import ExpertStore
4 4
5 5 def test_registry_olmoe(model_dir):
6 6 s = load_spec(model_dir)
7 − assert s.model_type == "olmoe" and s.num_layers == 16 and s.num_experts == 64 and s.top_k == 8
7 + if s.model_type != "olmoe":
8 + pytest.skip("OLMoE-specific constants")
9 + assert s.num_layers == 16 and s.num_experts == 64 and s.top_k == 8
8 10 assert s.quantization == {"group_size": 64, "bits": 4}
9 11 assert s.bytes_per_expert == 3538944
10 12 assert len(s.experts) == 1024
@@ -13,6 +15,8 @@ def test_registry_olmoe(model_dir):
13 15 def test_store_reads_match_mx_load(model_dir):
14 16 """Bytes read with pread at registry offsets must equal what mx.load returns for the same tensor."""
15 17 s = load_spec(model_dir)
18 + if s.layout != "per_expert":
19 + pytest.skip("stacked layout covered by test_registry_stacked.py")
16 20 st = ExpertStore(s)
17 21 ref = mx.load(f"{model_dir}/model.safetensors")
18 22 for (l, e) in [(0, 0), (7, 33), (15, 63)]:
modified titan/registry.py +14 −1
@@ -154,7 +154,20 @@ def load_spec(model_dir: str) -> ModelSpec:
154 154 tensors[f"{m}.{s}"] = locs[k]
155 155 per_expert_keys.add(k)
156 156 if not tensors:
157 − raise ValueError(f"no per-expert tensors found for layer {l} expert {e}; stacked layouts are not yet supported by the store")
157 + # stacked layout: switch_mlp.<m>_proj.<s> has shape [E, ...]; expert e is the contiguous slice e of the first axis
158 + for m in fmt["matrices"]:
159 + for s in ("weight", "scales", "biases"):
160 + k = fmt["stacked_key"].format(l=l, m=m, s=s)
161 + if k in locs:
162 + loc = locs[k]
163 + if loc.shape[0] != spec.num_experts:
164 + raise ValueError(f"{k}: first axis {loc.shape[0]} != num_experts {spec.num_experts}")
165 + per = loc.nbytes // spec.num_experts
166 + tensors[f"{m}.{s}"] = TensorLoc(f"{k}[{e}]", loc.dtype, tuple(loc.shape[1:]), loc.file, loc.offset + e * per, per)
167 + per_expert_keys.add(k)
168 + spec.layout = "stacked"
169 + if not tensors:
170 + raise ValueError(f"no expert tensors found for layer {l} expert {e} (neither per-expert nor stacked keys)")
158 171 spec.experts[(l, e)] = ExpertSpec(l, e, tensors)
159 172 spec.dense = {k: v for k, v in locs.items() if k not in per_expert_keys}
160 173 return spec
161 174
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere