1# TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon23**UQO Working Paper No. 33** — *A Measured Systems Study on a Heterogeneous Mac Cluster*45     67## TL;DR8- An MLX-native MoE inference runtime with bounded per-layer expert residency (slot tables), a `pread` NVMe tier, LRU/LFU eviction,9 causal prefetch predictors and an asynchronous prefetcher; the router's choices and the logits are those of mlx-lm by construction.10- Correctness: **bit-identical logits** (max |Δ| = 0.0) and token-identical greedy decoding vs mlx-lm on OLMoE-1B-7B-0125 4-bit, at 64/32/16/8 slots.11- All-resident control: **489 tok/s** vs mlx-lm 494 tok/s on an M5 Max laptop (−1 %). Bounded residency with demand loading from the page cache: 51–73 tok/s at 16 slots.12- Honest first result on H1: one-layer-ahead prefetch raises the hit rate (0.48→0.63) but **lowers** throughput; the paper derives the lead-time condition d·τ_layer ≥ λ_tier and tests deeper horizons.13- Live cluster inventory: 21 reachable Macs (M1–M6, 8–96 GB), **1 GbE everywhere**, no active Thunderbolt link → distributed gates use TCP; RDMA/JACCL blocked pending cabling.1415## Status16Version 0.1 (2026-11-01), **preliminary** working paper; not submitted. Gates A–E complete, Gate F (medium model, two nodes) and the17energy study running at the time of writing, Gate G audit done for the existing logs (see `research/LOG.md`). Nothing in the paper18is reported without a raw log under `benchmarks/raw/`; `artifacts/manifest.json` lists every artifact with its SHA-256.1920**Verdicts so far (all with bootstrap CIs, `benchmarks/processed/audit.json`):** H1 length-dependent on OLMoE/laptop NVMe (64-token runs: best predictor ties,21the install path is the bottleneck; 256-token runs: probe +32 % tok/s); H4 not supported (decode at batch 1 is launch-bound; fusion ±2 %); the remote22tier over 1 GbE is a capacity mechanism, not a latency one (−66 tok/s for odd-expert placement); the expert-contiguous storage layout23is the one clear win (cold NVMe 8 → 36 tok/s).2425## What this paper does26Designs, implements and measures TITAN-MoE against five pre-stated hypotheses (causal prefetch, topology-aware placement,27hierarchical caching, Metal kernels, adaptive orchestration on the latency–throughput–energy frontier), with an exact-semantics28constraint: placement decides speed, never which experts run.2930## Headline results (version 0.1, single node, OLMoE-1B-7B-0125-Instruct-4bit, M5 Max 48 GB)31| Configuration | decode tok/s | hit rate | source |32|---|---|---|---|33| mlx-lm (reference) | 491–496 | — | `benchmarks/raw/gateB/capacity_sweep_laptop/sweep.jsonl` |34| TITAN, all experts resident (64 slots) | 489 | 1.00 | same |35| TITAN, 32 slots, LRU, demand loading (page cache) | 56–80 | 0.73–0.81 | same (before lazy-install fix) |36| TITAN, 16 slots, LRU, demand loading (page cache) | 51–73 | 0.48–0.67 | `benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl` |37| TITAN, 16 slots + transition prefetch (d = 1, m = 8) | 35–56 | 0.63–0.73 | same |38| TITAN, 16 slots, cold NVMe, contiguous layout, no prefetch | 36.4 | 0.53 | `benchmarks/raw/gateC/cold_nvme_laptop/sweep.jsonl` |39| TITAN, 16 slots, cold NVMe + linear probe (recall@8 0.76) | 31.8 | 0.78 | `.../attribution.jsonl` |40| Qwen3-30B-A3B on M3U96a: mlx-lm / TITAN all-resident | 121.7 / 121.6 | — | `benchmarks/raw/gateC/qwen3_m3u96a/baseline.jsonl` |41| OLMoE two nodes (M3U96a + M3U96b, 1 GbE), 32 slots: local / odd experts remote | 81 / 15 | — | `benchmarks/raw/gateE/two_node_m3u96a_m3u96b.jsonl` |42| Energy, 256-token runs: all-resident 0.101 J/token (47 W); 16 slots cold 0.498 J/token (31.9 tok/s); + probe 0.441 J/token (**42.0 tok/s**, +10.1 [+5.2, +16.9]) | — | 0.63 / 0.84 | `benchmarks/raw/gateG/energy_laptop/` |4344## Repository layout45```46CLAUDE.md README.md LICENSE docs/MANDATE.md47titan/ registry.py store.py slots.py moe.py prefetch.py predictors.py runtime.py48metal/ (Gate D) transport/ (Gate E) adapters/ (formats beyond OLMoE/Qwen3-MoE)49tests/ test_registry_store.py test_numerical.py50benchmarks/ configs/ (workloads, models) scripts/ (bench_decode, collect_traces, simulate_cache, ssd_bandwidth) raw/ (immutable) processed/51research/ LOG.md DECISIONS.md RELATED_WORK.md prior_art.csv protocol.md52paper/ main.tex sections/ references.bib (UQO template; `cd paper && make`)53artifacts/ environment.json (sanitized live inventory) manifest.json54scripts/ cluster_setup.sh make_environment.py55third_party/ shallow clones of colibri / mlx / mlx-lm / llama.cpp (git-ignored; hashes in research/prior_art.csv)56```5758## Quick start59```bash60uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements-lock.txt61# model (3.9 GB, Apache-2.0, pinned revision) goes to the Hugging Face cache62.venv/bin/python -c "from huggingface_hub import snapshot_download as s; s('mlx-community/OLMoE-1B-7B-0125-Instruct-4bit', revision='e2384419')"63.venv/bin/python -m pytest tests -q # correctness vs mlx-lm (writes evidence to benchmarks/raw/tests/)64M=$(ls -d ~/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/*)65.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend mlx_lm --out /tmp/b.jsonl66.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend titan --capacity 16 --predictor transition2 --traces benchmarks/raw/gateC/traces_olmoe_v2_laptop --out /tmp/b.jsonl67cd paper && make # PDF68```69Local-only mode: everything above runs on one Mac; the distributed components are only enabled with an explicit host list.7071## Data72No external datasets. Workloads are synthetic prompts written for this study (`benchmarks/configs/workloads*.yaml`); routing73traces derived from them are in `benchmarks/raw/gateC/traces_*`. Models: OLMoE-1B-7B-0125-Instruct (Apache-2.0),74Qwen3-30B-A3B-Instruct-2507 (Apache-2.0), both as mlx-community 4-bit conversions with pinned revisions.7576## Methodology77See `research/protocol.md` (frozen at Gate C) and `paper/sections/formulation.tex`.7879## Figures & tables inventory80Generated by `scripts/make_figures.py` / `scripts/make_tables.py` from `benchmarks/raw/` (Gate G).8182## Limitations83Small model only in v0.1; distributed, Metal and energy experiments pending; 1 GbE topology; see paper Section "Limitations".8485## Citation86```bibtex87@techreport{boucher2026titanmoe,88 title = {TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon},89 author = {Boucher, Simon-Pierre},90 institution = {Université du Québec en Outaouais, Département des sciences administratives},91 type = {UQO Working Paper},92 number = {33},93 year = {2026}94}95```9697## Author & contact98Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca99100## Notes101- The mandate's repository name `wp33_titan_moe` is kept as the spbgit repository name (`wp33_uqo.git`); the folder follows the102 Recherche-UQO series layout (`ia/titan-moe`). Internal structure follows the mandate (D-001).103- Written with Claude Code (Claude Fable 5.1) acting as autonomous research engineer under `docs/MANDATE.md`; AI-assistance104 disclosure is included in the paper's reproducibility appendix.105