1# TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon23**UQO Working Paper No. 33** — *A Measured Systems Study on a Heterogeneous Mac Cluster*45     67## TL;DR8- An MLX-native MoE inference runtime with bounded per-layer expert residency (slot tables), a `pread` NVMe tier, LRU/LFU eviction,9 causal prefetch predictors and an asynchronous prefetcher; the router's choices and the logits are those of mlx-lm by construction.10- Correctness: **bit-identical logits** (max |Δ| = 0.0) and token-identical greedy decoding vs mlx-lm on OLMoE-1B-7B-0125 4-bit, at 64/32/16/8 slots.11- All-resident control: **489 tok/s** vs mlx-lm 494 tok/s on an M5 Max laptop (−1 %). Bounded residency with demand loading from the page cache: 51–73 tok/s at 16 slots.12- Honest first result on H1: one-layer-ahead prefetch raises the hit rate (0.48→0.63) but **lowers** throughput; the paper derives the lead-time condition d·τ_layer ≥ λ_tier and tests deeper horizons.13- Live cluster inventory: 21 reachable Macs (M1–M6, 8–96 GB), **1 GbE everywhere**, no active Thunderbolt link → distributed gates use TCP; RDMA/JACCL blocked pending cabling.1415## Status16Version 0.1 (2026-11-01), **protocol + preliminary** working paper; not submitted. Gates A–B complete, Gate C in progress, D–G pending17(see `research/LOG.md`). Nothing in the paper is reported without a raw log under `benchmarks/raw/`.1819## What this paper does20Designs, implements and measures TITAN-MoE against five pre-stated hypotheses (causal prefetch, topology-aware placement,21hierarchical caching, Metal kernels, adaptive orchestration on the latency–throughput–energy frontier), with an exact-semantics22constraint: placement decides speed, never which experts run.2324## Headline results (version 0.1, single node, OLMoE-1B-7B-0125-Instruct-4bit, M5 Max 48 GB)25| Configuration | decode tok/s | hit rate | source |26|---|---|---|---|27| mlx-lm (reference) | 491–496 | — | `benchmarks/raw/gateB/capacity_sweep_laptop/sweep.jsonl` |28| TITAN, all experts resident (64 slots) | 489 | 1.00 | same |29| TITAN, 32 slots, LRU, demand loading (page cache) | 56–80 | 0.73–0.81 | same (before lazy-install fix) |30| TITAN, 16 slots, LRU, demand loading (page cache) | 51–73 | 0.48–0.67 | `benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl` |31| TITAN, 16 slots + transition prefetch (d = 1, m = 8) | 35–56 | 0.63–0.73 | same |3233## Repository layout34```35CLAUDE.md README.md LICENSE docs/MANDATE.md36titan/ registry.py store.py slots.py moe.py prefetch.py predictors.py runtime.py37metal/ (Gate D) transport/ (Gate E) adapters/ (formats beyond OLMoE/Qwen3-MoE)38tests/ test_registry_store.py test_numerical.py39benchmarks/ configs/ (workloads, models) scripts/ (bench_decode, collect_traces, simulate_cache, ssd_bandwidth) raw/ (immutable) processed/40research/ LOG.md DECISIONS.md RELATED_WORK.md prior_art.csv protocol.md41paper/ main.tex sections/ references.bib (UQO template; `cd paper && make`)42artifacts/ environment.json (sanitized live inventory) manifest.json43scripts/ cluster_setup.sh make_environment.py44third_party/ shallow clones of colibri / mlx / mlx-lm / llama.cpp (git-ignored; hashes in research/prior_art.csv)45```4647## Quick start48```bash49uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements-lock.txt50# model (3.9 GB, Apache-2.0, pinned revision) goes to the Hugging Face cache51.venv/bin/python -c "from huggingface_hub import snapshot_download as s; s('mlx-community/OLMoE-1B-7B-0125-Instruct-4bit', revision='e2384419')"52.venv/bin/python -m pytest tests -q # correctness vs mlx-lm (writes evidence to benchmarks/raw/tests/)53M=$(ls -d ~/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/*)54.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend mlx_lm --out /tmp/b.jsonl55.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend titan --capacity 16 --predictor transition2 --traces benchmarks/raw/gateC/traces_olmoe_v2_laptop --out /tmp/b.jsonl56cd paper && make # PDF57```58Local-only mode: everything above runs on one Mac; the distributed components are only enabled with an explicit host list.5960## Data61No external datasets. Workloads are synthetic prompts written for this study (`benchmarks/configs/workloads*.yaml`); routing62traces derived from them are in `benchmarks/raw/gateC/traces_*`. Models: OLMoE-1B-7B-0125-Instruct (Apache-2.0),63Qwen3-30B-A3B-Instruct-2507 (Apache-2.0), both as mlx-community 4-bit conversions with pinned revisions.6465## Methodology66See `research/protocol.md` (frozen at Gate C) and `paper/sections/formulation.tex`.6768## Figures & tables inventory69Generated by `scripts/make_figures.py` / `scripts/make_tables.py` from `benchmarks/raw/` (Gate G).7071## Limitations72Small model only in v0.1; distributed, Metal and energy experiments pending; 1 GbE topology; see paper Section "Limitations".7374## Citation75```bibtex76@techreport{boucher2026titanmoe,77 title = {TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon},78 author = {Boucher, Simon-Pierre},79 institution = {Université du Québec en Outaouais, Département des sciences administratives},80 type = {UQO Working Paper},81 number = {33},82 year = {2026}83}84```8586## Author & contact87Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca8889## Notes90- The mandate's repository name `wp33_titan_moe` is kept as the spbgit repository name (`wp33_uqo.git`); the folder follows the91 Recherche-UQO series layout (`ia/titan-moe`). Internal structure follows the mandate (D-001).92- Written with Claude Code (Claude Fable 5.1) acting as autonomous research engineer under `docs/MANDATE.md`; AI-assistance93 disclosure is included in the paper's reproducibility appendix.94