# TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon

UQO Working Paper No. 33 — A Measured Systems Study on a Heterogeneous Mac Cluster

# TL;DR

• An MLX-native MoE inference runtime with bounded per-layer expert residency (slot tables), a pread NVMe tier, LRU/LFU eviction, causal prefetch predictors and an asynchronous prefetcher; the router's choices and the logits are those of mlx-lm by construction.
• Correctness: bit-identical logits (max |Δ| = 0.0) and token-identical greedy decoding vs mlx-lm on OLMoE-1B-7B-0125 4-bit, at 64/32/16/8 slots.
• All-resident control: 489 tok/s vs mlx-lm 494 tok/s on an M5 Max laptop (−1 %). Bounded residency with demand loading from the page cache: 51–73 tok/s at 16 slots.
• Honest first result on H1: one-layer-ahead prefetch raises the hit rate (0.48→0.63) but lowers throughput; the paper derives the lead-time condition d·τ_layer ≥ λ_tier and tests deeper horizons.
• Live cluster inventory: 21 reachable Macs (M1–M6, 8–96 GB), 1 GbE everywhere, no active Thunderbolt link → distributed gates use TCP; RDMA/JACCL blocked pending cabling.

# Status

Version 0.1 (2026-11-01), preliminary working paper; not submitted. Gates A–E complete, Gate F (medium model, two nodes) and the energy study running at the time of writing, Gate G audit done for the existing logs (see research/LOG.md). Nothing in the paper is reported without a raw log under benchmarks/raw/; artifacts/manifest.json lists every artifact with its SHA-256.

Verdicts so far (all with bootstrap CIs, benchmarks/processed/audit.json): H1 not supported on OLMoE/laptop NVMe (best predictor ties: the install path, not prediction, is the bottleneck); H4 not supported (decode at batch 1 is launch-bound; fusion ±2 %); the remote tier over 1 GbE is a capacity mechanism, not a latency one (−66 tok/s for odd-expert placement); the expert-contiguous storage layout is the one clear win (cold NVMe 8 → 36 tok/s).

# What this paper does

Designs, implements and measures TITAN-MoE against five pre-stated hypotheses (causal prefetch, topology-aware placement, hierarchical caching, Metal kernels, adaptive orchestration on the latency–throughput–energy frontier), with an exact-semantics constraint: placement decides speed, never which experts run.

# Headline results (version 0.1, single node, OLMoE-1B-7B-0125-Instruct-4bit, M5 Max 48 GB)

# Repository layout

CLAUDE.md  README.md  LICENSE  docs/MANDATE.md
titan/        registry.py store.py slots.py moe.py prefetch.py predictors.py runtime.py
metal/        (Gate D)         transport/  (Gate E)        adapters/ (formats beyond OLMoE/Qwen3-MoE)
tests/        test_registry_store.py test_numerical.py
benchmarks/   configs/ (workloads, models)  scripts/ (bench_decode, collect_traces, simulate_cache, ssd_bandwidth)  raw/ (immutable)  processed/
research/     LOG.md DECISIONS.md RELATED_WORK.md prior_art.csv protocol.md
paper/        main.tex sections/ references.bib (UQO template; `cd paper && make`)
artifacts/    environment.json (sanitized live inventory)  manifest.json
scripts/      cluster_setup.sh make_environment.py
third_party/  shallow clones of colibri / mlx / mlx-lm / llama.cpp (git-ignored; hashes in research/prior_art.csv)

# Quick start

uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements-lock.txt
# model (3.9 GB, Apache-2.0, pinned revision) goes to the Hugging Face cache
.venv/bin/python -c "from huggingface_hub import snapshot_download as s; s('mlx-community/OLMoE-1B-7B-0125-Instruct-4bit', revision='e2384419')"
.venv/bin/python -m pytest tests -q                      # correctness vs mlx-lm (writes evidence to benchmarks/raw/tests/)
M=$(ls -d ~/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/*)
.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend mlx_lm --out /tmp/b.jsonl
.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend titan --capacity 16 --predictor transition2 --traces benchmarks/raw/gateC/traces_olmoe_v2_laptop --out /tmp/b.jsonl
cd paper && make                                           # PDF

Local-only mode: everything above runs on one Mac; the distributed components are only enabled with an explicit host list.

# Data

No external datasets. Workloads are synthetic prompts written for this study (benchmarks/configs/workloads*.yaml); routing traces derived from them are in benchmarks/raw/gateC/traces_*. Models: OLMoE-1B-7B-0125-Instruct (Apache-2.0), Qwen3-30B-A3B-Instruct-2507 (Apache-2.0), both as mlx-community 4-bit conversions with pinned revisions.

# Methodology

See research/protocol.md (frozen at Gate C) and paper/sections/formulation.tex.

# Figures & tables inventory

Generated by scripts/make_figures.py / scripts/make_tables.py from benchmarks/raw/ (Gate G).

# Limitations

Small model only in v0.1; distributed, Metal and energy experiments pending; 1 GbE topology; see paper Section "Limitations".

# Citation

@techreport{boucher2026titanmoe,
  title       = {TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon},
  author      = {Boucher, Simon-Pierre},
  institution = {Université du Québec en Outaouais, Département des sciences administratives},
  type        = {UQO Working Paper},
  number      = {33},
  year        = {2026}
}

# Author & contact

Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca

# Notes

• The mandate's repository name wp33_titan_moe is kept as the spbgit repository name (wp33_uqo.git); the folder follows the Recherche-UQO series layout (ia/titan-moe). Internal structure follows the mandate (D-001).
• Written with Claude Code (Claude Fable 5.1) acting as autonomous research engineer under docs/MANDATE.md; AI-assistance disclosure is included in the paper's reproducibility appendix.