# TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon

**UQO Working Paper No. 33** — *A Measured Systems Study on a Heterogeneous Mac Cluster*

![Paper](https://img.shields.io/badge/paper-PDF-blue) ![Python](https://img.shields.io/badge/python-3.12-blue) ![MLX](https://img.shields.io/badge/MLX-0.32.4-black) ![Status](https://img.shields.io/badge/status-v0.1%20protocol%20%2B%20preliminary-orange) ![UQO](https://img.shields.io/badge/UQO-DSA-green) ![Author](https://img.shields.io/badge/author-S.--P.%20Boucher-lightgrey)

## TL;DR
- An MLX-native MoE inference runtime with bounded per-layer expert residency (slot tables), a `pread` NVMe tier, LRU/LFU eviction,
  causal prefetch predictors and an asynchronous prefetcher; the router's choices and the logits are those of mlx-lm by construction.
- Correctness: **bit-identical logits** (max |Δ| = 0.0) and token-identical greedy decoding vs mlx-lm on OLMoE-1B-7B-0125 4-bit, at 64/32/16/8 slots.
- All-resident control: **489 tok/s** vs mlx-lm 494 tok/s on an M5 Max laptop (−1 %). Bounded residency with demand loading from the page cache: 51–73 tok/s at 16 slots.
- Honest first result on H1: one-layer-ahead prefetch raises the hit rate (0.48→0.63) but **lowers** throughput; the paper derives the lead-time condition d·τ_layer ≥ λ_tier and tests deeper horizons.
- Live cluster inventory: 21 reachable Macs (M1–M6, 8–96 GB), **1 GbE everywhere**, no active Thunderbolt link → distributed gates use TCP; RDMA/JACCL blocked pending cabling.

## Status
Version 0.1 (2026-11-01), **preliminary** working paper; not submitted. Gates A–E complete, Gate F (medium model, two nodes) and the
energy study running at the time of writing, Gate G audit done for the existing logs (see `research/LOG.md`). Nothing in the paper
is reported without a raw log under `benchmarks/raw/`; `artifacts/manifest.json` lists every artifact with its SHA-256.

**Verdicts so far (all with bootstrap CIs, `benchmarks/processed/audit.json`):** H1 length-dependent on OLMoE/laptop NVMe (64-token runs: best predictor ties,
the install path is the bottleneck; 256-token runs: probe +32 % tok/s); H4 not supported (decode at batch 1 is launch-bound; fusion ±2 %); the remote
tier over 1 GbE is a capacity mechanism, not a latency one (−66 tok/s for odd-expert placement); the expert-contiguous storage layout
is the one clear win (cold NVMe 8 → 36 tok/s).

## What this paper does
Designs, implements and measures TITAN-MoE against five pre-stated hypotheses (causal prefetch, topology-aware placement,
hierarchical caching, Metal kernels, adaptive orchestration on the latency–throughput–energy frontier), with an exact-semantics
constraint: placement decides speed, never which experts run.

## Headline results (version 0.1, single node, OLMoE-1B-7B-0125-Instruct-4bit, M5 Max 48 GB)
| Configuration | decode tok/s | hit rate | source |
|---|---|---|---|
| mlx-lm (reference) | 491–496 | — | `benchmarks/raw/gateB/capacity_sweep_laptop/sweep.jsonl` |
| TITAN, all experts resident (64 slots) | 489 | 1.00 | same |
| TITAN, 32 slots, LRU, demand loading (page cache) | 56–80 | 0.73–0.81 | same (before lazy-install fix) |
| TITAN, 16 slots, LRU, demand loading (page cache) | 51–73 | 0.48–0.67 | `benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl` |
| TITAN, 16 slots + transition prefetch (d = 1, m = 8) | 35–56 | 0.63–0.73 | same |
| TITAN, 16 slots, cold NVMe, contiguous layout, no prefetch | 36.4 | 0.53 | `benchmarks/raw/gateC/cold_nvme_laptop/sweep.jsonl` |
| TITAN, 16 slots, cold NVMe + linear probe (recall@8 0.76) | 31.8 | 0.78 | `.../attribution.jsonl` |
| Qwen3-30B-A3B on M3U96a: mlx-lm / TITAN all-resident | 121.7 / 121.6 | — | `benchmarks/raw/gateC/qwen3_m3u96a/baseline.jsonl` |
| OLMoE two nodes (M3U96a + M3U96b, 1 GbE), 32 slots: local / odd experts remote | 81 / 15 | — | `benchmarks/raw/gateE/two_node_m3u96a_m3u96b.jsonl` |
| Energy, 256-token runs: all-resident 0.101 J/token (47 W); 16 slots cold 0.498 J/token (31.9 tok/s); + probe 0.441 J/token (**42.0 tok/s**, +10.1 [+5.2, +16.9]) | — | 0.63 / 0.84 | `benchmarks/raw/gateG/energy_laptop/` |

## Repository layout
```
CLAUDE.md  README.md  LICENSE  docs/MANDATE.md
titan/        registry.py store.py slots.py moe.py prefetch.py predictors.py runtime.py
metal/        (Gate D)         transport/  (Gate E)        adapters/ (formats beyond OLMoE/Qwen3-MoE)
tests/        test_registry_store.py test_numerical.py
benchmarks/   configs/ (workloads, models)  scripts/ (bench_decode, collect_traces, simulate_cache, ssd_bandwidth)  raw/ (immutable)  processed/
research/     LOG.md DECISIONS.md RELATED_WORK.md prior_art.csv protocol.md
paper/        main.tex sections/ references.bib (UQO template; `cd paper && make`)
artifacts/    environment.json (sanitized live inventory)  manifest.json
scripts/      cluster_setup.sh make_environment.py
third_party/  shallow clones of colibri / mlx / mlx-lm / llama.cpp (git-ignored; hashes in research/prior_art.csv)
```

## Quick start
```bash
uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements-lock.txt
# model (3.9 GB, Apache-2.0, pinned revision) goes to the Hugging Face cache
.venv/bin/python -c "from huggingface_hub import snapshot_download as s; s('mlx-community/OLMoE-1B-7B-0125-Instruct-4bit', revision='e2384419')"
.venv/bin/python -m pytest tests -q                      # correctness vs mlx-lm (writes evidence to benchmarks/raw/tests/)
M=$(ls -d ~/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/*)
.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend mlx_lm --out /tmp/b.jsonl
.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend titan --capacity 16 --predictor transition2 --traces benchmarks/raw/gateC/traces_olmoe_v2_laptop --out /tmp/b.jsonl
cd paper && make                                           # PDF
```
Local-only mode: everything above runs on one Mac; the distributed components are only enabled with an explicit host list.

## Data
No external datasets. Workloads are synthetic prompts written for this study (`benchmarks/configs/workloads*.yaml`); routing
traces derived from them are in `benchmarks/raw/gateC/traces_*`. Models: OLMoE-1B-7B-0125-Instruct (Apache-2.0),
Qwen3-30B-A3B-Instruct-2507 (Apache-2.0), both as mlx-community 4-bit conversions with pinned revisions.

## Methodology
See `research/protocol.md` (frozen at Gate C) and `paper/sections/formulation.tex`.

## Figures & tables inventory
Generated by `scripts/make_figures.py` / `scripts/make_tables.py` from `benchmarks/raw/` (Gate G).

## Limitations
Small model only in v0.1; distributed, Metal and energy experiments pending; 1 GbE topology; see paper Section "Limitations".

## Citation
```bibtex
@techreport{boucher2026titanmoe,
  title       = {TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon},
  author      = {Boucher, Simon-Pierre},
  institution = {Université du Québec en Outaouais, Département des sciences administratives},
  type        = {UQO Working Paper},
  number      = {33},
  year        = {2026}
}
```

## Author & contact
Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca

## Notes
- The mandate's repository name `wp33_titan_moe` is kept as the spbgit repository name (`wp33_uqo.git`); the folder follows the
  Recherche-UQO series layout (`ia/titan-moe`). Internal structure follows the mandate (D-001).
- Written with Claude Code (Claude Fable 5.1) acting as autonomous research engineer under `docs/MANDATE.md`; AI-assistance
  disclosure is included in the paper's reproducibility appendix.