1# WP33 — TITAN-MoE (mandate as received on 2026-11-01, verbatim)23Topology-Aware Intelligent Expert Orchestration for Distributed MoE Inference on Apple Silicon45Institution: Université du Québec en Outaouais (UQO) · Author: Simon-Pierre Boucher · Project: UQO Working Paper 33 · Agent: Claude Code · Language: English67## Mission89Act as an autonomous systems-research engineer and scientific author. Design, implement, benchmark and document TITAN-MoE, a native Apple Silicon inference runtime for sparse Mixture-of-Experts models, optimized for heterogeneous multi-Mac clusters. Produce working software, reproducible experiments, and a submission-quality LaTeX paper. Do not merely draft a speculative manuscript.1011Non-negotiable scientific integrity: Never invent measurements, sources, screenshots, benchmark results, hardware details or claims of novelty. Distinguish hypotheses from findings. Never claim a test passed without raw output. If hardware, permissions or models are unavailable, report blockers and mark experiments pending. Preserve all raw data and exact commands. Obtain authorization before remote execution, substantial downloads, paid APIs, security changes or destructive actions.1213## Core research question1415Can topology-aware expert placement, causal predictive prefetching, hierarchical memory caching and specialized Metal kernels improve MoE decode latency, prefill throughput, tokens per second, energy per token and feasible model capacity over strong MLX, llama.cpp and Colibri baselines on Apple Silicon?1617Hypotheses to TEST, not assume:18- H1: causal prefetch reduces expert loading stalls net of predictor overhead.19- H2: topology-aware placement outperforms uniform/static expert sharding on heterogeneous Mac nodes.20- H3: coordinated local RAM / remote RAM / SSD caching improves performance under fixed memory budgets.21- H4: selected fused Metal kernels improve end-to-end inference over stock MLX for matching shapes and precision.22- H5: adaptive orchestration improves the Pareto frontier of latency, throughput and energy, not just isolated microbenchmarks.2324## Phase 0 — Verify environment and prior art25261. Inventory authorized machines: exact Apple chip, GPU cores, RAM, SSD, macOS, Metal features, network links, link latency/bandwidth and power measurement tools. Never assume the available cluster matches historical notes.272. Examine primary code and documentation at https://github.com/JustVugg/colibri , https://github.com/ml-explore/mlx , https://github.com/ml-explore/mlx-lm , https://github.com/ggml-org/llama.cpp and relevant recent MoE inference research. Record repository commit hashes and dates.283. Investigate existing expert offloading, predictive expert caching, distributed MLX, Thunderbolt/RDMA, Metal MoE kernels and routing prediction. Build research/prior_art.csv with supported features, limitations, hardware, precision and evidence. Explicitly identify what is already known and revise novelty claims accordingly.294. Establish reproducible baselines and an experimental protocol BEFORE optimizing. Document model license and version, quantization, batch, prompt length and decoding settings.3031## System architecture3233Build modular components, with a reference MLX backend first and optimized C++/Metal/Swift only where justified:34351. Model and expert registry — Parse a supported MoE architecture's layers, experts, shared experts, top-k, routing semantics, weight layouts and quantization. Preserve router-selected expert identity, gating weights and aggregation exactly. Start with a small open-licensed MoE for correctness. Add larger architectures incrementally.362. Predictive expert cache — Implement no-prefetch, LRU, LFU, transition-frequency and lightweight learned predictors. Predict upcoming expert demand only from information genuinely available at decision time; avoid future-token or future-layer leakage. Prefetch asynchronously; the actual router always determines computation. Measure hit rate, wasted bytes, predictor overhead, stall time and throughput. Use held-out prompt families.373. Hierarchical memory — Manage resident experts across local unified memory, remote-node memory and local NVMe. Track bytes, capacity, eviction, fragmentation, cache warming and transfer costs. Do not presume remote memory is faster than SSD: measure each topology. Respect macOS memory pressure and safe limits.384. Distributed expert execution — Implement an authenticated transport and node discovery for explicitly authorized hosts. Start with TCP and two nodes. Evaluate supported faster transport (including Thunderbolt/RDMA) only after hardware/API verification. Compare transmitting activations to remote experts versus moving weights. Implement static sharding, frequency-based placement, capacity-aware placement, replication and topology-aware scheduling. Include failures, timeouts, backpressure and local fallback. Do not treat pooled Mac RAM as transparent shared memory.395. Native Metal kernels — Profile before optimization. Candidate kernels: quantized grouped GEMV/GEMM, fused dequantization, gated activations, expert dispatch/gather and weighted reductions. Test correctness against MLX logits and expert choices with explicit tolerances. Measure compile/warm-up separately from steady state. Benchmark prefill and decode separately; report end-to-end benefit, not only kernel speedups.406. Adaptive scheduler — Model placement decisions by measured compute, memory, SSD, link latency, network bandwidth, expert demand and power. Begin with a deterministic cost-aware heuristic; only then explore online learning or contextual bandits. Avoid cross-run contamination and account for scheduler overhead. Optimize a multi-objective function such as expected latency + lambda × energy + mu × communication cost under resource and correctness constraints.417. Optional Neural Engine feasibility — Explore Core ML/ANE only if operations and data transfers are supported and measurable. Do not promise ANE acceleration. Document negative results.4243## Formal research contribution4445Define expert e, layer l, node n and token step t. Introduce residency x[e,n,t], execution assignment z[e,n,t], and prefetch p[e,n,t]. Derive memory, routing, communication, latency and capacity constraints; model cache misses and topology. Formulate a stochastic constrained optimization problem, specify a practical online algorithm, analyze computational complexity and validate predicted costs against traces. Include a rigorously proved proposition only if one is defensible; do not fabricate theorems.4647## Experimental protocol4849- Models: at least small, medium and, if feasible, large licensed MoE models; log exact revisions, tokenizer, active/total parameters, expert count, top-k and quantization.50- Baselines: MLX-LM, llama.cpp and Colibri only when compatible on identical model/precision/hardware; otherwise explicitly flag non-comparability. Include no-prefetch, LRU/LFU, static placement and single-node controls.51- Workloads: cold and warm start, short and long context, code/reasoning/general prompts, batch/concurrency sweeps, varied output lengths, held-out workloads and changing expert popularity.52- Metrics: prefill tok/s, decode tok/s, time-to-first-token, p50/p95/p99 inter-token latency, total request latency, expert cache hit rate, stalls, SSD bytes, network bytes, peak RAM, GPU utilization when available, energy/token, correctness and stability.53- Procedure: fixed seeds where possible, exact commands, warm-up, randomized benchmark order, >=5 repeated runs when feasible, confidence intervals, thermal/power monitoring, hardware inventory, synchronized timers and raw immutable logs. Separate compilation from inference. Do not compare different precisions without clearly labeling quality and speed trade-offs.54- Ablations: predictive cache, eviction, replication, static/adaptive placement, network backend, quantization, Metal kernels, batching and scheduler overhead; test interactions.55- Include failures and cases where the proposed method loses.5657## Research outputs5859Produce data-driven figures: architecture, topology, expert activation distribution, prediction accuracy, stall breakdown, cache performance, throughput vs memory budget, scaling vs node count, latency breakdown, energy/throughput Pareto frontier, and factorial ablations. Generate every quantitative figure/table from logged measurements via scripts. Provide confidence intervals and raw-data references. No placeholder figures presented as evidence.6061## Manuscript6263Write paper/main.tex with a complete scientific paper titled "TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon". Author: Simon-Pierre Boucher, Université du Québec en Outaouais. Do not invent coauthors or funding. Include Abstract, Introduction, Background, Related Work, Mathematical Formulation, Architecture, Algorithms, Implementation, Experimental Setup, Results, Ablations, Robustness, Limitations, Conclusion, References and Reproducibility Appendix. Include original diagrams, pseudocode, tables, validated BibTeX and mathematical detail. Follow applicable venue AI-assistance disclosure policies. Compile PDF with latexmk when available; resolve citations and errors. If experiments remain pending, explicitly label paper a protocol/preliminary working paper and do not write fictional results.6465## Repository structure6667```68wp33_titan_moe/69 CLAUDE.md README.md LICENSE70 titan/ # scheduler, model registry, caching71 metal/ # Metal kernels72 transport/ # secure node transport73 adapters/ # supported MoE formats74 tests/ # unit, numerical and integration75 benchmarks/ configs/ scripts/ raw/ (immutable logs) processed/76 research/ LOG.md DECISIONS.md RELATED_WORK.md prior_art.csv protocol.md77 figures/78 paper/ main.tex references.bib sections/79 artifacts/ environment.json manifest.json80 scripts/81```8283Never commit model weights, credentials, private host inventories or sensitive data. Supply reproducible setup scripts and a safe local-only mode.8485## Sequential execution gates8687- Gate A: Verified hardware inventory, literature review, novelty matrix and benchmark protocol.88- Gate B: Single-node reference implementation, deterministic routing and numerical correctness tests.89- Gate C: Instrumented expert cache, predictor and controlled single-node experiments.90- Gate D: Metal optimization justified by profiling, with end-to-end comparisons.91- Gate E: Authorized two-node transport, static placement and distributed correctness.92- Gate F: Adaptive topology-aware scheduling, replication, larger node-count experiments and ablations.93- Gate G: Statistical audit, reproducibility package, final figures and compiled paper.9495After each gate update research/LOG.md, research/DECISIONS.md, README, test evidence and next steps. If blocked, proceed with independent local work but never silently skip a required validation. Make incremental commits where a repository exists; do not push without authorization.9697## Immediate instructions to Claude Code98991. Read this entire file and inspect the current repository and machine safely.1002. Create the project scaffold, research log and decision record.1013. Audit Colibri/MLX/llama.cpp and relevant peer-reviewed literature; identify credible novelty.1024. Inventory available Apple Silicon hardware and establish one reproducible small-MoE MLX baseline.1035. Propose a concrete first implementation milestone with measurable pass/fail criteria, then implement and test it.1046. Continue through gates autonomously within authorized resources. Ask only for genuinely required permissions, missing hardware access or scientific decisions.1057. Your final report must separate implemented, experimentally verified, unverified, and blocked deliverables and link to exact logs, code, figures and paper artifacts.106107Success means demonstrable, reproducible systems research—not an impressive but untested claim.108