Gate A/B: scaffold, mandate, inventory tooling, prior art (59 refs), registry/store/slot-table runtime, numerical tests bit-identical to mlx-lm

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 2 days ago (Nov 1, 2026) Browse files

38 changed files +2,215 −0

added .gitignore +39 −0

		`@@ -0,0 +1,39 @@`
	1	+# Third-party reference clones (shallow; commit hashes recorded in research/prior_art.csv)
	2	+third_party/
	3	+# Model weights are NEVER committed (Hugging Face cache lives outside the repo)
	4	+models/
	5	+*.safetensors
	6	+*.gguf
	7	+*.npz
	8	+# Python
	9	+.venv/
	10	+__pycache__/
	11	+*.pyc
	12	+.pytest_cache/
	13	+*.egg-info/
	14	+# Benchmark raw logs are immutable and committed, except very large traces
	15	+benchmarks/raw/*/.bin
	16	+benchmarks/raw/*/.npy
	17	+benchmarks/raw/*/.gz
	18	+# Private host inventories (artifacts/environment.json is the sanitized version)
	19	+artifacts/private/
	20	+# LaTeX build artifacts (paper/main.pdf IS committed)
	21	+paper/*.aux
	22	+paper/*.log
	23	+paper/*.out
	24	+paper/*.toc
	25	+paper/*.lof
	26	+paper/*.lot
	27	+paper/*.fls
	28	+paper/*.fdb_latexmk
	29	+paper/*.bbl
	30	+paper/*.blg
	31	+paper/*.spl
	32	+paper/*.synctex.gz
	33	+paper/*.run.xml
	34	+paper/*.bcf
	35	+# Logs
	36	+logs/
	37	+*.log
	38	+# macOS
	39	+.DS_Store

added CLAUDE.md +28 −0

		`@@ -0,0 +1,28 @@`
	1	+# TITAN-MoE — project instructions for Claude Code (WP33 UQO)
	2	+
	3	+Topology-Aware Intelligent Expert Orchestration for Distributed MoE Inference on Apple Silicon.
	4	+Author: Simon-Pierre Boucher (UQO). Agent: Claude Code. Language of code, docs and paper: English.
	5	+Full mandate: `docs/MANDATE.md`. Research log: `research/LOG.md`. Decisions: `research/DECISIONS.md`.
	6	+
	7	+## Non-negotiable scientific integrity
	8	+- Never invent measurements, sources, screenshots, benchmark results, hardware details or novelty claims.
	9	+- Distinguish hypotheses (H1–H5 in `docs/MANDATE.md`) from findings. A claim enters the paper only with a raw log under `benchmarks/raw/`.
	10	+- Never claim a test passed without its raw output (`tests/` run logs go to `benchmarks/raw/tests/`).
	11	+- If hardware, permissions or models are unavailable: record the blocker in `research/LOG.md`, mark the experiment PENDING, keep going on independent work.
	12	+- Preserve raw data and the exact command (every benchmark writes a JSON record with command line, git commit, environment hash, seed and timestamps).
	13	+- Authorization is required before: remote execution on nodes not yet listed in `artifacts/environment.json` as authorized, downloads > 20 GB, paid APIs, security changes (firewall, SSH, RDMA enable), destructive actions.
	14	+
	15	+## Layout
	16	+`titan/` scheduler, registry, cache · `metal/` Metal kernels · `transport/` authenticated node transport · `adapters/` MoE formats ·
	17	+`tests/` · `benchmarks/{configs,scripts,raw,processed}` · `research/` (LOG, DECISIONS, RELATED_WORK, prior_art.csv, protocol.md) ·
	18	+`figures/` · `paper/` (UQO template; `cd paper && make`) · `artifacts/` (environment.json, manifest.json) · `scripts/` (setup, reproduce) ·
	19	+`third_party/` (shallow clones of colibri / mlx / mlx-lm / llama.cpp, git-ignored; hashes in `research/prior_art.csv`).
	20	+
	21	+## Conventions
	22	+- Python 3.12 venv `.venv` (uv); MLX pinned in `requirements-lock.txt`. Run with `.venv/bin/python`.
	23	+- Models live in the Hugging Face cache (`~/.cache/huggingface`), never in the repo. Revisions are pinned in `benchmarks/configs/models.yaml`.
	24	+- Raw logs are immutable: write-once JSONL/JSON under `benchmarks/raw/<gate>/<experiment>/`; processed tables under `benchmarks/processed/`.
	25	+- Every figure/table in the paper is generated by `scripts/make_figures.py` / `scripts/make_tables.py` from `benchmarks/raw/`.
	26	+- Cluster nodes: short SSH aliases; LAN only; never store private keys on nodes; sudo via `echo '<pw>' \| sudo -S` only when a decision record authorizes it.
	27	+- Never commit weights, credentials, or private host inventories (`artifacts/private/` is ignored).
	28	+- Gates A–G are sequential; update LOG, DECISIONS, README and test evidence after each gate; incremental commits; no push without authorization.

added benchmarks/raw/gateA/smoke/mlx_lm_generate_laptop_20261101T011129.txt +7 −0

		`@@ -0,0 +1,7 @@`
	1	+ Fetching 6 files: 0%\| \| 0/6 [00:00<?, ?it/s] Fetching 6 files: 100%\|██████████\| 6/6 [00:00<00:00, 5124.38it/s]
	2	+==========
	3	+Mixture-of-experts models are memory-bound during decoding because they often involve processing large datasets that exceed the available GPU memory, leading to paging or swapping to disk, which significantly slows down computation. Each expert model may require significant model parameters and intermediate data storage, exacerbating memory constraints. Additionally, the frequent loading and unloading of models from memory during inference can introduce overhead, further limiting performance.
	4	+==========
	5	+Prompt: 33 tokens, 45.049 tokens-per-sec
	6	+Generation: 85 tokens, 493.904 tokens-per-sec
	7	+Peak memory: 4.110 GB

added benchmarks/raw/gateA/ssd/laptop-M5Max.json +28 −0

		`@@ -0,0 +1,28 @@`
	1	+{
	2	+ "host": "MacBookPro-de-simon-pierre-2.local",
	3	+ "file": "/Users/simon-pierreboucher/.cache/huggingface/hub/blobs/14/14a231265e6c537a2ca741df118b4a009bc8eeda3c50746cbc32eefeb8bed015",
	4	+ "file_bytes": 3893373229,
	5	+ "chunk_bytes": 3565158,
	6	+ "n": 300,
	7	+ "seed": 0,
	8	+ "machine": "arm64",
	9	+ "macos": "27.0",
	10	+ "chip": "Apple M5 Max",
	11	+ "timestamp": "2026-11-01T01:12:10",
	12	+ "cmd": "benchmarks/scripts/ssd_bandwidth.py /Users/simon-pierreboucher/.cache/huggingface/hub/blobs/14/14a231265e6c537a2ca741df118b4a009bc8eeda3c50746cbc32eefeb8bed015 --n 300 --out benchmarks/raw/gateA/ssd/laptop-M5Max.json",
	13	+ "random_nocache": {
	14	+ "GBps": 8.504955380875726,
	15	+ "lat_ms_p50": 0.43000001460313797,
	16	+ "lat_ms_p95": 0.46433298848569393,
	17	+ "lat_ms_p99": 0.4885419039055705,
	18	+ "lat_ms_mean": 0.418866811087355
	19	+ },
	20	+ "random_cached": {
	21	+ "GBps": 28.44024113777474,
	22	+ "lat_ms_p50": 0.1241665449924767
	23	+ },
	24	+ "sequential_nocache_8MB": {
	25	+ "GBps": 11.51973804557183,
	26	+ "bytes": 2147483648
	27	+ }
	28	+}
		\ No newline at end of file

added benchmarks/raw/gateB/design/slot_update_microbench_laptop.txt +6 −0

		`@@ -0,0 +1,6 @@`
	1	+setitem slot update (9 tensors, one expert 3.4 MB into 216 MB buffers): 0.305 ms
	2	+gather_qmm 1 token top-8 (3 kernels): 0.260 ms per layer
	3	+per-expert quantized_matmul loop (24 kernels): 0.428 ms per layer
	4	+stack 8 experts + gather_qmm: 0.502 ms per layer
	5	+mx.load lazy header: 10.6 ms, 9527 tensors
	6	+lazy expert load (page-cached file): first 0.528 ms, repeat 0.006 ms

added benchmarks/raw/tests/greedy_cap16_20261101T011813.json +14 −0

		`@@ -0,0 +1,14 @@`
	1	+[
	2	+ {
	3	+ "prompt": "The capital of Canada is",
	4	+ "ref": "Ottawa.\n\nOttawa is the capital of Canada, and it is located in the province of Ontario. It is situated on the south bank of the Ottawa",
	5	+ "titan": "Ottawa.\n\nOttawa is the capital of Canada, and it is located in the province of Ontario. It is situated on the south bank of the Ottawa",
	6	+ "equal": true
	7	+ },
	8	+ {
	9	+ "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
	10	+ "ref": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n",
	11	+ "titan": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n",
	12	+ "equal": true
	13	+ }
	14	+]
		\ No newline at end of file

added benchmarks/raw/tests/numerical_cap16_2026-11-01T011806.json +48 −0

		`@@ -0,0 +1,48 @@`
	1	+{
	2	+ "capacity": 16,
	3	+ "prompts": [
	4	+ {
	5	+ "prompt": "The capital of Canada is",
	6	+ "tokens": 5,
	7	+ "max_abs_dlogit": 0.0,
	8	+ "argmax_equal": true,
	9	+ "hit_rate": 0.2580101180438449,
	10	+ "misses": 440,
	11	+ "stall_s": 0.1681120308348909,
	12	+ "layers_routed": 46
	13	+ },
	14	+ {
	15	+ "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
	16	+ "tokens": 21,
	17	+ "max_abs_dlogit": 0.0,
	18	+ "argmax_equal": true,
	19	+ "hit_rate": 0.42512908777969016,
	20	+ "misses": 1002,
	21	+ "stall_s": 0.3916150442091748,
	22	+ "layers_routed": 127
	23	+ },
	24	+ {
	25	+ "prompt": "Q: If a train travels 60 km in 45 minute",
	26	+ "tokens": 25,
	27	+ "max_abs_dlogit": 0.0,
	28	+ "argmax_equal": true,
	29	+ "hit_rate": 0.41904344010530936,
	30	+ "misses": 1324,
	31	+ "stall_s": 0.5106099748518318,
	32	+ "layers_routed": 161
	33	+ },
	34	+ {
	35	+ "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
	36	+ "tokens": 26,
	37	+ "max_abs_dlogit": 0.0,
	38	+ "argmax_equal": true,
	39	+ "hit_rate": 0.45278969957081544,
	40	+ "misses": 1275,
	41	+ "stall_s": 0.49254042911343277,
	42	+ "layers_routed": 164
	43	+ }
	44	+ ],
	45	+ "tol": 0.001,
	46	+ "ts": "2026-11-01T01:18:06",
	47	+ "max_abs_dlogit_worst": 0.0
	48	+}
		\ No newline at end of file

added benchmarks/raw/tests/numerical_cap32_2026-11-01T011805.json +48 −0

		`@@ -0,0 +1,48 @@`
	1	+{
	2	+ "capacity": 32,
	3	+ "prompts": [
	4	+ {
	5	+ "prompt": "The capital of Canada is",
	6	+ "tokens": 5,
	7	+ "max_abs_dlogit": 0.0,
	8	+ "argmax_equal": true,
	9	+ "hit_rate": 0.0,
	10	+ "misses": 430,
	11	+ "stall_s": 0.15632070356514305,
	12	+ "layers_routed": 16
	13	+ },
	14	+ {
	15	+ "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
	16	+ "tokens": 21,
	17	+ "max_abs_dlogit": 0.0,
	18	+ "argmax_equal": true,
	19	+ "hit_rate": 0.6060606060606061,
	20	+ "misses": 429,
	21	+ "stall_s": 0.16922229016199708,
	22	+ "layers_routed": 40
	23	+ },
	24	+ {
	25	+ "prompt": "Q: If a train travels 60 km in 45 minute",
	26	+ "tokens": 25,
	27	+ "max_abs_dlogit": 0.0,
	28	+ "argmax_equal": true,
	29	+ "hit_rate": 0.6090458488228006,
	30	+ "misses": 631,
	31	+ "stall_s": 0.24928946851287037,
	32	+ "layers_routed": 58
	33	+ },
	34	+ {
	35	+ "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
	36	+ "tokens": 26,
	37	+ "max_abs_dlogit": 0.0,
	38	+ "argmax_equal": true,
	39	+ "hit_rate": 0.6224188790560472,
	40	+ "misses": 512,
	41	+ "stall_s": 0.20007895666640252,
	42	+ "layers_routed": 49
	43	+ }
	44	+ ],
	45	+ "tol": 0.001,
	46	+ "ts": "2026-11-01T01:18:05",
	47	+ "max_abs_dlogit_worst": 0.0
	48	+}
		\ No newline at end of file

added benchmarks/raw/tests/numerical_cap64_2026-11-01T011726.json +48 −0

		`@@ -0,0 +1,48 @@`
	1	+{
	2	+ "capacity": 64,
	3	+ "prompts": [
	4	+ {
	5	+ "prompt": "The capital of Canada is",
	6	+ "tokens": 5,
	7	+ "max_abs_dlogit": 0.0,
	8	+ "argmax_equal": true,
	9	+ "hit_rate": 0.0,
	10	+ "misses": 430,
	11	+ "stall_s": 0.15794904483482242,
	12	+ "layers_routed": 16
	13	+ },
	14	+ {
	15	+ "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
	16	+ "tokens": 21,
	17	+ "max_abs_dlogit": 0.0,
	18	+ "argmax_equal": true,
	19	+ "hit_rate": 0.4637904468412943,
	20	+ "misses": 348,
	21	+ "stall_s": 0.13284112873952836,
	22	+ "layers_routed": 16
	23	+ },
	24	+ {
	25	+ "prompt": "Q: If a train travels 60 km in 45 minute",
	26	+ "tokens": 25,
	27	+ "max_abs_dlogit": 0.0,
	28	+ "argmax_equal": true,
	29	+ "hit_rate": 0.823076923076923,
	30	+ "misses": 138,
	31	+ "stall_s": 0.05005849804729223,
	32	+ "layers_routed": 16
	33	+ },
	34	+ {
	35	+ "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
	36	+ "tokens": 26,
	37	+ "max_abs_dlogit": 0.0,
	38	+ "argmax_equal": true,
	39	+ "hit_rate": 0.9234828496042217,
	40	+ "misses": 58,
	41	+ "stall_s": 0.021775670000351965,
	42	+ "layers_routed": 16
	43	+ }
	44	+ ],
	45	+ "tol": 0.001,
	46	+ "ts": "2026-11-01T01:17:26",
	47	+ "max_abs_dlogit_worst": 0.0
	48	+}
		\ No newline at end of file

added benchmarks/raw/tests/numerical_cap64_2026-11-01T011753.json +48 −0

		`@@ -0,0 +1,48 @@`
	1	+{
	2	+ "capacity": 64,
	3	+ "prompts": [
	4	+ {
	5	+ "prompt": "The capital of Canada is",
	6	+ "tokens": 5,
	7	+ "max_abs_dlogit": 0.0,
	8	+ "argmax_equal": true,
	9	+ "hit_rate": 0.0,
	10	+ "misses": 430,
	11	+ "stall_s": 0.1560894971480593,
	12	+ "layers_routed": 16
	13	+ },
	14	+ {
	15	+ "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
	16	+ "tokens": 21,
	17	+ "max_abs_dlogit": 0.0,
	18	+ "argmax_equal": true,
	19	+ "hit_rate": 0.4637904468412943,
	20	+ "misses": 348,
	21	+ "stall_s": 0.12562509416602552,
	22	+ "layers_routed": 16
	23	+ },
	24	+ {
	25	+ "prompt": "Q: If a train travels 60 km in 45 minute",
	26	+ "tokens": 25,
	27	+ "max_abs_dlogit": 0.0,
	28	+ "argmax_equal": true,
	29	+ "hit_rate": 0.823076923076923,
	30	+ "misses": 138,
	31	+ "stall_s": 0.049756161286495626,
	32	+ "layers_routed": 16
	33	+ },
	34	+ {
	35	+ "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
	36	+ "tokens": 26,
	37	+ "max_abs_dlogit": 0.0,
	38	+ "argmax_equal": true,
	39	+ "hit_rate": 0.9234828496042217,
	40	+ "misses": 58,
	41	+ "stall_s": 0.02132466295734048,
	42	+ "layers_routed": 16
	43	+ }
	44	+ ],
	45	+ "tol": 0.001,
	46	+ "ts": "2026-11-01T01:17:53",
	47	+ "max_abs_dlogit_worst": 0.0
	48	+}
		\ No newline at end of file

added benchmarks/raw/tests/numerical_cap64_2026-11-01T011805.json +48 −0

		`@@ -0,0 +1,48 @@`
	1	+{
	2	+ "capacity": 64,
	3	+ "prompts": [
	4	+ {
	5	+ "prompt": "The capital of Canada is",
	6	+ "tokens": 5,
	7	+ "max_abs_dlogit": 0.0,
	8	+ "argmax_equal": true,
	9	+ "hit_rate": 0.0,
	10	+ "misses": 430,
	11	+ "stall_s": 0.15764980018138885,
	12	+ "layers_routed": 16
	13	+ },
	14	+ {
	15	+ "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
	16	+ "tokens": 21,
	17	+ "max_abs_dlogit": 0.0,
	18	+ "argmax_equal": true,
	19	+ "hit_rate": 0.4637904468412943,
	20	+ "misses": 348,
	21	+ "stall_s": 0.1283369231969118,
	22	+ "layers_routed": 16
	23	+ },
	24	+ {
	25	+ "prompt": "Q: If a train travels 60 km in 45 minute",
	26	+ "tokens": 25,
	27	+ "max_abs_dlogit": 0.0,
	28	+ "argmax_equal": true,
	29	+ "hit_rate": 0.823076923076923,
	30	+ "misses": 138,
	31	+ "stall_s": 0.049926698906347156,
	32	+ "layers_routed": 16
	33	+ },
	34	+ {
	35	+ "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
	36	+ "tokens": 26,
	37	+ "max_abs_dlogit": 0.0,
	38	+ "argmax_equal": true,
	39	+ "hit_rate": 0.9234828496042217,
	40	+ "misses": 58,
	41	+ "stall_s": 0.02137274609412998,
	42	+ "layers_routed": 16
	43	+ }
	44	+ ],
	45	+ "tol": 0.001,
	46	+ "ts": "2026-11-01T01:18:05",
	47	+ "max_abs_dlogit_worst": 0.0
	48	+}
		\ No newline at end of file

added benchmarks/raw/tests/numerical_cap8_2026-11-01T011808.json +48 −0

		`@@ -0,0 +1,48 @@`
	1	+{
	2	+ "capacity": 8,
	3	+ "prompts": [
	4	+ {
	5	+ "prompt": "The capital of Canada is",
	6	+ "tokens": 5,
	7	+ "max_abs_dlogit": 0.0,
	8	+ "argmax_equal": true,
	9	+ "hit_rate": 0.1625,
	10	+ "misses": 536,
	11	+ "stall_s": 0.20198134251404554,
	12	+ "layers_routed": 80
	13	+ },
	14	+ {
	15	+ "prompt": "def fibonacci(n):\n \"\"\"Return the n-th",
	16	+ "tokens": 21,
	17	+ "max_abs_dlogit": 0.0,
	18	+ "argmax_equal": true,
	19	+ "hit_rate": 0.4425981873111782,
	20	+ "misses": 1476,
	21	+ "stall_s": 0.57237828662619,
	22	+ "layers_routed": 331
	23	+ },
	24	+ {
	25	+ "prompt": "Q: If a train travels 60 km in 45 minute",
	26	+ "tokens": 25,
	27	+ "max_abs_dlogit": 0.0,
	28	+ "argmax_equal": true,
	29	+ "hit_rate": 0.375,
	30	+ "misses": 1995,
	31	+ "stall_s": 0.7614422921324149,
	32	+ "layers_routed": 399
	33	+ },
	34	+ {
	35	+ "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent",
	36	+ "tokens": 26,
	37	+ "max_abs_dlogit": 0.0,
	38	+ "argmax_equal": true,
	39	+ "hit_rate": 0.3878048780487805,
	40	+ "misses": 2008,
	41	+ "stall_s": 0.7666637990623713,
	42	+ "layers_routed": 410
	43	+ }
	44	+ ],
	45	+ "tol": 0.001,
	46	+ "ts": "2026-11-01T01:18:08",
	47	+ "max_abs_dlogit_worst": 0.0
	48	+}
		\ No newline at end of file

added benchmarks/raw/tests/pytest_gateB_20261101T011707.txt +41 −0

		`@@ -0,0 +1,41 @@`
	1	+F
	2	+=================================== FAILURES ===================================
	3	+_________________________ test_logits_match_mlx_lm[64] _________________________
	4	+
	5	+model_dir = '/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b'
	6	+ref = (Model(
	7	+ (model): OlmoeModel(
	8	+ (embed_tokens): QuantizedEmbedding(50304, 2048, group_size=64, bits=4, mode=affine)...4, bias=False, group_size=64, bits=4, mode=affine)
	9	+), <mlx_lm.tokenizer_utils.TokenizerWrapper object at 0x7093d97fe0>)
	10	+capacity = 64
	11	+
	12	+ @pytest.mark.parametrize("capacity", [64, 32, 16, 8])
	13	+ def test_logits_match_mlx_lm(model_dir, ref, capacity):
	14	+ refm, tok = ref
	15	+ tm = TitanModel(model_dir, capacity=capacity)
	16	+ trace = _route_trace(tm)
	17	+ rec = {"capacity": capacity, "prompts": [], "tol": TOL, "ts": time.strftime("%Y-%m-%dT%H:%M:%S")}
	18	+ worst = 0.0
	19	+ for p in PROMPTS:
	20	+ ids = mx.array(tok.encode(p))[None]
	21	+ ref_logits = refm(ids); mx.eval(ref_logits)
	22	+ trace.clear(); tm.reset_stats()
	23	+ tl = tm(ids); mx.eval(tl)
	24	+ d = float(mx.max(mx.abs(tl.astype(mx.float32) - ref_logits.astype(mx.float32))))
	25	+ same_argmax = bool(mx.array_equal(mx.argmax(tl, -1), mx.argmax(ref_logits, -1)))
	26	+ # reference expert choices: recompute the router exactly as mlx-lm does, from the reference model's own hidden states
	27	+ worst = max(worst, d)
	28	+ st = tm.cache_stats()
	29	+ rec["prompts"].append({"prompt": p[:40], "tokens": int(ids.shape[1]), "max_abs_dlogit": d, "argmax_equal": same_argmax,
	30	+ "hit_rate": st["hit_rate"], "misses": st["misses"], "stall_s": st["stall_time_s"],
	31	+ "layers_routed": len(trace)})
	32	+ assert same_argmax, (capacity, p)
	33	+> assert d <= TOL, (capacity, p, d)
	34	+E AssertionError: (64, 'The capital of Canada is', 0.01953125)
	35	+E assert 0.01953125 <= 0.001
	36	+
	37	+tests/test_numerical.py:74: AssertionError
	38	+=========================== short test summary info ============================
	39	+FAILED tests/test_numerical.py::test_logits_match_mlx_lm[64] - AssertionError...
	40	+!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
	41	+1 failed in 1.12s

added benchmarks/raw/tests/pytest_gateB_20261101T011725.txt +71 −0

		`@@ -0,0 +1,71 @@`
	1	+.F
	2	+=================================== FAILURES ===================================
	3	+_________________________ test_logits_match_mlx_lm[32] _________________________
	4	+
	5	+model_dir = '/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b'
	6	+ref = (Model(
	7	+ (model): OlmoeModel(
	8	+ (embed_tokens): QuantizedEmbedding(50304, 2048, group_size=64, bits=4, mode=affine)...4, bias=False, group_size=64, bits=4, mode=affine)
	9	+), <mlx_lm.tokenizer_utils.TokenizerWrapper object at 0x7094fe7fe0>)
	10	+capacity = 32
	11	+
	12	+ @pytest.mark.parametrize("capacity", [64, 32, 16, 8])
	13	+ def test_logits_match_mlx_lm(model_dir, ref, capacity):
	14	+ refm, tok = ref
	15	+ tm = TitanModel(model_dir, capacity=capacity)
	16	+ trace = _route_trace(tm)
	17	+ rec = {"capacity": capacity, "prompts": [], "tol": TOL, "ts": time.strftime("%Y-%m-%dT%H:%M:%S")}
	18	+ worst = 0.0
	19	+ for p in PROMPTS:
	20	+ ids = mx.array(tok.encode(p))[None]
	21	+ ref_logits = refm(ids); mx.eval(ref_logits)
	22	+ trace.clear(); tm.reset_stats()
	23	+> tl = tm(ids); mx.eval(tl)
	24	+ ^^^^^^^
	25	+
	26	+tests/test_numerical.py:64:
	27	+_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
	28	+titan/runtime.py:55: in __call__
	29	+ return self.model(inputs, cache=cache)
	30	+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	31	+.venv/lib/python3.12/site-packages/mlx_lm/models/olmoe.py:190: in __call__
	32	+ out = self.model(inputs, cache)
	33	+ ^^^^^^^^^^^^^^^^^^^^^^^^^
	34	+.venv/lib/python3.12/site-packages/mlx_lm/models/olmoe.py:172: in __call__
	35	+ h = layer(h, mask, cache=c)
	36	+ ^^^^^^^^^^^^^^^^^^^^^^^
	37	+.venv/lib/python3.12/site-packages/mlx_lm/models/olmoe.py:145: in __call__
	38	+ x = x + self.mlp(self.post_attention_layernorm(x))
	39	+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	40	+.venv/lib/python3.12/site-packages/mlx_lm/models/olmoe.py:123: in __call__
	41	+ y = self.switch_mlp(x_flat, indices)
	42	+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	43	+titan/moe.py:40: in __call__
	44	+ slot_of = dict(zip(uniq, self.table.ensure(uniq)))
	45	+ ^^^^^^^^^^^^^^^^^^^^^^^
	46	+titan/slots.py:134: in ensure
	47	+ self.load(e, protected=needed)
	48	+titan/slots.py:112: in load
	49	+ slot = self._install(expert, tensors, protected)
	50	+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	51	+titan/slots.py:97: in _install
	52	+ victim = self.policy.victim(protected)
	53	+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	54	+_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
	55	+
	56	+self = <titan.slots.EvictionPolicy object at 0x70959f0080>
	57	+protected = {0, 2, 3, 4, 5, 6, ...}
	58	+
	59	+ def victim(self, protected: Iterable[int]) -> int:
	60	+ prot = set(protected)
	61	+ for e in self.order:
	62	+ if e not in prot:
	63	+ return e
	64	+> raise RuntimeError("no evictable expert")
	65	+E RuntimeError: no evictable expert
	66	+
	67	+titan/slots.py:36: RuntimeError
	68	+=========================== short test summary info ============================
	69	+FAILED tests/test_numerical.py::test_logits_match_mlx_lm[32] - RuntimeError: ...
	70	+!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
	71	+1 failed, 1 passed in 1.94s

added benchmarks/raw/tests/pytest_gateB_20261101T011752.txt +44 −0

		`@@ -0,0 +1,44 @@`
	1	+.F
	2	+=================================== FAILURES ===================================
	3	+_________________________ test_logits_match_mlx_lm[32] _________________________
	4	+
	5	+model_dir = '/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b'
	6	+ref = (Model(
	7	+ (model): OlmoeModel(
	8	+ (embed_tokens): QuantizedEmbedding(50304, 2048, group_size=64, bits=4, mode=affine)...4, bias=False, group_size=64, bits=4, mode=affine)
	9	+), <mlx_lm.tokenizer_utils.TokenizerWrapper object at 0x7095c4bfe0>)
	10	+capacity = 32
	11	+
	12	+ @pytest.mark.parametrize("capacity", [64, 32, 16, 8])
	13	+ def test_logits_match_mlx_lm(model_dir, ref, capacity):
	14	+ refm, tok = ref
	15	+ tm = TitanModel(model_dir, capacity=capacity)
	16	+ trace = _route_trace(tm)
	17	+ rec = {"capacity": capacity, "prompts": [], "tol": TOL, "ts": time.strftime("%Y-%m-%dT%H:%M:%S")}
	18	+ worst = 0.0
	19	+ for p in PROMPTS:
	20	+ ids = mx.array(tok.encode(p))[None]
	21	+ ref_logits = refm(ids); mx.eval(ref_logits)
	22	+ trace.clear(); tm.reset_stats()
	23	+ tl = tm(ids); mx.eval(tl)
	24	+ d = float(mx.max(mx.abs(tl.astype(mx.float32) - ref_logits.astype(mx.float32))))
	25	+ same_argmax = bool(mx.array_equal(mx.argmax(tl, -1), mx.argmax(ref_logits, -1)))
	26	+ # reference expert choices: recompute the router exactly as mlx-lm does, from the reference model's own hidden states
	27	+ worst = max(worst, d)
	28	+ st = tm.cache_stats()
	29	+ rec["prompts"].append({"prompt": p[:40], "tokens": int(ids.shape[1]), "max_abs_dlogit": d, "argmax_equal": same_argmax,
	30	+ "hit_rate": st["hit_rate"], "misses": st["misses"], "stall_s": st["stall_time_s"],
	31	+ "layers_routed": len(trace)})
	32	+ assert same_argmax, (capacity, p)
	33	+ assert d <= TOL, (capacity, p, d)
	34	+> assert len(trace) == tm.spec.num_layers
	35	+E AssertionError: assert 40 == 16
	36	+E + where 40 = len([(0, (57, 55, 33, 35, 51, 7, ...)), (0, (6, 2, 8, 25, 53, 52, ...)), (0, (25, 44, 7, 21, 39, 60, ...)), (1, (15, 18, 32, 25, 11, 61, ...)), (1, (18, 42, 24, 34, 16, 25, ...)), (1, (15, 52, 25, 47, 38, 43, ...)), ...])
	37	+E + and 16 = ModelSpec(path='/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bi...shots/e23844197887b031e7ddddbb0b8959c5a6853a7b/model.safetensors', offset=52664621, nbytes=4096)}, layout='per_expert').num_layers
	38	+E + where ModelSpec(path='/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bi...shots/e23844197887b031e7ddddbb0b8959c5a6853a7b/model.safetensors', offset=52664621, nbytes=4096)}, layout='per_expert') = <titan.runtime.TitanModel object at 0x7095446270>.spec
	39	+
	40	+tests/test_numerical.py:75: AssertionError
	41	+=========================== short test summary info ============================
	42	+FAILED tests/test_numerical.py::test_logits_match_mlx_lm[32] - AssertionError...
	43	+!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
	44	+1 failed, 1 passed in 1.96s

added benchmarks/raw/tests/pytest_gateB_20261101T011804.txt +2 −0

		`@@ -0,0 +1,2 @@`
	1	+....... [100%]
	2	+7 passed in 9.48s

added benchmarks/scripts/ssd_bandwidth.py +64 −0

		`@@ -0,0 +1,64 @@`
	1	+#!/usr/bin/env python3
	2	+"""Measure local SSD read bandwidth at expert-granularity (random reads) and sequential reads,
	3	+bypassing the page cache with F_NOCACHE (macOS). Writes a JSON record to benchmarks/raw/<out>.
	4	+Usage: ssd_bandwidth.py <file> --chunk-mb 3.4 --n 200 --out benchmarks/raw/gateA/ssd/<host>.json
	5	+"""
	6	+import argparse, fcntl, json, os, platform, random, socket, subprocess, time, statistics, sys
	7	+
	8	+F_NOCACHE = 48 # from <sys/fcntl.h> on macOS
	9	+
	10	+def read_chunks(path, offsets, size, nocache=True):
	11	+ fd = os.open(path, os.O_RDONLY)
	12	+ if nocache:
	13	+ fcntl.fcntl(fd, F_NOCACHE, 1)
	14	+ lat = []
	15	+ nbytes = 0
	16	+ t0 = time.perf_counter()
	17	+ for off in offsets:
	18	+ t = time.perf_counter()
	19	+ b = os.pread(fd, size, off)
	20	+ lat.append(time.perf_counter() - t)
	21	+ nbytes += len(b)
	22	+ tot = time.perf_counter() - t0
	23	+ os.close(fd)
	24	+ return nbytes, tot, lat
	25	+
	26	+def main():
	27	+ ap = argparse.ArgumentParser()
	28	+ ap.add_argument("file")
	29	+ ap.add_argument("--chunk-mb", type=float, default=3.4)
	30	+ ap.add_argument("--n", type=int, default=200)
	31	+ ap.add_argument("--seq-mb", type=int, default=2048)
	32	+ ap.add_argument("--seed", type=int, default=0)
	33	+ ap.add_argument("--out", required=True)
	34	+ a = ap.parse_args()
	35	+ size = int(a.chunk_mb * 1024 * 1024)
	36	+ fsize = os.path.getsize(a.file)
	37	+ rng = random.Random(a.seed)
	38	+ offsets = [rng.randrange(0, fsize - size) // 4096 * 4096 for _ in range(a.n)]
	39	+ rec = {"host": socket.gethostname(), "file": a.file, "file_bytes": fsize, "chunk_bytes": size, "n": a.n,
	40	+ "seed": a.seed, "machine": platform.machine(), "macos": platform.mac_ver()[0],
	41	+ "chip": subprocess.run(["sysctl", "-n", "machdep.cpu.brand_string"], capture_output=True, text=True).stdout.strip(),
	42	+ "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"), "cmd": " ".join(sys.argv)}
	43	+ # random expert-sized reads, uncached
	44	+ nb, tot, lat = read_chunks(a.file, offsets, size, nocache=True)
	45	+ lat_ms = sorted(x * 1e3 for x in lat)
	46	+ rec["random_nocache"] = {"GBps": nb / tot / 1e9, "lat_ms_p50": statistics.median(lat_ms),
	47	+ "lat_ms_p95": lat_ms[int(0.95 * len(lat_ms)) - 1], "lat_ms_p99": lat_ms[int(0.99 * len(lat_ms)) - 1],
	48	+ "lat_ms_mean": statistics.mean(lat_ms)}
	49	+ # same offsets, page-cache allowed (second pass is warm)
	50	+ read_chunks(a.file, offsets, size, nocache=False)
	51	+ nb, tot, lat = read_chunks(a.file, offsets, size, nocache=False)
	52	+ rec["random_cached"] = {"GBps": nb / tot / 1e9, "lat_ms_p50": statistics.median(x * 1e3 for x in lat)}
	53	+ # sequential uncached
	54	+ seq = min(a.seq_mb * 1024 * 1024, fsize)
	55	+ offs = list(range(0, seq, 8 * 1024 * 1024))
	56	+ nb, tot, _ = read_chunks(a.file, offs, 8 * 1024 * 1024, nocache=True)
	57	+ rec["sequential_nocache_8MB"] = {"GBps": nb / tot / 1e9, "bytes": nb}
	58	+ os.makedirs(os.path.dirname(a.out), exist_ok=True)
	59	+ with open(a.out, "w") as f:
	60	+ json.dump(rec, f, indent=2)
	61	+ print(json.dumps(rec, indent=2))
	62	+
	63	+if __name__ == "__main__":
	64	+ main()

added docs/MANDATE.md +107 −0

		`@@ -0,0 +1,107 @@`
	1	+# WP33 — TITAN-MoE (mandate as received on 2026-11-01, verbatim)
	2	+
	3	+Topology-Aware Intelligent Expert Orchestration for Distributed MoE Inference on Apple Silicon
	4	+
	5	+Institution: Université du Québec en Outaouais (UQO) · Author: Simon-Pierre Boucher · Project: UQO Working Paper 33 · Agent: Claude Code · Language: English
	6	+
	7	+## Mission
	8	+
	9	+Act as an autonomous systems-research engineer and scientific author. Design, implement, benchmark and document TITAN-MoE, a native Apple Silicon inference runtime for sparse Mixture-of-Experts models, optimized for heterogeneous multi-Mac clusters. Produce working software, reproducible experiments, and a submission-quality LaTeX paper. Do not merely draft a speculative manuscript.
	10	+
	11	+Non-negotiable scientific integrity: Never invent measurements, sources, screenshots, benchmark results, hardware details or claims of novelty. Distinguish hypotheses from findings. Never claim a test passed without raw output. If hardware, permissions or models are unavailable, report blockers and mark experiments pending. Preserve all raw data and exact commands. Obtain authorization before remote execution, substantial downloads, paid APIs, security changes or destructive actions.
	12	+
	13	+## Core research question
	14	+
	15	+Can topology-aware expert placement, causal predictive prefetching, hierarchical memory caching and specialized Metal kernels improve MoE decode latency, prefill throughput, tokens per second, energy per token and feasible model capacity over strong MLX, llama.cpp and Colibri baselines on Apple Silicon?
	16	+
	17	+Hypotheses to TEST, not assume:
	18	+- H1: causal prefetch reduces expert loading stalls net of predictor overhead.
	19	+- H2: topology-aware placement outperforms uniform/static expert sharding on heterogeneous Mac nodes.
	20	+- H3: coordinated local RAM / remote RAM / SSD caching improves performance under fixed memory budgets.
	21	+- H4: selected fused Metal kernels improve end-to-end inference over stock MLX for matching shapes and precision.
	22	+- H5: adaptive orchestration improves the Pareto frontier of latency, throughput and energy, not just isolated microbenchmarks.
	23	+
	24	+## Phase 0 — Verify environment and prior art
	25	+
	26	+1. Inventory authorized machines: exact Apple chip, GPU cores, RAM, SSD, macOS, Metal features, network links, link latency/bandwidth and power measurement tools. Never assume the available cluster matches historical notes.
	27	+2. Examine primary code and documentation at https://github.com/JustVugg/colibri , https://github.com/ml-explore/mlx , https://github.com/ml-explore/mlx-lm , https://github.com/ggml-org/llama.cpp and relevant recent MoE inference research. Record repository commit hashes and dates.
	28	+3. Investigate existing expert offloading, predictive expert caching, distributed MLX, Thunderbolt/RDMA, Metal MoE kernels and routing prediction. Build research/prior_art.csv with supported features, limitations, hardware, precision and evidence. Explicitly identify what is already known and revise novelty claims accordingly.
	29	+4. Establish reproducible baselines and an experimental protocol BEFORE optimizing. Document model license and version, quantization, batch, prompt length and decoding settings.
	30	+
	31	+## System architecture
	32	+
	33	+Build modular components, with a reference MLX backend first and optimized C++/Metal/Swift only where justified:
	34	+
	35	+1. Model and expert registry — Parse a supported MoE architecture's layers, experts, shared experts, top-k, routing semantics, weight layouts and quantization. Preserve router-selected expert identity, gating weights and aggregation exactly. Start with a small open-licensed MoE for correctness. Add larger architectures incrementally.
	36	+2. Predictive expert cache — Implement no-prefetch, LRU, LFU, transition-frequency and lightweight learned predictors. Predict upcoming expert demand only from information genuinely available at decision time; avoid future-token or future-layer leakage. Prefetch asynchronously; the actual router always determines computation. Measure hit rate, wasted bytes, predictor overhead, stall time and throughput. Use held-out prompt families.
	37	+3. Hierarchical memory — Manage resident experts across local unified memory, remote-node memory and local NVMe. Track bytes, capacity, eviction, fragmentation, cache warming and transfer costs. Do not presume remote memory is faster than SSD: measure each topology. Respect macOS memory pressure and safe limits.
	38	+4. Distributed expert execution — Implement an authenticated transport and node discovery for explicitly authorized hosts. Start with TCP and two nodes. Evaluate supported faster transport (including Thunderbolt/RDMA) only after hardware/API verification. Compare transmitting activations to remote experts versus moving weights. Implement static sharding, frequency-based placement, capacity-aware placement, replication and topology-aware scheduling. Include failures, timeouts, backpressure and local fallback. Do not treat pooled Mac RAM as transparent shared memory.
	39	+5. Native Metal kernels — Profile before optimization. Candidate kernels: quantized grouped GEMV/GEMM, fused dequantization, gated activations, expert dispatch/gather and weighted reductions. Test correctness against MLX logits and expert choices with explicit tolerances. Measure compile/warm-up separately from steady state. Benchmark prefill and decode separately; report end-to-end benefit, not only kernel speedups.
	40	+6. Adaptive scheduler — Model placement decisions by measured compute, memory, SSD, link latency, network bandwidth, expert demand and power. Begin with a deterministic cost-aware heuristic; only then explore online learning or contextual bandits. Avoid cross-run contamination and account for scheduler overhead. Optimize a multi-objective function such as expected latency + lambda × energy + mu × communication cost under resource and correctness constraints.
	41	+7. Optional Neural Engine feasibility — Explore Core ML/ANE only if operations and data transfers are supported and measurable. Do not promise ANE acceleration. Document negative results.
	42	+
	43	+## Formal research contribution
	44	+
	45	+Define expert e, layer l, node n and token step t. Introduce residency x[e,n,t], execution assignment z[e,n,t], and prefetch p[e,n,t]. Derive memory, routing, communication, latency and capacity constraints; model cache misses and topology. Formulate a stochastic constrained optimization problem, specify a practical online algorithm, analyze computational complexity and validate predicted costs against traces. Include a rigorously proved proposition only if one is defensible; do not fabricate theorems.
	46	+
	47	+## Experimental protocol
	48	+
	49	+- Models: at least small, medium and, if feasible, large licensed MoE models; log exact revisions, tokenizer, active/total parameters, expert count, top-k and quantization.
	50	+- Baselines: MLX-LM, llama.cpp and Colibri only when compatible on identical model/precision/hardware; otherwise explicitly flag non-comparability. Include no-prefetch, LRU/LFU, static placement and single-node controls.
	51	+- Workloads: cold and warm start, short and long context, code/reasoning/general prompts, batch/concurrency sweeps, varied output lengths, held-out workloads and changing expert popularity.
	52	+- Metrics: prefill tok/s, decode tok/s, time-to-first-token, p50/p95/p99 inter-token latency, total request latency, expert cache hit rate, stalls, SSD bytes, network bytes, peak RAM, GPU utilization when available, energy/token, correctness and stability.
	53	+- Procedure: fixed seeds where possible, exact commands, warm-up, randomized benchmark order, >=5 repeated runs when feasible, confidence intervals, thermal/power monitoring, hardware inventory, synchronized timers and raw immutable logs. Separate compilation from inference. Do not compare different precisions without clearly labeling quality and speed trade-offs.
	54	+- Ablations: predictive cache, eviction, replication, static/adaptive placement, network backend, quantization, Metal kernels, batching and scheduler overhead; test interactions.
	55	+- Include failures and cases where the proposed method loses.
	56	+
	57	+## Research outputs
	58	+
	59	+Produce data-driven figures: architecture, topology, expert activation distribution, prediction accuracy, stall breakdown, cache performance, throughput vs memory budget, scaling vs node count, latency breakdown, energy/throughput Pareto frontier, and factorial ablations. Generate every quantitative figure/table from logged measurements via scripts. Provide confidence intervals and raw-data references. No placeholder figures presented as evidence.
	60	+
	61	+## Manuscript
	62	+
	63	+Write paper/main.tex with a complete scientific paper titled "TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon". Author: Simon-Pierre Boucher, Université du Québec en Outaouais. Do not invent coauthors or funding. Include Abstract, Introduction, Background, Related Work, Mathematical Formulation, Architecture, Algorithms, Implementation, Experimental Setup, Results, Ablations, Robustness, Limitations, Conclusion, References and Reproducibility Appendix. Include original diagrams, pseudocode, tables, validated BibTeX and mathematical detail. Follow applicable venue AI-assistance disclosure policies. Compile PDF with latexmk when available; resolve citations and errors. If experiments remain pending, explicitly label paper a protocol/preliminary working paper and do not write fictional results.
	64	+
	65	+## Repository structure
	66	+
	67	+```
	68	+wp33_titan_moe/
	69	+ CLAUDE.md README.md LICENSE
	70	+ titan/ # scheduler, model registry, caching
	71	+ metal/ # Metal kernels
	72	+ transport/ # secure node transport
	73	+ adapters/ # supported MoE formats
	74	+ tests/ # unit, numerical and integration
	75	+ benchmarks/ configs/ scripts/ raw/ (immutable logs) processed/
	76	+ research/ LOG.md DECISIONS.md RELATED_WORK.md prior_art.csv protocol.md
	77	+ figures/
	78	+ paper/ main.tex references.bib sections/
	79	+ artifacts/ environment.json manifest.json
	80	+ scripts/
	81	+```
	82	+
	83	+Never commit model weights, credentials, private host inventories or sensitive data. Supply reproducible setup scripts and a safe local-only mode.
	84	+
	85	+## Sequential execution gates
	86	+
	87	+- Gate A: Verified hardware inventory, literature review, novelty matrix and benchmark protocol.
	88	+- Gate B: Single-node reference implementation, deterministic routing and numerical correctness tests.
	89	+- Gate C: Instrumented expert cache, predictor and controlled single-node experiments.
	90	+- Gate D: Metal optimization justified by profiling, with end-to-end comparisons.
	91	+- Gate E: Authorized two-node transport, static placement and distributed correctness.
	92	+- Gate F: Adaptive topology-aware scheduling, replication, larger node-count experiments and ablations.
	93	+- Gate G: Statistical audit, reproducibility package, final figures and compiled paper.
	94	+
	95	+After each gate update research/LOG.md, research/DECISIONS.md, README, test evidence and next steps. If blocked, proceed with independent local work but never silently skip a required validation. Make incremental commits where a repository exists; do not push without authorization.
	96	+
	97	+## Immediate instructions to Claude Code
	98	+
	99	+1. Read this entire file and inspect the current repository and machine safely.
	100	+2. Create the project scaffold, research log and decision record.
	101	+3. Audit Colibri/MLX/llama.cpp and relevant peer-reviewed literature; identify credible novelty.
	102	+4. Inventory available Apple Silicon hardware and establish one reproducible small-MoE MLX baseline.
	103	+5. Propose a concrete first implementation milestone with measurable pass/fail criteria, then implement and test it.
	104	+6. Continue through gates autonomously within authorized resources. Ask only for genuinely required permissions, missing hardware access or scientific decisions.
	105	+7. Your final report must separate implemented, experimentally verified, unverified, and blocked deliverables and link to exact logs, code, figures and paper artifacts.
	106	+
	107	+Success means demonstrable, reproducible systems research—not an impressive but untested claim.

added paper/.latexmkrc +5 −0

		`@@ -0,0 +1,5 @@`
	1	+$pdf_mode = 1;
	2	+$pdflatex = 'pdflatex -interaction=nonstopmode -halt-on-error -synctex=1 %O %S';
	3	+$bibtex_use = 2;
	4	+$clean_ext = 'synctex.gz run.xml bbl bcf fdb_latexmk fls log aux out toc lof lot blg';
	5	+@default_files = ('main.tex');

added paper/Makefile +11 −0

		`@@ -0,0 +1,11 @@`
	1	+.PHONY: all clean distclean
	2	+
	3	+all:
	4	+ latexmk
	5	+
	6	+clean:
	7	+ latexmk -c
	8	+
	9	+distclean:
	10	+ latexmk -C
	11	+ rm -f .synctex.gz .run.xml .bbl .bcf

added paper/references.bib +396 −0

		`@@ -0,0 +1,396 @@`
	1	+% Verified by web lookup on 2026-11-01 (see research/prior_art.csv, column evidence_url_seen).
	2	+% Each entry was checked on its arXiv/publisher/GitHub page by the literature sub-agent; re-verified during Gate G.
	3	+
	4	+@article{eliseev2023fast,
	5	+ title={Fast Inference of Mixture-of-Experts Language Models with Offloading},
	6	+ author={Eliseev, Artyom and Mazur, Denis},
	7	+ journal={arXiv preprint arXiv:2312.17238},
	8	+ year={2023}
	9	+}
	10	+@article{xue2024moeinfinity,
	11	+ title={{MoE-Infinity}: Efficient {MoE} Inference on Personal Machines with Sparsity-Aware Expert Cache},
	12	+ author={Xue, Leyang and Fu, Yao and Lu, Zhan and Mai, Luo and Marina, Mahesh},
	13	+ journal={arXiv preprint arXiv:2401.14361},
	14	+ year={2024}
	15	+}
	16	+@inproceedings{hwang2024pregated,
	17	+ title={Pre-gated {MoE}: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference},
	18	+ author={Hwang, Ranggi and Wei, Jianyu and Cao, Shijie and Hwang, Changho and Tang, Xiaohu and Cao, Ting and Yang, Mao},
	19	+ booktitle={Proceedings of the 51st ACM/IEEE Annual International Symposium on Computer Architecture (ISCA)},
	20	+ year={2024},
	21	+ note={arXiv:2308.12066}
	22	+}
	23	+@article{yi2025edgemoe,
	24	+ title={{EdgeMoE}: Empowering Sparse Large Language Models on Mobile Devices},
	25	+ author={Yi, Rongjie and Guo, Liwei and Wei, Shiyun and Zhou, Ao and Wang, Shangguang and Xu, Mengwei},
	26	+ journal={IEEE Transactions on Mobile Computing},
	27	+ volume={24},
	28	+ number={8},
	29	+ pages={7059--7073},
	30	+ year={2025},
	31	+ doi={10.1109/TMC.2025.3546466},
	32	+ note={arXiv:2308.14352}
	33	+}
	34	+@article{zhong2024adapmoe,
	35	+ title={{AdapMoE}: Adaptive Sensitivity-based Expert Gating and Management for Efficient {MoE} Inference},
	36	+ author={Zhong, Shuzhang and Liang, Ling and Wang, Yuan and Wang, Runsheng and Huang, Ru and Li, Meng},
	37	+ journal={arXiv preprint arXiv:2408.10284},
	38	+ year={2024},
	39	+ note={Related DOI 10.1145/3676536.3676741 (ICCAD 2024, not verified on ACM page)}
	40	+}
	41	+@article{tang2024hobbit,
	42	+ title={{HOBBIT}: A Mixed Precision Expert Offloading System for Fast {MoE} Inference},
	43	+ author={Tang, Peng and Liu, Jiacheng and Hou, Xiaofeng and Pu, Yifei and Wang, Jing and Heng, Pheng-Ann and Li, Chao and Guo, Minyi},
	44	+ journal={arXiv preprint arXiv:2411.01433},
	45	+ year={2024}
	46	+}
	47	+@inproceedings{kamahori2025fiddler,
	48	+ title={Fiddler: {CPU-GPU} Orchestration for Fast Inference of Mixture-of-Experts Models},
	49	+ author={Kamahori, Keisuke and Tang, Tian and Gu, Yile and Zhu, Kan and Kasikci, Baris},
	50	+ booktitle={International Conference on Learning Representations (ICLR)},
	51	+ year={2025},
	52	+ note={arXiv:2402.07033}
	53	+}
	54	+@inproceedings{cao2025moelightning,
	55	+ title={{MoE-Lightning}: High-Throughput {MoE} Inference on Memory-constrained {GPUs}},
	56	+ author={Cao, Shiyi and Liu, Shu and Griggs, Tyler and Schafhalter, Peter and Liu, Xiaoxuan and Sheng, Ying and Gonzalez, Joseph E. and Zaharia, Matei and Stoica, Ion},
	57	+ booktitle={Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)},
	58	+ year={2025},
	59	+ note={arXiv:2411.11217; ACM DOI not retrieved}
	60	+}
	61	+@inproceedings{du2024sida,
	62	+ title={{SiDA-MoE}: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models},
	63	+ author={Du, Zhixu and Li, Shiyu and Wu, Yuhao and Jiang, Xiangyu and Sun, Jingwei and Zheng, Qilin and Wu, Yongkai and Li, Ang and Li, Hai and Chen, Yiran},
	64	+ booktitle={Proceedings of Machine Learning and Systems (MLSys)},
	65	+ year={2024},
	66	+ note={arXiv:2310.18859}
	67	+}
	68	+@article{song2024promoe,
	69	+ title={{ProMoE}: Fast {MoE}-based {LLM} Serving using Proactive Caching},
	70	+ author={Song, Xiaoniu and Zhong, Zihang and Chen, Rong and Chen, Haibo},
	71	+ journal={arXiv preprint arXiv:2410.22134},
	72	+ year={2024}
	73	+}
	74	+@inproceedings{he2026expertflow,
	75	+ title={{ExpertFlow}: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling},
	76	+ author={He, Xin and Zhang, Shunkang and Tang, Kaijie and Shi, Shaohuai and Wang, Yuxin and Zeng, Zihao and Tang, Zhenheng and Chu, Xiaowen and Yin, Haiyan and Tsang, Ivor W. and Ong, Yew Soon},
	77	+ booktitle={Proceedings of the Design Automation Conference (DAC)},
	78	+ year={2026},
	79	+ note={arXiv:2410.17954}
	80	+}
	81	+@article{yu2025finemoe,
	82	+ title={Taming Latency-Memory Trade-Off in {MoE}-Based {LLM} Serving via Fine-Grained Expert Offloading},
	83	+ author={Yu, Hanfei and Cui, Xingqi and Zhang, Hong and Wang, Hao and Wang, Hao},
	84	+ journal={arXiv preprint arXiv:2502.05370},
	85	+ year={2025}
	86	+}
	87	+@inproceedings{zhang2025daop,
	88	+ title={{DAOP}: Data-Aware Offloading and Predictive Pre-Calculation for Efficient {MoE} Inference},
	89	+ author={Zhang, Yujie and Aggarwal, Shivam and Mitra, Tulika},
	90	+ booktitle={Design, Automation \& Test in Europe Conference (DATE)},
	91	+ year={2025},
	92	+ note={arXiv:2501.10375}
	93	+}
	94	+@article{xu2025moegen,
	95	+ title={{MoE-Gen}: High-Throughput {MoE} Inference on a Single {GPU} with Module-Based Batching},
	96	+ author={Xu, Tairan and Xue, Leyang and Lu, Zhan and Jackson, Adrian and Mai, Luo},
	97	+ journal={arXiv preprint arXiv:2503.09716},
	98	+ year={2025}
	99	+}
	100	+@article{fang2025klotski,
	101	+ title={Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline},
	102	+ author={Fang, Zhiyuan and Huang, Yuegui and Hong, Zicong and Lyu, Yufeng and Chen, Wuhui and Yu, Yue and Yu, Fan and Zheng, Zibin},
	103	+ journal={arXiv preprint arXiv:2502.06888},
	104	+ year={2025}
	105	+}
	106	+@article{tairin2025emoe,
	107	+ title={{eMoE}: Task-aware Memory Efficient Mixture-of-Experts-Based ({MoE}) Model Inference},
	108	+ author={Tairin, Suraiya and Mahmud, Shohaib and Shen, Haiying and Iyer, Anand},
	109	+ journal={arXiv preprint arXiv:2503.06823},
	110	+ year={2025}
	111	+}
	112	+@inproceedings{suo2025coserve,
	113	+ title={{CoServe}: Efficient Collaboration-of-Experts ({CoE}) Model Inference with Limited Memory},
	114	+ author={Suo, Jiashun and Liao, Xiaojian and Xiao, Limin and Ruan, Li and Wang, Jinquan and Su, Xiao and Huo, Zhisheng},
	115	+ booktitle={Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)},
	116	+ year={2025},
	117	+ note={arXiv:2503.02354}
	118	+}
	119	+@article{skliar2025cacheaware,
	120	+ title={Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference},
	121	+ author={Skliar, Andrii and van Rozendaal, Ties and Lepert, Romain and Boinovski, Todor and van Baalen, Mart and Nagel, Markus and Whatmough, Paul and Ehteshami Bejnordi, Babak},
	122	+ journal={Transactions on Machine Learning Research},
	123	+ year={2025},
	124	+ note={arXiv:2412.00099}
	125	+}
	126	+@inproceedings{alizadeh2024flash,
	127	+ title={{LLM} in a flash: Efficient Large Language Model Inference with Limited Memory},
	128	+ author={Alizadeh, Keivan and Mirzadeh, Iman and Belenko, Dmitry and Khatamifard, Karen and Cho, Minsik and Del Mundo, Carlo C and Rastegari, Mohammad and Farajtabar, Mehrdad},
	129	+ booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)},
	130	+ year={2024},
	131	+ note={arXiv:2312.11514}
	132	+}
	133	+@inproceedings{song2024powerinfer,
	134	+ title={{PowerInfer}: Fast Large Language Model Serving with a Consumer-grade {GPU}},
	135	+ author={Song, Yixin and Mi, Zeyu and Xie, Haotong and Chen, Haibo},
	136	+ booktitle={Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (SOSP)},
	137	+ year={2024},
	138	+ note={arXiv:2312.12456}
	139	+}
	140	+@article{huang2023towards,
	141	+ title={Towards {MoE} Deployment: Mitigating Inefficiencies in Mixture-of-Expert ({MoE}) Inference},
	142	+ author={Huang, Haiyang and Ardalani, Newsha and Sun, Anna and Ke, Liu and Lee, Hsien-Hsin S. and Sridhar, Anjali and Bhosale, Shruti and Wu, Carole-Jean and Lee, Benjamin},
	143	+ journal={arXiv preprint arXiv:2303.06182},
	144	+ year={2023}
	145	+}
	146	+@inproceedings{cai2024readme,
	147	+ title={{Read-ME}: Refactorizing {LLMs} as Router-Decoupled Mixture of Experts with System Co-Design},
	148	+ author={Cai, Ruisi and Ro, Yeonju and Kim, Geon-Woo and Wang, Peihao and Ehteshami Bejnordi, Babak and Akella, Aditya and Wang, Zhangyang},
	149	+ booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
	150	+ year={2024},
	151	+ note={arXiv:2410.19123}
	152	+}
	153	+@article{yao2024exflow,
	154	+ title={Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference},
	155	+ author={Yao, Jinghan and Anthony, Quentin and Shafi, Aamir and Subramoni, Hari and Panda, Dhabaleswar K.},
	156	+ journal={arXiv preprint arXiv:2401.08383},
	157	+ year={2024}
	158	+}
	159	+@article{fang2025fate,
	160	+ title={Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate},
	161	+ author={Fang, Zhiyuan and Hong, Zicong and Huang, Yuegui and Lyu, Yufeng and Chen, Wuhui and Yu, Yue and Yu, Fan and Zheng, Zibin},
	162	+ journal={arXiv preprint arXiv:2502.12224},
	163	+ year={2025}
	164	+}
	165	+@inproceedings{liang2026notall,
	166	+ title={Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models},
	167	+ author={Liang, Jingcong and Wang, Siyuan and Tian, Miren and Li, Yitong and Tang, Duyu and Wei, Zhongyu},
	168	+ booktitle={International Conference on Learning Representations (ICLR)},
	169	+ year={2026},
	170	+ note={arXiv:2505.16056}
	171	+}
	172	+@inproceedings{zhong2025hybrimoe,
	173	+ title={{HybriMoE}: Hybrid {CPU-GPU} Scheduling and Cache Management for Efficient {MoE} Inference},
	174	+ author={Zhong, Shuzhang and Sun, Yanfan and Liang, Ling and Wang, Runsheng and Huang, Ru and Li, Meng},
	175	+ booktitle={Proceedings of the Design Automation Conference (DAC)},
	176	+ year={2025},
	177	+ note={arXiv:2504.05897}
	178	+}
	179	+@inproceedings{huang2026cpugpu,
	180	+ title={Efficient {CPU-GPU} Collaborative Inference for {MoE}-based {LLMs} on Memory-Limited Systems},
	181	+ author={Huang, En-Ming and Lin, Li-Shang and Lee, Chun-Yi},
	182	+ booktitle={Asia and South Pacific Design Automation Conference (ASP-DAC)},
	183	+ year={2026},
	184	+ note={arXiv:2512.16473}
	185	+}
	186	+@inproceedings{zhao2026mobile,
	187	+ title={{MoBiLE}: Efficient Mixture-of-Experts Inference on Consumer {GPU} with Mixture of Big Little Experts},
	188	+ author={Zhao, Yushu and Qin, Yubin and Wang, Yang and Yang, Xiaolong and Han, Huiming and Wei, Shaojun and Hu, Yang and Yin, Shouyi},
	189	+ booktitle={Asia and South Pacific Design Automation Conference (ASP-DAC)},
	190	+ year={2026},
	191	+ note={arXiv:2510.12357}
	192	+}
	193	+@inproceedings{yu2026layerscope,
	194	+ title={{LayerScope}: Predictive Cross-Layer Scheduling for Efficient Multi-Batch {MoE} Inference on Legacy Servers},
	195	+ author={Yu, Enda and Dong, Dezun and Zhang, Zhaoning and Bai, Zhe and Yang, Weiling and Wang, Haojie and Li, Dongsheng and Wu, Yongwei and Liao, Xiangke},
	196	+ booktitle={Proceedings of the International Conference on Supercomputing (ICS)},
	197	+ year={2026},
	198	+ note={arXiv:2509.23638}
	199	+}
	200	+@article{shen2025expertflow,
	201	+ title={{ExpertFlow}: Adaptive Expert Scheduling and Memory Coordination for Efficient {MoE} Inference},
	202	+ author={Shen, Zixu and Chu, Kexin and Zhang, Yifan and Xiang, Dawei and Wu, Runxin and Zhang, Wei},
	203	+ journal={arXiv preprint arXiv:2510.26730},
	204	+ year={2025}
	205	+}
	206	+@article{zhang2025duoserve,
	207	+ title={{DuoServe-MoE}: Dual-Phase Expert Prefetch and Caching for {LLM} Inference {QoS} Assurance},
	208	+ author={Zhang, Yuning and Pinkert, Grant and Yang, Nan and Li, Yanli and Yuan, Dong},
	209	+ journal={arXiv preprint arXiv:2509.07379},
	210	+ year={2025}
	211	+}
	212	+@article{kim2026flashmoe,
	213	+ title={{FlashMoE}: Reducing {SSD} {I/O} Bottlenecks via {ML}-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices},
	214	+ author={Kim, Byeongju and Lee, Jungwan and Han, Donghyeon and Yoo, Hoi-Jun and Kim, Sangyeob},
	215	+ journal={arXiv preprint arXiv:2601.17063},
	216	+ year={2026}
	217	+}
	218	+@inproceedings{pan2026trimoe,
	219	+ title={{TriMoE}: Augmenting {GPU} with {AMX}-Enabled {CPU} and {DIMM-NDP} for High-Throughput {MoE} Inference via Offloading},
	220	+ author={Pan, Yudong and He, Yintao and Han, Tianhua and Liu, Lian and Zhao, Shixin and Chen, Zhirong and Wang, Mengdi and Li, Cangyuan and Han, Yinhe and Wang, Ying},
	221	+ booktitle={Proceedings of the Design Automation Conference (DAC)},
	222	+ year={2026},
	223	+ note={arXiv:2603.01058}
	224	+}
	225	+@article{kong2026specprefetch,
	226	+ title={{SpecPrefetch}: Parameter-Efficient Expert Prefetching for Sparse {MoE} Foundation Models},
	227	+ author={Kong, Jinwei and Meng, Runqi and Wang, Fanyi and Qiu, Wentao and Hu, Haotian and Zhou, Yongjian and Ge, Zhenhua},
	228	+ journal={arXiv preprint arXiv:2607.24787},
	229	+ year={2026}
	230	+}
	231	+@article{wang2026seqmoe,
	232	+ title={{SeqMoE}: Toward Full-Load Performance via Predictive and Graph-Compatible {MoE} Offloading},
	233	+ author={Wang, Zihan and Wang, Yuqi and Gong, Lei and Tang, Cheng and Lou, Wenqi and Wang, Teng and Wang, Chao and Zhou, Xuehai},
	234	+ journal={arXiv preprint arXiv:2609.12978},
	235	+ year={2026}
	236	+}
	237	+@article{liang2026ssdllama,
	238	+ title={{SSD-LLaMA}: {SSD}-Native Inference for Trillion-Parameter {MoE} at 1+ Token/s on a Consumer {PC}},
	239	+ author={Liang, Fangzhou and Shen, Yibin and Hu, Jianmin and Xu, Jiayang and Gao, Hanchi and Xu, Minxian and Meng, Zili},
	240	+ journal={arXiv preprint arXiv:2609.18110},
	241	+ year={2026}
	242	+}
	243	+@article{yang2026moecore,
	244	+ title={{MoE-CORE}: Coordinated Expert Offloading and Residency for Memory-Constrained {MoE} Inference},
	245	+ author={Yang, Ke and Gao, Yongji and Li, Xushi and Luo, Kui and Zhang, Sicheng and Zhou, Tianming and Liu, Keyi and Lu, Shufang and Chen, Aoxuan and Meng, Jie and Gao, Jingchun and Li, Dan and You, Xinkai and Xia, Zhixiang and Shi, Yan and Liu, Yang and Zeng, Yanjia and Feng, Liangjun},
	246	+ journal={arXiv preprint arXiv:2610.01950},
	247	+ year={2026}
	248	+}
	249	+@article{yuan2026ordermoe,
	250	+ title={{OrderMoE}: An expert similarity driven distributed edge {MoE} inference},
	251	+ author={Yuan, Xin and Li, Ning and Chen, Quan and Xu, Wenchao and Guo, Song},
	252	+ journal={arXiv preprint arXiv:2607.17154},
	253	+ year={2026}
	254	+}
	255	+@article{benazir2026npumoe,
	256	+ title={Efficient Mixture-of-Experts {LLM} Inference with {Apple Silicon} {NPUs}},
	257	+ author={Benazir, Afsara and Lin, Felix Xiaozhu},
	258	+ journal={arXiv preprint arXiv:2604.18788},
	259	+ year={2026}
	260	+}
	261	+@article{alfarizy2026doesmoe,
	262	+ title={Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study},
	263	+ author={Alfarizy, Alfarizy and Nguyen, Hung Truong Thanh and Richard, Ren{\'e} and Razavi-Far, Roozbeh and Cao, Hung},
	264	+ journal={arXiv preprint arXiv:2606.21428},
	265	+ year={2026}
	266	+}
	267	+@inproceedings{kannan2026greenbench,
	268	+ title={{GreenBench}: Benchmarking Energy Efficiency and Carbon Footprint of Open-Source {LLM} Inference on {Apple Silicon}},
	269	+ author={Kannan, Rajeswari and Firke, Raj and Bengle, Shreya and Deshmukh, Srushti},
	270	+ booktitle={IEEE International Conference on Computing, Communication, Control and Automation (ICCUBEA)},
	271	+ year={2026},
	272	+ note={arXiv:2608.28667}
	273	+}
	274	+@article{javat2026siliconshowdown,
	275	+ title={Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade {LLM} Inference},
	276	+ author={Javat, Abdurrahman and Kazakov, Allan},
	277	+ journal={arXiv preprint arXiv:2605.00519},
	278	+ year={2026}
	279	+}
	280	+@article{rajesh2025productiongrade,
	281	+ title={Production-Grade Local {LLM} Inference on {Apple Silicon}: A Comparative Study of {MLX}, {MLC-LLM}, {Ollama}, {llama.cpp}, and {PyTorch MPS}},
	282	+ author={Rajesh, Varun and Jodhpurkar, Om and Anbuselvan, Pooja and Singh, Mantinder and Jallepali, Ashok and Godbole, Shantanu and Sharma, Pradeep Kumar and Shrivastava, Hritvik},
	283	+ journal={arXiv preprint arXiv:2511.05502},
	284	+ year={2025}
	285	+}
	286	+@article{rathnayaka2026basert,
	287	+ title={{BaseRT}: Best-in-Class {LLM} Inference on {Apple Silicon} via Native {Metal}},
	288	+ author={Rathnayaka, Prabod and Waschkowski, Fabian and Wesemann, Lukas},
	289	+ journal={arXiv preprint arXiv:2607.00501},
	290	+ year={2026}
	291	+}
	292	+@article{wilkins2024hybrid,
	293	+ title={Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of {LLM} Inference Workloads},
	294	+ author={Wilkins, Grant and Keshav, Srinivasan and Mortier, Richard},
	295	+ journal={arXiv preprint arXiv:2407.00010},
	296	+ year={2024}
	297	+}
	298	+@inproceedings{borzunov2023petals,
	299	+ title={Petals: Collaborative Inference and Fine-tuning of Large Models},
	300	+ author={Borzunov, Alexander and Baranchuk, Dmitry and Dettmers, Tim and Riabinin, Maksim and Belkada, Younes and Chumachenko, Artem and Samygin, Pavel and Raffel, Colin},
	301	+ booktitle={Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
	302	+ pages={558--568},
	303	+ year={2023},
	304	+ doi={10.18653/v1/2023.acl-demo.54}
	305	+}
	306	+@article{li2025primacpp,
	307	+ title={{Prima.cpp}: Fast 30-70B {LLM} Inference on Heterogeneous and Low-Resource Home Clusters},
	308	+ author={Li, Zonghang and Li, Tao and Feng, Wenjiao and Xiao, Rongxing and She, Jianshu and Huang, Hong and Guizani, Mohsen and Yu, Hongfang and Ho, Qirong and Xiang, Wei and Liu, Xue},
	309	+ journal={arXiv preprint arXiv:2504.08791},
	310	+ year={2025}
	311	+}
	312	+@misc{exo2025,
	313	+ title={exo: Run frontier {AI} locally},
	314	+ author={{exo labs}},
	315	+ howpublished={\url{https://github.com/exo-explore/exo}},
	316	+ year={2025},
	317	+ note={Accessed 2026-11-01}
	318	+}
	319	+@misc{llamacpp2026,
	320	+ title={llama.cpp: {LLM} inference in {C/C++}},
	321	+ author={Gerganov, Georgi and {ggml-org contributors}},
	322	+ howpublished={\url{https://github.com/ggml-org/llama.cpp}},
	323	+ year={2026},
	324	+ note={Accessed 2026-11-01; RPC backend documented at tools/rpc/README.md}
	325	+}
	326	+@misc{hannun2023mlx,
	327	+ title={{MLX}: Efficient and flexible machine learning on {Apple silicon}},
	328	+ author={Hannun, Awni and Digani, Jagrit and Katharopoulos, Angelos and Collobert, Ronan},
	329	+ howpublished={\url{https://github.com/ml-explore/mlx}},
	330	+ year={2023},
	331	+ note={Accessed 2026-11-01; distributed docs: \url{https://ml-explore.github.io/mlx/build/html/usage/distributed.html}}
	332	+}
	333	+@misc{apple2026mlxdistributed,
	334	+ title={Local and Distributed Large {LLM} Inference with {MLX} and {RDMA} on {Mac Studio}},
	335	+ author={{Apple Machine Learning Research}},
	336	+ howpublished={\url{https://machinelearning.apple.com/research/mlx-distributed-inference}},
	337	+ year={2026},
	338	+ note={Published 2026-10-15. Accessed 2026-11-01}
	339	+}
	340	+@misc{apple2026wwdc233,
	341	+ title={Explore distributed inference and training with {MLX} ({WWDC26} session 233)},
	342	+ author={{Apple Inc.}},
	343	+ howpublished={\url{https://developer.apple.com/videos/play/wwdc2026/233/}},
	344	+ year={2026},
	345	+ note={Accessed 2026-11-01}
	346	+}
	347	+@misc{geerling2025rdma,
	348	+ title={1.5 {TB} of {VRAM} on {Mac Studio} -- {RDMA} over {Thunderbolt 5}},
	349	+ author={Geerling, Jeff},
	350	+ howpublished={\url{https://jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5}},
	351	+ year={2025},
	352	+ note={Blog post, 2025-12-18. Accessed 2026-11-01}
	353	+}
	354	+@inproceedings{rajbhandari2022deepspeedmoe,
	355	+ title={{DeepSpeed-MoE}: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation {AI} Scale},
	356	+ author={Rajbhandari, Samyam and Li, Conglong and Yao, Zhewei and Zhang, Minjia and Aminabadi, Reza Yazdani and Awan, Ammar Ahmad and Rasley, Jeff and He, Yuxiong},
	357	+ booktitle={Proceedings of the 39th International Conference on Machine Learning (ICML)},
	358	+ year={2022},
	359	+ note={arXiv:2201.05596}
	360	+}
	361	+@misc{deepep2025,
	362	+ title={{DeepEP}: an efficient expert-parallel communication library},
	363	+ author={Zhao, Chenggang and others},
	364	+ howpublished={\url{https://github.com/deepseek-ai/DeepEP}},
	365	+ year={2025},
	366	+ note={Accessed 2026-11-01; author list abbreviated from the repository's own BibTeX}
	367	+}
	368	+@inproceedings{li2023lina,
	369	+ title={Accelerating Distributed {MoE} Training and Inference with {Lina}},
	370	+ author={Li, Jiamin and Jiang, Yimin and Zhu, Yibo and Wang, Cong and Xu, Hong},
	371	+ booktitle={Proceedings of the USENIX Annual Technical Conference (USENIX ATC)},
	372	+ pages={945--959},
	373	+ year={2023},
	374	+ note={arXiv:2210.17223}
	375	+}
	376	+@article{hwang2023tutel,
	377	+ title={Tutel: Adaptive Mixture-of-Experts at Scale},
	378	+ author={Hwang, Changho and Cui, Wei and Xiong, Yifan and Yang, Ziyue and Liu, Ze and Hu, Han and Wang, Zilong and Salas, Rafael and Jose, Jithin and Ram, Prabhat and Chau, Joe and Cheng, Peng and Yang, Fan and Yang, Mao and Xiong, Yongqiang},
	379	+ journal={arXiv preprint arXiv:2206.03382},
	380	+ year={2023},
	381	+ note={Venue (MLSys 2023) not shown on arXiv page; unverified}
	382	+}
	383	+@article{gale2023megablocks,
	384	+ title={{MegaBlocks}: Efficient Sparse Training with Mixture-of-Experts},
	385	+ author={Gale, Trevor and Narayanan, Deepak and Young, Cliff and Zaharia, Matei},
	386	+ journal={arXiv preprint arXiv:2211.15841},
	387	+ year={2023},
	388	+ note={Venue (MLSys 2023) not shown on arXiv page; unverified}
	389	+}
	390	+@misc{colibri2026,
	391	+ title={colibri: Tiny engine, immense model},
	392	+ author={Fornaro, Vincenzo},
	393	+ howpublished={\url{https://github.com/JustVugg/colibri}},
	394	+ year={2026},
	395	+ note={Version 2.1.0, Apache-2.0. Accessed 2026-11-01. No paper or citation provided by the project}
	396	+}

added paper/sections/titlepage.tex +77 −0

		`@@ -0,0 +1,77 @@`
	1	+% ============================================================================
	2	+% Title Page
	3	+% ============================================================================
	4	+\thispagestyle{empty}
	5	+
	6	+\begin{center}
	7	+
	8	+% --- Logo ---
	9	+\includegraphics[width=3.5cm]{uq_logo.jpg}
	10	+
	11	+\vspace{0.6cm}
	12	+
	13	+{\footnotesize\textsc{Université du Québec en Outaouais}}\\[0.15cm]
	14	+{\footnotesize\textsc{Département des sciences administratives}}
	15	+
	16	+\vspace{0.8cm}
	17	+
	18	+{\footnotesize\textsc{Working Paper No.~\WPnumber}}
	19	+
	20	+\vspace{1.2cm}
	21	+
	22	+% --- Title ---
	23	+{\LARGE\bfseries \WPtitle\par}
	24	+
	25	+\ifx\WPsubtitle\empty\else
	26	+ \vspace{0.3cm}
	27	+ {\large\itshape \WPsubtitle\par}
	28	+\fi
	29	+
	30	+\vspace{1.2cm}
	31	+
	32	+% --- Author ---
	33	+{\large \WPauthor}\\[0.3cm]
	34	+{\normalsize \WPaffiliation}\\[0.15cm]
	35	+{\normalsize \href{mailto:\WPemail}{\WPemail}}\\[0.15cm]
	36	+{\small \WPaddress}
	37	+
	38	+\vspace{0.8cm}
	39	+
	40	+% --- Date & Version ---
	41	+{\normalsize \WPdate}\\[0.1cm]
	42	+{\small Version~\WPversion}
	43	+
	44	+\end{center}
	45	+
	46	+\vfill
	47	+
	48	+\newpage
	49	+
	50	+% ============================================================================
	51	+% Abstract Page
	52	+% ============================================================================
	53	+\thispagestyle{empty}
	54	+
	55	+\vspace*{1cm}
	56	+
	57	+\noindent\rule{\textwidth}{0.4pt}
	58	+\vspace{0.3cm}
	59	+
	60	+\noindent\textbf{Abstract}
	61	+
	62	+\vspace{0.15cm}
	63	+
	64	+\noindent\WPabstract
	65	+
	66	+\vspace{0.4cm}
	67	+
	68	+\noindent\textbf{Keywords:} \WPkeywords
	69	+
	70	+\vspace{0.15cm}
	71	+
	72	+\noindent\textbf{JEL Classification:} \WPjel
	73	+
	74	+\vspace{0.3cm}
	75	+\noindent\rule{\textwidth}{0.4pt}
	76	+
	77	+\newpage

added paper/sections/titlepage_anon.tex +53 −0

		`@@ -0,0 +1,53 @@`
	1	+% ============================================================================
	2	+% ANONYMOUS TITLE PAGE — for double-anonymized journal submission
	3	+% Used by paper/main_anon.tex (same macros as main.tex, no author block,
	4	+% no institution, no logo). Abstract/keywords/JEL page identical to titlepage.tex.
	5	+% ============================================================================
	6	+\thispagestyle{empty}
	7	+\begin{center}
	8	+
	9	+\vspace*{3cm}
	10	+
	11	+{\LARGE\bfseries \WPtitle\par}
	12	+
	13	+\ifx\WPsubtitle\empty\else
	14	+ \vspace{0.3cm}
	15	+ {\large\itshape \WPsubtitle\par}
	16	+\fi
	17	+
	18	+\vspace{1.5cm}
	19	+
	20	+{\normalsize [Author identity withheld for review]}
	21	+
	22	+\vspace{0.8cm}
	23	+
	24	+{\normalsize \WPdate}
	25	+
	26	+\end{center}
	27	+
	28	+\vfill
	29	+
	30	+\newpage
	31	+\thispagestyle{empty}
	32	+\vspace*{1cm}
	33	+
	34	+\noindent\rule{\textwidth}{0.4pt}
	35	+\vspace{0.3cm}
	36	+
	37	+\noindent\textbf{Abstract}
	38	+\vspace{0.15cm}
	39	+
	40	+\noindent\WPabstract
	41	+
	42	+\vspace{0.4cm}
	43	+
	44	+\noindent\textbf{Keywords:} \WPkeywords
	45	+
	46	+\vspace{0.15cm}
	47	+
	48	+\noindent\textbf{JEL Classification:} \WPjel
	49	+
	50	+\vspace{0.3cm}
	51	+\noindent\rule{\textwidth}{0.4pt}
	52	+
	53	+\newpage

added paper/uq_logo.jpg +0 −0

Binary file not shown.

added research/DECISIONS.md +40 −0

		`@@ -0,0 +1,40 @@`
	1	+# Decision record (WP33 TITAN-MoE)
	2	+
	3	+Format: D-NNN · date · decision · rationale · alternatives considered · consequences.
	4	+
	5	+## D-001 · 2026-11-01 · Project location and numbering
	6	+Decision: the project is UQO Working Paper No. 33, stored at `Recherche-UQO/ia/titan-moe/` (category `ia/`, after WP30/31),
	7	+future bare repo `gitsrv:srv/git/wp33_uqo.git`. Inside, the directory structure follows the mandate (`titan/`, `metal/`,
	8	+`transport/`, `adapters/`, `tests/`, `benchmarks/`, `research/`, `figures/`, `paper/`, `artifacts/`, `scripts/`).
	9	+Rationale: one numbered folder per paper is the series standard (`HARMONISATION.md`); the mandate's `wp33_titan_moe/` name is
	10	+honoured as the repository name. Deviation noted in README "Notes".
	11	+
	12	+## D-002 · 2026-11-01 · Reference MLX backend first; third-party code as shallow git-ignored clones
	13	+Decision: Gate B is a pure-MLX reference implementation on top of `mlx.core` ops (`gather_qmm`, `gather_mm`), not a fork of
	14	+mlx-lm. Colibri, mlx, mlx-lm and llama.cpp are shallow-cloned under `third_party/` for audit only; hashes are recorded.
	15	+Rationale: mandate ("reference MLX backend first; optimized C++/Metal/Swift only where justified"); keeps the repo small.
	16	+
	17	+## D-003 · 2026-11-01 · First model: OLMoE-1B-7B-0125-Instruct, 4-bit MLX conversion
	18	+Decision: `mlx-community/OLMoE-1B-7B-0125-Instruct-4bit` (rev `e2384419`, Apache-2.0, 16 layers, 64 experts, top-8, no shared expert).
	19	+Rationale: small (3.9 GB), fully open licence, supported by mlx-lm (`olmoe.py`), llama.cpp (GGUF) and Colibri (OLMoE engine),
	20	+so three baselines can be compared on the same architecture; 64 experts × 16 layers gives a non-trivial routing/caching problem.
	21	+Alternatives: Qwen1.5-MoE-A2.7B (non-Apache licence), gpt-oss-20b (MXFP4, larger). Medium/large candidates (Qwen3-30B-A3B,
	22	+gpt-oss-120b, Qwen3-235B) are deferred to Gate C/F and need an explicit download authorization above 20 GB.
	23	+
	24	+## D-004 · 2026-11-01 · Unreachable nodes are excluded, not assumed
	25	+Decision: nodes that did not answer SSH within 15 s at inventory time are excluded from WP33 experiments until re-probed;
	26	+the paper's hardware table only lists machines whose specifications were read live (`sysctl`, `system_profiler`).
	27	+
	28	+## D-005 · 2026-11-01 · Slot-table execution with `gather_qmm` on stacked slot buffers
	29	+Decision: each layer owns a `SlotTable` of `capacity` resident experts in stacked MLX buffers; expert ids are remapped to
	30	+slot ids and executed with the same `mx.gather_qmm` kernel mlx-lm uses, so the compute path is identical for resident and
	31	+just-loaded experts. Loading = in-place slice assignment (`buf[slot] = tensor`), measured at 0.305 ms for the 9 tensors of
	32	+one OLMoE expert into 216 MB buffers (`benchmarks/raw/gateB/design/slot_update_microbench_laptop.txt`), i.e. MLX reuses the
	33	+buffer rather than copying it. Prefill rows whose unique-expert union exceeds the capacity are chunked greedily by rows (exact).
	34	+Alternatives measured: per-expert `quantized_matmul` loop 0.43 ms/layer vs 0.26 ms for gather_qmm (1 token, top-8);
	35	+re-stacking 8 experts per call 0.50 ms/layer. Consequence: the host must read the router's choice (`tolist()`) before each
	36	+layer's expert kernels — the unavoidable synchronization of demand loading, shared with Colibri; its cost is measured in Gate C.
	37	+
	38	+## D-006 · 2026-11-01 · Activation parity
	39	+Decision: use mlx-lm's compiled `swiglu` (silu(gate)·up in one fused kernel). Rationale: an unfused fp16 `up·sigmoid(gate)·gate`
	40	+gave max \|Δlogit\| = 0.0195 vs mlx-lm; the fused version gives 0.0 (bit-identical) — see `benchmarks/raw/tests/numerical_*.json`.

added research/LOG.md +39 −0

		`@@ -0,0 +1,39 @@`
	1	+# TITAN-MoE research log (WP33 UQO)
	2	+
	3	+Chronological, append-only. Times are local (America/Toronto). Every entry names the artifact it produced.
	4	+
	5	+## 2026-11-01 — Session 1: mandate received, Gate A started
	6	+
	7	+- 13:xx Mandate pasted by the author (verbatim copy in `docs/MANDATE.md`). Project placed in the UQO working-paper series as
	8	+ WP33, directory `Recherche-UQO/ia/titan-moe/` (series convention; the mandate's `wp33_titan_moe/` name is kept as the
	9	+ repository name `wp33_uqo.git` on spbgit — see DECISIONS D-001).
	10	+- Scaffold created from `_modele-working-paper` (UQO LaTeX template, Makefile, .latexmkrc, title pages). Git repo initialised (`main`).
	11	+- Python 3.12 venv (`uv`): mlx 0.32.4, mlx-lm 0.32.2, numpy 2.5.4 (laptop). Lock file to be written at Gate B.
	12	+- Reference repositories shallow-cloned into `third_party/` (git-ignored) with commit hashes recorded in `research/prior_art.csv`:
	13	+ colibri `0a5504a` (2026-10-31), mlx-lm `736ab0d` (2026-10-30), mlx `bd63d38` (2026-11-01, version header 0.32.5), llama.cpp `a91eab6` (2026-10-31).
	14	+- Hardware inventory of the MacLustr LAN launched (29 aliases). Results in `artifacts/environment.json` (sanitized) and
	15	+ `artifacts/private/inventory-2026-11-01.txt` (not committed). 7 aliases unreachable at probe time (SSH timeout):
	16	+ M2M32, M2M32b, M4M36, m4ma, m4mb, m4mc, m2m16; M4BP48 answered with an empty record. Treated as unavailable for WP33 unless
	17	+ they come back; not assumed from historical notes.
	18	+- Literature search delegated to a web-research sub-agent (output → `research/RELATED_WORK.md`, `research/prior_art.csv`, `paper/references.bib`),
	19	+ to be verified entry by entry before use.
	20	+- Small open-licensed MoE chosen for Gate B: `mlx-community/OLMoE-1B-7B-0125-Instruct-4bit` (Apache-2.0, 3.9 GB, revision `e2384419`),
	21	+ download started 2026-11-01 (laptop HF cache).
	22	+- OLMoE-1B-7B-0125-Instruct-4bit downloaded (33 s). Layout read from `model.safetensors` (`benchmarks/raw/gateA/smoke/`):
	23	+ per-expert tensors `model.layers.L.mlp.experts.E.{gate,up,down}_proj.{weight,scales,biases}` (affine 4-bit, group 64;
	24	+ gate/up weight U32 [1024,256] = 1.05 MB + scales/biases F16 2×64 KB; down [2048,128] = 1.05 MB + 2×64 KB) → 3.375 MiB per expert,
	25	+ 1 024 experts = 3.625 GB; attention 0.151 GB; embeddings/head 0.116 GB (tied). Per decode token: 16 layers × top-8 = 128 experts = 432 MiB of weights touched.
	26	+- Smoke baseline mlx-lm `generate` on the laptop (M5 Max, 48 GB, mlx 0.32.4, mlx-lm 0.32.2, temp 0, 33-token prompt, 85 generated):
	27	+ prompt 45.0 tok/s (includes first-call compilation), generation 493.9 tok/s, peak memory 4.11 GB. Raw: `benchmarks/raw/gateA/smoke/mlx_lm_generate_laptop_*.txt`.
	28	+- SSD probe `benchmarks/scripts/ssd_bandwidth.py` (single-threaded `pread`, F_NOCACHE after `sudo purge`): laptop random 3.4 MB reads
	29	+ 8.50 GB/s (p50 0.43 ms / chunk, p99 0.49 ms), page-cached 28.4 GB/s, sequential 8 MB reads 11.5 GB/s. Lesson: F_NOCACHE alone does
	30	+ not evict already-cached pages — `purge` is mandatory before a cold measurement (first attempt read 37 GB/s = cache). Raw: `benchmarks/raw/gateA/ssd/laptop-M5Max.json`.
	31	+- Design microbenchmark for the slot table (`benchmarks/raw/gateB/design/slot_update_microbench_laptop.txt`): in-place slot
	32	+ update 0.305 ms; gather_qmm 0.260 ms/layer; per-expert loop 0.428 ms; restack 0.502 ms; lazy mmap expert load 0.53 ms (cached).
	33	+- Gate B implemented: `titan/registry.py` (safetensors byte ranges, routing semantics), `titan/store.py` (pread SSD tier with
	34	+ accounting), `titan/slots.py` (LRU/LFU slot tables, stall accounting), `titan/moe.py` (TitanSwitchGLU, exact row chunking),
	35	+ `titan/runtime.py` (mlx-lm host model with experts spliced out). Tests `tests/` (7 passed, raw `benchmarks/raw/tests/pytest_gateB_20261101T011805.txt`):
	36	+ registry; pread bytes == `mx.load` bytes; logits vs mlx-lm max \|Δ\| = 0.0 for capacity 64/32/16/8 on 4 prompts (argmax equal);
	37	+ greedy 32-token decode token-identical with 16 slots. First unfused activation attempt gave Δ = 0.0195 (fixed, D-006).
	38	+- Literature pass completed: 59 verified works → `research/prior_art.csv`, `paper/references.bib` (59 entries), synthesis and
	39	+ novelty matrix in `research/RELATED_WORK.md`.

added research/RELATED_WORK.md +60 −0

		`@@ -0,0 +1,60 @@`
	1	+# Related work and novelty matrix (Gate A, 2026-11-01)
	2	+
	3	+Sources: `research/prior_art.csv` (59 works, each verified on its arXiv / publisher / GitHub page on 2026-11-01 by a
	4	+web-research pass; evidence URL per row) and a direct code/doc audit of the four reference repositories shallow-cloned in
	5	+`third_party/` (hashes below). Venue fields left empty in the CSV were not verifiable (ACM DL blocked fetches); they are
	6	+cited as arXiv preprints until re-verified at Gate G.
	7	+
	8	+\| repo \| commit \| date \| what we read \|
	9	+\|---\|---\|---\|---\|
	10	+\| JustVugg/colibri \| `0a5504a1dc64f86fcd0fe7db298b0448c730e7e6` \| 2026-10-31 \| README (How it works, Benchmarks), docs/cluster.md, docs/CACHE_ROUTE.md, docs/metal.md, docs/METAL-M5MAX-PERF-REPORT.md, docs/routing-telemetry.md (titles) \|
	11	+\| ml-explore/mlx \| `bd63d3803008edda03ce74080955d9879e2ef1eb` (version.h 0.32.5) \| 2026-11-01 \| mlx/distributed/{ring,mpi,jaccl,nccl}, docs/src/usage/distributed.rst (JACCL = RDMA over Thunderbolt 5, macOS ≥ 26.2, `rdma_ctl enable`), ops `gather_qmm`/`gather_mm` \|
	12	+\| ml-explore/mlx-lm \| `736ab0d8c1d0edc79c3d798044fe6904e2fa23be` \| 2026-10-30 \| models/switch_layers.py (SwitchGLU → `gather_qmm`, sort when ≥ 64 rows), models/olmoe.py, qwen3_moe.py (PipelineMixin), generate.py (wired limit, async eval), 50+ MoE architectures \|
	13	+\| ggml-org/llama.cpp \| `a91eab67d8389d979d29fdf6633b2edd0718eb4b` \| 2026-10-31 \| common/arg.cpp (`--cpu-moe`, `--n-cpu-moe`, `--override-tensor`, `--rpc`), tools/rpc, ggml-metal backend \|
	14	+
	15	+## 1. What is already known (and therefore not claimed)
	16	+
	17	+Expert offloading and SSD streaming. Per-expert offloading with an LRU cache was established by Eliseev & Mazur (2023) and
	18	+MoE-Infinity (Xue et al. 2024); SSD-tier streaming is demonstrated in Colibri (C engine: per-layer LRU + learned pinned hot set
	19	+`.coli_usage`, one `pread` per expert, O_DIRECT, loader pool, Metal backend opt-in, 2.24 tok/s GLM-5.2 int4 on an M5 Max 128 GB),
	20	+SSD-LLaMA (Liang et al. 2026) and FlashMoE (Kim et al. 2026, learned replacement beating LRU/LFU). Colibri explicitly states the
	21	+design principle we adopt: placement only decides speed, never which experts run.
	22	+
	23	+Expert prediction. Pre-gated MoE (Hwang et al., ISCA 2024) and Read-ME (Cai et al. 2024) change the architecture to route one
	24	+layer ahead; Fate (Fang et al. 2025), ProMoE, ExpertFlow (He et al. 2026), LayerScope, DuoServe, SpecPrefetch and SeqMoE
	25	+(2025–2026) learn cross-layer predictors on GPUs/NPUs; Colibri's `PILOT` prefetches the next layer from the router lookahead
	26	+(GLM-5.2: 71.6 % predictable one layer ahead, per README). Liang et al. (ICLR 2026) quantify routing locality and warn that
	27	+not all models suit offloading. Cache-aware routing (Skliar et al. 2025, arXiv 2412.00099; Colibri `CACHE_ROUTE`) changes the
	28	+expert choice and is therefore excluded from TITAN-MoE by construction (correctness constraint).
	29	+
	30	+Distributed inference on Macs. MLX ships ring (TCP/Thunderbolt), MPI and JACCL (RDMA over Thunderbolt 5, macOS ≥ 26.2,
	31	+Mac Studio); Apple reports 816B-parameter MoE inference across 4 × M5 Ultra (tensor/pipeline sharding of whole layers). exo,
	32	+prima.cpp, Petals and llama.cpp RPC shard layers or tensors, not experts. Colibri's cluster mode delegates routed experts to
	33	+remote "expert workers" over persistent TCP with a deterministic weighted hash (weights ∝ measured disk speed), batch-union
	34	+requests, q8 activations (not token-exact) and timeouts; it has no topology or latency model and no replication.
	35	+
	36	+Apple Silicon characterization. powermetrics-based energy per token for dense models (GreenBench, Silicon Showdown),
	37	+MoE on M-series with all experts resident (NPUMoE/ANE, BaseRT Metal, Alfarizy et al.). Silicon Showdown reports MoE on Apple
	38	+Silicon is latency-bound rather than bandwidth-bound when resident — a finding our Gate B baseline is consistent with
	39	+(OLMoE 4-bit: 494 tok/s decode on M5 Max with 432 MiB of expert weights touched per token, i.e. ≈ 0.2 TB/s effective).
	40	+
	41	+## 2. Novelty matrix (hypotheses vs prior art)
	42	+
	43	+\| TITAN-MoE element \| closest prior art \| already shown there? \| what remains open (our claim, to be tested) \|
	44	+\|---\|---\|---\|---\|
	45	+\| H1 causal cross-layer/cross-token prefetch on Apple Silicon with an SSD tier \| Colibri PILOT (1 layer ahead), Fate, SeqMoE, FlashMoE (GPU/x86) \| prediction works on GPUs; Colibri has a single heuristic \| head-to-head of no-prefetch / LRU / LFU / transition / learned predictors under identical Metal kernels and SSD, net of predictor overhead, on held-out prompt families \|
	46	+\| H2 topology-aware expert placement on heterogeneous Macs \| ExFlow (GPU affinity), OrderMoE (edge servers), Colibri cluster (hash by disk speed) \| datacenter GPUs only; Colibri has no topology model \| cost-model placement using measured per-node compute, RAM, SSD, link latency/bandwidth; comparison with static/uniform/frequency/capacity placement \|
	47	+\| H3 remote RAM vs local SSD as the second tier \| MLX JACCL, exo, llama.cpp RPC (whole shards) ; SSD-LLaMA, Klotski (local SSD) \| never compared as expert cache tiers \| measured tier model (10 GbE / Thunderbolt vs NVMe) and a coordinated 3-tier policy under fixed budgets \|
	48	+\| H4 fused Metal kernels for quantized grouped MoE \| MLX `gather_qmm`, llama.cpp `mul_mat_id`, BaseRT, Colibri Metal \| strong stock kernels exist \| gains only if profiling shows a bottleneck; end-to-end (not kernel-only) comparison with explicit tolerances \|
	49	+\| H5 latency × throughput × energy Pareto with adaptive orchestration \| GreenBench, Silicon Showdown (dense, single node) \| no energy for offloaded/distributed MoE on Macs \| powermetrics-based J/token across budgets, nodes and policies \|
	50	+
	51	+Honest expectations recorded before experiments: on a 48–96 GB node a small MoE is fully resident and no cache policy can
	52	+beat the all-resident baseline (H1/H3 only matter when the expert set exceeds the budget); distribution over Ethernet adds
	53	+≥ 100 µs per layer of RTT, so a 16-layer model would lose ≥ 1.6 ms per token versus 2 ms per token local decode — H2 can only
	54	+win at scales where a single node cannot hold the model or where SSD streaming is slower than the link. These are the regimes
	55	+the protocol targets (medium/large tiers, memory budgets < 100 %).
	56	+
	57	+## 3. Baselines and comparability (from the audit)
	58	+- mlx-lm: same weights, same kernels → exact comparison (Gate B shows bit-identical logits).
	59	+- llama.cpp Metal: different quantization (K-quants) → speed reported with a precision flag + quality check.
	60	+- Colibri on macOS: OLMoE engine is CPU int8 (Metal backend covers GLM/Inkling/Kimi engines only) → reference only.

added research/prior_art.csv +60 −0

		`@@ -0,0 +1,60 @@`
	1	+key,authors,year,title,venue,url,category,hardware,precision,offload_tiers,prediction_method,distributed,metal_or_apple,limitations_or_notes,evidence_url_seen
	2	+eliseev2023fast,Artyom Eliseev; Denis Mazur,2023,Fast Inference of Mixture-of-Experts Language Models with Offloading,arXiv technical report (no venue),https://arxiv.org/abs/2312.17238,expert offloading,"consumer desktop GPU, free-tier Colab T4",mixed quantization (HQQ; attention and experts quantized separately),"GPU <- CPU RAM (per-expert, per-layer)",LRU expert cache across adjacent tokens; speculative expert prefetch listed as future work in repo,no,no,"Mixtral-8x7B only; CUDA; repo github.com/dvmazur/mixtral-offloading (MIT, ~2.3k stars)",https://arxiv.org/abs/2312.17238 ; https://github.com/dvmazur/mixtral-offloading
	3	+xue2024moeinfinity,Leyang Xue; Yao Fu; Zhan Lu; Luo Mai; Mahesh Marina,2024,MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache,(not stated on arXiv page; v3 Mar 2025),https://arxiv.org/abs/2401.14361,expert offloading,"personal machine, single GPU, batch size 1",not stated,GPU <- host memory,sparsity-aware expert cache built from traced expert-activation traces (representative traces guide replacement + prefetch),no,no,venue empty (unverified),https://arxiv.org/abs/2401.14361
	4	+hwang2024pregated,Ranggi Hwang; Jianyu Wei; Shijie Cao; Changho Hwang; Xiaohu Tang; Ting Cao; Mao Yang,2024,Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference,ISCA 2024 (51st IEEE/ACM ISCA; DOI 10.1109/ISCA59077.2024.00078 per UNIST record + MSR camera-ready PDF),https://arxiv.org/abs/2308.12066,predictive routing / offloading,single GPU + CPU memory,not stated,GPU <- CPU memory,"pre-gating: layer l gate computed from layer l-1 activations (model architecture change, requires fine-tuning)",no,no,requires modifying the MoE architecture; DOI seen only in secondary record,https://arxiv.org/abs/2308.12066 ; https://www.microsoft.com/en-us/research/uploads/prodnew/2024/05/isca24_pregated_moe_camera_ready.pdf
	5	+yi2025edgemoe,Rongjie Yi; Liwei Guo; Shiyun Wei; Ao Zhou; Shangguang Wang; Mengwei Xu,2025,EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices,"IEEE Transactions on Mobile Computing 24(8):7059-7073, DOI 10.1109/TMC.2025.3546466 (arXiv v1 2023)",https://arxiv.org/abs/2308.14352,expert offloading (mobile),mobile/edge devices,per-expert bitwidth adaptation,device memory <- external storage (flash),"expert preloading: predicts likely experts, compute-I/O pipeline",no,no,mobile SoCs not Apple,https://arxiv.org/abs/2308.14352 ; https://api.openalex.org/works/doi:10.1109/TMC.2025.3546466
	6	+zhong2024adapmoe,Shuzhang Zhong; Ling Liang; Yuan Wang; Runsheng Wang; Ru Huang; Meng Li,2024,AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference,"ICCAD 2024 (inferred from related DOI 10.1145/3676536.3676741; ACM page returned 403, unverified)",https://arxiv.org/abs/2408.10284,expert offloading / adaptive gating,edge GPU,not stated,GPU <- host,sensitivity-based dynamic number of activated experts + prefetch + cache management,no,no,venue inferred from DOI prefix only,https://arxiv.org/abs/2408.10284
	7	+tang2024hobbit,Peng Tang; Jiacheng Liu; Xiaofeng Hou; Yifei Pu; Jing Wang; Pheng-Ann Heng; Chao Li; Minyi Guo,2024,HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference,(not stated; venue unverified),https://arxiv.org/abs/2411.01433,expert offloading,memory-limited edge devices; built on llama.cpp,mixed precision (low-precision substitutes for cache-miss experts),device <- host/storage via llama.cpp,"token-level dynamic loading, layer-level adaptive prefetch, sequence-level multidimensional cache",no,indirectly (llama.cpp codebase) — no Apple results stated,up to 9.93x decode speedup claimed,https://arxiv.org/abs/2411.01433
	8	+kamahori2025fiddler,Keisuke Kamahori; Tian Tang; Yile Gu; Kan Zhu; Baris Kasikci,2025,Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models,ICLR 2025,https://arxiv.org/abs/2402.07033,CPU-GPU hybrid execution,single consumer GPU + CPU,not stated,GPU + CPU (compute experts on CPU instead of moving weights),none (executes misses on CPU),no,no,x86 CPU + NVIDIA,https://arxiv.org/abs/2402.07033
	9	+cao2025moelightning,Shiyi Cao; Shu Liu; Tyler Griggs; Peter Schafhalter; Xiaoxuan Liu; Ying Sheng; Joseph E. Gonzalez; Matei Zaharia; Ion Stoica,2025,MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs,ASPLOS 2025 (per Zaharia publication list; ACM DOI not obtained),https://arxiv.org/abs/2411.11217,throughput-oriented offloading,T4 16GB and 2-4 T4,not stated,"GPU <- CPU <- disk I/O (CGOPipe, paged weights)",Hierarchical Roofline performance model (no routing prediction),multi-GPU single host,no,"batch-throughput, not latency",https://arxiv.org/abs/2411.11217
	10	+du2024sida,Zhixu Du; Shiyu Li; Yuhao Wu; Xiangyu Jiang; Jingwei Sun; Qilin Zheng; Yongkai Wu; Ang Li; Hai Li; Yiran Chen,2024,SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models,MLSys 2024,https://arxiv.org/abs/2310.18859,predictive offloading,GPU + main memory,not stated,GPU <- main memory,offline-trained hash/predictor of expert activation from input data,no,no,,https://arxiv.org/abs/2310.18859
	11	+song2024promoe,Xiaoniu Song; Zihang Zhong; Rong Chen; Haibo Chen,2024,ProMoE: Fast MoE-based LLM Serving using Proactive Caching,(not stated; v3 Sep 2025),https://arxiv.org/abs/2410.22134,predictive prefetching,edge device with limited GPU memory,not stated,GPU <- host memory,"learned predictor using intermediate results to prefetch future-layer experts (proactive, off critical path)",no,no,,https://arxiv.org/abs/2410.22134
	12	+he2026expertflow,Xin He; Shunkang Zhang; Kaijie Tang; Shaohuai Shi; Yuxin Wang; Zihao Zeng; Zhenheng Tang; Xiaowen Chu; Haiyan Yin; Ivor W. Tsang; Yew Soon Ong,2026,ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling,DAC 2026 (arXiv Comments),https://arxiv.org/abs/2410.17954,predictive caching,single GPU,not stated,GPU <- CPU memory,transformer-based routing-path predictor for all layers in one pass + token scheduler,no,no,"arXiv v1 2024, v2 Apr 2026",https://arxiv.org/abs/2410.17954
	13	+yu2025finemoe,Hanfei Yu; Xingqi Cui; Hong Zhang; Hao Wang; Hao Wang,2025,Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading,(not stated),https://arxiv.org/abs/2502.05370,fine-grained offloading (fMoE/FineMoE),"6-GPU testbed, HF Transformers",not stated,GPU <- CPU,fine-grained expert-selection patterns + semantic hints from prompts,multi-GPU host,no,"author list shows ""Hao Wang"" twice on arXiv",https://arxiv.org/abs/2502.05370
	14	+zhang2025daop,Yujie Zhang; Shivam Aggarwal; Tulika Mitra,2025,DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference,DATE 2025 (arXiv Comments),https://arxiv.org/abs/2501.10375,CPU-GPU hybrid + prediction,CPU + GPU,not stated,GPU + CPU,per-sequence activation pattern assignment; predicted experts pre-computed on CPU; graceful degradation,no,no,,https://arxiv.org/abs/2501.10375
	15	+xu2025moegen,Tairan Xu; Leyang Xue; Zhan Lu; Adrian Jackson; Luo Mai,2025,MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching,(not stated),https://arxiv.org/abs/2503.09716,throughput offloading,single GPU + host memory,not stated,GPU <- host,none (module-based batching),no,no,"throughput, not single-user latency",https://arxiv.org/abs/2503.09716
	16	+fang2025klotski,Zhiyuan Fang; Yuegui Huang; Zicong Hong; Yufeng Lyu; Wuhui Chen; Yue Yu; Fan Yu; Zibin Zheng,2025,Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline,(not stated),https://arxiv.org/abs/2502.06888,offloading pipeline,GPU + CPU + disk,not stated,GPU <- CPU <- disk,correlation-aware expert prefetcher,no,no,,https://arxiv.org/abs/2502.06888
	17	+tairin2025emoe,Suraiya Tairin; Shohaib Mahmud; Haiying Shen; Anand Iyer,2025,eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference,(not stated),https://arxiv.org/abs/2503.06823,predictive loading,GPU,not stated,not stated on abstract,recurring routing-pattern predictor run every few prompts; task-aware skip,no,no,,https://arxiv.org/abs/2503.06823
	18	+suo2025coserve,Jiashun Suo; Xiaojian Liao; Limin Xiao; Li Ruan; Jinquan Wang; Xiao Su; Zhisheng Huo,2025,CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory,ASPLOS 2025 (arXiv journal-ref),https://arxiv.org/abs/2503.02354,CoE offloading,heterogeneous CPU+GPU,not stated,memory <-> storage,expert-dependency-aware scheduler + offline profiler for processor/device allocation,no,no,"Collaboration-of-Experts, not classic token-routed MoE",https://arxiv.org/abs/2503.02354
	19	+skliar2025cacheaware,Andrii Skliar; Ties van Rozendaal; Romain Lepert; Todor Boinovski; Mart van Baalen; Markus Nagel; Paul Whatmough; Babak Ehteshami Bejnordi,2025,Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference,TMLR (June 2025),https://arxiv.org/abs/2412.00099,cache-aware routing,"mobile device, DRAM-limited",not stated,DRAM <- storage,training-free cache-aware routing that biases router toward cached experts (~2x speedup),no,no,Qualcomm; no Apple,https://arxiv.org/abs/2412.00099
	20	+alizadeh2024flash,Keivan Alizadeh; Iman Mirzadeh; Dmitry Belenko; Karen Khatamifard; Minsik Cho; Carlo C Del Mundo; Mohammad Rastegari; Mehrdad Farajtabar,2024,LLM in a flash: Efficient Large Language Model Inference with Limited Memory,ACL 2024,https://arxiv.org/abs/2312.11514,"flash/SSD streaming (dense, Apple)",Apple M1 Max + consumer GPU (per paper; abstract says flash memory),not stated,DRAM <- flash,windowing (recent-neuron reuse) + row-column bundling; flash cost model,no,yes (Apple authors; M1 Max evaluation in paper body — not on abstract),"dense FFN sparsity, not MoE experts",https://arxiv.org/abs/2312.11514
	21	+song2024powerinfer,Yixin Song; Zeyu Mi; Haotong Xie; Haibo Chen,2024,PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU,SOSP 2024,https://arxiv.org/abs/2312.12456,hot/cold neuron locality,consumer GPU + CPU,not stated,GPU (hot neurons) + CPU (cold),adaptive activation predictors,no,no,"dense ReLU-sparsity, context for locality arguments",https://arxiv.org/abs/2312.12456
	22	+huang2023towards,Haiyang Huang; Newsha Ardalani; Anna Sun; Liu Ke; Hsien-Hsin S. Lee; Anjali Sridhar; Shruti Bhosale; Carole-Jean Wu; Benjamin Lee,2023,Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference,(not stated),https://arxiv.org/abs/2303.06182,characterization / expert buffering,GPU + CPU,not stated,GPU (frequent experts) <- CPU memory,dynamic gating + expert buffering by frequency,no,no,Meta; early locality characterization,https://arxiv.org/abs/2303.06182
	23	+cai2024readme,Ruisi Cai; Yeonju Ro; Geon-Woo Kim; Peihao Wang; Babak Ehteshami Bejnordi; Aditya Akella; Zhangyang Wang,2024,Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design,NeurIPS 2024,https://arxiv.org/abs/2410.19123,pre-gating router / lookahead,GPU,not stated,not stated,"decoupled pre-gating router computed ahead -> lookahead scheduling, expert-aware batching/caching",no,no,requires refactorized model,https://arxiv.org/abs/2410.19123
	24	+yao2024exflow,Jinghan Yao; Quentin Anthony; Aamir Shafi; Hari Subramoni; Dhabaleswar K. Panda,2024,Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference,(not stated),https://arxiv.org/abs/2401.08383,expert-parallel placement (ExFlow),multi-GPU,not stated,none,conditional inter-layer expert affinity -> integer-programming expert-to-GPU placement,yes (expert parallel),no,"placement, not offloading",https://arxiv.org/abs/2401.08383
	25	+fang2025fate,Zhiyuan Fang; Zicong Hong; Yuegui Huang; Yufeng Lyu; Wuhui Chen; Yue Yu; Fan Yu; Zibin Zheng,2025,Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate,(not stated),https://arxiv.org/abs/2502.12224,cross-layer expert prediction,edge device,quantization for cache/I/O,device <- host/storage,adjacent-layer gate inputs predict next-layer experts (97% prefetch acc.) + shallow-favoring cache,no,no,,https://arxiv.org/abs/2502.12224
	26	+liang2026notall,Jingcong Liang; Siyuan Wang; Miren Tian; Yitong Li; Duyu Tang; Zhongyu Wei,2026,Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models,ICLR 2026 (iclr.cc poster page; arXiv Comments empty),https://arxiv.org/abs/2505.16056,routing locality study,n/a (20 MoE LLMs analysed),n/a,n/a,metrics SRP/SCH; finds cache ~2x active experts; domain-specialized experts raise consistency,no,no,analysis only,https://arxiv.org/abs/2505.16056 ; https://iclr.cc/virtual/2026/poster/10011732
	27	+zhong2025hybrimoe,Shuzhang Zhong; Yanfan Sun; Ling Liang; Runsheng Wang; Ru Huang; Meng Li,2025,HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference,DAC 2025 (arXiv Comments),https://arxiv.org/abs/2504.05897,CPU-GPU hybrid,CPU + GPU,not stated,GPU <- host,impact-driven prefetch + score-based cache,no,no,,https://arxiv.org/abs/2504.05897
	28	+huang2026cpugpu,En-Ming Huang; Li-Shang Lin; Chun-Yi Lee,2026,Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems,ASP-DAC 2026 (arXiv Comments),https://arxiv.org/abs/2512.16473,CPU-GPU hybrid,consumer CPU + GPU,not stated,GPU cache + CPU execution of misses,frequency cache; argues prefetch cannot hide transfer,no,no,,https://arxiv.org/abs/2512.16473
	29	+zhao2026mobile,Yushu Zhao; Yubin Qin; Yang Wang; Xiaolong Yang; Huiming Han; Shaojun Wei; Yang Hu; Shouyi Yin,2026,MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts,ASP-DAC 2026 (arXiv Comments),https://arxiv.org/abs/2510.12357,offloading + expert substitution,consumer GPU,not stated,GPU <- CPU,"small ""little"" experts for unimportant tokens + prefetch/fallback",no,no,,https://arxiv.org/abs/2510.12357
	30	+yu2026layerscope,Enda Yu; Dezun Dong; Zhaoning Zhang; Zhe Bai; Weiling Yang; Haojie Wang; Dongsheng Li; Yongwei Wu; Xiangke Liao,2026,LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers,ICS 2026 (arXiv Comments),https://arxiv.org/abs/2509.23638,cross-layer prediction,"legacy GPU servers, PCIe",not stated,GPU <- CPU,learnable layer-aware predictor + cross-layer prefetch scheduling,no,no,"arXiv title earlier ""PreScope""",https://arxiv.org/abs/2509.23638
	31	+shen2025expertflow,Zixu Shen; Kexin Chu; Yifan Zhang; Dawei Xiang; Runxin Wu; Wei Zhang,2025,ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference,(not stated),https://arxiv.org/abs/2510.26730,adaptive prefetch horizon,memory-constrained GPU,not stated,GPU <- host,pregating + intermediate states; runtime-adaptive prefetch distance; cache-aware routing,no,no,name collision with he2026expertflow,https://arxiv.org/abs/2510.26730
	32	+zhang2025duoserve,Yuning Zhang; Grant Pinkert; Nan Yang; Yanli Li; Dong Yuan,2025,DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance,(not stated),https://arxiv.org/abs/2509.07379,prefill/decode-phase prefetch,CUDA GPU,not stated,GPU <- host,offline-trained layer-level predictor for decode; two-stream prefill pipeline,no,no,,https://arxiv.org/abs/2509.07379
	33	+kim2026flashmoe,Byeongju Kim; Jungwan Lee; Donghyeon Han; Hoi-Jun Yoo; Sangyeob Kim,2026,FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices,(not stated),https://arxiv.org/abs/2601.17063,SSD streaming + learned cache,user-grade desktop,not stated,RAM <- SSD,learned replacement (recency+frequency) beats LRU/LFU by up to 51% hit rate,no,no,"closest ""LRU vs learned"" comparison found",https://arxiv.org/abs/2601.17063
	34	+pan2026trimoe,Yudong Pan; Yintao He; Tianhua Han; Lian Liu; Shixin Zhao; Zhirong Chen; Mengdi Wang; Cangyuan Li; Yinhe Han; Ying Wang,2026,TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading,DAC 2026 (arXiv Comments),https://arxiv.org/abs/2603.01058,heterogeneous offloading,GPU + AMX CPU + DIMM NDP,not stated,GPU / CPU / near-memory,hot/warm/cold expert classification + prediction-driven rebalancing,no,no,server-class,https://arxiv.org/abs/2603.01058
	35	+kong2026specprefetch,Jinwei Kong; Runqi Meng; Fanyi Wang; Wentao Qiu; Haotian Hu; Yongjian Zhou; Zhenhua Ge,2026,SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models,(not stated),https://arxiv.org/abs/2607.24787,cross-layer prediction (mobile),Snapdragon 8 Elite,not stated,accelerator <- host/storage,small shared adapter predicts next-layer experts; window-aware transfer scheduler,no,no,,https://arxiv.org/abs/2607.24787
	36	+wang2026seqmoe,Zihan Wang; Yuqi Wang; Lei Gong; Cheng Tang; Wenqi Lou; Teng Wang; Chao Wang; Xuehai Zhou,2026,SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading,(not stated),https://arxiv.org/abs/2609.12978,multi-step prediction,device with ~45% experts resident,not stated,device <- host,seq2seq model forecasting experts multiple steps and layers ahead; probabilistic Belady eviction,no,no,,https://arxiv.org/abs/2609.12978
	37	+liang2026ssdllama,Fangzhou Liang; Yibin Shen; Jianmin Hu; Jiayang Xu; Hanchi Gao; Minxian Xu; Zili Meng,2026,SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC,(not stated),https://arxiv.org/abs/2609.18110,SSD streaming,RTX 5090 + <=32 GB RAM,quantized (unspecified) + lossless compression,VRAM <- RAM <- SSD (3 tiers),not stated; argues prefetch alone insufficient,no,no,,https://arxiv.org/abs/2609.18110
	38	+yang2026moecore,Ke Yang; Yongji Gao; Xushi Li; Kui Luo; Sicheng Zhang; Tianming Zhou; Keyi Liu; Shufang Lu; Aoxuan Chen; Jie Meng; Jingchun Gao; Dan Li; Xinkai You; Zhixiang Xia; Yan Shi; Yang Liu; Yanjia Zeng; Liangjun Feng,2026,MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference,(not stated),https://arxiv.org/abs/2610.01950,cross-layer prefetch + residency,NPU with 84 GB cap,not stated,device <- host,cross-layer prefetcher (1 expert/layer default) + routing-history-aware per-layer cache,no,no,"""Dan Li"" listed twice on arXiv",https://arxiv.org/abs/2610.01950
	39	+yuan2026ordermoe,Xin Yuan; Ning Li; Quan Chen; Wenchao Xu; Song Guo,2026,OrderMoE: An expert similarity driven distributed edge MoE inference,(not stated),https://arxiv.org/abs/2607.17154,distributed expert placement,"edge servers, bandwidth-limited",not stated,none,expert-similarity grouping -> placement; runtime local-substitute selection,yes (multi-server expert placement),no,no device heterogeneity model,https://arxiv.org/abs/2607.17154
	40	+benazir2026npumoe,Afsara Benazir; Felix Xiaozhu Lin,2026,Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs,(not stated),https://arxiv.org/abs/2604.18788,Apple Silicon MoE runtime (ANE),Apple M2 / M2 Ultra (Neural Engine),not stated,unified memory (all experts resident),offline calibration of expert capacity/popularity; load-aware graph residency,no,"yes (ANE + CPU/GPU fallback, energy mJ/token)",assumes all experts fit in unified memory — no SSD tier,https://arxiv.org/abs/2604.18788
	41	+alfarizy2026doesmoe,Alfarizy Alfarizy; Hung Truong Thanh Nguyen; Rene Richard; Roozbeh Razavi-Far; Hung Cao,2026,Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study,submitted to FAIEMA 2026 (arXiv Comments),https://arxiv.org/abs/2606.21428,Apple Silicon MoE energy characterization,Apple M2 Pro laptop; Jetson Orin Nano,llama.cpp GGUF,none,none,no,"yes (M2 Pro, llama.cpp)",OLMoE only; energy tracks total params,https://arxiv.org/abs/2606.21428
	42	+kannan2026greenbench,Rajeswari Kannan; Raj Firke; Shreya Bengle; Srushti Deshmukh,2026,GreenBench: Benchmarking Energy Efficiency and Carbon Footprint of Open-Source LLM Inference on Apple Silicon,IEEE ICCUBEA 2026 (arXiv Comments),https://arxiv.org/abs/2608.28667,Apple Silicon energy (powermetrics),Apple M4 Pro 48 GB,Ollama defaults,none,none,no,yes (powermetrics),"dense 3-9B only, no MoE",https://arxiv.org/abs/2608.28667
	43	+javat2026siliconshowdown,Abdurrahman Javat; Allan Kazakov,2026,"Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference",(not stated),https://arxiv.org/abs/2605.00519,Apple vs NVIDIA tokens/joule,"M3 Ultra, M4 Pro vs Blackwell","NVFP4 vs BF16, quantized",CPU offload penalties measured,none,no,yes (powermetrics per search snippet; MoE latency-bound finding),powermetrics not on abstract page,https://arxiv.org/abs/2605.00519
	44	+rajesh2025productiongrade,Varun Rajesh; Om Jodhpurkar; Pooja Anbuselvan; Mantinder Singh; Ashok Jallepali; Shantanu Godbole; Pradeep Kumar Sharma; Hritvik Shrivastava,2025,"Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS",(not stated),https://arxiv.org/abs/2511.05502,Apple Silicon runtime characterization,Mac Studio M2 Ultra 192 GB,various,none,none,no,"yes (MLX, llama.cpp Metal)",no energy measurement; dense Qwen-2.5,https://arxiv.org/abs/2511.05502
	45	+rathnayaka2026basert,Prabod Rathnayaka; Fabian Waschkowski; Lukas Wesemann,2026,BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal,(not stated),https://arxiv.org/abs/2607.00501,Metal kernel runtime,"Apple M3, M4 Pro",not stated,unified memory,none,no,yes (native Metal kernel fusion; MoE prefill gains),no offloading,https://arxiv.org/abs/2607.00501
	46	+wilkins2024hybrid,Grant Wilkins; Srinivasan Keshav; Richard Mortier,2024,Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads,(not stated),https://arxiv.org/abs/2407.00010,energy scheduling,heterogeneous processors,not stated,none,cost-based scheduler,cluster-level,Apple Silicon/powermetrics not confirmed on abstract page (search snippet only),7.5% energy reduction,https://arxiv.org/abs/2407.00010
	47	+borzunov2023petals,Alexander Borzunov; Dmitry Baranchuk; Tim Dettmers; Maksim Riabinin; Younes Belkada; Artem Chumachenko; Pavel Samygin; Colin Raffel,2023,Petals: Collaborative Inference and Fine-tuning of Large Models,"ACL 2023 System Demonstrations, pp. 558-568, DOI 10.18653/v1/2023.acl-demo.54",https://aclanthology.org/2023.acl-demo.54/,distributed inference (volunteer),consumer GPUs over Internet,8-bit (per paper),none,none,yes (pipeline over WAN),no,dense BLOOM-176B,https://aclanthology.org/2023.acl-demo.54/ ; https://arxiv.org/abs/2209.01188
	48	+li2025primacpp,Zonghang Li; Tao Li; Wenjiao Feng; Rongxing Xiao; Jianshu She; Hong Huang; Mohsen Guizani; Hongfang Yu; Qirong Ho; Wei Xiang; Xue Liu,2025,Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters,"(not stated; ""38 pages"")",https://arxiv.org/abs/2504.08791,heterogeneous home cluster,"mixed CPUs/GPUs, Wi-Fi, slow disks",llama.cpp quants,disk I/O overlapped via pipelined-ring,Halda scheduler (per-device CPU/GPU split under memory limits),yes (pipelined ring),not stated on abstract,dense models; no expert-level placement,https://arxiv.org/abs/2504.08791
	49	+exo2025,exo labs (exo-explore),2025,exo: Run frontier AI locally (GitHub repository),GitHub,https://github.com/exo-explore/exo,distributed Apple cluster,"Apple Silicon Macs (M4 Pro mini, M4 Max Studio/MBP, M3 Ultra Studio)",MLX quants,none,none,"yes (topology-aware auto-parallel, tensor + pipeline, RDMA over TB5)","yes (MLX backend; macOS 15.1+, RDMA needs macOS 26.2)","no expert-level offload; RDMA requires full mesh and identical macOS builds; ~48k stars, Apache-2.0",https://github.com/exo-explore/exo
	50	+llamacpp2026,Georgi Gerganov and ggml-org contributors,2026,llama.cpp: LLM inference in C/C++ (GitHub repository; RPC backend),GitHub,https://github.com/ggml-org/llama.cpp,Metal kernels + RPC distributed,"Apple Silicon (Metal), CUDA, CPU",GGUF quant families,none (full model split across devices),none,yes (RPC: memory-proportional tensor split; RDMA over TB5 on macOS 26.2+),yes (Metal first-class),"RPC ""proof-of-concept"", insecure; no MoE expert placement; MIT, ~133k stars",https://github.com/ggml-org/llama.cpp ; https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md
	51	+hannun2023mlx,Awni Hannun; Jagrit Digani; Angelos Katharopoulos; Ronan Collobert,2023,MLX: Efficient and flexible machine learning on Apple silicon,GitHub software (repo citation block),https://github.com/ml-explore/mlx,Apple array framework,Apple Silicon unified memory,various,none,none,"yes (MPI / ring over Thunderbolt / JACCL RDMA backends, docs v0.32.4)",yes,"JACCL requires TB5 + macOS 26.2, rdma_ctl enable in Recovery, fully connected mesh only",https://github.com/ml-explore/mlx ; https://ml-explore.github.io/mlx/build/html/usage/distributed.html
	52	+apple2026mlxdistributed,Apple Machine Learning Research (no authors listed),2026,Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio,"Apple ML Research blog, 2026-10-15",https://machinelearning.apple.com/research/mlx-distributed-inference,official Apple distributed MoE result,Mac Studio M3 Ultra / M5 Ultra 512 GB; 2 and 4 M5 Ultra TB5 mesh,4-bit NVFP4 (Muse Spark Open ~493 GB),none (all in unified memory),none,yes (pipeline + tensor parallel via JACCL),yes,"Muse Spark Open 816B/42B-active: 15.7 (2 PP) / 25.0 (2 TP) / 36.7 tok/s (4 TP); MoE handled by TP, no expert parallel; no energy figures",https://machinelearning.apple.com/research/mlx-distributed-inference
	53	+apple2026wwdc233,Apple (WWDC26 session 233),2026,Explore distributed inference and training with MLX,WWDC 2026 session 233,https://developer.apple.com/videos/play/wwdc2026/233/,official RDMA/JACCL source,Mac clusters over Thunderbolt 5,n/a,none,none,"yes (mesh/ring topologies, tensor + pipeline parallel)",yes (RDMA over TB5 from macOS 26.2; JACCL open-source),"video session, no numbers extracted",https://developer.apple.com/videos/play/wwdc2026/233/
	54	+geerling2025rdma,Jeff Geerling,2025,1.5 TB of VRAM on Mac Studio - RDMA over Thunderbolt 5,"blog post, 2025-12-18",https://jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5,independent Mac cluster measurement,"4x M3 Ultra Mac Studio (2x512 GB + 2x256 GB), TB5 full mesh",MLX/llama.cpp quants,none,none,yes (exo 1.0 vs llama.cpp RPC),yes (macOS 26.2),"TB5 ~50-60 Gbps real; RDMA latency ~300us -> <50us; Qwen3-235B ~32 tok/s, Kimi K2 ~30 tok/s on 4 nodes; idle <10 W per Studio but no energy/token",https://jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5
	55	+rajbhandari2022deepspeedmoe,Samyam Rajbhandari; Conglong Li; Zhewei Yao; Minjia Zhang; Reza Yazdani Aminabadi; Ammar Ahmad Awan; Jeff Rasley; Yuxiong He,2022,DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale,ICML 2022 (arXiv Comments),https://arxiv.org/abs/2201.05596,expert-parallel serving (context),multi-GPU datacenter,FP16/INT8 (per paper),none,none,yes (expert + tensor + data parallel),no,,https://arxiv.org/abs/2201.05596
	56	+deepep2025,Chenggang Zhao et al. (DeepSeek-AI),2025,DeepEP: an efficient expert-parallel communication library (GitHub repository),GitHub (README BibTeX key deepep2025),https://github.com/deepseek-ai/DeepEP,expert-parallel all-to-all kernels,"NVIDIA Hopper+, NVLink + RDMA",FP8 dispatch,none,none,"yes (intra/inter-node all-to-all, low-latency decode kernels)",no,Linux/CUDA only; MIT,https://github.com/deepseek-ai/DeepEP
	57	+li2023lina,Jiamin Li; Yimin Jiang; Yibo Zhu; Cong Wang; Hong Xu,2023,Accelerating Distributed MoE Training and Inference with Lina,USENIX ATC 2023 (pp. 945-959),https://www.usenix.org/conference/atc23/presentation/li-jiamin,expert-parallel serving,A100 + 100 Gbps InfiniBand,not stated,none,expert-popularity-aware resource scheduling for inference,yes,no,,https://www.usenix.org/conference/atc23/presentation/li-jiamin ; https://arxiv.org/abs/2210.17223
	58	+hwang2023tutel,Changho Hwang; Wei Cui; Yifan Xiong; Ziyue Yang; Ze Liu; Han Hu; Zilong Wang; Rafael Salas; Jithin Jose; Prabhat Ram; Joe Chau; Peng Cheng; Fan Yang; Mao Yang; Yongqiang Xiong,2023,Tutel: Adaptive Mixture-of-Experts at Scale,(MLSys 2023 believed; NOT shown on arXiv page — unverified),https://arxiv.org/abs/2206.03382,adaptive parallelism (context),multi-GPU,not stated,none,none,yes,no,venue unverified,https://arxiv.org/abs/2206.03382
	59	+gale2023megablocks,Trevor Gale; Deepak Narayanan; Cliff Young; Matei Zaharia,2023,MegaBlocks: Efficient Sparse Training with Mixture-of-Experts,(MLSys 2023 believed; NOT shown on arXiv page — unverified),https://arxiv.org/abs/2211.15841,block-sparse MoE kernels (context),GPU,not stated,none,none,no,no,training-focused; venue unverified,https://arxiv.org/abs/2211.15841
	60	+colibri2026,Vincenzo Fornaro (GitHub: JustVugg),2026,"colibri: Tiny engine, immense model — local inference engine streaming routed MoE experts from disk (GitHub repository, v2.1.0)","GitHub, Apache-2.0, ~45k stars, 5,150 commits",https://github.com/JustVugg/colibri,SSD expert streaming engine (C),x86 benchmark machines (e.g. Ryzen AI Max+ 395 128 GB); Apple Silicon via Metal backend (no Mac benchmarks listed),"GGUF IQ1_S..Q8_0, F16, BF16, MXFP4; int4/int8 containers; fp4 routed experts (DeepSeek V4.1 Flash)",RAM (pinned hot set + per-layer LRU) <- disk; GPU via Metal/CUDA/Vulkan,"router-lookahead thread prefetches next layer (""71.6% predictable one layer ahead"" for GLM-5.2); pinned hot set learned from user's conversations (.coli_usage)","yes (cluster mode: TCP, coordinator + disk-backed expert workers, experts hashed by expert ID with disk-speed-weighted shares; no topology awareness)",yes (COLI_METAL=1; macOS builds),"Terms ""PILOT"" and ""CACHE_ROUTE"" NOT found in README, docs site, or cluster.md — do not cite them as Colibri features without checking source; no paper, no BibTeX, no citation; cluster benchmark only prefill 618s->454s on 2 SATA workers",https://github.com/JustVugg/colibri ; https://raw.githubusercontent.com/JustVugg/colibri/main/README.md ; https://github.com/JustVugg/colibri/blob/main/docs/cluster.md ; https://justvugg.github.io/colibri

added research/protocol.md +52 −0

		`@@ -0,0 +1,52 @@`
	1	+# Benchmark protocol (WP33 TITAN-MoE) — v0.1, 2026-11-01 (Gate A draft; frozen at Gate C before any optimization claim)
	2	+
	3	+## 1. Models (exact revisions pinned in `benchmarks/configs/models.yaml`)
	4	+\| tier \| model \| HF repo / revision \| licence \| layers \| experts \| top-k \| shared \| active / total params \| quantization \|
	5	+\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	6	+\| small \| OLMoE-1B-7B-0125-Instruct \| `mlx-community/OLMoE-1B-7B-0125-Instruct-4bit` @ `e2384419` \| Apache-2.0 \| 16 \| 64 \| 8 \| none \| 1.3B / 6.9B \| affine 4-bit, g64 (MLX) \|
	7	+\| medium (Gate C+, needs download authorization ≤ 20 GB → granted by default rule in CLAUDE.md: 17.2 GB) \| Qwen3-30B-A3B-Instruct-2507 \| `mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit` @ `e9675aa3` \| Apache-2.0 \| 48 \| 128 \| 8 \| none \| 3.3B / 30.5B \| affine 4-bit, g64 \|
	8	+\| large (Gate F, requires explicit authorization: 62 GB) \| gpt-oss-120b \| `mlx-community/gpt-oss-120b-MXFP4-Q4` @ `bce781be` \| Apache-2.0 \| 36 \| 128 \| 4 \| none \| 5.1B / 117B \| MXFP4 experts \|
	9	+
	10	+Tokenizers: the ones shipped in each repo. Decoding: greedy (temperature 0) for correctness and throughput; sampling only in
	11	+quality checks with fixed seeds.
	12	+
	13	+## 2. Baselines
	14	+- MLX-LM (`python -m mlx_lm generate`, same venv, same weights, same revision) — primary, exactly comparable precision.
	15	+- llama.cpp (`llama-bench`, `llama-cli`, Metal backend, commit `a91eab6`) on a GGUF of the same base checkpoint; the
	16	+ quantization scheme differs (Q4_K_M / Q4_0 vs MLX affine g64) → reported with a NON-COMPARABLE PRECISION flag and a quality
	17	+ check (perplexity on the same held-out text) next to the speed numbers.
	18	+- Colibri (`0a5504a`): OLMoE engine (int8 conversion, CPU; Vulkan/CUDA not on macOS; Metal backend only for GLM/Inkling/Kimi
	19	+ engines) → OLMoE on Colibri is CPU-only int8: reported as a different-precision, different-device reference, never as a
	20	+ same-conditions competitor. For the large tier (GLM-5.2 int4) Colibri is compared only if TITAN-MoE runs GLM-5.2 (not planned).
	21	+- Internal controls: TITAN reference backend with all experts resident (upper bound), no-prefetch demand loading, LRU, LFU,
	22	+ static placement, single-node.
	23	+
	24	+## 3. Workloads (`benchmarks/configs/workloads.yaml`)
	25	+Prompt families (held-out split by family, never used to fit predictors): general chat (Alpaca-style instructions), code
	26	+(HumanEval-like prompts), reasoning (GSM8K-style), long-context summarization (2k–8k tokens), and a "drift" family that
	27	+switches domain mid-session to test changing expert popularity. Output lengths 64 / 256 / 1024. Batch 1 (latency) and 4 / 8
	28	+(throughput) where the backend supports batching. Cold start = process start with purged page cache; warm = second run.
	29	+
	30	+## 4. Metrics (all logged per request as JSON lines in `benchmarks/raw/<gate>/<exp>/<run>.jsonl`)
	31	+prefill tok/s · decode tok/s · TTFT · inter-token latency p50/p95/p99 · total latency · expert hit rate (local RAM / remote RAM / SSD)
	32	+· stall time (ms waiting for weights) · bytes read from SSD · bytes over network · peak RSS and `mx.get_peak_memory()` ·
	33	+GPU utilization (`powermetrics --samplers gpu_power`, sudo) · energy per token (powermetrics CPU+GPU+ANE package power × wall time
	34	+/ tokens) · correctness (argmax agreement and max \|Δlogit\| vs MLX-LM reference) · stability (variance across repeats).
	35	+
	36	+## 5. Procedure
	37	+Fixed seeds; exact command recorded in each JSON; warm-up run discarded; benchmark order randomized per block (seeded);
	38	+≥ 5 repeats (bootstrap 95 % CI, 10 000 resamples); temperature/thermal pressure sampled every 1 s during runs
	39	+(`powermetrics --samplers thermal`); hardware inventory hash attached; timers = `time.perf_counter_ns` on each node with a
	40	+one-way clock-offset estimate from an NTP-style ping exchange before each distributed run; compilation/warm-up measured
	41	+separately from steady state; different precisions never pooled in the same table without a precision column.
	42	+
	43	+## 6. Ablation plan (factorial where feasible)
	44	+predictor ∈ {none, LRU, LFU, transition, learned} × prefetch depth ∈ {0, 1, 2 layers} × memory budget ∈ {25 %, 50 %, 75 %, 100 %
	45	+of expert bytes} × placement ∈ {static-uniform, frequency, capacity-aware, topology-aware} × replication ∈ {0, 1 hot set} ×
	46	+transport ∈ {TCP LAN, (Thunderbolt/JACCL if cabled)} × kernels ∈ {stock MLX, TITAN Metal} × batch ∈ {1, 4, 8}.
	47	+Interactions reported, including losses.
	48	+
	49	+## 7. Statistical audit (Gate G)
	50	+Pre-registered primary endpoints: H1 decode tok/s at 50 % budget (predictor vs no-prefetch); H2 decode tok/s 2–4 nodes
	51	+(topology-aware vs static); H3 tok/s vs budget curve AUC; H4 end-to-end decode tok/s (TITAN kernels vs stock); H5 hypervolume of
	52	+the (latency, throughput, energy) Pareto set. Effects reported with CIs; p-values secondary.

added tests/conftest.py +15 −0

Large diff (15 lines) is not loaded by default to keep the page fast.Load diff

added tests/test_numerical.py +93 −0

Large diff (93 lines) is not loaded by default to keep the page fast.Load diff

added tests/test_registry_store.py +26 −0

Large diff (26 lines) is not loaded by default to keep the page fast.Load diff

added titan/__init__.py +2 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

added titan/moe.py +71 −0

Large diff (71 lines) is not loaded by default to keep the page fast.Load diff

added titan/registry.py +166 −0

Large diff (166 lines) is not loaded by default to keep the page fast.Load diff

added titan/runtime.py +75 −0

Large diff (75 lines) is not loaded by default to keep the page fast.Load diff

added titan/slots.py +150 −0

Large diff (150 lines) is not loaded by default to keep the page fast.Load diff

added titan/store.py +85 −0

Large diff (86 lines) is not loaded by default to keep the page fast.Load diff

spb/wp33_uqo

Gate A/B: scaffold, mandate, inventory tooling, prior art (59 refs), registry/store/slot-table runtime, numerical tests bit-identical to mlx-lm

Theme