# TITAN-MoE research log (WP33 UQO)

Chronological, append-only. Times are local (America/Toronto). Every entry names the artifact it produced.

# 2026-11-01 — Session 1: mandate received, Gate A started

• 13:xx Mandate pasted by the author (verbatim copy in docs/MANDATE.md). Project placed in the UQO working-paper series as WP33, directory Recherche-UQO/ia/titan-moe/ (series convention; the mandate's wp33_titan_moe/ name is kept as the repository name wp33_uqo.git on spbgit — see DECISIONS D-001).
• Scaffold created from _modele-working-paper (UQO LaTeX template, Makefile, .latexmkrc, title pages). Git repo initialised (main).
• Python 3.12 venv (uv): mlx 0.32.4, mlx-lm 0.32.2, numpy 2.5.4 (laptop). Lock file to be written at Gate B.
• Reference repositories shallow-cloned into third_party/ (git-ignored) with commit hashes recorded in research/prior_art.csv: colibri 0a5504a (2026-10-31), mlx-lm 736ab0d (2026-10-30), mlx bd63d38 (2026-11-01, version header 0.32.5), llama.cpp a91eab6 (2026-10-31).
• Hardware inventory of the MacLustr LAN launched (29 aliases). Results in artifacts/environment.json (sanitized) and artifacts/private/inventory-2026-11-01.txt (not committed). 7 aliases unreachable at probe time (SSH timeout): M2M32, M2M32b, M4M36, m4ma, m4mb, m4mc, m2m16; M4BP48 answered with an empty record. Treated as unavailable for WP33 unless they come back; not assumed from historical notes.
• Literature search delegated to a web-research sub-agent (output → research/RELATED_WORK.md, research/prior_art.csv, paper/references.bib), to be verified entry by entry before use.
• Small open-licensed MoE chosen for Gate B: mlx-community/OLMoE-1B-7B-0125-Instruct-4bit (Apache-2.0, 3.9 GB, revision e2384419), download started 2026-11-01 (laptop HF cache).
• OLMoE-1B-7B-0125-Instruct-4bit downloaded (33 s). Layout read from model.safetensors (benchmarks/raw/gateA/smoke/): per-expert tensors model.layers.L.mlp.experts.E.{gate,up,down}_proj.{weight,scales,biases} (affine 4-bit, group 64; gate/up weight U32 [1024,256] = 1.05 MB + scales/biases F16 2×64 KB; down [2048,128] = 1.05 MB + 2×64 KB) → 3.375 MiB per expert, 1 024 experts = 3.625 GB; attention 0.151 GB; embeddings/head 0.116 GB (tied). Per decode token: 16 layers × top-8 = 128 experts = 432 MiB of weights touched.
• Smoke baseline mlx-lm generate on the laptop (M5 Max, 48 GB, mlx 0.32.4, mlx-lm 0.32.2, temp 0, 33-token prompt, 85 generated): prompt 45.0 tok/s (includes first-call compilation), generation 493.9 tok/s, peak memory 4.11 GB. Raw: benchmarks/raw/gateA/smoke/mlx_lm_generate_laptop_*.txt.
• SSD probe benchmarks/scripts/ssd_bandwidth.py (single-threaded pread, F_NOCACHE after sudo purge): laptop random 3.4 MB reads 8.50 GB/s (p50 0.43 ms / chunk, p99 0.49 ms), page-cached 28.4 GB/s, sequential 8 MB reads 11.5 GB/s. Lesson: F_NOCACHE alone does not evict already-cached pages — purge is mandatory before a cold measurement (first attempt read 37 GB/s = cache). Raw: benchmarks/raw/gateA/ssd/laptop-M5Max.json.
• Design microbenchmark for the slot table (benchmarks/raw/gateB/design/slot_update_microbench_laptop.txt): in-place slot update 0.305 ms; gather_qmm 0.260 ms/layer; per-expert loop 0.428 ms; restack 0.502 ms; lazy mmap expert load 0.53 ms (cached).
• Gate B implemented: titan/registry.py (safetensors byte ranges, routing semantics), titan/store.py (pread SSD tier with accounting), titan/slots.py (LRU/LFU slot tables, stall accounting), titan/moe.py (TitanSwitchGLU, exact row chunking), titan/runtime.py (mlx-lm host model with experts spliced out). Tests tests/ (7 passed, raw benchmarks/raw/tests/pytest_gateB_20261101T011805.txt): registry; pread bytes == mx.load bytes; logits vs mlx-lm max |Δ| = 0.0 for capacity 64/32/16/8 on 4 prompts (argmax equal); greedy 32-token decode token-identical with 16 slots. First unfused activation attempt gave Δ = 0.0195 (fixed, D-006).
• Literature pass completed: 59 verified works → research/prior_art.csv, paper/references.bib (59 entries), synthesis and novelty matrix in research/RELATED_WORK.md.
• Capacity sweep on the laptop (benchmarks/raw/gateB/capacity_sweep_laptop/sweep.jsonl, 64-token greedy, general+code prompts): mlx-lm 491–496 tok/s decode, prefill 1 216–1 567 tok/s. TITAN capacity 64 without preloading: 201–203 tok/s — every layer paid a host synchronisation (tolist() of the router choice) because some rare experts were never requested, so no table was "complete". Fix: a sync-free fast path when a layer's table holds every expert (device-side slot map, titan/moe.py), plus warm_all() for the all-resident control → 489 tok/s (−1 % vs mlx-lm; raw model-forward microbench benchmarks/raw/gateB/design/fastpath_profile_laptop.txt: TITAN 479 vs mlx-lm 469 tok/s async, 380 vs 370 sync). Demand loading from the page cache with LRU: capacity 32 → 56–80 tok/s (hit 0.73–0.81, stall 0.7–1.0 s per 64 tokens); capacity 16 → 36–45 tok/s (hit 0.48–0.60). This is the no-prefetch control for H1.
• Link probe from the nodes (artifacts/private/linkprobe-2026-11-01.txt): every node negotiates 1000baseT on en0 (the LAN switch is 1 GbE even for the Mac Studios' 10 GbE ports) → ≈ 28 ms to move one 3.4 MB OLMoE expert over the LAN vs 0.43 ms from the laptop's SSD. The laptop is not on the LAN (ping to 192.168.2.x fails; SSH goes through Tailscale, ~30 ms RTT to the MacBook Airs) → all distributed experiments must run between LAN nodes, with M3U96a as coordinator. rdma_ctl exists on every node (macOS 26.4–27.0); Thunderbolt cabling between nodes still to be verified (netprobe running).
• Routing traces collected with the mlx-lm model (benchmarks/raw/gateC/traces_olmoe_laptop/, 18 prompts, 4 families × fit/heldout, 3 686 tokens incl. 447 prompt tokens; per token and layer: float16 router probabilities over 64 experts + top-8 ids).
• Network probe from M3U96a (artifacts/private/netprobe-2026-11-01.txt): RTT to the 17 LAN nodes 0.53–1.02 ms mean (20 pings, min 0.36–0.62 ms); iperf3 M3U96a→M3U96b 0.945 Gbit/s (5 s, TCP) = the 1 GbE ceiling; iperf3 not installed on M2U64/M4M64c, failed to parse on M4M64a (to rerun). No Thunderbolt link is active on any Studio (all Thunderbolt N interfaces status: inactive on M3U96a/M3U96b/M4M64a/M2U64/M4M64c/M5M36) → Thunderbolt/JACCL RDMA is blocked pending physical cabling (user action); the distributed gates (E/F) will use TCP over 1 GbE, which is the honest topology of this cluster.