# TITAN-MoE research log (WP33 UQO)

Chronological, append-only. Times are local (America/Toronto). Every entry names the artifact it produced.

## 2026-11-01 — Session 1: mandate received, Gate A started

- 13:xx Mandate pasted by the author (verbatim copy in `docs/MANDATE.md`). Project placed in the UQO working-paper series as
  **WP33**, directory `Recherche-UQO/ia/titan-moe/` (series convention; the mandate's `wp33_titan_moe/` name is kept as the
  repository name `wp33_uqo.git` on spbgit — see DECISIONS D-001).
- Scaffold created from `_modele-working-paper` (UQO LaTeX template, Makefile, .latexmkrc, title pages). Git repo initialised (`main`).
- Python 3.12 venv (`uv`): mlx 0.32.4, mlx-lm 0.32.2, numpy 2.5.4 (laptop). Lock file to be written at Gate B.
- Reference repositories shallow-cloned into `third_party/` (git-ignored) with commit hashes recorded in `research/prior_art.csv`:
  colibri `0a5504a` (2026-10-31), mlx-lm `736ab0d` (2026-10-30), mlx `bd63d38` (2026-11-01, version header 0.32.5), llama.cpp `a91eab6` (2026-10-31).
- Hardware inventory of the MacLustr LAN launched (29 aliases). Results in `artifacts/environment.json` (sanitized) and
  `artifacts/private/inventory-2026-11-01.txt` (not committed). 7 aliases unreachable at probe time (SSH timeout):
  M2M32, M2M32b, M4M36, m4ma, m4mb, m4mc, m2m16; M4BP48 answered with an empty record. Treated as unavailable for WP33 unless
  they come back; not assumed from historical notes.
- Literature search delegated to a web-research sub-agent (output → `research/RELATED_WORK.md`, `research/prior_art.csv`, `paper/references.bib`),
  to be verified entry by entry before use.
- Small open-licensed MoE chosen for Gate B: `mlx-community/OLMoE-1B-7B-0125-Instruct-4bit` (Apache-2.0, 3.9 GB, revision `e2384419`),
  download started 2026-11-01 (laptop HF cache).
- OLMoE-1B-7B-0125-Instruct-4bit downloaded (33 s). Layout read from `model.safetensors` (`benchmarks/raw/gateA/smoke/`):
  per-expert tensors `model.layers.L.mlp.experts.E.{gate,up,down}_proj.{weight,scales,biases}` (affine 4-bit, group 64;
  gate/up weight U32 [1024,256] = 1.05 MB + scales/biases F16 2×64 KB; down [2048,128] = 1.05 MB + 2×64 KB) → **3.375 MiB per expert**,
  1 024 experts = 3.625 GB; attention 0.151 GB; embeddings/head 0.116 GB (tied). Per decode token: 16 layers × top-8 = 128 experts = 432 MiB of weights touched.
- Smoke baseline mlx-lm `generate` on the laptop (M5 Max, 48 GB, mlx 0.32.4, mlx-lm 0.32.2, temp 0, 33-token prompt, 85 generated):
  prompt 45.0 tok/s (includes first-call compilation), generation **493.9 tok/s**, peak memory 4.11 GB. Raw: `benchmarks/raw/gateA/smoke/mlx_lm_generate_laptop_*.txt`.
- SSD probe `benchmarks/scripts/ssd_bandwidth.py` (single-threaded `pread`, F_NOCACHE after `sudo purge`): laptop random 3.4 MB reads
  **8.50 GB/s** (p50 0.43 ms / chunk, p99 0.49 ms), page-cached 28.4 GB/s, sequential 8 MB reads 11.5 GB/s. Lesson: F_NOCACHE alone does
  not evict already-cached pages — `purge` is mandatory before a cold measurement (first attempt read 37 GB/s = cache). Raw: `benchmarks/raw/gateA/ssd/laptop-M5Max.json`.
- Design microbenchmark for the slot table (`benchmarks/raw/gateB/design/slot_update_microbench_laptop.txt`): in-place slot
  update 0.305 ms; gather_qmm 0.260 ms/layer; per-expert loop 0.428 ms; restack 0.502 ms; lazy mmap expert load 0.53 ms (cached).
- **Gate B implemented**: `titan/registry.py` (safetensors byte ranges, routing semantics), `titan/store.py` (pread SSD tier with
  accounting), `titan/slots.py` (LRU/LFU slot tables, stall accounting), `titan/moe.py` (TitanSwitchGLU, exact row chunking),
  `titan/runtime.py` (mlx-lm host model with experts spliced out). Tests `tests/` (7 passed, raw `benchmarks/raw/tests/pytest_gateB_20261101T011805.txt`):
  registry; pread bytes == `mx.load` bytes; logits vs mlx-lm **max |Δ| = 0.0** for capacity 64/32/16/8 on 4 prompts (argmax equal);
  greedy 32-token decode token-identical with 16 slots. First unfused activation attempt gave Δ = 0.0195 (fixed, D-006).
- Literature pass completed: 59 verified works → `research/prior_art.csv`, `paper/references.bib` (59 entries), synthesis and
  novelty matrix in `research/RELATED_WORK.md`.
- Capacity sweep on the laptop (`benchmarks/raw/gateB/capacity_sweep_laptop/sweep.jsonl`, 64-token greedy, general+code prompts):
  mlx-lm 491–496 tok/s decode, prefill 1 216–1 567 tok/s. TITAN capacity 64 *without* preloading: 201–203 tok/s — every layer paid a
  host synchronisation (`tolist()` of the router choice) because some rare experts were never requested, so no table was "complete".
  Fix: a sync-free fast path when a layer's table holds every expert (device-side slot map, `titan/moe.py`), plus `warm_all()` for the
  all-resident control → **489 tok/s** (−1 % vs mlx-lm; raw model-forward microbench `benchmarks/raw/gateB/design/fastpath_profile_laptop.txt`:
  TITAN 479 vs mlx-lm 469 tok/s async, 380 vs 370 sync). Demand loading from the page cache with LRU: capacity 32 → 56–80 tok/s
  (hit 0.73–0.81, stall 0.7–1.0 s per 64 tokens); capacity 16 → 36–45 tok/s (hit 0.48–0.60). This is the no-prefetch control for H1.
- Link probe from the nodes (`artifacts/private/linkprobe-2026-11-01.txt`): **every node negotiates 1000baseT** on en0 (the LAN switch is
  1 GbE even for the Mac Studios' 10 GbE ports) → ≈ 28 ms to move one 3.4 MB OLMoE expert over the LAN vs 0.43 ms from the laptop's
  SSD. The laptop is *not* on the LAN (ping to 192.168.2.x fails; SSH goes through Tailscale, ~30 ms RTT to the MacBook Airs) → all
  distributed experiments must run between LAN nodes, with M3U96a as coordinator. `rdma_ctl` exists on every node (macOS 26.4–27.0);
  Thunderbolt cabling between nodes still to be verified (netprobe running).
- Routing traces collected with the mlx-lm model (`benchmarks/raw/gateC/traces_olmoe_laptop/`, 18 prompts, 4 families × fit/heldout,
  3 686 tokens incl. 447 prompt tokens; per token and layer: float16 router probabilities over 64 experts + top-8 ids).
- Network probe from M3U96a (`artifacts/private/netprobe-2026-11-01.txt`): RTT to the 17 LAN nodes 0.53–1.02 ms mean (20 pings,
  min 0.36–0.62 ms); iperf3 M3U96a→M3U96b **0.945 Gbit/s** (5 s, TCP) = the 1 GbE ceiling; iperf3 not installed on M2U64/M4M64c, failed
  to parse on M4M64a (to rerun). **No Thunderbolt link is active on any Studio** (all `Thunderbolt N` interfaces `status: inactive`
  on M3U96a/M3U96b/M4M64a/M2U64/M4M64c/M5M36) → Thunderbolt/JACCL RDMA is **blocked pending physical cabling** (user action); the
  distributed gates (E/F) will use TCP over 1 GbE, which is the honest topology of this cluster.