# TITAN-MoE research log (WP33 UQO)

Chronological, append-only. Times are local (America/Toronto). Every entry names the artifact it produced.

# 2026-11-01 — Session 1: mandate received, Gate A started

• 13:xx Mandate pasted by the author (verbatim copy in docs/MANDATE.md). Project placed in the UQO working-paper series as WP33, directory Recherche-UQO/ia/titan-moe/ (series convention; the mandate's wp33_titan_moe/ name is kept as the repository name wp33_uqo.git on spbgit — see DECISIONS D-001).
• Scaffold created from _modele-working-paper (UQO LaTeX template, Makefile, .latexmkrc, title pages). Git repo initialised (main).
• Python 3.12 venv (uv): mlx 0.32.4, mlx-lm 0.32.2, numpy 2.5.4 (laptop). Lock file to be written at Gate B.
• Reference repositories shallow-cloned into third_party/ (git-ignored) with commit hashes recorded in research/prior_art.csv: colibri 0a5504a (2026-10-31), mlx-lm 736ab0d (2026-10-30), mlx bd63d38 (2026-11-01, version header 0.32.5), llama.cpp a91eab6 (2026-10-31).
• Hardware inventory of the MacLustr LAN launched (29 aliases). Results in artifacts/environment.json (sanitized) and artifacts/private/inventory-2026-11-01.txt (not committed). 7 aliases unreachable at probe time (SSH timeout): M2M32, M2M32b, M4M36, m4ma, m4mb, m4mc, m2m16; M4BP48 answered with an empty record. Treated as unavailable for WP33 unless they come back; not assumed from historical notes.
• Literature search delegated to a web-research sub-agent (output → research/RELATED_WORK.md, research/prior_art.csv, paper/references.bib), to be verified entry by entry before use.
• Small open-licensed MoE chosen for Gate B: mlx-community/OLMoE-1B-7B-0125-Instruct-4bit (Apache-2.0, 3.9 GB, revision e2384419), download started 2026-11-01 (laptop HF cache).
• OLMoE-1B-7B-0125-Instruct-4bit downloaded (33 s). Layout read from model.safetensors (benchmarks/raw/gateA/smoke/): per-expert tensors model.layers.L.mlp.experts.E.{gate,up,down}_proj.{weight,scales,biases} (affine 4-bit, group 64; gate/up weight U32 [1024,256] = 1.05 MB + scales/biases F16 2×64 KB; down [2048,128] = 1.05 MB + 2×64 KB) → 3.375 MiB per expert, 1 024 experts = 3.625 GB; attention 0.151 GB; embeddings/head 0.116 GB (tied). Per decode token: 16 layers × top-8 = 128 experts = 432 MiB of weights touched.
• Smoke baseline mlx-lm generate on the laptop (M5 Max, 48 GB, mlx 0.32.4, mlx-lm 0.32.2, temp 0, 33-token prompt, 85 generated): prompt 45.0 tok/s (includes first-call compilation), generation 493.9 tok/s, peak memory 4.11 GB. Raw: benchmarks/raw/gateA/smoke/mlx_lm_generate_laptop_*.txt.
• SSD probe benchmarks/scripts/ssd_bandwidth.py (single-threaded pread, F_NOCACHE after sudo purge): laptop random 3.4 MB reads 8.50 GB/s (p50 0.43 ms / chunk, p99 0.49 ms), page-cached 28.4 GB/s, sequential 8 MB reads 11.5 GB/s. Lesson: F_NOCACHE alone does not evict already-cached pages — purge is mandatory before a cold measurement (first attempt read 37 GB/s = cache). Raw: benchmarks/raw/gateA/ssd/laptop-M5Max.json.
• Design microbenchmark for the slot table (benchmarks/raw/gateB/design/slot_update_microbench_laptop.txt): in-place slot update 0.305 ms; gather_qmm 0.260 ms/layer; per-expert loop 0.428 ms; restack 0.502 ms; lazy mmap expert load 0.53 ms (cached).
• Gate B implemented: titan/registry.py (safetensors byte ranges, routing semantics), titan/store.py (pread SSD tier with accounting), titan/slots.py (LRU/LFU slot tables, stall accounting), titan/moe.py (TitanSwitchGLU, exact row chunking), titan/runtime.py (mlx-lm host model with experts spliced out). Tests tests/ (7 passed, raw benchmarks/raw/tests/pytest_gateB_20261101T011805.txt): registry; pread bytes == mx.load bytes; logits vs mlx-lm max |Δ| = 0.0 for capacity 64/32/16/8 on 4 prompts (argmax equal); greedy 32-token decode token-identical with 16 slots. First unfused activation attempt gave Δ = 0.0195 (fixed, D-006).
• Literature pass completed: 59 verified works → research/prior_art.csv, paper/references.bib (59 entries), synthesis and novelty matrix in research/RELATED_WORK.md.
• Capacity sweep on the laptop (benchmarks/raw/gateB/capacity_sweep_laptop/sweep.jsonl, 64-token greedy, general+code prompts): mlx-lm 491–496 tok/s decode, prefill 1 216–1 567 tok/s. TITAN capacity 64 without preloading: 201–203 tok/s — every layer paid a host synchronisation (tolist() of the router choice) because some rare experts were never requested, so no table was "complete". Fix: a sync-free fast path when a layer's table holds every expert (device-side slot map, titan/moe.py), plus warm_all() for the all-resident control → 489 tok/s (−1 % vs mlx-lm; raw model-forward microbench benchmarks/raw/gateB/design/fastpath_profile_laptop.txt: TITAN 479 vs mlx-lm 469 tok/s async, 380 vs 370 sync). Demand loading from the page cache with LRU: capacity 32 → 56–80 tok/s (hit 0.73–0.81, stall 0.7–1.0 s per 64 tokens); capacity 16 → 36–45 tok/s (hit 0.48–0.60). This is the no-prefetch control for H1.
• Link probe from the nodes (artifacts/private/linkprobe-2026-11-01.txt): every node negotiates 1000baseT on en0 (the LAN switch is 1 GbE even for the Mac Studios' 10 GbE ports) → ≈ 28 ms to move one 3.4 MB OLMoE expert over the LAN vs 0.43 ms from the laptop's SSD. The laptop is not on the LAN (ping to 192.168.2.x fails; SSH goes through Tailscale, ~30 ms RTT to the MacBook Airs) → all distributed experiments must run between LAN nodes, with M3U96a as coordinator. rdma_ctl exists on every node (macOS 26.4–27.0); Thunderbolt cabling between nodes still to be verified (netprobe running).
• Routing traces collected with the mlx-lm model (benchmarks/raw/gateC/traces_olmoe_laptop/, 18 prompts, 4 families × fit/heldout, 3 686 tokens incl. 447 prompt tokens; per token and layer: float16 router probabilities over 64 experts + top-8 ids).
• Network probe from M3U96a (artifacts/private/netprobe-2026-11-01.txt): RTT to the 17 LAN nodes 0.53–1.02 ms mean (20 pings, min 0.36–0.62 ms); iperf3 M3U96a→M3U96b 0.945 Gbit/s (5 s, TCP) = the 1 GbE ceiling; iperf3 not installed on M2U64/M4M64c, failed to parse on M4M64a (to rerun). No Thunderbolt link is active on any Studio (all Thunderbolt N interfaces status: inactive on M3U96a/M3U96b/M4M64a/M2U64/M4M64c/M5M36) → Thunderbolt/JACCL RDMA is blocked pending physical cabling (user action); the distributed gates (E/F) will use TCP over 1 GbE, which is the honest topology of this cluster.
• Second network probe (artifacts/private/netprobe2-2026-11-01.txt, iperf3 installed on M2U64 and M4M64c via Homebrew): M3U96a → M4M64a / M2U64 / M4M64c / M3U96b all 0.945–0.947 Gbit/s; RTT M4M64a→M3U96b 0.87 ms, M4M64a→M2U64 0.69 ms.
• Offline simulation v1 (benchmarks/processed/gateC/sim_olmoe_laptop_ssd_v1.json, 18-prompt traces, per-layer tables, cold per prompt, 1 loader, t_read 0.43 ms, t_layer 0.128 ms): LRU hit rate 0.57 (cap 16) / 0.80 (32) / 0.93 (48); LFU ≈ LRU; one-layer-ahead transition predictor recall@8 = 0.50 held-out (0.53 fit), raises hit rate to 0.66 / 0.83 / 0.94 at the cost of +40–60 % loads per token; the linear probe on previous-layer log-probabilities underperforms (recall 0.16, too little data → v2 traces); the oracle upper bound reaches 0.97 (cap 32). prev_token is a no-op by construction with per-layer tables (the previous token's experts are still resident), and prev_rank with m ≤ K prefetched nothing (fixed: ranks K+1..K+m).
• Online prefetch smoke test at 16 slots (held-out general/code, page-cached file, 2 reader threads, m = 8): transition raised the hit rate (0.48→0.63, 0.67→0.73) but lowered decode throughput (51→35, 73→56 tok/s) — a one-layer lead (≈ 0.13 ms) cannot hide a 0.21 ms fetch, late prefetches are waited for, and wrong predictions add traffic. H1 at this tier/model is therefore not supported by a 1-layer horizon; deeper horizons (transition2/3/4) and the token horizon are being simulated (sim v2 on the 65-prompt traces).
• Routing traces v2 collected (benchmarks/raw/gateC/traces_olmoe_v2_laptop/, 65 prompts, 6 families incl. long-context and drift, up to 384 generated tokens each).
• Qwen3-30B-A3B-Instruct-2507-4bit on M3U96a (M3 Ultra 96 GB): checkpoint is in the stacked switch_mlp layout → registry extended (per-expert byte ranges = slices of the first axis; tests/test_registry_stacked.py byte-equal to mx.load). 2.53 MiB per expert, 16.3 GB of experts, 0.87 GB dense, 384 experts (0.97 GiB) per decode token; routers are 8-bit. Tests on M3U96a (benchmarks/raw/tests/m3u96a/pytest_qwen3_m3u96a_20261101T014415.txt, 6 passed): logits max |Δ| = 0.0 at 128/64/32/16 slots, greedy 32-token decode identical at 32 slots. Baselines (benchmarks/raw/gateC/qwen3_m3u96a/baseline.jsonl, 64 tokens, 3 repeats): mlx-lm 121.3–121.6 tok/s decode, prefill 339–434 tok/s; TITAN all-resident (128 slots, preloaded) 121.5–121.9 tok/s — parity.
• Cluster venvs ready on M3U96a, M3U96b, M4M64a, M2U64, M4M64c (scripts/cluster_setup.sh, mlx 0.32.4 / mlx-lm 0.32.2 everywhere).
• Paper skeleton v0.1 compiles (24 pages; tab:sim pending the v2 simulation). Cold-NVMe prefetch sweep launched on the laptop (benchmarks/scripts/run_cold_nvme_laptop.sh: purge + F_NOCACHE, 16/32 slots, none/transition{1,2,3}/prev_rank, held-out).
• Offline simulation v2 (benchmarks/processed/gateC/sim_olmoe_v2_ssd043_m8.json, 65 prompts, per-layer LRU, t_read 0.43 ms, 1 loader), held-out: LRU hit 0.581 / 0.706 / 0.806 / 0.933 at C = 16 / 24 / 32 / 48. Transition predictor recall@8 0.534 at depth 1, 0.525 at depth 2, 0.517 at depth 3 → a 3-layer lead costs almost no accuracy, which is what the lead-time condition needs on a cold NVMe (d·τ_layer ≥ λ → d ≥ 4 at 0.43 ms). Hit rate with transition prefetch 0.686 / 0.771 / 0.840 / 0.942 (+0.01–0.10) at +60–100 % loads; prev_rank (m = 16) hurts (prefetch precision 0.18–0.47, evicts useful experts); the linear probe was broken by a label-alignment bug (rows not offset across traces) → fixed, re-simulated (sim_olmoe_v2_ssd043_m8_probe_fixed.json); oracle 0.94–0.99.
• Cold NVMe tier (purge + F_NOCACHE), 16 slots, held-out general prompts, laptop: v1 (9 pread per expert, serial) 8–10 tok/s, stall 7–8 s / 64 tokens (benchmarks/raw/gateC/cold_nvme_laptop/sweep_v1_9preads_serial.jsonl); v2 parallel demand reads (4 reader threads) 18 tok/s, stall 3.0 s (quick_v2.jsonl); finding: in the mlx-community OLMoE file the 9 tensors of one expert are scattered over a 1.5 GB span (hash order of keys), so every miss costs 9 random reads. v3: scripts/convert_layout.py writes an expert-contiguous blob (one 4 KiB-aligned 3.4 MB record per expert, 3.6 GB written in 3.2 s; ~/.cache/titan/<hash>/), the registry points at it transparently, tests verify byte equality with the original (tests/test_registry_store.py, one pread per expert) → 36 tok/s, stall 1.2 s / 64 tokens (quick_v3.jsonl), vs 51–73 tok/s from the page cache. Cold sweep restarted with v3.
• Gate E (transport) implemented: transport/protocol.py (TX1 frames, HMAC-SHA256 on every frame with a pre-shared token kept in ~/.titan_token, mode 600, never in the repo), transport/worker.py (resident expert subset, exact gather_qmm + swiglu per (row, k) pair, IP allow-list, timeouts), transport/client.py, titan/distributed.py (dispatch to every worker before waiting on any, local execution of the rest, assembly in original (row, k) order, local fallback on failure). Localhost two-process test on M3U96a (benchmarks/raw/tests/m3u96a/pytest_distributed_localhost_*.txt, distributed_localhost_olmoe_*.json): odd experts of every layer executed by the worker → logits max |Δ| = 0.0 vs mlx-lm, greedy 24 tokens identical, 16 layers remote per token, localhost RTT 0.11–0.14 ms. Two-node runs (coordinator M3U96a ↔ worker M3U96b over 1 GbE) launched: benchmarks/raw/gateE/two_node_m3u96a_m3u96b.jsonl (local control at 32 slots: 68–88 tok/s page-cached; odd placement at 32 and 64 slots pending).
• Cold sweep v3 (in progress, 16 slots): none ≈ 36 tok/s; transition2 31.7–34.1 (stall 0.82–0.92 s vs ≈ 1.2 s), transition3 26.8–28.0, prev_rank (m = 16) 18–19 tok/s → deeper horizons cut the stall by ~30 % but the host-side cost of prefetch (installs, GIL contention of reader threads, eviction churn) cancels the gain on this small model; to be attributed with per-token timers after the sweep.
• Two-node result (Gate E, H2/H3 first evidence) — coordinator M3U96a, worker M3U96b, OLMoE, 32 slots, page-cached, 64-token greedy, held-out general/code (benchmarks/raw/gateE/two_node_m3u96a_m3u96b.jsonl): local control 70.1 / 87.0 tok/s (median, hit 0.74 / 0.84); odd experts executed remotely: 15.0 / 14.9 tok/s, logits still bit-identical (Δ = 0.0), remote wait 2.5 s per 64 tokens = ≈ 2.5 ms per layer round trip (TCP RTT 0.5–0.65 ms + worker-side array creation, gather_qmm dispatch, eval and serialisation) × 16 layers ≈ 40 ms per token. On this cluster remote execution over 1 GbE therefore costs more per layer than a cold-NVMe demand miss (≈ 1.2 ms per layer at 16 slots) and far more than local compute (0.13 ms): as the formulation predicts, the remote tier cannot be a latency win for a model whose experts fit on local NVMe; it can only add capacity. Worker-side cost breakdown to be measured once the sweep finishes.
• Cold-NVMe sweep v3 complete (benchmarks/raw/gateC/cold_nvme_laptop/sweep.jsonl, 12 runs per cell, held-out general/code/reasoning, 64 tokens, purge + F_NOCACHE, 4 readers, contiguous layout): 16 slots — none 36.4 tok/s (hit 0.53, 267 MB/token from SSD), transition d=1 27.9, d=2 35.0 (hit 0.65, stall −33 %: 0.77 s vs 1.15 s per 64 tokens), d=3 29.1, transition2 m=16 25.9, prev_rank m=16 19.1; 32 slots — none 50.6, transition 41.3, transition2 42.0, transition2 m=16 41.0, transition3 41.0, prev_rank 28.4.
• Host-time attribution (attribution.jsonl, per token, 16 slots, cold): none = sync 14.7 ms (GPU graph incl. the installs of missed experts) + reads 19.9 ms; transition2 = sync 20.4 + prefetch bookkeeping 1.9 + reads 18.8 ms with 58 prefetches/token (1.8 late). Confidence thresholds (transition2t50, t70: 54 and 31 prefetches/token) leave sync at 20.0–20.3 ms. Mechanism: every installed expert costs ≈ 9 lazy slice writes (≈ 0.1 ms of graph work on the critical path); with ≈ 50 % prefetch precision the installs double while demand reads fall only 30 % → H1 not supported on OLMoE/laptop NVMe: hit +0.14, throughput −4 to −15 %. The oracle (hit 0.94) shows the ceiling; the lever is install cost per expert and prediction precision, not lead time (d = 2 already suffices).
• Gate D profiling on M4M64a (benchmarks/raw/gateD/profile_olmoe_m4m64a.json, launch_overhead_m4m64a.txt): decode step 2.99 ms sync (334 tok/s), effective 151 GB/s on 452 MB of expert bytes per token (≈ 28 % of the M4 Max's nominal bandwidth); isolated per-layer timings are dominated by the ≈ 0.15–0.17 ms synchronisation floor; in-graph elementwise ops cost 4.7–7.2 µs each; a gather_qmm for k = 1…16 experts takes the same 0.17 ms (latency-bound), 250 GB/s only at k = 64. Decode at batch 1 is therefore launch/latency-bound: fusing up+gate into one gather_qmm saves one dispatch per layer (≈ 16 × ~10 µs per token, ≤ 5 %); larger gains need token batching, not kernels. H4 is tested only with the cheap fusion (next entry).
• Remote cost breakdown M3U96a→M3U96b (benchmarks/raw/gateE/remote_cost_breakdown_m3u96a_m3u96b.txt): application ping 0.74–1.17 ms; exec round trip 2.09 ms (1 pair), 1.99 (4), 2.59 (8 pairs = one decode layer), 7.27 (8 rows × 64 pairs), 19.1 ms (32 rows × 256 pairs, 1 MB returned ≈ 9 ms on the wire) → ≈ 1.2–1.7 ms of worker-side overhead per call on top of the RTT.
• H4 cheap fusion (titan/slots.py fused_upgate, tests/test_fused_upgate.py → logits Δ = 0.0 at 64 and 16 slots; benchmarks/raw/gateD/fused_upgate_laptop.jsonl): one gather_qmm over a [C, 2H, D] up+gate buffer vs two kernels: all-resident 475.6 vs 476.7 tok/s (12 runs, −0.2 %), cold 16 slots 35.9 vs 35.0 tok/s (4 runs, +2.6 %, within the run-to-run range). As the profiling predicted, saving one dispatch per layer is immaterial at batch 1 → H4 not supported for this fusion; no custom Metal kernel is justified by the profile (decode is launch/latency-bound, 28 % of nominal bandwidth).
• Linear probe fixed and evaluated (sim_olmoe_v2_ssd043_m8_probe_fixed.json): one-layer-ahead logistic probe on the previous layer's router log-probabilities → recall@8 0.761 held-out (0.779 fit), prefetch precision 0.74–0.83, simulated hit rate 0.81 / 0.86 / 0.90 / 0.97 at C = 16 / 24 / 32 / 48 (none: 0.58 / 0.71 / 0.81 / 0.93; oracle 0.94 / 0.96 / 0.97 / 0.99); modelled stall 10.0 ms vs 23.1 ms per token at C = 16. Depth sweep (probe2–4) running (sim_olmoe_v2_ssd043_m8_probe_depths.json).
• Online probe on the cold tier (16 slots, held-out general, attribution.jsonl): probe d=1 hit 0.775, reads 12.8 ms/token (none 19.9) but sync 19.7 + prefetch 3.3 ms (6.6 late waits/token) → 31.8 tok/s vs 33.0 none; probe m=16 hit 0.847, reads 8.3 ms, but 127 prefetches/token and 25 late waits → 25.3; probe2 31.3 (hit 0.74), probe3 30.7 (hit 0.72). Every predictor converts 7–11 ms of saved reads into ≈ 5 ms of extra GPU-side install work plus 1–3 ms of late waits: on this model and tier the install path (9 lazy slice writes ≈ 0.13 ms per expert) is the bottleneck, not prediction accuracy. Identified fix for v0.2: single-copy record install.
• Probe depth simulation (sim_olmoe_v2_ssd043_m8_probe_depths.json, held-out, C = 16): recall@8 0.761 / 0.743 / 0.731 at depth 1 / 2 / 3, hit 0.81 / 0.78 / 0.76 → the probe also keeps its accuracy with a longer lead.
• Gate F preparation: Qwen3-30B routing traces collected on M3U96a (benchmarks/raw/gateF/traces_qwen3_m3u96a, 65 prompts, 128 tokens); contiguous layout built on M3U96a (6 144 records, 16.3 GB). The LAN copy M3U96a → M3U96b first failed on a stale SSH host key for 192.168.2.85 in M3U96a's known_hosts (the LAN was renumbered by DHCP on 2026-09-21). Verified that the key presented on the LAN (SHA256:zcdc123MTDYreNavJ9lUAS1OWSFSzxvxipqaBSygHsA) is M3U96b's actual /etc/ssh/ssh_host_ed25519_key.pub fingerprint before replacing the entry (backup ~/.ssh/known_hosts.bak-20261101-wp33 on M3U96a); copy and layout on M3U96b restarted.
• Energy run (H5, single node) (benchmarks/raw/gateG/energy_laptop/, powermetrics 1 s, 411 samples, idle 13.2 W, 256-token greedy, held-out general/code, 12 runs per configuration): mlx-lm 463 tok/s at 47.2 W → 0.101 J/token gross (0.073 net of idle); TITAN all-resident 462 tok/s, 46.4 W, 0.101 J/token; TITAN 16 slots cold NVMe 31.9 tok/s at 15.9 W → 0.498 J/token gross (0.086 net); 16 slots cold + probe prefetch 42.0 tok/s at 18.4 W → 0.441 J/token gross (0.119 net). New H1 evidence: with 256-token generations the probe beats demand loading by +32 % (the 64-token runs showed a tie) — see audit CIs; the hit rate reaches 0.84 over longer sequences while the per-token install overhead is unchanged. Energy favours prefetch per token in gross terms (the machine is on anyway) but not net of idle; all-resident decoding is 4–5× cheaper per token than any cold-tier configuration.
• Gate F, first sweep discarded for exactness (benchmarks/raw/gateF/two_node_qwen3_m3u96a_m3u96b_v1_fp16wire.jsonl, run_gateF_v1_fp16wire.log): with Qwen3-30B the remote path gave max |Δlogit| = 0.65 (odd) / 0.43 (popular) vs mlx-lm because the transport cast bfloat16 activations and results to float16 on the wire (OLMoE is float16-native, hence its Δ = 0.0). Fix: activations and results travel in the model's dtype (bfloat16 as uint16 bit patterns, dtype tag in the header); localhost Qwen3 test on M3U96a now Δ = 0.0, greedy identical (benchmarks/raw/tests/m3u96a/pytest_distributed_localhost_qwen3_*). Throughputs of the discarded sweep, kept for the record (cold local tier on M3U96a, 64 tokens): none 8.7–8.9 tok/s at 32 slots (stall 65 ms/token), 15–20 at 64 slots; odd 5.4–5.8 / 4.1; popular 6.6–7.0 / 4.5–5.5; remote wait 70–146 ms/token (48 layers × 1.5–3 ms). Sweep relaunched with the fix.
• Earlier the relaunched sweep had also crashed on "There is no Stream(gpu, N) in current thread": MLX arrays created in reader threads cannot be evaluated in another thread; fixed by returning numpy from reader threads and creating MLX arrays on the consuming thread (titan/store.py to_mx, titan/prefetch.py).
• Gate F (exact) complete (benchmarks/raw/gateF/two_node_qwen3_m3u96a_m3u96b.jsonl, 16 runs per cell, Δlogit = 0.0 everywhere): Qwen3-30B-A3B, coordinator M3U96a with cold NVMe local tier (purge + F_NOCACHE), worker M3U96b over 1 GbE. 32 slots: local 8.8 tok/s (hit 0.78, stall 66 ms/token), odd 4.2 (remote wait 112 ms/token, 14 183 remote pairs), popular 5.1 (67 ms, 9 161 pairs); 64 slots: local 16.3, odd 5.2, popular 6.0. CIs: popular − local −3.6 [−4.4, −2.9] (32), −10.2 [−13.6, −8.8] (64). H2 weak form supported (frequency-based > static), strong form not (remote < local NVMe on 1 GbE). Paper updated (Section Gate F, figure gateF_qwen3.pdf, table tab:gateF, conclusion rewritten).