1# TITAN-MoE research log (WP33 UQO)23Chronological, append-only. Times are local (America/Toronto). Every entry names the artifact it produced.45## 2026-11-01 — Session 1: mandate received, Gate A started67- 13:xx Mandate pasted by the author (verbatim copy in `docs/MANDATE.md`). Project placed in the UQO working-paper series as8 **WP33**, directory `Recherche-UQO/ia/titan-moe/` (series convention; the mandate's `wp33_titan_moe/` name is kept as the9 repository name `wp33_uqo.git` on spbgit — see DECISIONS D-001).10- Scaffold created from `_modele-working-paper` (UQO LaTeX template, Makefile, .latexmkrc, title pages). Git repo initialised (`main`).11- Python 3.12 venv (`uv`): mlx 0.32.4, mlx-lm 0.32.2, numpy 2.5.4 (laptop). Lock file to be written at Gate B.12- Reference repositories shallow-cloned into `third_party/` (git-ignored) with commit hashes recorded in `research/prior_art.csv`:13 colibri `0a5504a` (2026-10-31), mlx-lm `736ab0d` (2026-10-30), mlx `bd63d38` (2026-11-01, version header 0.32.5), llama.cpp `a91eab6` (2026-10-31).14- Hardware inventory of the MacLustr LAN launched (29 aliases). Results in `artifacts/environment.json` (sanitized) and15 `artifacts/private/inventory-2026-11-01.txt` (not committed). 7 aliases unreachable at probe time (SSH timeout):16 M2M32, M2M32b, M4M36, m4ma, m4mb, m4mc, m2m16; M4BP48 answered with an empty record. Treated as unavailable for WP33 unless17 they come back; not assumed from historical notes.18- Literature search delegated to a web-research sub-agent (output → `research/RELATED_WORK.md`, `research/prior_art.csv`, `paper/references.bib`),19 to be verified entry by entry before use.20- Small open-licensed MoE chosen for Gate B: `mlx-community/OLMoE-1B-7B-0125-Instruct-4bit` (Apache-2.0, 3.9 GB, revision `e2384419`),21 download started 2026-11-01 (laptop HF cache).22- OLMoE-1B-7B-0125-Instruct-4bit downloaded (33 s). Layout read from `model.safetensors` (`benchmarks/raw/gateA/smoke/`):23 per-expert tensors `model.layers.L.mlp.experts.E.{gate,up,down}_proj.{weight,scales,biases}` (affine 4-bit, group 64;24 gate/up weight U32 [1024,256] = 1.05 MB + scales/biases F16 2×64 KB; down [2048,128] = 1.05 MB + 2×64 KB) → **3.375 MiB per expert**,25 1 024 experts = 3.625 GB; attention 0.151 GB; embeddings/head 0.116 GB (tied). Per decode token: 16 layers × top-8 = 128 experts = 432 MiB of weights touched.26- Smoke baseline mlx-lm `generate` on the laptop (M5 Max, 48 GB, mlx 0.32.4, mlx-lm 0.32.2, temp 0, 33-token prompt, 85 generated):27 prompt 45.0 tok/s (includes first-call compilation), generation **493.9 tok/s**, peak memory 4.11 GB. Raw: `benchmarks/raw/gateA/smoke/mlx_lm_generate_laptop_*.txt`.28- SSD probe `benchmarks/scripts/ssd_bandwidth.py` (single-threaded `pread`, F_NOCACHE after `sudo purge`): laptop random 3.4 MB reads29 **8.50 GB/s** (p50 0.43 ms / chunk, p99 0.49 ms), page-cached 28.4 GB/s, sequential 8 MB reads 11.5 GB/s. Lesson: F_NOCACHE alone does30 not evict already-cached pages — `purge` is mandatory before a cold measurement (first attempt read 37 GB/s = cache). Raw: `benchmarks/raw/gateA/ssd/laptop-M5Max.json`.31- Design microbenchmark for the slot table (`benchmarks/raw/gateB/design/slot_update_microbench_laptop.txt`): in-place slot32 update 0.305 ms; gather_qmm 0.260 ms/layer; per-expert loop 0.428 ms; restack 0.502 ms; lazy mmap expert load 0.53 ms (cached).33- **Gate B implemented**: `titan/registry.py` (safetensors byte ranges, routing semantics), `titan/store.py` (pread SSD tier with34 accounting), `titan/slots.py` (LRU/LFU slot tables, stall accounting), `titan/moe.py` (TitanSwitchGLU, exact row chunking),35 `titan/runtime.py` (mlx-lm host model with experts spliced out). Tests `tests/` (7 passed, raw `benchmarks/raw/tests/pytest_gateB_20261101T011805.txt`):36 registry; pread bytes == `mx.load` bytes; logits vs mlx-lm **max |Δ| = 0.0** for capacity 64/32/16/8 on 4 prompts (argmax equal);37 greedy 32-token decode token-identical with 16 slots. First unfused activation attempt gave Δ = 0.0195 (fixed, D-006).38- Literature pass completed: 59 verified works → `research/prior_art.csv`, `paper/references.bib` (59 entries), synthesis and39 novelty matrix in `research/RELATED_WORK.md`.40- Capacity sweep on the laptop (`benchmarks/raw/gateB/capacity_sweep_laptop/sweep.jsonl`, 64-token greedy, general+code prompts):41 mlx-lm 491–496 tok/s decode, prefill 1 216–1 567 tok/s. TITAN capacity 64 *without* preloading: 201–203 tok/s — every layer paid a42 host synchronisation (`tolist()` of the router choice) because some rare experts were never requested, so no table was "complete".43 Fix: a sync-free fast path when a layer's table holds every expert (device-side slot map, `titan/moe.py`), plus `warm_all()` for the44 all-resident control → **489 tok/s** (−1 % vs mlx-lm; raw model-forward microbench `benchmarks/raw/gateB/design/fastpath_profile_laptop.txt`:45 TITAN 479 vs mlx-lm 469 tok/s async, 380 vs 370 sync). Demand loading from the page cache with LRU: capacity 32 → 56–80 tok/s46 (hit 0.73–0.81, stall 0.7–1.0 s per 64 tokens); capacity 16 → 36–45 tok/s (hit 0.48–0.60). This is the no-prefetch control for H1.47- Link probe from the nodes (`artifacts/private/linkprobe-2026-11-01.txt`): **every node negotiates 1000baseT** on en0 (the LAN switch is48 1 GbE even for the Mac Studios' 10 GbE ports) → ≈ 28 ms to move one 3.4 MB OLMoE expert over the LAN vs 0.43 ms from the laptop's49 SSD. The laptop is *not* on the LAN (ping to 192.168.2.x fails; SSH goes through Tailscale, ~30 ms RTT to the MacBook Airs) → all50 distributed experiments must run between LAN nodes, with M3U96a as coordinator. `rdma_ctl` exists on every node (macOS 26.4–27.0);51 Thunderbolt cabling between nodes still to be verified (netprobe running).52- Routing traces collected with the mlx-lm model (`benchmarks/raw/gateC/traces_olmoe_laptop/`, 18 prompts, 4 families × fit/heldout,53 3 686 tokens incl. 447 prompt tokens; per token and layer: float16 router probabilities over 64 experts + top-8 ids).54- Network probe from M3U96a (`artifacts/private/netprobe-2026-11-01.txt`): RTT to the 17 LAN nodes 0.53–1.02 ms mean (20 pings,55 min 0.36–0.62 ms); iperf3 M3U96a→M3U96b **0.945 Gbit/s** (5 s, TCP) = the 1 GbE ceiling; iperf3 not installed on M2U64/M4M64c, failed56 to parse on M4M64a (to rerun). **No Thunderbolt link is active on any Studio** (all `Thunderbolt N` interfaces `status: inactive`57 on M3U96a/M3U96b/M4M64a/M2U64/M4M64c/M5M36) → Thunderbolt/JACCL RDMA is **blocked pending physical cabling** (user action); the58 distributed gates (E/F) will use TCP over 1 GbE, which is the honest topology of this cluster.59- Second network probe (`artifacts/private/netprobe2-2026-11-01.txt`, iperf3 installed on M2U64 and M4M64c via Homebrew):60 M3U96a → M4M64a / M2U64 / M4M64c / M3U96b all **0.945–0.947 Gbit/s**; RTT M4M64a→M3U96b 0.87 ms, M4M64a→M2U64 0.69 ms.61- Offline simulation v1 (`benchmarks/processed/gateC/sim_olmoe_laptop_ssd_v1.json`, 18-prompt traces, per-layer tables, cold per62 prompt, 1 loader, t_read 0.43 ms, t_layer 0.128 ms): LRU hit rate 0.57 (cap 16) / 0.80 (32) / 0.93 (48); LFU ≈ LRU;63 one-layer-ahead transition predictor recall@8 = 0.50 held-out (0.53 fit), raises hit rate to 0.66 / 0.83 / 0.94 at the cost of64 +40–60 % loads per token; the linear probe on previous-layer log-probabilities underperforms (recall 0.16, too little data → v265 traces); the oracle upper bound reaches 0.97 (cap 32). `prev_token` is a no-op by construction with per-layer tables (the previous66 token's experts are still resident), and `prev_rank` with m ≤ K prefetched nothing (fixed: ranks K+1..K+m).67- Online prefetch smoke test at 16 slots (held-out general/code, page-cached file, 2 reader threads, m = 8): transition raised the hit rate68 (0.48→0.63, 0.67→0.73) but **lowered decode throughput** (51→35, 73→56 tok/s) — a one-layer lead (≈ 0.13 ms) cannot hide a 0.21 ms69 fetch, late prefetches are waited for, and wrong predictions add traffic. H1 at this tier/model is therefore *not supported by a70 1-layer horizon*; deeper horizons (transition2/3/4) and the token horizon are being simulated (sim v2 on the 65-prompt traces).71- Routing traces v2 collected (`benchmarks/raw/gateC/traces_olmoe_v2_laptop/`, 65 prompts, 6 families incl. long-context and drift,72 up to 384 generated tokens each).73- **Qwen3-30B-A3B-Instruct-2507-4bit on M3U96a** (M3 Ultra 96 GB): checkpoint is in the *stacked* `switch_mlp` layout → registry74 extended (per-expert byte ranges = slices of the first axis; `tests/test_registry_stacked.py` byte-equal to `mx.load`). 2.53 MiB per75 expert, 16.3 GB of experts, 0.87 GB dense, 384 experts (0.97 GiB) per decode token; routers are 8-bit. Tests on M3U96a76 (`benchmarks/raw/tests/m3u96a/pytest_qwen3_m3u96a_20261101T014415.txt`, 6 passed): logits **max |Δ| = 0.0** at 128/64/32/16 slots,77 greedy 32-token decode identical at 32 slots. Baselines (`benchmarks/raw/gateC/qwen3_m3u96a/baseline.jsonl`, 64 tokens, 3 repeats):78 mlx-lm **121.3–121.6 tok/s** decode, prefill 339–434 tok/s; TITAN all-resident (128 slots, preloaded) **121.5–121.9 tok/s** — parity.79- Cluster venvs ready on M3U96a, M3U96b, M4M64a, M2U64, M4M64c (`scripts/cluster_setup.sh`, mlx 0.32.4 / mlx-lm 0.32.2 everywhere).80- Paper skeleton v0.1 compiles (24 pages; `tab:sim` pending the v2 simulation). Cold-NVMe prefetch sweep launched on the laptop81 (`benchmarks/scripts/run_cold_nvme_laptop.sh`: purge + F_NOCACHE, 16/32 slots, none/transition{1,2,3}/prev_rank, held-out).82- Offline simulation v2 (`benchmarks/processed/gateC/sim_olmoe_v2_ssd043_m8.json`, 65 prompts, per-layer LRU, t_read 0.43 ms, 1 loader),83 held-out: LRU hit 0.581 / 0.706 / 0.806 / 0.933 at C = 16 / 24 / 32 / 48. Transition predictor recall@8 **0.534 at depth 1, 0.52584 at depth 2, 0.517 at depth 3** → a 3-layer lead costs almost no accuracy, which is what the lead-time condition needs on a cold NVMe85 (d·τ_layer ≥ λ → d ≥ 4 at 0.43 ms). Hit rate with transition prefetch 0.686 / 0.771 / 0.840 / 0.942 (+0.01–0.10) at +60–100 % loads;86 prev_rank (m = 16) *hurts* (prefetch precision 0.18–0.47, evicts useful experts); the linear probe was broken by a label-alignment87 bug (rows not offset across traces) → fixed, re-simulated (`sim_olmoe_v2_ssd043_m8_probe_fixed.json`); oracle 0.94–0.99.88- Cold NVMe tier (purge + F_NOCACHE), 16 slots, held-out general prompts, laptop: **v1** (9 `pread` per expert, serial) 8–10 tok/s,89 stall 7–8 s / 64 tokens (`benchmarks/raw/gateC/cold_nvme_laptop/sweep_v1_9preads_serial.jsonl`); **v2** parallel demand reads90 (4 reader threads) 18 tok/s, stall 3.0 s (`quick_v2.jsonl`); finding: in the mlx-community OLMoE file the 9 tensors of one expert are91 scattered over a 1.5 GB span (hash order of keys), so every miss costs 9 random reads. **v3**: `scripts/convert_layout.py` writes an92 expert-contiguous blob (one 4 KiB-aligned 3.4 MB record per expert, 3.6 GB written in 3.2 s; `~/.cache/titan/<hash>/`), the registry93 points at it transparently, tests verify byte equality with the original (`tests/test_registry_store.py`, one pread per expert) →94 **36 tok/s**, stall 1.2 s / 64 tokens (`quick_v3.jsonl`), vs 51–73 tok/s from the page cache. Cold sweep restarted with v3.95- **Gate E (transport)** implemented: `transport/protocol.py` (TX1 frames, HMAC-SHA256 on every frame with a pre-shared token kept in96 `~/.titan_token`, mode 600, never in the repo), `transport/worker.py` (resident expert subset, exact gather_qmm + swiglu per (row, k)97 pair, IP allow-list, timeouts), `transport/client.py`, `titan/distributed.py` (dispatch to every worker before waiting on any, local98 execution of the rest, assembly in original (row, k) order, local fallback on failure). Localhost two-process test on M3U96a99 (`benchmarks/raw/tests/m3u96a/pytest_distributed_localhost_*.txt`, `distributed_localhost_olmoe_*.json`): odd experts of every layer100 executed by the worker → logits **max |Δ| = 0.0** vs mlx-lm, greedy 24 tokens identical, 16 layers remote per token, localhost RTT101 0.11–0.14 ms. Two-node runs (coordinator M3U96a ↔ worker M3U96b over 1 GbE) launched: `benchmarks/raw/gateE/two_node_m3u96a_m3u96b.jsonl`102 (local control at 32 slots: 68–88 tok/s page-cached; `odd` placement at 32 and 64 slots pending).103- Cold sweep v3 (in progress, 16 slots): none ≈ 36 tok/s; transition2 31.7–34.1 (stall 0.82–0.92 s vs ≈ 1.2 s), transition3 26.8–28.0,104 prev_rank (m = 16) 18–19 tok/s → deeper horizons cut the stall by ~30 % but the host-side cost of prefetch (installs, GIL contention of105 reader threads, eviction churn) cancels the gain on this small model; to be attributed with per-token timers after the sweep.106- **Two-node result (Gate E, H2/H3 first evidence)** — coordinator M3U96a, worker M3U96b, OLMoE, 32 slots, page-cached, 64-token107 greedy, held-out general/code (`benchmarks/raw/gateE/two_node_m3u96a_m3u96b.jsonl`): local control 70.1 / 87.0 tok/s (median,108 hit 0.74 / 0.84); **odd experts executed remotely: 15.0 / 14.9 tok/s**, logits still bit-identical (Δ = 0.0), remote wait109 2.5 s per 64 tokens = **≈ 2.5 ms per layer round trip** (TCP RTT 0.5–0.65 ms + worker-side array creation, gather_qmm dispatch,110 eval and serialisation) × 16 layers ≈ 40 ms per token. On this cluster remote execution over 1 GbE therefore costs more per layer111 than a cold-NVMe demand miss (≈ 1.2 ms per layer at 16 slots) and far more than local compute (0.13 ms): as the formulation112 predicts, the remote tier cannot be a latency win for a model whose experts fit on local NVMe; it can only add *capacity*.113 Worker-side cost breakdown to be measured once the sweep finishes.114- **Cold-NVMe sweep v3 complete** (`benchmarks/raw/gateC/cold_nvme_laptop/sweep.jsonl`, 12 runs per cell, held-out general/code/reasoning,115 64 tokens, purge + F_NOCACHE, 4 readers, contiguous layout): 16 slots — none **36.4 tok/s** (hit 0.53, 267 MB/token from SSD),116 transition d=1 27.9, **d=2 35.0** (hit 0.65, stall −33 %: 0.77 s vs 1.15 s per 64 tokens), d=3 29.1, transition2 m=16 25.9,117 prev_rank m=16 19.1; 32 slots — none **50.6**, transition 41.3, transition2 42.0, transition2 m=16 41.0, transition3 41.0, prev_rank 28.4.118- **Host-time attribution** (`attribution.jsonl`, per token, 16 slots, cold): none = sync 14.7 ms (GPU graph incl. the installs of missed119 experts) + reads 19.9 ms; transition2 = sync 20.4 + prefetch bookkeeping 1.9 + reads 18.8 ms with 58 prefetches/token (1.8 late).120 Confidence thresholds (`transition2t50`, `t70`: 54 and 31 prefetches/token) leave sync at 20.0–20.3 ms. Mechanism: every installed121 expert costs ≈ 9 lazy slice writes (≈ 0.1 ms of graph work on the critical path); with ≈ 50 % prefetch precision the installs double122 while demand reads fall only 30 % → **H1 not supported on OLMoE/laptop NVMe**: hit +0.14, throughput −4 to −15 %. The oracle123 (hit 0.94) shows the ceiling; the lever is install cost per expert and prediction precision, not lead time (d = 2 already suffices).124- **Gate D profiling** on M4M64a (`benchmarks/raw/gateD/profile_olmoe_m4m64a.json`, `launch_overhead_m4m64a.txt`): decode step 2.99 ms125 sync (334 tok/s), effective 151 GB/s on 452 MB of expert bytes per token (≈ 28 % of the M4 Max's nominal bandwidth); isolated126 per-layer timings are dominated by the ≈ 0.15–0.17 ms synchronisation floor; in-graph elementwise ops cost 4.7–7.2 µs each; a127 `gather_qmm` for k = 1…16 experts takes the same 0.17 ms (latency-bound), 250 GB/s only at k = 64. Decode at batch 1 is therefore128 launch/latency-bound: fusing up+gate into one gather_qmm saves one dispatch per layer (≈ 16 × ~10 µs per token, ≤ 5 %); larger129 gains need token batching, not kernels. H4 is tested only with the cheap fusion (next entry).130- **Remote cost breakdown** M3U96a→M3U96b (`benchmarks/raw/gateE/remote_cost_breakdown_m3u96a_m3u96b.txt`): application ping 0.74–1.17 ms;131 exec round trip 2.09 ms (1 pair), 1.99 (4), 2.59 (8 pairs = one decode layer), 7.27 (8 rows × 64 pairs), 19.1 ms (32 rows × 256 pairs,132 1 MB returned ≈ 9 ms on the wire) → ≈ 1.2–1.7 ms of worker-side overhead per call on top of the RTT.133- **H4 cheap fusion** (`titan/slots.py` `fused_upgate`, `tests/test_fused_upgate.py` → logits Δ = 0.0 at 64 and 16 slots;134 `benchmarks/raw/gateD/fused_upgate_laptop.jsonl`): one gather_qmm over a [C, 2H, D] up+gate buffer vs two kernels:135 all-resident 475.6 vs 476.7 tok/s (12 runs, −0.2 %), cold 16 slots 35.9 vs 35.0 tok/s (4 runs, +2.6 %, within the run-to-run range).136 As the profiling predicted, saving one dispatch per layer is immaterial at batch 1 → **H4 not supported** for this fusion; no custom137 Metal kernel is justified by the profile (decode is launch/latency-bound, 28 % of nominal bandwidth).138- **Linear probe fixed and evaluated** (`sim_olmoe_v2_ssd043_m8_probe_fixed.json`): one-layer-ahead logistic probe on the previous139 layer's router log-probabilities → recall@8 **0.761 held-out** (0.779 fit), prefetch precision 0.74–0.83, simulated hit rate 0.81 /140 0.86 / 0.90 / 0.97 at C = 16 / 24 / 32 / 48 (none: 0.58 / 0.71 / 0.81 / 0.93; oracle 0.94 / 0.96 / 0.97 / 0.99); modelled stall 10.0 ms vs141 23.1 ms per token at C = 16. Depth sweep (probe2–4) running (`sim_olmoe_v2_ssd043_m8_probe_depths.json`).142- **Online probe on the cold tier** (16 slots, held-out general, `attribution.jsonl`): probe d=1 hit 0.775, reads 12.8 ms/token (none 19.9)143 but sync 19.7 + prefetch 3.3 ms (6.6 late waits/token) → **31.8 tok/s vs 33.0 none**; probe m=16 hit 0.847, reads 8.3 ms, but 127144 prefetches/token and 25 late waits → 25.3; probe2 31.3 (hit 0.74), probe3 30.7 (hit 0.72). Every predictor converts 7–11 ms of saved145 reads into ≈ 5 ms of extra GPU-side install work plus 1–3 ms of late waits: on this model and tier the install path (9 lazy slice146 writes ≈ 0.13 ms per expert) is the bottleneck, not prediction accuracy. Identified fix for v0.2: single-copy record install.147- Probe depth simulation (`sim_olmoe_v2_ssd043_m8_probe_depths.json`, held-out, C = 16): recall@8 0.761 / 0.743 / 0.731 at depth 1 / 2 / 3,148 hit 0.81 / 0.78 / 0.76 → the probe also keeps its accuracy with a longer lead.149- Gate F preparation: Qwen3-30B routing traces collected on M3U96a (`benchmarks/raw/gateF/traces_qwen3_m3u96a`, 65 prompts, 128 tokens);150 contiguous layout built on M3U96a (6 144 records, 16.3 GB). The LAN copy M3U96a → M3U96b first failed on a **stale SSH host key** for151 192.168.2.85 in M3U96a's `known_hosts` (the LAN was renumbered by DHCP on 2026-09-21). Verified that the key presented on the LAN152 (`SHA256:zcdc123MTDYreNavJ9lUAS1OWSFSzxvxipqaBSygHsA`) is M3U96b's actual `/etc/ssh/ssh_host_ed25519_key.pub` fingerprint before153 replacing the entry (backup `~/.ssh/known_hosts.bak-20261101-wp33` on M3U96a); copy and layout on M3U96b restarted.154- **Energy run (H5, single node)** (`benchmarks/raw/gateG/energy_laptop/`, powermetrics 1 s, 411 samples, idle 13.2 W, 256-token greedy,155 held-out general/code, 12 runs per configuration): mlx-lm 463 tok/s at 47.2 W → **0.101 J/token** gross (0.073 net of idle);156 TITAN all-resident 462 tok/s, 46.4 W, 0.101 J/token; TITAN 16 slots cold NVMe 31.9 tok/s at 15.9 W → 0.498 J/token gross (0.086 net);157 **16 slots cold + probe prefetch 42.0 tok/s** at 18.4 W → 0.441 J/token gross (0.119 net). New H1 evidence: with 256-token generations158 the probe beats demand loading by +32 % (the 64-token runs showed a tie) — see audit CIs; the hit rate reaches 0.84 over longer159 sequences while the per-token install overhead is unchanged. Energy favours prefetch per token in gross terms (the machine is on160 anyway) but not net of idle; all-resident decoding is 4–5× cheaper per token than any cold-tier configuration.161- **Gate F, first sweep discarded for exactness** (`benchmarks/raw/gateF/two_node_qwen3_m3u96a_m3u96b_v1_fp16wire.jsonl`,162 `run_gateF_v1_fp16wire.log`): with Qwen3-30B the remote path gave max |Δlogit| = 0.65 (odd) / 0.43 (popular) vs mlx-lm because the163 transport cast bfloat16 activations and results to float16 on the wire (OLMoE is float16-native, hence its Δ = 0.0). Fix: activations164 and results travel in the model's dtype (bfloat16 as uint16 bit patterns, dtype tag in the header); localhost Qwen3 test on M3U96a now165 Δ = 0.0, greedy identical (`benchmarks/raw/tests/m3u96a/pytest_distributed_localhost_qwen3_*`). Throughputs of the discarded sweep,166 kept for the record (cold local tier on M3U96a, 64 tokens): none 8.7–8.9 tok/s at 32 slots (stall 65 ms/token), 15–20 at 64 slots;167 odd 5.4–5.8 / 4.1; popular 6.6–7.0 / 4.5–5.5; remote wait 70–146 ms/token (48 layers × 1.5–3 ms). Sweep relaunched with the fix.168- Earlier the relaunched sweep had also crashed on "There is no Stream(gpu, N) in current thread": MLX arrays created in reader threads169 cannot be evaluated in another thread; fixed by returning numpy from reader threads and creating MLX arrays on the consuming thread170 (`titan/store.py` `to_mx`, `titan/prefetch.py`).171- **Gate F (exact) complete** (`benchmarks/raw/gateF/two_node_qwen3_m3u96a_m3u96b.jsonl`, 16 runs per cell, Δlogit = 0.0 everywhere):172 Qwen3-30B-A3B, coordinator M3U96a with cold NVMe local tier (purge + F_NOCACHE), worker M3U96b over 1 GbE. 32 slots: local **8.8** tok/s173 (hit 0.78, stall 66 ms/token), odd 4.2 (remote wait 112 ms/token, 14 183 remote pairs), popular 5.1 (67 ms, 9 161 pairs); 64 slots:174 local **16.3**, odd 5.2, popular 6.0. CIs: popular − local −3.6 [−4.4, −2.9] (32), −10.2 [−13.6, −8.8] (64). H2 weak form supported175 (frequency-based > static), strong form not (remote < local NVMe on 1 GbE). Paper updated (Section Gate F, figure `gateF_qwen3.pdf`,176 table `tab:gateF`, conclusion rewritten).177