1# WP33 TITAN-MoE — final report of session 1 (2026-11-01)23Everything below links to a file in this repository; nothing is claimed without one. Status words: **implemented** (code exists and is4exercised by a test or a benchmark), **verified** (a raw log under `benchmarks/raw/` supports the stated number), **unverified**5(implemented, no measurement yet), **blocked** (needs a resource or authorization the agent does not have).67## 1. Implemented and verified89| Deliverable | Evidence |10|---|---|11| Live hardware inventory (21 reachable Macs; 8 aliases unreachable), 1 GbE everywhere, no active Thunderbolt link | `artifacts/environment.json`; raw probes in `artifacts/private/` (not committed); `research/LOG.md` 2026-11-01 |12| Network characterization: iperf3 0.945–0.947 Gbit/s between Studios, RTT 0.5–1.0 ms, app-level ping 0.74–1.17 ms | `artifacts/private/netprobe*.txt`, `benchmarks/raw/gateE/remote_cost_breakdown_m3u96a_m3u96b.txt` |13| Prior-art audit: 59 verified works + code audit of colibri/mlx/mlx-lm/llama.cpp at recorded commits; novelty matrix | `research/prior_art.csv`, `research/RELATED_WORK.md`, `paper/references.bib` |14| Expert registry (per-expert and stacked safetensors layouts; byte ranges; routing semantics) | `titan/registry.py`, `tests/test_registry_store.py`, `tests/test_registry_stacked.py` (M3U96a) |15| SSD tier: `pread` store, parallel demand reads, expert-contiguous TITAN layout (one read per expert) | `titan/store.py`, `scripts/convert_layout.py`; cold 16-slot decode 8–10 → 18 → 36 tok/s (`benchmarks/raw/gateC/cold_nvme_laptop/`) |16| Slot tables (LRU/LFU), exact execution block over `gather_qmm`, sync-free fast path, lazy batched installs | `titan/slots.py`, `titan/moe.py`; bit-identical logits vs mlx-lm (OLMoE 64/32/16/8 slots; Qwen3-30B 128/64/32/16): `benchmarks/raw/tests/` |17| All-resident parity: 489.1 [488.2, 489.5] vs 495.0 tok/s (laptop); 121.6 vs 121.7 (M3U96a, Qwen3-30B) | `benchmarks/processed/audit.json`, `benchmarks/raw/gateB/capacity_sweep_laptop/`, `benchmarks/raw/gateC/qwen3_m3u96a/` |18| Routing traces (18 + 65 prompts OLMoE; 65 prompts Qwen3-30B) and trace-driven simulator | `benchmarks/raw/gateC/traces_*`, `benchmarks/raw/gateF/traces_qwen3_m3u96a`, `benchmarks/scripts/simulate_cache.py`, `benchmarks/processed/gateC/` |19| Predictors (prev-rank, transition depth 1–4 ± confidence threshold, logistic probe depth 1–4, oracle) and async prefetcher | `titan/predictors.py`, `titan/prefetch.py`; probe recall@8 0.761 held-out |20| **H1 verdict (not supported on OLMoE / laptop NVMe)** with full host-time attribution | `benchmarks/raw/gateC/cold_nvme_laptop/{sweep,attribution}.jsonl`, `figures/attribution.pdf`, audit CIs |21| Gate D profiling (launch-bound decode, 151 GB/s = 28 % of nominal) and fused up+gate option (bit-identical, ±2 %) → **H4 not supported** | `benchmarks/raw/gateD/`, `tests/test_fused_upgate.py` |22| Authenticated TCP transport (HMAC per frame, allow-list, timeouts), expert worker, remote executor with local fallback; bit-identical on localhost and across M3U96a↔M3U96b | `transport/`, `titan/distributed.py`, `tests/test_distributed_local.py`, `benchmarks/raw/tests/m3u96a/distributed_localhost_*.json`, `benchmarks/raw/gateE/two_node_m3u96a_m3u96b.jsonl` |23| **Two-node result**: odd experts remote over 1 GbE = 15 vs 81 tok/s local (−66 [−72, −55]); 2.5 ms per layer round trip | same + `remote_cost_breakdown_*.txt` |24| **Gate F (Qwen3-30B, two Studios, exact)**: frequency-based placement > static (5.1 vs 4.2; 6.0 vs 5.2 tok/s) but < local cold NVMe (8.8; 16.3) → H2 weak form only | `benchmarks/raw/gateF/two_node_qwen3_m3u96a_m3u96b.jsonl`, `figures/gateF_qwen3.pdf`, audit CIs |25| **H5 energy (laptop)**: 0.101 J/token all-resident (47 W); 0.498 J/token 16 slots cold; 0.441 with probe (42.0 vs 31.9 tok/s at 256 tokens: H1 length-dependent) | `benchmarks/raw/gateG/energy_laptop/`, `benchmarks/processed/energy_laptop.json` |26| Statistical audit (bootstrap CIs for every comparison), figures and tables generated from logs, manifest with SHA-256 | `scripts/audit_stats.py`, `scripts/make_figures.py`, `scripts/make_tables.py`, `artifacts/manifest.json` |27| Paper v0.1 (30 pages, compiles, 0 undefined references), preliminary/negative results labelled | `paper/main.pdf` |2829## 2. Implemented, running or unverified at the time of writing30| Item | State |31|---|---|32| (moved to §1) Gate F and H5 energy | see §1 |33| Adaptive scheduler (online cost re-estimation, bandits) | not implemented (D-012): the measured cost model predicts no winning placement for a model that fits local NVMe over 1 GbE; the static frequency-based placement is exercised by Gate F |34| Replication of a hot set on the coordinator | not implemented |35| Colibri and llama.cpp speed baselines | not run (non-comparable precision; flagged in the protocol) |36| Core ML / ANE feasibility | not explored (documented as out of scope for v0.1) |3738## 3. Blocked (needs the author)39| Item | Why |40|---|---|41| Thunderbolt 5 / RDMA (JACCL) tier | no Thunderbolt cable between any two nodes (all TB interfaces inactive); physical cabling required |42| Large model tier (gpt-oss-120b, 62 GB) — the regime where a node cannot hold the experts | download > 20 GB requires explicit authorization (D-003, D-008) |43| 8 unreachable aliases (M2M32, M2M32b, M4M36, m4ma, m4mb, m4mc, m2m16, M4BP48) | SSH timeout at inventory time |44| Push to spbgit (`wp33_uqo.git`) | mandate: no push without authorization; local commits only |4546## 4. What the evidence says, in one paragraph47The runtime is exact and free when everything fits; when it does not, the storage layout and the number of host-side operations48per missed expert dominate, not prediction accuracy and not kernel fusion; and gigabit Ethernet makes remote execution a capacity49tool rather than a speed tool. The next version should (i) install an expert with one copy instead of nine slice writes, then re-run50the probe experiments; (ii) run the medium model on the 36–48 GB nodes with experts beyond their budget; (iii) measure energy on the51cluster, not only on the laptop; (iv) if cabled, add the Thunderbolt tier.52