# WP33 TITAN-MoE — final report of session 1 (2026-11-01)

Everything below links to a file in this repository; nothing is claimed without one. Status words: **implemented** (code exists and is
exercised by a test or a benchmark), **verified** (a raw log under `benchmarks/raw/` supports the stated number), **unverified**
(implemented, no measurement yet), **blocked** (needs a resource or authorization the agent does not have).

## 1. Implemented and verified

| Deliverable | Evidence |
|---|---|
| Live hardware inventory (21 reachable Macs; 8 aliases unreachable), 1 GbE everywhere, no active Thunderbolt link | `artifacts/environment.json`; raw probes in `artifacts/private/` (not committed); `research/LOG.md` 2026-11-01 |
| Network characterization: iperf3 0.945–0.947 Gbit/s between Studios, RTT 0.5–1.0 ms, app-level ping 0.74–1.17 ms | `artifacts/private/netprobe*.txt`, `benchmarks/raw/gateE/remote_cost_breakdown_m3u96a_m3u96b.txt` |
| Prior-art audit: 59 verified works + code audit of colibri/mlx/mlx-lm/llama.cpp at recorded commits; novelty matrix | `research/prior_art.csv`, `research/RELATED_WORK.md`, `paper/references.bib` |
| Expert registry (per-expert and stacked safetensors layouts; byte ranges; routing semantics) | `titan/registry.py`, `tests/test_registry_store.py`, `tests/test_registry_stacked.py` (M3U96a) |
| SSD tier: `pread` store, parallel demand reads, expert-contiguous TITAN layout (one read per expert) | `titan/store.py`, `scripts/convert_layout.py`; cold 16-slot decode 8–10 → 18 → 36 tok/s (`benchmarks/raw/gateC/cold_nvme_laptop/`) |
| Slot tables (LRU/LFU), exact execution block over `gather_qmm`, sync-free fast path, lazy batched installs | `titan/slots.py`, `titan/moe.py`; bit-identical logits vs mlx-lm (OLMoE 64/32/16/8 slots; Qwen3-30B 128/64/32/16): `benchmarks/raw/tests/` |
| All-resident parity: 489.1 [488.2, 489.5] vs 495.0 tok/s (laptop); 121.6 vs 121.7 (M3U96a, Qwen3-30B) | `benchmarks/processed/audit.json`, `benchmarks/raw/gateB/capacity_sweep_laptop/`, `benchmarks/raw/gateC/qwen3_m3u96a/` |
| Routing traces (18 + 65 prompts OLMoE; 65 prompts Qwen3-30B) and trace-driven simulator | `benchmarks/raw/gateC/traces_*`, `benchmarks/raw/gateF/traces_qwen3_m3u96a`, `benchmarks/scripts/simulate_cache.py`, `benchmarks/processed/gateC/` |
| Predictors (prev-rank, transition depth 1–4 ± confidence threshold, logistic probe depth 1–4, oracle) and async prefetcher | `titan/predictors.py`, `titan/prefetch.py`; probe recall@8 0.761 held-out |
| **H1 verdict (not supported on OLMoE / laptop NVMe)** with full host-time attribution | `benchmarks/raw/gateC/cold_nvme_laptop/{sweep,attribution}.jsonl`, `figures/attribution.pdf`, audit CIs |
| Gate D profiling (launch-bound decode, 151 GB/s = 28 % of nominal) and fused up+gate option (bit-identical, ±2 %) → **H4 not supported** | `benchmarks/raw/gateD/`, `tests/test_fused_upgate.py` |
| Authenticated TCP transport (HMAC per frame, allow-list, timeouts), expert worker, remote executor with local fallback; bit-identical on localhost and across M3U96a↔M3U96b | `transport/`, `titan/distributed.py`, `tests/test_distributed_local.py`, `benchmarks/raw/tests/m3u96a/distributed_localhost_*.json`, `benchmarks/raw/gateE/two_node_m3u96a_m3u96b.jsonl` |
| **Two-node result**: odd experts remote over 1 GbE = 15 vs 81 tok/s local (−66 [−72, −55]); 2.5 ms per layer round trip | same + `remote_cost_breakdown_*.txt` |
| Statistical audit (bootstrap CIs for every comparison), figures and tables generated from logs, manifest with SHA-256 | `scripts/audit_stats.py`, `scripts/make_figures.py`, `scripts/make_tables.py`, `artifacts/manifest.json` |
| Paper v0.1 (30 pages, compiles, 0 undefined references), preliminary/negative results labelled | `paper/main.pdf` |

## 2. Implemented, running or unverified at the time of writing
| Item | State |
|---|---|
| Gate F: Qwen3-30B two-node sweep (local cold NVMe vs odd vs frequency-based placement, 32/64 slots) | script `benchmarks/scripts/run_gateF_qwen3.sh`; worker on M3U96b; **running / to be read from `benchmarks/raw/gateF/`** |
| H5 energy per token (powermetrics) for 4 single-node configurations | `benchmarks/scripts/run_energy_laptop.sh`, `scripts/analyze_energy.py`; **running** |
| Adaptive scheduler (online cost re-estimation, bandits) | not implemented; the deterministic heuristic (Algorithm 2) is specified and its static version is exercised by Gate F |
| Replication of a hot set on the coordinator | not implemented |
| Colibri and llama.cpp speed baselines | not run (non-comparable precision; flagged in the protocol) |
| Core ML / ANE feasibility | not explored (documented as out of scope for v0.1) |

## 3. Blocked (needs the author)
| Item | Why |
|---|---|
| Thunderbolt 5 / RDMA (JACCL) tier | no Thunderbolt cable between any two nodes (all TB interfaces inactive); physical cabling required |
| Large model tier (gpt-oss-120b, 62 GB) — the regime where a node cannot hold the experts | download > 20 GB requires explicit authorization (D-003, D-008) |
| 8 unreachable aliases (M2M32, M2M32b, M4M36, m4ma, m4mb, m4mc, m2m16, M4BP48) | SSH timeout at inventory time |
| Push to spbgit (`wp33_uqo.git`) | mandate: no push without authorization; local commits only |

## 4. What the evidence says, in one paragraph
The runtime is exact and free when everything fits; when it does not, the storage layout and the number of host-side operations
per missed expert dominate, not prediction accuracy and not kernel fusion; and gigabit Ethernet makes remote execution a capacity
tool rather than a speed tool. The next version should (i) install an expert with one copy instead of nine slice writes, then re-run
the probe experiments; (ii) run the medium model on the 36–48 GB nodes with experts beyond their budget; (iii) measure energy on the
cluster, not only on the laptop; (iv) if cabled, add the Thunderbolt tier.