1\section{Results}2\label{sec:results}34Every number below has a raw log in \code{benchmarks/raw/}; the path is given with each result. Results marked5\textsc{pending} have a protocol but no log yet in this version.67\subsection{Correctness}8\label{sec:results-correctness}9On four prompts (general, code, reasoning, French) \titan{} reproduces mlx-lm's logits with $\max|\Delta| = 0.0$ (float16) at10capacities 64, 32, 16 and 8 slots per layer, with identical argmax at every position and identical greedy continuations of1132 tokens at 16 slots (\code{benchmarks/raw/tests/numerical\_*.json}, \code{greedy\_*.json}, \code{pytest\_gateB\_*.txt},127 tests passed). The registry's byte ranges read with \code{pread} are byte-equal to \code{mx.load} for sampled experts of both the13per-expert layout (OLMoE) and the stacked layout (Qwen3-30B-A3B, on M3U96a). Qwen3-30B-A3B logit parity:14\textsc{pending} (test launched on M3U96a; result to be inserted from its log).1516\subsection{All-resident control and demand loading}17\label{sec:results-capacity}18Table~\ref{tab:capacity} summarises the laptop capacity sweep (64-token greedy generations, general and code prompts,19page-cached checkpoint, \code{benchmarks/raw/gateB/capacity\_sweep\_laptop/sweep.jsonl} and20\code{benchmarks/raw/gateC/prefetch\_online\_laptop/smoke\_lazyinstall.jsonl}).2122\begin{table}[t]23\centering\small24\caption{OLMoE-1B-7B 4-bit on the M5 Max laptop: decode throughput vs residency (2 repeats after warm-up; ranges over prompts).}25\label{tab:capacity}26\begin{tabular}{llrrr}27\toprule28Backend & Residency & decode \tps{} & hit rate & stall / 64 tokens \\29\midrule30mlx-lm & all & 491--496 & -- & -- \\31\titan{} & 64 slots, preloaded & 489 & 1.00 & 0 \\32\titan{} & 64 slots, not preloaded (sync path) & 201--203 & 1.00 & 0 \\33\titan{} & 32 slots, LRU, demand (v1 installs) & 56--80 & 0.73--0.81 & 0.7--1.0 s \\34\titan{} & 16 slots, LRU, demand (v1 installs) & 36--45 & 0.48--0.60 & 1.7--2.0 s \\35\titan{} & 16 slots, LRU, demand (lazy installs) & 51--73 & 0.48--0.67 & 0.6--0.9 s \\36\titan{} & 16 slots, LRU + transition prefetch ($d{=}1$, $m{=}8$) & 35--56 & 0.63--0.73 & 0.6--1.1 s \\37\bottomrule38\end{tabular}39\end{table}4041Two observations. The all-resident control matches mlx-lm within one percent only when every layer's table is complete;42otherwise each layer pays a host synchronisation, which alone costs 2.4$\times$ in throughput on this model and is the floor43of any demand-loading design, including colibri's. And the demand-loading rows are the H1 \emph{control}: at 16 slots the44model does 53 misses per token (Prop.~\ref{prop:capacity}), i.e.\ about 11~ms of fetch per 2~ms of compute.4546\subsection{Routing locality}47\label{sec:results-locality}48From the v1 traces (18 prompts, 3\,686 tokens), per-layer LRU hit rates are 0.57 at 16 slots, 0.80 at 32 and 0.93 at 48 (LFU49within 0.01), nearly identical on fit and held-out prompts (\code{benchmarks/processed/gateC/sim\_olmoe\_laptop\_ssd\_v1.json});50a one-layer-ahead transition predictor has recall@8 of 0.50 on held-out prompts. On the v2 traces (65 prompts, 6 families,51\code{benchmarks/processed/gateC/}) the transition predictor keeps almost all of its accuracy when it looks further ahead52(recall@8 0.534 / 0.525 / 0.517 at depth 1 / 2 / 3), and a per-layer logistic probe on the previous layer's router53log-probabilities reaches recall@8 of \textbf{0.761} held-out (0.779 fit) with prefetch precision 0.74--0.83, lifting the simulated54hit rate at 16 slots from 0.58 (LRU) to 0.81 (oracle 0.94). A first version of the probe scored 0.16 because of a label-alignment55bug across traces; the fix is recorded in the log and the buggy rows are excluded from Table~\ref{tab:sim} and Figure~\ref{fig:sim}.56The rank-based previous-token predictor hurts at every capacity (precision 0.18--0.47, it evicts useful experts).5758\input{tables/sim}5960\begin{figure}[t]61\centering62\includegraphics[width=\textwidth]{sim_hit_vs_capacity.pdf}63\caption{Trace-driven simulation on the v2 held-out prompts: per-layer hit rate (left) and modelled stall per token (right)64by slot capacity and predictor. Generated by \code{scripts/make\_figures.py} from \code{benchmarks/processed/gateC/}.}65\label{fig:sim}66\end{figure}6768\subsection{Prefetching (H1)}69\label{sec:results-prefetch}70\paragraph{Page-cache tier, one-layer horizon.} The online prefetcher raised the hit rate from 0.48 to 0.63 (general) and from710.67 to 0.73 (code) but reduced throughput from 51 to 35 and from 73 to 56~\tps{} (Table~\ref{tab:capacity}): the lead time72$d\,\tau_\ell = 0.13$~ms is shorter than the 0.21~ms fetch, so predicted experts arrive late, and the 50\,\% of wrong predictions73add traffic. This is the regime excluded by condition~\eqref{eq:condition}.7475\paragraph{Cold-NVMe tier.} The SSD tier itself had to be engineered before the question could be asked: in the published76OLMoE checkpoint the nine tensors of one expert are scattered over a 1.5~GB span of the file (hash order of keys), so each miss77cost nine random reads and 16-slot decoding ran at 8--10~\tps{}; parallel demand reads raised it to 18 and the expert-contiguous78layout of Section~\ref{sec:architecture} (one 4~KiB-aligned record per expert, one \code{pread}) to 36~\tps{}79(\code{benchmarks/raw/gateC/cold\_nvme\_laptop/}). Table~\ref{tab:cold} then compares predictors on that tier with the page cache80purged before every configuration: at 16 slots the depth-2 transition predictor cuts the stall by a third (0.77 vs 1.15~s per 6481tokens) and ties with demand loading (35.0 vs 36.4~\tps{}); every other predictor, including depth 1 and 3 and the rank-based one,82loses; at 32 slots all of them lose to demand loading (41--42 vs 50.6~\tps{}).8384\paragraph{Attribution.} Figure~\ref{fig:attribution} and Table~\ref{tab:attr} split the host time per token into the wait for85the GPU graph (which includes the slice writes that install experts), prefetch bookkeeping and late waits, and demand reads.86The probe raises the hit rate to 0.78 and cuts demand reads from 19.9 to 12.8~ms per token, but the GPU wait grows from 14.787to 19.7~ms and late waits add 3~ms: each installed expert costs nine lazy slice writes (about 0.13~ms of graph work on the88critical path), and the prefetcher installs more experts than it saves reads for. Confidence thresholds and deeper horizons89change the mix, not the sum. \textbf{H1 is therefore not supported on this model and tier}: the best predictor ties with no90prefetch. The attribution is positive knowledge: the bottleneck is the install path, not prediction accuracy (the probe is91already within 0.16 of the oracle's hit rate at 16 slots), so a single-copy install (one record per expert written in one92operation) is the change that would make the probe's 7~ms of saved reads count, and it is the first item of version~0.2.9394\input{tables/attr}9596\begin{figure}[t]97\centering98\includegraphics[width=\textwidth]{attribution.pdf}99\caption{Host time per token on the cold-NVMe tier (OLMoE, 16 slots, held-out general prompts): GPU wait including installs,100prefetch bookkeeping and late waits, demand reads. Labels give the median decode throughput and hit rate.}101\label{fig:attribution}102\end{figure}103104\input{tables/cold}105106\begin{figure}[t]107\centering108\includegraphics[width=\textwidth]{capacity_sweep.pdf}109\caption{Decode throughput of every single-node configuration measured on the laptop (median over repeats, bar = range):110mlx-lm, \titan{} all-resident with and without the synchronisation-free path, bounded residency on the page-cache tier, and the111cold-NVMe tier with and without prefetch. Source files are named in brackets.}112\label{fig:capacity}113\end{figure}114115\subsection{Medium model (Qwen3-30B-A3B)}116On M3U96a (M3 Ultra, 96~GB) the stacked-layout registry reads byte-equal slices, logits are bit-identical to mlx-lm at 128, 64,11732 and 16 slots per layer and greedy decoding is token-identical at 32 slots118(\code{benchmarks/raw/tests/m3u96a/}). Table~\ref{tab:qwen3} gives the baselines: mlx-lm 121.4~\tps{} and \titan{} all-resident119121.7~\tps{} (medians of 3 repeats), i.e.\ parity on a model whose decode step touches 0.97~GiB of expert weights.120121\input{tables/qwen3}122123\subsection{Two-node execution over gigabit Ethernet (H2, H3: first evidence)}124\label{sec:results-twonode}125With M3U96a as coordinator and M3U96b as worker (OLMoE, 32 slots, page-cached checkpoint on both, held-out general and code126prompts, \code{benchmarks/raw/gateE/two\_node\_m3u96a\_m3u96b.jsonl}), executing the odd-numbered experts of every layer on the127worker keeps the logits bit-identical to mlx-lm but reduces decode throughput from 70 / 87~\tps{} (local control) to12815 / 15~\tps{} (Figure~\ref{fig:twonode}). The remote wait is 2.5~s per 64 tokens, i.e.\ about 2.5~ms per layer: a 0.5--0.65~ms TCP129round trip plus the worker's array creation, kernel dispatch, evaluation and serialisation, paid 16 times per token. This is130larger than a cold-NVMe demand miss at 16 slots (about 1.2~ms per layer) and twenty times the local compute per layer. On this131topology the remote tier is therefore a capacity mechanism, not a latency one, exactly as Section~\ref{sec:prefetch-condition}132anticipates; H2 can only be tested in the regime where a node cannot hold the experts it needs locally.133134\begin{figure}[t]135\centering136\includegraphics[width=0.6\textwidth]{two_node.pdf}137\caption{Two Mac Studios over 1~GbE: local bounded residency vs remote execution of half the experts (medians).}138\label{fig:twonode}139\end{figure}140141The round trip was measured directly (\code{benchmarks/raw/gateE/remote\_cost\_breakdown\_m3u96a\_m3u96b.txt}): application-level142ping 0.74--1.17~ms; one layer's request (one row, eight pairs) 2.59~ms; eight rows with 64 pairs 7.3~ms; 32 rows with 256 pairs14319.1~ms, of which about 9~ms is the 1~MB reply on the wire. The worker therefore adds 1.2--1.7~ms per call to the network144latency, which bounds what a faster transport could recover without also changing the worker's execution path.145146\subsection{Metal kernels (H4)}147\label{sec:results-metal}148Profiling on M4M64a (\code{benchmarks/raw/gateD/}) shows that a decode step with all experts resident takes 2.99~ms149(334~\tps{} synchronous) and moves 452~MB of expert weights, i.e.\ 151~GB/s or about 28\,\% of the chip's nominal bandwidth;150a \gatherqmm{} over 1 to 16 experts takes the same 0.17~ms (latency-bound; 250~GB/s only at 64 experts), and in-graph151elementwise ops cost 5--7~$\mu$s each against a 0.16~ms synchronisation floor. The step is launch-bound, so the most a kernel152fusion can save is a few dispatches per layer. We tested the cheapest one, a single \gatherqmm{} over a stacked up+gate buffer153(bit-identical logits, \code{tests/test\_fused\_upgate.py}): $-0.2\,\%$ with all experts resident and $+2.6\,\%$ on the cold tier,154both within the run-to-run range (Table~\ref{tab:fused}). \textbf{H4 is not supported} at batch size one on this model; the profile155says that batching tokens, not kernels, is what would change the regime.156157\input{tables/fused}158159\subsection{Energy per token (H5, single node)}160\label{sec:results-energy}161Table~\ref{tab:energy} reports package power sampled at 1~s by \code{powermetrics} while four configurations decoded 256 tokens on162the laptop (\code{benchmarks/raw/gateG/energy\_laptop/}). All-resident decoding costs 0.101~J per token at 47~W whether mlx-lm or163\titan{} runs it; bounded residency on the cold tier costs five times more per token in gross terms (0.50~J at 16~W) because the164machine waits on the disk most of the time; the probe prefetcher brings this to 0.44~J at 18~W. Net of the 13.2~W idle power the165ordering reverses between the two cold configurations (0.086 vs 0.119~J), which is the usual ambiguity of attributing idle power.166The same run also gives the first \emph{positive} H1 evidence: over 256-token generations the probe reaches a hit rate of 0.84 and16742.0~\tps{} against 31.9 for demand loading ($+10.1$ [$+5.2$, $+16.9$]~\tps{}, Table~\ref{tab:audit}), whereas the 64-token runs168of Section~\ref{sec:results-prefetch} showed a tie: the per-token install overhead is constant while the benefit of a warm, well-predicted169table grows with sequence length. H1 is therefore \emph{length-dependent} on this hardware, and the cluster-level Pareto study remains170pending.171172\input{tables/energy}173