| 47 |
47 |
\label{sec:results-locality} |
| 48 |
48 |
From the v1 traces (18 prompts, 3\,686 tokens), per-layer LRU hit rates are 0.57 at 16 slots, 0.80 at 32 and 0.93 at 48 (LFU |
| 49 |
49 |
within 0.01), nearly identical on fit and held-out prompts (\code{benchmarks/processed/gateC/sim\_olmoe\_laptop\_ssd\_v1.json}); |
| 50 |
|
−a one-layer-ahead transition predictor has recall@8 of 0.50 on held-out prompts; a linear probe on the previous layer's |
| 51 |
|
−log-probabilities only 0.16 with these few traces. Table~\ref{tab:sim} is generated from the simulation JSON (v2 traces when available, else v1). |
|
50 |
+a one-layer-ahead transition predictor has recall@8 of 0.50 on held-out prompts. On the v2 traces (65 prompts, 6 families, |
|
51 |
+\code{benchmarks/processed/gateC/}) the transition predictor keeps almost all of its accuracy when it looks further ahead |
|
52 |
+(recall@8 0.534 / 0.525 / 0.517 at depth 1 / 2 / 3), and a per-layer logistic probe on the previous layer's router |
|
53 |
+log-probabilities reaches recall@8 of \textbf{0.761} held-out (0.779 fit) with prefetch precision 0.74--0.83, lifting the simulated |
|
54 |
+hit rate at 16 slots from 0.58 (LRU) to 0.81 (oracle 0.94). A first version of the probe scored 0.16 because of a label-alignment |
|
55 |
+bug across traces; the fix is recorded in the log and the buggy rows are excluded from Table~\ref{tab:sim} and Figure~\ref{fig:sim}. |
|
56 |
+The rank-based previous-token predictor hurts at every capacity (precision 0.18--0.47, it evicts useful experts). |
| 52 |
57 |
|
| 53 |
58 |
\input{tables/sim} |
| 54 |
59 |
|
| 62 |
67 |
|
| 63 |
68 |
\subsection{Prefetching (H1)} |
| 64 |
69 |
\label{sec:results-prefetch} |
| 65 |
|
−With the one-layer horizon the online prefetcher raised the hit rate from 0.48 to 0.63 (general) and from 0.67 to 0.73 |
| 66 |
|
−(code) but reduced throughput from 51 to 35 and from 73 to 56~\tps{} (Table~\ref{tab:capacity}): the lead time |
| 67 |
|
−$d\,\tau_\ell = 0.13$~ms is shorter than the 0.21~ms fetch, so predicted experts arrive late and are waited for, and the |
| 68 |
|
−50\,\% of wrong predictions add traffic. This is exactly the regime excluded by condition~\eqref{eq:condition}. |
| 69 |
|
−Depth-$d$ and token-horizon prefetching on the cold-NVMe tier are reported in Table~\ref{tab:cold} (generated from |
| 70 |
|
−\code{benchmarks/raw/gateC/cold\_nvme\_laptop/sweep.jsonl}; rows present only for configurations that have run). |
|
70 |
+\paragraph{Page-cache tier, one-layer horizon.} The online prefetcher raised the hit rate from 0.48 to 0.63 (general) and from |
|
71 |
+0.67 to 0.73 (code) but reduced throughput from 51 to 35 and from 73 to 56~\tps{} (Table~\ref{tab:capacity}): the lead time |
|
72 |
+$d\,\tau_\ell = 0.13$~ms is shorter than the 0.21~ms fetch, so predicted experts arrive late, and the 50\,\% of wrong predictions |
|
73 |
+add traffic. This is the regime excluded by condition~\eqref{eq:condition}. |
|
74 |
+ |
|
75 |
+\paragraph{Cold-NVMe tier.} The SSD tier itself had to be engineered before the question could be asked: in the published |
|
76 |
+OLMoE checkpoint the nine tensors of one expert are scattered over a 1.5~GB span of the file (hash order of keys), so each miss |
|
77 |
+cost nine random reads and 16-slot decoding ran at 8--10~\tps{}; parallel demand reads raised it to 18 and the expert-contiguous |
|
78 |
+layout of Section~\ref{sec:architecture} (one 4~KiB-aligned record per expert, one \code{pread}) to 36~\tps{} |
|
79 |
+(\code{benchmarks/raw/gateC/cold\_nvme\_laptop/}). Table~\ref{tab:cold} then compares predictors on that tier with the page cache |
|
80 |
+purged before every configuration: at 16 slots the depth-2 transition predictor cuts the stall by a third (0.77 vs 1.15~s per 64 |
|
81 |
+tokens) and ties with demand loading (35.0 vs 36.4~\tps{}); every other predictor, including depth 1 and 3 and the rank-based one, |
|
82 |
+loses; at 32 slots all of them lose to demand loading (41--42 vs 50.6~\tps{}). |
|
83 |
+ |
|
84 |
+\paragraph{Attribution.} Figure~\ref{fig:attribution} and Table~\ref{tab:attr} split the host time per token into the wait for |
|
85 |
+the GPU graph (which includes the slice writes that install experts), prefetch bookkeeping and late waits, and demand reads. |
|
86 |
+The probe raises the hit rate to 0.78 and cuts demand reads from 19.9 to 12.8~ms per token, but the GPU wait grows from 14.7 |
|
87 |
+to 19.7~ms and late waits add 3~ms: each installed expert costs nine lazy slice writes (about 0.13~ms of graph work on the |
|
88 |
+critical path), and the prefetcher installs more experts than it saves reads for. Confidence thresholds and deeper horizons |
|
89 |
+change the mix, not the sum. \textbf{H1 is therefore not supported on this model and tier}: the best predictor ties with no |
|
90 |
+prefetch. The attribution is positive knowledge: the bottleneck is the install path, not prediction accuracy (the probe is |
|
91 |
+already within 0.16 of the oracle's hit rate at 16 slots), so a single-copy install (one record per expert written in one |
|
92 |
+operation) is the change that would make the probe's 7~ms of saved reads count, and it is the first item of version~0.2. |
|
93 |
+ |
|
94 |
+\input{tables/attr} |
|
95 |
+ |
|
96 |
+\begin{figure}[t] |
|
97 |
+\centering |
|
98 |
+\includegraphics[width=\textwidth]{attribution.pdf} |
|
99 |
+\caption{Host time per token on the cold-NVMe tier (OLMoE, 16 slots, held-out general prompts): GPU wait including installs, |
|
100 |
+prefetch bookkeeping and late waits, demand reads. Labels give the median decode throughput and hit rate.} |
|
101 |
+\label{fig:attribution} |
|
102 |
+\end{figure} |
| 71 |
103 |
|
| 72 |
104 |
\input{tables/cold} |
| 73 |
105 |
|
| 106 |
138 |
\label{fig:twonode} |
| 107 |
139 |
\end{figure} |
| 108 |
140 |
|
| 109 |
|
−\subsection{Metal kernels (H4), energy frontier (H5)} |
| 110 |
|
−\textsc{pending}: Gates D and the energy study have protocols (Section~\ref{sec:algorithms}, \code{research/protocol.md}) and no |
| 111 |
|
−logs in this version. |
|
141 |
+The round trip was measured directly (\code{benchmarks/raw/gateE/remote\_cost\_breakdown\_m3u96a\_m3u96b.txt}): application-level |
|
142 |
+ping 0.74--1.17~ms; one layer's request (one row, eight pairs) 2.59~ms; eight rows with 64 pairs 7.3~ms; 32 rows with 256 pairs |
|
143 |
+19.1~ms, of which about 9~ms is the 1~MB reply on the wire. The worker therefore adds 1.2--1.7~ms per call to the network |
|
144 |
+latency, which bounds what a faster transport could recover without also changing the worker's execution path. |
|
145 |
+ |
|
146 |
+\subsection{Metal kernels (H4)} |
|
147 |
+\label{sec:results-metal} |
|
148 |
+Profiling on M4M64a (\code{benchmarks/raw/gateD/}) shows that a decode step with all experts resident takes 2.99~ms |
|
149 |
+(334~\tps{} synchronous) and moves 452~MB of expert weights, i.e.\ 151~GB/s or about 28\,\% of the chip's nominal bandwidth; |
|
150 |
+a \gatherqmm{} over 1 to 16 experts takes the same 0.17~ms (latency-bound; 250~GB/s only at 64 experts), and in-graph |
|
151 |
+elementwise ops cost 5--7~$\mu$s each against a 0.16~ms synchronisation floor. The step is launch-bound, so the most a kernel |
|
152 |
+fusion can save is a few dispatches per layer. We tested the cheapest one, a single \gatherqmm{} over a stacked up+gate buffer |
|
153 |
+(bit-identical logits, \code{tests/test\_fused\_upgate.py}): $-0.2\,\%$ with all experts resident and $+2.6\,\%$ on the cold tier, |
|
154 |
+both within the run-to-run range (Table~\ref{tab:fused}). \textbf{H4 is not supported} at batch size one on this model; the profile |
|
155 |
+says that batching tokens, not kernels, is what would change the regime. |
|
156 |
+ |
|
157 |
+\input{tables/fused} |
|
158 |
+ |
|
159 |
+\subsection{Energy frontier (H5)} |
|
160 |
+\textsc{pending}: the \code{powermetrics} sampler (\code{benchmarks/scripts/power\_sampler.py}) is written; no energy run has been |
|
161 |
+logged in this version. |