SPB Git

spb/wp30_uqo

Public
5.0 KB · 50 lines latex
1\subsection{Two-node reuse and locality (E2/E6)}2\label{sec:network}3Figure~\ref{fig:breakeven} and Table~\ref{tab:network} report genuine two-node measurements: the sender node (M3U96a) prefills a4test document with Qwen2.5-3B, serializes the cache with its manifest, optionally quantizes or compresses it, and sends it over the5LAN to a receiver worker (M3U96b) that checks the manifest, installs the cache, prefills only the fresh question and decodes the6answer; the alternative sends the token ids and lets the worker prefill the whole document. Each point is the median of 18 runs7(6 test documents $\times$ 3 repeats, after one warm-up on both ends); the effective transport rate was8\EsixBONoneEZZZGbps{}~Gb/s on a link that iperf3 measured at 0.94~Gb/s.910\begin{figure}[t]\centering11\includegraphics[width=\textwidth]{fig6_breakeven.pdf}12\caption{Reuse/network break-even. Left: measured round trip to a remote answer (medians, p25--p75 bars) for moving the native bf16 cache13(B1), the zstd-compressed cache, the 8-bit and 4-bit packed caches (B2), versus sending the text and prefilling remotely; the grey line is14the sender's own local prefill for scale. Right: analytic transfer time from the measured bandwidth (solid) and a hypothetical 10~Gb/s link15(dashed) against the measured remote prefill slope. Qwen2.5-3B; synthetic test documents; M3U96a $\to$ M3U96b.}\label{fig:breakeven}16\end{figure}1718\begin{table}[t]\centering\scriptsize19\caption{Two-node transfer (medians of 18 runs; ms unless stated). ``Send'' is the sender-side socket time; ``load'' is manifest check plus20deserialization at the worker; the round trip includes the worker's fresh-query prefill and 32 decoded tokens. 4-bit rows transfer a cache21that yields quality~0 for Qwen2.5 (Section~\ref{sec:results-frontier}) and are shown only for byte and time accounting.}\label{tab:network}22\begin{tabular}{@{}llrrrrr@{}}\toprule23Context class & Representation & payload (MB) & send & load & remote prefill & round trip \\\midrule242k  & text $\to$ remote prefill & 0.01 & -- & -- & \EsixTextRemotePrefillNoneTZZZPrefill & \EsixTextRemotePrefillNoneTZZZRt \\252k  & B1 bf16 cache & \EsixBONoneTZZZBytes & \EsixBONoneTZZZSend & \EsixBONoneTZZZLoad & 0 & \EsixBONoneTZZZRt \\262k  & B2 8-bit cache & \EsixBTENoneTZZZBytes & \EsixBTENoneTZZZSend & \EsixBTENoneTZZZLoad & 0 & \EsixBTENoneTZZZRt \\278k  & text $\to$ remote prefill & 0.03 & -- & -- & \EsixTextRemotePrefillNoneEZZZPrefill & \EsixTextRemotePrefillNoneEZZZRt \\288k  & B1 bf16 cache & \EsixBONoneEZZZBytes & \EsixBONoneEZZZSend & \EsixBONoneEZZZLoad & 0 & \EsixBONoneEZZZRt \\298k  & B2 8-bit cache & \EsixBTENoneEZZZBytes & \EsixBTENoneEZZZSend & \EsixBTENoneEZZZLoad & 0 & \EsixBTENoneEZZZRt \\3016k (${\approx}$11k tokens) & text $\to$ remote prefill & 0.06 & -- & -- & \EsixTextRemotePrefillNoneOSZZZPrefill & \EsixTextRemotePrefillNoneOSZZZRt \\3116k & B1 bf16 cache & \EsixBONoneOSZZZBytes & \EsixBONoneOSZZZSend & \EsixBONoneOSZZZLoad & 0 & \EsixBONoneOSZZZRt \\3216k & B2 8-bit cache & \EsixBTENoneOSZZZBytes & \EsixBTENoneOSZZZSend & \EsixBTENoneOSZZZLoad & 0 & \EsixBTENoneOSZZZRt \\\bottomrule33\end{tabular}34\end{table}3536Three observations follow. First, the serialized bf16 cache is three to four orders of magnitude larger than the text it replaces37(\EsixBONoneEZZZBytes{}~MB versus 29~kB at 8k tokens), and zstd recovers only ${\approx}20$\,\% of it (high-entropy payload; H8), while packed388-bit quantization halves it. Second, on this link and with this receiver, moving the cache is nevertheless \emph{faster} than re-reading39the text at every context length we measured: at 8k tokens the bf16 transfer answers in \EsixBONoneEZZZRt{}~ms against40\EsixTextRemotePrefillNoneEZZZRt{}~ms for remote prefill, and the 8-bit cache in \EsixBTENoneEZZZRt{}~ms, because the M3~Ultra receiver prefills41Qwen2.5-3B at ${\approx}0.44$~ms per token (\EsixTextRemotePrefillNoneEZZZPrefill{}~ms for 8k) whereas 1~GbE moves the per-token cache42(${\approx}26$~kB bf16, ${\approx}14$~kB at 8 bits) in ${\approx}0.23$ and ${\approx}0.12$~ms. The break-even therefore depends on the ratio43of receiver prefill time to link time per token, which the right panel makes explicit: a faster receiver (the laptop prefills 8k tokens in44${\approx}1$~s) or a slower link would move the crossing to longer contexts, and a 10~Gb/s link would make transfer dominant everywhere. Third,45the cost of building the cache is paid by the sender and is counted once per document: when the cache is reused for $N$ queries at the46receiver, the per-query cost is the fresh-query prefill plus decoding (${\approx}$\EsixBONoneEZZZLoad{}~ms load and ${\approx}190$~ms to the47first token at 8k), while the text alternative pays the full prefill on each cold query unless the receiver keeps its own cache---at which48point the two strategies converge and the question becomes where to schedule the work. Equation~\eqref{eq:amortized} with the measured49components gives the break-even reuse count for any bandwidth; these are measured points on one link, not a universal bound.50
Files · 3c3d27f14e6841b389fd503bcab914bda201aa79
Menu

Theme