SPB Git

spb/wp30_uqo

Public
11.7 KB · 107 lines latex
1\subsection{Single-handoff compression frontier (E1)}2\label{sec:results-frontier}3Table~\ref{tab:primary} and Figure~\ref{fig:frontier} summarize the core campaign: 160 test documents, 22 conditions, two model pairs,413\,280 scheduled trials, all executed; 74 attempts ended as \emph{invalid output} (empty or non-text generations, all from 4-bit Qwen caches and5random eviction) and 240 as \emph{implementation error} (LLMLingua-2 on the Llama pair: the cluster node lacked the PyTorch runtime; those6trials were retried on the laptop and both attempts are kept, Appendix~\ref{app:tables}). Failures count as zero quality.78\begin{figure}[p]\centering\includegraphics[width=\textwidth]{fig2_quality_vs_bytes.pdf}9\caption{Quality against measured serialized payload bytes (log scale), by task family (columns) and model pair (rows). Each point is one10condition (median payload, mean quality, 95\,\% cluster-bootstrap CI). Text methods cluster at $10^2$--$10^4$ bytes, cache methods at11$10^7$--$10^9$ bytes: a 3B-parameter cache of a document is three to four orders of magnitude larger than the document. Question-aware and12question-blind conditions are both shown here; Appendix figures separate them.}\label{fig:frontier}\end{figure}1314\begin{sidewaystable}[p]\centering\scriptsize15\caption{Primary comparison: quality (\%, with 95\,\% cluster-bootstrap CI) per condition, family and pair. Synthetic = private-fact accuracy16(5 questions per document: single fact, exact copy, multi-hop), exec = hidden-test pass (all 5 tests), QASPER/2Wiki = LongBench F1.17``blind''/``aware'' = question access of the compressor. Latency and bytes for the synthetic family in Table~\ref{tab:cost}.}\label{tab:primary}18\adjustbox{max width=\textwidth}{\input{tables/table_primary}}19\end{sidewaystable}2021\paragraph{Lossless representations are interchangeable; nothing else is.} Full text (A1), zstd-compressed text (A7), the native cache22(B0), the serialized and reloaded cache (B1) and the 8-bit packed cache (B2-8) give the same answers within noise on every family and pair23(T2: B2-8 vs B0 \DiffTW{} points, Holm $p=\PTW$; T6: A1 vs B0 $p=\PTX$; $n=\NTW$ paired instances each). This is the expected24consistency result---the cache \emph{is} the document as the receiver read it---and it is also the only region of the frontier where25representation choice is free of quality cost. The A7 result confirms H8: the codec shrinks the text payload ${\approx}3.5\times$26(\BSyntheticQwenAO{} bytes raw for an average synthetic document) with no effect on quality or receiver time, while on bf16 caches zstd saves27${\approx}20$\,\% (Section~\ref{sec:network}).2829\paragraph{Packed 4-bit quantization is model-dependent (H2 confirmed).} At 4 bits with 64-element groups, Llama-3.2-3B keeps30\QSyntheticLlamaBWBitsF{}\,\% accuracy on private facts and \QExecLlamaBWBitsF{}\,\% on executable tasks, whereas Qwen2.5-3B collapses to31\QSyntheticQwenBWBitsF{}\,\% and \QExecQwenBWBitsF{}\,\% (T3, both pairs pooled: \DiffTH{} points, Holm $p=\PTH$). The failure is total32(garbled tokens, 29 empty generations), it reproduces in the native \texttt{mlx-lm} path, and it is the per-token affine quantization of Qwen's33outlier key channels that KIVI's per-channel scheme was designed to avoid. Eight bits are safe for both models; halving the cache is therefore34free for both, quartering it is free for one.3536\paragraph{Eviction without the question is destructive; eviction with the question is nearly free---for retrieval, not for code.} Keeping a37quarter of the positions by the sink-plus-recent rule (B3) leaves \QSyntheticQwenBHK{}\,\% (Qwen) and \QSyntheticLlamaBHK{}\,\% (Llama) on38private facts, close to the share of facts that happen to sit in the retained window; random retention (B4) is at zero and produces39incoherent output on both models. The SnapKV-style rule with the question as observation window (B5-aware) recovers \QSyntheticQwenBVKAware{}\,\%40(Qwen) and \QSyntheticLlamaBVKAware{}\,\% (Llama) at the same 1/4 budget (T4 vs B3: \DiffTF{} points, Holm $p=\PTF$, H3 confirmed in its41first half), and the pyramid allocation (B6) and the selection-plus-8-bit composition (B9) behave like B5. With a generic window instead of42the question (B5-blind) the same algorithm falls to \QSyntheticQwenBVKBlind{}\,\%/\QSyntheticLlamaBVKBlind{}\,\%: attention-based selection43is a \emph{question-aware} method and must be compared as one (H7). On the executable family every eviction method, aware or not, scores44zero on both models: a code specification cannot be quartered, because every rule in the ticket matters for the hidden tests. On the public45QA subsets eviction costs little (the answers are short spans), but the F1 ceiling of 3B receivers is low (A1 $\approx$ 0.25--0.39).4647\paragraph{Text compression at a 512-token budget: summaries beat structured evidence, extraction beats both when the question is known.}48Question-blind summaries (A4) retain \QSyntheticQwenAFBBlind{}\,\% (Qwen) and \QSyntheticLlamaAFBBlind{}\,\% (Llama) of the private-fact49answers; structured JSON evidence with spans (A5) only \QSyntheticQwenAVBBlind{}\,\%/\QSyntheticLlamaAVBBlind{}\,\% (T1: \DiffTO{} points,50Holm $p=\PTO$, $n=\NTO$). H1's expectation that typed facts would protect identifiers and numbers is \emph{rejected} at this budget: the51sender spends its 512 output tokens on exact quotes (provenance) and emits ${\approx}8$ facts out of 62 per document, so most answers are52simply absent from the message; the receiver answers ``unknown'' (correctly abstaining) rather than hallucinating. References plus metered53retrieval (A6) repair little (\QSyntheticQwenAXBRBlind{}\,\%/\QSyntheticLlamaAXBRBlind{}\,\%) because the reference index inherits the same54coverage. Deterministic extraction (A3) is the strongest cheap method: blind centrality selection keeps \QSyntheticQwenAHBBlind{}\,\%, and with55the question (BM25) \QSyntheticQwenAHBAware{}\,\%/\QSyntheticLlamaAHBAware{}\,\%---indistinguishable from full text at 1/15 of the bytes.56Head/tail truncation (A2) keeps exactly the facts that fall in the kept windows (\QSyntheticQwenAWB{}\,\%). The question types that the57five-question core did not cover (negation, revision/precedence, temporal order, aggregation, units, unanswerable) were run as an exploratory58extension after the core and are reported, labelled as such, in Appendix~\ref{app:qtypes}; the headline there is that timestamped revisions are59the facts that summaries lose most. On the executable family the60ordering changes: blind extractive selection keeps the whole specification (\QExecQwenAHBBlind{}\,\%/\QExecLlamaAHBBlind{}\,\%) because the61rules are dense in digits and identifiers, and the Llama summary preserves the rules better than the Qwen summary62(\QExecLlamaAFBBlind{} vs \QExecQwenAFBBlind{}\,\%). LLMLingua-2 (A8) at the same rate removes the very tokens the questions ask for63(\QSyntheticQwenAEBBlind{}\,\% on private facts, 0 on code) while being competitive on QASPER (\QQasperQwenAEBBlind{} vs64\QQasperQwenAO{}\,\% F1 for full text): a compressor trained on meeting transcripts is tuned for paraphrasable content, not for exact values.6566\paragraph{Cost accounting changes the ranking (RQ3).} Table~\ref{tab:cost} gives, for the synthetic family, the median serialized bytes and67the cold and warm per-query latencies assembled from the recorded spans. In bytes, every text method beats every cache method by68$10^3$--$10^4$. In cold latency on the laptop (Qwen pair), full text costs \TcoldSyntheticQwenAO{}~ms per question (prefill of the whole69document each time), the native cache \TcoldSyntheticQwenBZ{}~ms (document prefill once plus the fresh question), and a blind summary70\TcoldSyntheticQwenAFBBlind{}~ms---the sender's generation dominates. In the warm, reused regime the ranking inverts: the cache answers a new71question in \TwarmSyntheticQwenBZ{}~ms and the 8-bit cache in \TwarmSyntheticQwenBWBitsE{}~ms, the summary in \TwarmSyntheticQwenAFBBlind{}~ms72(it is short), while full text still pays \TwarmSyntheticQwenAO{}~ms. Peak memory separates two phases (Table~\ref{tab:primary} sources; Figure~\ref{fig:latency}, bottom): the \emph{prefill} peak is the73same for every cache method as for full text (7.16~GB on the laptop with both models resident), because compression happens after the74document has been read---a method that compresses after prefill cannot lower the prefill peak; the \emph{decode-phase} peak then reflects the75resident state (6.90~GB for the dense Qwen cache, 6.74 at 8 bits, 6.55 at 1/4 retention, 6.3--6.6 for text messages), i.e., ${\approx}0.3$~GB76for an 8k-token Qwen cache and three times that for Llama's eight KV heads. No single denominator ranks the methods the same way; the frontier maps (Figures~\ref{fig:frontier} and77\ref{fig:latency}) are the result.7879\begin{table}[t]\centering\scriptsize80\caption{Resource accounting, synthetic family (medians over trials). Cold = build + compress/serialize + load + receiver; warm = load +81receiver (document state already built); bytes = serialized payload including manifest. Qwen pair on the laptop (M5 Max); Llama pair on82M3U96a (M3 Ultra) for this family---compare within a pair only.}\label{tab:cost}83\begin{tabular}{@{}lrrrrrr@{}}\toprule84 & \multicolumn{3}{c}{Qwen2.5-3B (laptop)} & \multicolumn{3}{c}{Llama-3.2-3B (M3U96a)} \\85Condition & bytes & cold ms & warm ms & bytes & cold ms & warm ms \\\midrule86A1 full text & \BSyntheticQwenAO & \TcoldSyntheticQwenAO & \TwarmSyntheticQwenAO & \BSyntheticLlamaAO & \TcoldSyntheticLlamaAO & \TwarmSyntheticLlamaAO \\87A4 summary (blind) & \BSyntheticQwenAFBBlind & \TcoldSyntheticQwenAFBBlind & \TwarmSyntheticQwenAFBBlind & \BSyntheticLlamaAFBBlind & \TcoldSyntheticLlamaAFBBlind & \TwarmSyntheticLlamaAFBBlind \\88A5 structured (blind) & \BSyntheticQwenAVBBlind & \TcoldSyntheticQwenAVBBlind & \TwarmSyntheticQwenAVBBlind & \BSyntheticLlamaAVBBlind & \TcoldSyntheticLlamaAVBBlind & \TwarmSyntheticLlamaAVBBlind \\89A7 zstd text & \BSyntheticQwenASZstd & \TcoldSyntheticQwenASZstd & \TwarmSyntheticQwenASZstd & \BSyntheticLlamaASZstd & \TcoldSyntheticLlamaASZstd & \TwarmSyntheticLlamaASZstd \\90B0 native cache & \BSyntheticQwenBZ & \TcoldSyntheticQwenBZ & \TwarmSyntheticQwenBZ & \BSyntheticLlamaBZ & \TcoldSyntheticLlamaBZ & \TwarmSyntheticLlamaBZ \\91B1 serialized reuse & \BSyntheticQwenBO & \TcoldSyntheticQwenBO & \TwarmSyntheticQwenBO & \BSyntheticLlamaBO & \TcoldSyntheticLlamaBO & \TwarmSyntheticLlamaBO \\92B2 8-bit & \BSyntheticQwenBWBitsE & \TcoldSyntheticQwenBWBitsE & \TwarmSyntheticQwenBWBitsE & \BSyntheticLlamaBWBitsE & \TcoldSyntheticLlamaBWBitsE & \TwarmSyntheticLlamaBWBitsE \\93B3 sink+recent 1/4 & \BSyntheticQwenBHK & \TcoldSyntheticQwenBHK & \TwarmSyntheticQwenBHK & \BSyntheticLlamaBHK & \TcoldSyntheticLlamaBHK & \TwarmSyntheticLlamaBHK \\94B5 attention 1/4 (aware) & \BSyntheticQwenBVKAware & \TcoldSyntheticQwenBVKAware & \TwarmSyntheticQwenBVKAware & \BSyntheticLlamaBVKAware & \TcoldSyntheticLlamaBVKAware & \TwarmSyntheticLlamaBVKAware \\95B6 pyramid 1/4 (aware) & \BSyntheticQwenBXKAware & \TcoldSyntheticQwenBXKAware & \TwarmSyntheticQwenBXKAware & \BSyntheticLlamaBXKAware & \TcoldSyntheticLlamaBXKAware & \TwarmSyntheticLlamaBXKAware \\\bottomrule96\end{tabular}97\end{table}9899\begin{figure}[t]\centering\includegraphics[width=\textwidth]{fig3_quality_vs_latency_memory.pdf}100\caption{Quality against cold end-to-end latency (top) and peak MLX memory during the trial (bottom), by family, both pairs pooled.}101\label{fig:latency}\end{figure}102103\begin{figure}[t]\centering\includegraphics[width=\textwidth]{fig4_latency_decomposition.pdf}104\caption{Latency decomposition (medians of recorded spans, synthetic family, cold scenario; document build counted once). Sender generation105dominates the text methods that need a model; document prefill dominates the cache methods; selection and serialization are minor; the106pyramid cache pays a token-by-token query prefill.}\label{fig:decomp}\end{figure}107
Files · main
Menu

Theme