SPB Git

spb/wp30_uqo

Public
5.0 KB · 54 lines latex
1\section{Introduction}2\label{sec:intro}34Agents built on large language models rarely work alone. A reader agent summarizes a filing for an analyst agent; a crawler hands5extracted facts to an aggregator; a long document is processed once and then queried many times by different workers. Every such6handoff chooses a \emph{representation} for the information that crosses the boundary between two model executions: the raw text,7a shortened text, a prose summary, a structured record with provenance, a pointer to shared memory, or---when the two executions8share a checkpoint---the attention state (the key--value, or KV, cache) that the first model built while reading. A growing9literature proposes to compress these states \citep{liu2024kivi,li2024snapkv,cai2024pyramidkv,zhang2023h2o,xiao2024streamingllm,10zweiger2026compaction}, to reuse them across requests, and, most recently, to translate them between different checkpoints or model11families so that a receiver can ``inherit'' a sender's reading without re-reading the source \citep{fu2026c2c,zou2026latentmas,12alabdulkareem2026swat}.1314The engineering appeal is clear: a receiver that loads a cache skips the prefill of the document, and a compressed cache shrinks15memory. The scientific question is less clear. Compression methods are evaluated with different models, different information-access16assumptions (does the compressor see the question?), different budgets (tokens, positions, bits, bytes), and different measurement17boundaries (kernel time versus end-to-end time, resident tensors versus serialized payloads). Text baselines are often weak or absent.18Cache sizes are usually quoted in positions or bits per value rather than in the bytes that actually cross a network, and the receiver19side of a handoff---deserialization, loading, fresh-query processing---is seldom counted. Nor is the behaviour of these representations20studied over \emph{repeated} handoffs, where small distortions compound.2122This paper asks a single question under a common measurement harness: \emph{under which task, model, cache, network and reuse23conditions does each communication representation preserve useful information at the lowest end-to-end resource cost?} We24answer it empirically, with experiments executed from a single research operator (Claude Code, an agentic coding assistant) on25commodity Apple-Silicon hardware, using the MLX runtime, whose KV caches are ordinary tensors that can be inspected, sliced, quantized,26serialized and loaded back. All quality numbers come from real model generations; all byte counts are measured on serialized payloads;27all latencies are wall-clock spans with device synchronization; and the complete per-attempt record of every trial is released.2829\paragraph{Contributions.}30\begin{enumerate}[leftmargin=1.4em]31\item An \textbf{experimentally grounded taxonomy} of communication representations (Table~\ref{tab:taxonomy}) that separates32  text compression, KV compression, cache reuse, cache transfer and parametric transfer, and records for each method its33  question-access regime, its compatibility requirements and whether it composes across sources.34\item A \textbf{common harness} (\texttt{wp30}) with a validated cache adapter layer (identity round trip, prefix equivalence,35  position preservation, packed-byte accounting, manifest-checked loading), a per-attempt record schema, deterministic scorers,36  evaluator-isolated answer keys, and document-level cluster bootstrap statistics.37\item \textbf{Quality/resource trade-off maps} for twenty-two conditions on two 3B-parameter model families and three task38  families (synthetic private-fact ledgers, public long-document QA, executable tasks), under question-blind and question-aware39  regimes, with measured bytes, latency decompositions and peak memory.40\item A study of \textbf{repeated handoffs} (1 to 8 transfers) with an evaluator-only fact ledger that measures retention,41  unsupported additions and the cost of metered repair, and a \textbf{fan-in} study with conflicting evidence.42\item \textbf{Genuine two-node cache transfer} measurements on a 1~GbE LAN against remote text prefill and local recompute,43  with a measured break-even model.44\item A \textbf{same-family learned KV mapper} (Qwen2.5-1.5B~$\to$~Qwen2.5-3B) trained on train-split caches only, evaluated45  against text handoffs and an identity-transfer negative control; and a \textbf{development-fitted adaptive router} evaluated on46  held-out data against the best development-selected static method.47\end{enumerate}4849Several results are negative or conditional, and we report them as such. Cross-family cache transfer was \emph{not} executed:50the only published cross-family method with code \citep{alabdulkareem2026swat} has no released checkpoint and no licence, and51our budget did not permit training a mapper of its size; the study is therefore a \emph{partial} transfer study in that respect52(Section~\ref{sec:limitations}). We also do not claim that any receiver \emph{learned} anything: every effect measured here is53contextual, i.e., it vanishes when the message or cache is removed.54
Files · main
Menu

Theme