SPB Git

spb/wp30_uqo

Public
2.6 KB · 27 lines latex
1\subsection{Hosted receivers (Track D)}2\label{sec:hosted}3Table~\ref{tab:hosted} reports the same text messages, built by the local Qwen2.5-3B sender, answered by two hosted models through their4documented APIs on the first eight test documents of three families (5 questions per synthetic document; 672 trials; no failures;5estimated spend 0.53~USD at the published list prices). Both hosted receivers answer every private-fact question correctly from the full6document (A1), confirming that the synthetic facts are recoverable from the text; from the 512-token blind summary (A4) they recover only728--30\,\% of the answers, and from the structured JSON (A5) 12.5\,\%---the same ordering, and nearly the same levels, as the local 3B8receivers (Section~\ref{sec:results-frontier}). The bottleneck is therefore the sender's lossy message, not the receiver's capacity: a frontier9receiver cannot recover a number that the message does not contain, and the reference-plus-retrieval variant (A6) does not repair it under10the 2\,000-byte retrieval budget. On QASPER the hosted models score higher F1 than the local receivers from the same material (0.48 versus110.35 for A1), i.e., receiver capacity matters when the material is complete.1213\begin{table}[H]\centering\scriptsize14\caption{Hosted receivers on messages built by the local sender (test split, first 8 documents per family). Quality = accuracy (synthetic,15exec) or F1 (QASPER) with 95\,\% cluster-bootstrap CI; cost = estimated per-query cost from provider price tables (m\$ = 0.001~USD); TTFT16= time to first streamed content token (Anthropic only; the OpenAI Responses call was not streamed). ``+cache'' = documented prompt caching17(Anthropic \texttt{cache\_control}; OpenAI automatic prefix caching). Provider sampling defaults (no temperature control, D-014).}\label{tab:hosted}18\adjustbox{max width=\textwidth}{\input{tables/table_hosted}}19\end{table}2021\paragraph{Provider prefix caching is a billing object with a minimum size.} With explicit cache breakpoints, Anthropic reported22\texttt{cache\_read\_input\_tokens}~$\approx$~5\,000 on the second and later questions of the 8k-token documents and the per-query estimated23cost fell from 6.3 to 2.2~m\$, but \emph{nothing} was cached for the 2k-token documents: the documented minimum cacheable prefix for this24model is 4\,096 tokens, so a 2\,000-token document never qualifies. OpenAI's automatic caching reported cached tokens on 2k and 8k documents25alike. Measured TTFT did not change with caching at this size (medians 0.72 versus 0.75~s on the 8k documents). None of these fields reveals26cache bytes, precision or residency; they are reported as usage accounting only.27
Files · d5e182fa7d7d5118aa92e61c47c40ed848da0ecf
Menu

Theme