SPB Git

spb/wp30_uqo

Public
4.8 KB · 45 lines latex
1\section{Adaptive representation and placement (E8)}2\label{sec:adaptive}3The router uses only deployment-observable features---task family, receiver context length (three bins), model pair and question-access4regime---and a table fitted on the development pilot: for each cell and each candidate action (fifteen static conditions spanning text,5evidence and cache representations) it stores the development mean quality, its lower 95\,\% bootstrap bound and the mean amortized latency;6it picks the cheapest action whose lower bound is within $\tau=0.9$ of the best lower bound in the cell, and falls back to the development-best7static action when a cell is missing (Appendix~\ref{app:pseudo}). The table was frozen before any test item was evaluated; on test, the8chosen action's outcome is read from the full-factorial core records of that instance (an off-policy evaluation that is exact because every9action was run on every instance), and the router's own lookup time (${\approx}1$~ms) is added. Figure~\ref{fig:router} and10Table~\ref{tab:router} compare it with the development-best static action, full context, the native cache, the question-aware attention11cache and the infeasible oracle that picks the best action per instance.1213\begin{figure}[t]\centering\includegraphics[width=\textwidth]{fig7_router.pdf}14\caption{Dev-fitted router against static policies on held-out test instances (both pairs, all families; $n=640$ per regime). The oracle is15an infeasible upper reference.}\label{fig:router}\end{figure}1617\begin{table}[t]\centering\footnotesize18\caption{Router evaluation on held-out test instances. Quality in percent with 95\,\% cluster-bootstrap CI; cost = mean amortized latency per19query (ms, document build divided by the number of reusing questions); fallback = share of instances routed by the nearest-cell fallback.}\label{tab:router}20\begin{tabular}{@{}llrrrr@{}}\toprule21Regime & Policy & quality (\%) & 95\,\% CI & cost (ms) & fallback (\%) \\\midrule22blind & router & \EeightRouterBlindQ & [\EeightRouterBlindQlo, \EeightRouterBlindQhi] & \EeightRouterBlindCost & \EeightRouterBlindFallback \\23blind & best static (dev) = full context (A1) & \EeightBestStaticDevBlindQ & [\EeightBestStaticDevBlindQlo, \EeightBestStaticDevBlindQhi] & \EeightBestStaticDevBlindCost & 0 \\24blind & native cache (B0) & \EeightNativeCacheBlindQ & [\EeightNativeCacheBlindQlo, \EeightNativeCacheBlindQhi] & \EeightNativeCacheBlindCost & 0 \\25blind & structured evidence (A5) & \EeightStructuredBlindQ & [\EeightStructuredBlindQlo, \EeightStructuredBlindQhi] & \EeightStructuredBlindCost & 0 \\26blind & oracle (infeasible) & \EeightOracleInfeasibleBlindQ & [\EeightOracleInfeasibleBlindQlo, \EeightOracleInfeasibleBlindQhi] & \EeightOracleInfeasibleBlindCost & 0 \\\midrule27aware & router & \EeightRouterAwareQ & [\EeightRouterAwareQlo, \EeightRouterAwareQhi] & \EeightRouterAwareCost & \EeightRouterAwareFallback \\28aware & best static (dev) = extractive (A3-aware) & \EeightBestStaticDevAwareQ & [\EeightBestStaticDevAwareQlo, \EeightBestStaticDevAwareQhi] & \EeightBestStaticDevAwareCost & 0 \\29aware & attention cache (B5-aware) & \EeightAttentionCacheAwareAwareQ & [\EeightAttentionCacheAwareAwareQlo, \EeightAttentionCacheAwareAwareQhi] & \EeightAttentionCacheAwareAwareCost & 0 \\30aware & oracle (infeasible) & \EeightOracleInfeasibleAwareQ & [\EeightOracleInfeasibleAwareQlo, \EeightOracleInfeasibleAwareQhi] & \EeightOracleInfeasibleAwareCost & 0 \\\bottomrule31\end{tabular}32\end{table}3334In the question-blind regime the router matches the quality of the development-best static policy (paired difference35\EeightDiffBlindQuality{} points, 95\,\% CI [\EeightLoBlindQuality, \EeightHiBlindQuality], $p=\EeightPBlindQuality$) at half its amortized36latency (\EeightDiffBlindCostMs{}~ms, CI [\EeightLoBlindCostMs, \EeightHiBlindCostMs]): it learned on development data that the native cache37(B0) and the 8-bit cache are as accurate as full text for the private-fact and executable families and cheaper once the document is reused, and38that nothing beats full text on the public QA subsets. In the question-aware regime the router does not help: its quality equals the39development-best static action (\EeightDiffAwareQuality{} points, CI [\EeightLoAwareQuality, \EeightHiAwareQuality]) but its cost is40\emph{higher} (\EeightDiffAwareCostMs{}~ms), because in several cells it selects attention-selected caches whose per-query compression cost on41the test documents exceeded the development estimate, and because the cheap aware extractive baseline (A3) was already near the oracle. The42fallback rate was \EeightRouterBlindFallback{}\,\% (context bins unseen on dev). Taken together, H9 is confirmed for the aware regime and43rejected for the blind regime, with the caveat that the gain is a latency gain at equal quality and that it depends on reuse: at $N=1$ the44cache build cost is not amortized and full text would be preferred.45
Files · 2ccc6b7e52385b2438c0f3e62e88226009fb4f9f
Menu

Theme