spb/wp13_uqo

1% ============================================================================2\section{Methodology: Orchestration Mechanisms}3\label{sec:methodology}4% ============================================================================56This section is the technical core of the paper. We describe, mechanism by mechanism, how7VQuant drives a language model through a complete analysis: the streaming transport and its8event protocol, the recursive tool-calling loop, the batching policy, the context-management9machinery, and the token-accounting correction. Each mechanism is described as it is actually10implemented, so that the design can be reproduced. Throughout, the guiding principle is a strict11division of labor: the model \emph{plans and synthesizes}, while deterministic server code12\emph{transports, schedules, computes, and accounts}.1314\subsection{The streaming transport and event protocol}15\label{subsec:transport}1617VQuant does not use a model-provider SDK. It opens a raw HTTP connection to the provider's18messages endpoint with a streaming response type, and parses the resulting Server-Sent Event19(SSE) byte stream itself. This choice gives the server full control over how partial output is20interpreted and re-emitted.2122The parser maintains a single growing string buffer. On each network chunk it appends the bytes,23splits the buffer on newlines, and \emph{retains the final, possibly incomplete, line} in the24buffer for the next chunk---a detail that is essential for correctness, because SSE frames do not25align with TCP segment boundaries. Each complete line beginning with the \texttt{data:} prefix is26stripped and parsed as JSON into a typed event. The parser dispatches on the event's27\texttt{type} field; the eight relevant cases are summarized in Table~\ref{tab:events} and the28overall flow is shown in Figure~\ref{fig:streaming}.2930\input{figures/streaming}3132\input{tables/events}3334Three behaviors of this dispatcher deserve emphasis. First, the model's output travels on35\emph{two independent channels}: ordinary answer text arrives as \texttt{text\_delta} events and36is accumulated into the answer while being forwarded to the client, whereas the model's private37reasoning arrives as \texttt{thinking\_delta} events on a separate callback that the client38renders in a distinct panel. Second, a tool call is not delivered atomically: the tool's name39and identifier arrive in a \texttt{content\_block\_start}, but its arguments stream in as a40sequence of \texttt{input\_json\_delta} fragments that the parser concatenates into a partial41JSON string; only at \texttt{content\_block\_stop} is that string parsed into the structured42argument object. The loop must therefore wait for the block to close before a tool becomes43executable. Third, every forwarded event is flushed to the client immediately, so that the44user perceives output token by token rather than in batches.4546\subsection{The recursive tool-calling loop}47\label{subsec:loop}4849The agentic loop is implemented as \emph{recursion} rather than iteration. A single routine50issues one streaming model call, and when that call resolves it either terminates---if the model51requested no tools---or executes the requested tools, appends their results to the conversation,52and \emph{calls itself again}. Algorithm~\ref{alg:loop} states the procedure and53Figure~\ref{fig:loop} depicts it.5455\input{figures/loop}5657\begin{algorithm}[htbp]58\caption{Recursive tool-calling loop (\textsc{ProcessResponse})}59\label{alg:loop}60\begin{algorithmic}[1]61\Require message list $M$ (mutated in place), model $m$, tool catalogue $\mathcal{T}$62\State $(\textit{text}, R, \textit{usage}) \gets \textsc{StreamModel}(m, M, \mathcal{T})$63  \Comment{Section~\ref{subsec:transport}}64\State accumulate \textit{usage}; stream \textit{text} and thinking to the client65\If{$R = \varnothing$} \Comment{no tool calls $\Rightarrow$ final answer}66  \State \Return67\EndIf68\State append $\langle\text{assistant}: \text{tool\_use blocks of } R\rangle$ to $M$69\ForAll{batch $B$ in $\textsc{Chunk}(R, 5)$} \Comment{Section~\ref{subsec:batching}}70  \ForAll{tool call $r \in B$}71    \State $\textit{res} \gets \textsc{Execute}(r)$; stream a typed result event to the client72    \State collect $\textit{res}$ as a tool\_result block73  \EndFor74  \State \textsc{Wait}(500\,\text{ms})75\EndFor76\State append $\langle\text{user}: \text{all tool\_result blocks}\rangle$ to $M$77\State \textsc{ProcessResponse}($M, m, \mathcal{T}$) \Comment{recurse with augmented context}78\end{algorithmic}79\end{algorithm}8081The conversation is represented as a list of messages that is mutated as the loop proceeds. When82the model requests tools, the server first appends an \emph{assistant} message containing the83model's tool-use blocks, then---after execution---appends a \emph{user} message whose content is84the corresponding array of tool-result blocks, each keyed to its call identifier. This85strict pairing of tool-use and tool-result blocks is what the provider's protocol requires, and86maintaining it correctly across the recursion is the loop's central invariant. Recursion87terminates naturally when a model call returns no tool requests, at which point the accumulated88text is the report. Multimodal inputs are handled at the entry point: an uploaded image is89attached to the first user message as a base64 block alongside the text query.9091\subsection{Batched tool execution}92\label{subsec:batching}9394Within a single turn the model frequently requests many tools at once---a comprehensive report95may trigger a dozen or more. Executing all of them simultaneously would risk provider rate96limits and connection timeouts. VQuant therefore partitions the requested calls into97\emph{batches of at most five} and executes the batches \emph{sequentially}, inserting a98500-millisecond pause between consecutive batches. Results from all batches are accumulated and99returned to the model together in the single tool-result message of Algorithm~\ref{alg:loop}, so100that the model sees a complete result set on the next turn rather than a fragmented one. The101batch boundary thus serves purely as a rate-limiting throttle; it does not change what the model102observes. When more than one batch is required the server emits a progress event naming the103tools in flight, so the throttling is visible to the user rather than appearing as a stall.104105\subsection{System-prompt routing}106\label{subsec:routing}107108The behavior of the loop is shaped by a system prompt that is assembled fresh on every call.109Two parts are computed at request time: the current date and time (in both a human-readable and110an ISO form) are injected at the top so that relative date expressions resolve against live data111rather than the model's training cutoff; the remainder is a static policy block. That policy112instructs the model to detect and match the user's language; to act without procedural narration113and, in particular, to perform an entire custom analysis in a single execution114(Section~\ref{subsec:onecall}); to calibrate effort to request complexity---one or two tools for115a simple lookup, a handful for a moderate request, ten or more for a comprehensive report; and to116follow output-formatting conventions for downloadable artifacts. A final guardrail directs the117model to refuse attempts to extract the platform's own implementation. This prompt-level routing118is the mechanism by which a catalogue of more than two hundred tools (Table~\ref{tab:tools}) is119navigated without an explicit planner: the model is taught \emph{when not to call} tools as much120as when to call them.121122\input{tables/tools}123124\subsection{Adaptive reasoning}125\label{subsec:thinking}126127For models that support it, the orchestration service enables \emph{adaptive extended thinking}128by attaching a thinking directive to the request and asking for a summarized display. A small129capability list determines which models receive this directive, so the same code path serves the130full model range. As noted in Section~\ref{subsec:transport}, the resulting reasoning summary131arrives on its own delta channel and is streamed to a dedicated client panel. Surfacing132reasoning makes the agent's plan legible to the user and provides an audit trail of how a report133was assembled, at no change to the loop itself.134135\subsection{Context management under a token budget}136\label{subsec:context}137138A persistent challenge for conversational agents is the bounded context window: accumulated139history---especially verbose tool results---eventually exceeds what the model can attend to.140VQuant addresses this with a two-stage mechanism applied before every model call.141142\paragraph{Result truncation.} First, the server walks every message and caps any individual143tool-result block at a fixed character budget (5{,}000 characters), appending a truncation144marker. Raw data payloads are frequently far larger than the analysis requires, and this step145removes that bloat before it is ever re-sent on a subsequent turn.146147\paragraph{Adaptive summarization.} Second, the server estimates the total token count of the148prepared messages. The estimate uses a deliberately cheap heuristic---approximately one token per149four characters, summed over text and tool-result blocks---because it is only a trigger, not an150accounting figure. If the estimate exceeds a conservative threshold (set at $10^{5}$ tokens, well151below the hard context limit to leave room for the response and new tool results), the server152retains the most recent five messages verbatim and replaces the older history with a153model-generated summary, reinserted as a single synthetic user message. Algorithm~\ref{alg:context}154states the procedure and Figure~\ref{fig:context} illustrates the transformation.155156\input{figures/context}157158\begin{algorithm}[htbp]159\caption{Context management via truncation and adaptive summarization}160\label{alg:context}161\begin{algorithmic}[1]162\Require message list $M$, model $m$, threshold $\tau = 10^{5}$, recency $k = 5$, cap $c = 5000$ chars163\State cap every tool-result block in $M$ at $c$ characters164\State $t \gets \textsc{EstimateTokens}(M)$ \Comment{cheap $\approx$ chars$/4$ heuristic}165\If{$t > \tau$}166  \State $M_{\text{recent}} \gets$ last $k$ messages of $M$;\quad $M_{\text{old}} \gets$ the rest167  \If{$M_{\text{old}} \neq \varnothing$}168    \State $T \gets \textsc{ExtractText}(M_{\text{old}})$ \Comment{drop tool-use blocks; keep last 10k chars}169    \State $s \gets \textsc{Summarize}(T, m)$ \Comment{separate non-streaming call}170    \State $M \gets [\,\langle\text{user: ``[summary]: ''}+s\rangle\,] \mathbin{\Vert} M_{\text{recent}}$171  \EndIf172\EndIf173\State \Return $M$174\end{algorithmic}175\end{algorithm}176177The summary is produced by a dedicated, \emph{non-streaming} model call with its own system178prompt, a low temperature, and a tight output cap, instructed to preserve the analytically179relevant context---the instruments under discussion, prior conclusions, and user preferences.180Before summarizing, a helper extracts only the natural-language turns (discarding tool-use181blocks) and keeps at most the last ten thousand characters, bounding the cost of summarization182itself. Should the summarization call fail, the mechanism degrades gracefully to a truncated183literal excerpt rather than aborting the request. The net effect is that conversation length is,184for practical purposes, unbounded, at the cost of one additional model call whenever the185threshold is crossed.186187\subsection{Token accounting and its correction}188\label{subsec:accounting}189190Accurate token accounting matters both for the budget trigger above and for per-session cost191reporting, yet the streaming protocol does not reliably report output-token counts. VQuant192therefore applies a correction mechanism. Input tokens are read from the provider's193\texttt{message\_start} event; output tokens are taken from streaming usage updates \emph{but194reconciled} against a local count obtained by running the accumulated answer text through a195tokenizer at the end of the stream. The system records the \emph{maximum} of the provider's196figure and the local count, guaranteeing that cost is never understated, and falls back to the197provider's figure only if local encoding fails. Cost is then computed from a simple linear198schedule (a fixed price per million input tokens and a higher price per million output tokens)199and accumulated across all calls in a session, including the recursive tool-driven turns and any200summarization calls. This separation---a cheap heuristic for the budget trigger, an exact201tokenizer for billing---keeps the hot path inexpensive while keeping the ledger accurate.202203\subsection{The single-call computation paradigm}204\label{subsec:onecall}205206The final orchestration mechanism concerns \emph{how} numerical work is delegated. A naive agent207decomposes a quantitative task into a sequence of tool calls---fetch data, then in a separate208turn analyze it, then in a third turn plot it---each of which incurs a full recursion through209Algorithm~\ref{alg:loop} and therefore a full model round trip. VQuant instead instructs the210model, through the system prompt, to perform an entire analysis in a \emph{single} execution of211the Python tool: acquire data inside the sandbox, compute, and render all figures in one call212(Listing~\ref{lst:onecall}).213214\begin{lstlisting}[language=Python,caption={Single-call execution: data acquisition,215computation, and visualization are consolidated into one sandboxed run, collapsing what would216otherwise be three recursive model round trips into one.},label={lst:onecall}]217# One tool call performs the whole analysis:218data = fmp.get_historical_prices("NVDA")      # live fetch inside the sandbox219df = pd.DataFrame(data["historical"]).sort_values("date")220df["ret"] = df["close"].pct_change()221222plt.figure(figsize=(12, 6))                    # figure 1: price path223plt.plot(pd.to_datetime(df["date"]), df["close"]); plt.tight_layout()224225plt.figure(figsize=(12, 6))                    # figure 2: return distribution226plt.hist(df["ret"].dropna(), bins=50); plt.tight_layout()227228save_excel(df, "nvda.xlsx")                     # spreadsheet export in the same run229\end{lstlisting}230231Three rules, enforced through the prompt, make this paradigm effective. First, the model must232\emph{never} hard-code data arrays: inlining hundreds of values both bloats the generated tokens233(which must be streamed and accounted) and defeats reproducibility; data must be fetched234programmatically through the in-process client. Second, each visualization must be a separate235figure rather than a subplot grid, which yields cleanly captionable, individually referenceable236images. Third, all artifacts---figures and spreadsheets---are produced within the same execution.237By collapsing the recursion depth for an analytical task from three or more to one, this paradigm238removes the dominant source of latency in the loop. The substrate that executes this code is the239subject of the next section.240