TeX 93.9%
Python 6%
1% ============================================================================2\section{Orchestration Mechanisms}3\label{sec:methodology}4% ============================================================================56This section is the technical core of the paper. We describe, mechanism by mechanism, how VQuant7drives a language model through a complete analysis: the streaming transport and its event8protocol, the recursive tool-calling loop, the batching policy, the routing prompt, the9context-management procedure, the token-accounting correction, and the single-call computation10convention. Each mechanism is described as implemented at the audited commit, with its actual11constants, so that the design can be reproduced; Appendix~\ref{app:audit} gives the source12location of every statement.1314\subsection{Streaming transport and event protocol}15\label{subsec:transport}1617On the reference path VQuant does not use the provider's software development kit. It opens a18raw HTTP connection to the provider's messages endpoint with a streaming response and parses the19resulting SSE byte stream itself, which gives the server full control over how partial output is20interpreted and re-emitted.2122The parser maintains a single growing string buffer. On each network chunk it appends the bytes,23splits the buffer on newlines, and \emph{retains the final, possibly incomplete, line} for the next24chunk---a detail that is essential for correctness because SSE frames do not align with TCP25segment boundaries. Each complete line beginning with the \texttt{data:} prefix is stripped and26parsed as JSON into a typed event, and the parser dispatches on the event's \texttt{type} field.27The eight relevant cases are summarised in Table~\ref{tab:events} and the flow is shown in28Figure~\ref{fig:streaming}.2930\input{figures/streaming}3132\input{tables/events}3334Three behaviours of this dispatcher deserve emphasis. First, the model's output travels on two35independent channels: answer text arrives as \texttt{text\_delta} events and is accumulated into36the answer while being forwarded to the client, whereas the model's reasoning summary arrives as37\texttt{thinking\_delta} events on a separate callback that the client renders in a distinct38panel. Second, a tool call is not delivered atomically: the tool's name and identifier arrive in a39\texttt{content\_block\_start}, its arguments stream in as a sequence of40\texttt{input\_json\_delta} fragments that the parser concatenates into a partial JSON string, and41only at \texttt{content\_block\_stop} is that string parsed into the structured argument object.42The loop must therefore wait for the block to close before a tool becomes executable. Third, every43forwarded event is flushed to the client immediately, so the user perceives output as it is44generated rather than in batches.4546\subsection{The recursive tool-calling loop}47\label{subsec:loop}4849The agentic loop is implemented as recursion. A single routine issues one streaming model call;50when the call resolves it either terminates---if the model requested no tools---or executes the51requested tools, appends their results to the conversation, and calls itself again.52Algorithm~\ref{alg:loop} states the procedure and Figure~\ref{fig:loop} depicts it.5354\input{figures/loop}5556\begin{algorithm}[htbp]57\caption{Recursive tool-calling loop (\textsc{ProcessResponse})}58\label{alg:loop}59\begin{algorithmic}[1]60\Require message list $M$ (mutated in place), model $m$, tool catalogue $\mathcal{T}$ ($|\mathcal{T}|=245$)61\State $M \gets \textsc{PrepareContext}(M, m)$ \Comment{Algorithm~\ref{alg:context}}62\State $(\textit{text}, R, \textit{usage}) \gets \textsc{StreamModel}(m, M, \mathcal{T})$ \Comment{Section~\ref{subsec:transport}}63\State accumulate \textit{usage}; stream \textit{text} and reasoning summary to the client64\If{$R = \varnothing$} \Comment{no tool calls $\Rightarrow$ final answer}65 \State \Return66\EndIf67\State append $\langle\text{assistant}: \text{tool\_use blocks of } R\rangle$ to $M$ \Comment{tool-use blocks only; no text or thinking blocks}68\ForAll{batch $B$ in $\textsc{Chunk}(R, 5)$} \Comment{Section~\ref{subsec:batching}}69 \State emit a \texttt{status} event naming the tools in $B$ if more than one batch70 \ForAll{tool call $r \in B$}71 \State $\textit{res} \gets \textsc{Execute}(r)$; stream a typed result event to the client72 \State $\textit{res} \gets \textsc{Truncate}(\textit{res}, c_1(r))$ \Comment{$c_1 = 3{,}000$ chars; $15{,}000$ for extraction and Python tools}73 \State collect $\textit{res}$ as a tool\_result block keyed to $r$'s identifier74 \EndFor75 \State \textsc{Wait}(500\,\text{ms}) if another batch follows76\EndFor77\State append $\langle\text{user}: \text{all tool\_result blocks}\rangle$ to $M$78\State \textsc{ProcessResponse}($M, m, \mathcal{T}$) \Comment{recurse with augmented context}79\end{algorithmic}80\end{algorithm}8182The conversation is represented as a list of messages mutated as the loop proceeds. When the model83requests tools, the server first appends an \emph{assistant} message containing the model's84tool-use blocks, then---after execution---appends a \emph{user} message whose content is the array85of tool-result blocks, each keyed to its call identifier. This pairing of tool-use and tool-result86blocks is what the provider's protocol requires, and maintaining it across the recursion is the87loop's central invariant. Two consequences of the implementation should be noted. The assistant88message carries only the tool-use blocks: any answer text the model emitted in the same turn is89streamed to the user but not re-sent to the model, and reasoning blocks are not replayed, so the90model re-derives its plan from the tool results on the next turn rather than from its own prior91deliberation. And because the routine recurses without a depth bound, termination relies on the92model eventually returning no tool requests; the first version of this paper did not state this,93and a maximum recursion depth is a reasonable hardening (Section~\ref{subsec:threats}). Recursion94terminates when a call returns no tool requests, at which point the accumulated text is the95report. Multimodal input is handled at the entry point: an uploaded image is attached to the96first user message as a base64 block alongside the query.9798\subsection{Batched tool execution}99\label{subsec:batching}100101Within a single turn the model frequently requests many tools at once; a comprehensive report may102trigger a dozen or more. Executing all of them simultaneously would risk provider rate limits and103connection timeouts. VQuant therefore partitions the requested calls into batches of at most104five, executes the batches sequentially, and inserts a 500-millisecond pause between consecutive105batches. Within a batch the calls are executed one after another, not concurrently; the batch is106a unit of pacing rather than of parallelism. Results from all batches are accumulated and returned107to the model together in the single tool-result message of Algorithm~\ref{alg:loop}, so the model108sees a complete result set on the next turn. This matters for a second reason: the provider's109guidance is that parallel tool calls answered across several user messages lead the model to stop110issuing them in parallel, so the single-message return preserves the model's tendency to batch its111own requests. When more than one batch is required the server emits a \texttt{status} event naming112the tools in flight, so the throttling is visible to the user rather than appearing as a stall.113114\subsection{System-prompt routing}115\label{subsec:routing}116117The behaviour of the loop is shaped by a system prompt assembled on every call. Its first lines118inject the current date and time, in a human-readable and an ISO form, so that relative date119expressions resolve against live data rather than against the model's training cut-off; the120remainder is a static policy block whose structure is given in Appendix~\ref{app:prompt}. The121policy instructs the model to answer in the language of the request; to act without procedural122narration and, in particular, to perform an entire custom analysis in a single Python execution123(Section~\ref{subsec:onecall}); to calibrate effort to request complexity---one or two tools for a124simple lookup, three to five for a moderate request, ten or more for a comprehensive report; to125prefer the primary provider for United States equities and the second for international126instruments; and to follow output-formatting conventions for downloadable artifacts, including a127preference for live spreadsheet formulas over constants. A final block directs the model to128refuse, with a fixed bilingual notice, attempts to extract the platform's own implementation.129130This prompt-level routing is the mechanism by which the 245-tool catalogue131(Table~\ref{tab:tools}) is navigated without an explicit planner or a tool retriever of the kind132proposed by \citet{patil2024gorilla} and \citet{qin2024toolllm}: the model is taught \emph{when133not to call} tools as much as when to call them, and the per-tool schema descriptions carry the134rest. The prompt names only about sixty tools explicitly; the others are discoverable solely from135their schemas. Two of the tools the prompt does name---a Monte Carlo simulator and a136volatility-surface generator---are not in the catalogue at the audited commit137(Section~\ref{subsec:numerical}), an inconsistency between prompt and catalogue that the model138would experience as a tool that does not exist.139140\input{tables/tools}141142\subsection{Adaptive reasoning}143\label{subsec:thinking}144145For models that support it, the orchestration service enables adaptive extended thinking by146attaching a thinking directive to the request and asking for a summarised display. A short147capability list determines which models receive the directive, so that the same code path serves148the full model range. As noted in Section~\ref{subsec:transport}, the resulting reasoning summary149arrives on its own delta channel and is streamed to a dedicated client panel. Surfacing the150summary makes the agent's plan legible to the user and provides a partial audit trail of how a151report was assembled; it does not alter the loop, and nothing downstream parses or depends on the152summary's content.153154\subsection{Context management under a token budget}155\label{subsec:context}156157A persistent challenge for conversational agents is the bounded context window: accumulated158history---especially verbose tool results---eventually exceeds what the model can attend to, and159attention over long contexts degrades before the hard limit is reached \citep{liu2024lost}.160VQuant addresses this with a three-stage procedure. The first stage operates at execution time161inside Algorithm~\ref{alg:loop}; the other two are applied immediately before every model call162and are stated as Algorithm~\ref{alg:context}.163164\paragraph{Stage 1: per-result truncation at execution.} When a tool result is serialised for the165model, it is cut at 3{,}000 characters, or 15{,}000 characters for the fourteen web-extraction166tools and the Python tool, whose outputs carry extracted text or figure references that must not167be lost; a marker recording the original length is appended. The full result is retained for168session persistence and for the client.169170\paragraph{Stage 2: re-send cap.} Before each call the server walks every message and caps any171individual tool-result block at 5{,}000 characters. This second cap is reached only by results172that were admitted under the higher 15{,}000-character limit at Stage~1 and are now being re-sent173on a later turn; it removes bulk that the analysis no longer needs.174175\paragraph{Stage 3: adaptive summarisation.} The server then estimates the token count of the176prepared messages with a deliberately cheap heuristic---one token per four characters, summed over177text and tool-result blocks---because the estimate is a trigger, not an accounting figure. If it178exceeds a threshold of $10^{5}$ tokens, the server retains the most recent five messages verbatim179and replaces the older history with a model-generated summary, reinserted as a single synthetic180user message. The threshold was chosen as a conservative fraction of a 200{,}000-token window; the181reference models now offer windows five times larger, so the constant is tunable and its effect182is one of the ablations proposed in Section~\ref{subsec:protocol}. Figure~\ref{fig:context}183illustrates the transformation.184185\input{figures/context}186187\begin{algorithm}[htbp]188\caption{Context preparation before each model call (\textsc{PrepareContext})}189\label{alg:context}190\begin{algorithmic}[1]191\Require message list $M$, model $m$, threshold $\tau = 10^{5}$ tokens, recency $k = 5$, cap $c_2 = 5{,}000$ chars, summary input cap $c_3 = 10{,}000$ chars192\State cap every tool-result block in $M$ at $c_2$ characters \Comment{Stage 2}193\State $t \gets \textsc{EstimateTokens}(M)$ \Comment{$\lceil \text{chars}/4 \rceil$ over text and tool-result blocks}194\If{$t > \tau$} \Comment{Stage 3}195 \State $M_{\text{recent}} \gets$ last $k$ messages of $M$;\quad $M_{\text{old}} \gets$ the rest196 \If{$M_{\text{old}} \neq \varnothing$}197 \State $T \gets \textsc{ExtractText}(M_{\text{old}})$ \Comment{natural-language turns only; last $c_3$ characters}198 \State $s \gets \textsc{Summarize}(T, m)$ \Comment{non-streaming call; 2{,}000-token cap; temperature 0.5}199 \If{the call fails} $s \gets$ first 500 characters of $T$ \Comment{graceful degradation}200 \EndIf201 \State $M \gets [\,\langle\text{user: ``[Previous conversation summary]: ''}+s\rangle\,] \mathbin{\Vert} M_{\text{recent}}$202 \EndIf203\EndIf204\State \Return $M$205\end{algorithmic}206\end{algorithm}207208The summary is produced by a dedicated non-streaming call to the same model with its own system209prompt, a temperature of 0.5, and a 2{,}000-token output cap, instructed to preserve the210analytically relevant context---instruments under discussion, prior conclusions, and user211preferences. Before summarising, a helper extracts only the natural-language turns (discarding212tool-use and tool-result blocks) and keeps at most the last 10{,}000 characters, bounding the cost213of summarisation itself. Should the call fail, the mechanism degrades to a literal excerpt rather214than aborting the request. The net effect is that conversation length is, for practical purposes,215unbounded, at the cost of one additional model call whenever the threshold is crossed and of216whatever the summary omits. The procedure is lossy by construction, and the paper makes no claim217about how much analytically relevant information survives it; that is a measurable quantity and218is included in the evaluation protocol.219220\subsection{Token accounting and its correction}221\label{subsec:accounting}222223Token accounting matters for per-session cost reporting, and the streaming protocol does not224always report final output-token counts reliably. VQuant therefore applies a correction. Input225tokens are read from the provider's \texttt{message\_start} event; output tokens are taken from226streaming usage updates but reconciled, at \texttt{message\_stop}, against a local count obtained227by running the accumulated answer text through a tokenizer. The system records the \emph{maximum}228of the provider's figure and the local count, so that cost is never understated, and falls back to229the provider's figure if local encoding fails.230231Two qualifications correct the first version of this paper. First, the local tokenizer is a232general-purpose byte-pair encoder for OpenAI models (the \texttt{gpt-tokenizer} package), not the233provider's tokenizer; its count is an approximation whose direction of error is not fixed, so the234``exact tokenizer for billing'' of the earlier description overstated what the mechanism235achieves. Second, the local count covers answer text only, not the reasoning tokens that the236provider bills, so the maximum rule guards against under-reporting of text but not of thinking.237Cost is then computed from a single linear schedule---\$3 per million input tokens and \$15 per238million output tokens---and accumulated across all calls in a session, including recursive turns239and summarisation calls. The schedule is the same for every selectable model, so the ledger240mis-states cost whenever a model with different list prices is selected; a per-model price table241is a straightforward correction. The separation of a cheap heuristic for the context trigger from a242tokenizer-based count for the ledger is sound; the earlier description's statement that the243tokenizer also drives the context trigger was incorrect.244245\subsection{The single-call computation convention}246\label{subsec:onecall}247248The final mechanism concerns \emph{how} numerical work is delegated. A naive agent decomposes a249quantitative task into a sequence of tool calls---fetch data, then in a separate turn analyse it,250then in a third turn plot it---each of which incurs a full recursion through251Algorithm~\ref{alg:loop} and therefore a full model round trip. VQuant instead instructs the252model, through the system prompt and the Python tool's own description, to perform an entire253analysis in a single execution of the Python tool: acquire data inside the sandbox, compute, and254render all figures in one call (Listing~\ref{lst:onecall}).255256\begin{lstlisting}[language=Python,caption={Single-call execution: data acquisition,257computation, and visualisation are consolidated into one sandboxed run, so that what would258otherwise be three recursive model round trips becomes one. The \texttt{fmp} client,259\texttt{pd}, \texttt{plt}, and \texttt{save\_excel} are injected by the sandbox260(Section~\ref{subsec:sandbox}).},label={lst:onecall}]261# One tool call performs the whole analysis:262data = fmp.get_historical_prices("NVDA") # live fetch inside the sandbox263df = pd.DataFrame(data["historical"]).sort_values("date")264df["ret"] = df["close"].pct_change()265266plt.figure(figsize=(12, 6)) # figure 1: price path267plt.plot(pd.to_datetime(df["date"]), df["close"]); plt.tight_layout()268269plt.figure(figsize=(12, 6)) # figure 2: return distribution270plt.hist(df["ret"].dropna(), bins=50); plt.tight_layout()271272save_excel(df, "nvda.xlsx") # spreadsheet export in the same run273\end{lstlisting}274275Three rules, enforced through the prompt, make the convention effective. First, the model must not276hard-code data arrays: inlining hundreds of values bloats the generated tokens, which must be277streamed and paid for, and defeats reproducibility, since the provenance of the numbers is lost;278data must be fetched programmatically through the injected client. Second, each visualisation279must be a separate figure rather than a subplot grid, which yields individually captionable280images under the capture sweep of Section~\ref{subsec:sandbox}. Third, all artifacts---figures281and spreadsheets---are produced within the same execution. The convention reduces the recursion282depth for an analytical task from three or more to one. We call it a convention rather than a283mechanism because it is enforced by instruction, not by code: the model can disobey it, and the284rate at which it does so is one of the quantities the evaluation protocol is designed to measure.285Its cost is a loss of granularity---a single failed execution forfeits the whole analysis, whereas286a decomposed agent can recover at a finer level---which Section~\ref{subsec:threats} discusses.287