% ============================================================================
\section{Orchestration Mechanisms}
\label{sec:methodology}
% ============================================================================

This section is the technical core of the paper. We describe, mechanism by mechanism, how VQuant
drives a language model through a complete analysis: the streaming transport and its event
protocol, the recursive tool-calling loop, the batching policy, the routing prompt, the
context-management procedure, the token-accounting correction, and the single-call computation
convention. Each mechanism is described as implemented at the audited commit, with its actual
constants, so that the design can be reproduced; Appendix~\ref{app:audit} gives the source
location of every statement.

\subsection{Streaming transport and event protocol}
\label{subsec:transport}

On the reference path VQuant does not use the provider's software development kit. It opens a
raw HTTP connection to the provider's messages endpoint with a streaming response and parses the
resulting SSE byte stream itself, which gives the server full control over how partial output is
interpreted and re-emitted.

The parser maintains a single growing string buffer. On each network chunk it appends the bytes,
splits the buffer on newlines, and \emph{retains the final, possibly incomplete, line} for the next
chunk---a detail that is essential for correctness because SSE frames do not align with TCP
segment boundaries. Each complete line beginning with the \texttt{data:} prefix is stripped and
parsed as JSON into a typed event, and the parser dispatches on the event's \texttt{type} field.
The eight relevant cases are summarised in Table~\ref{tab:events} and the flow is shown in
Figure~\ref{fig:streaming}.

\input{figures/streaming}

\input{tables/events}

Three behaviours of this dispatcher deserve emphasis. First, the model's output travels on two
independent channels: answer text arrives as \texttt{text\_delta} events and is accumulated into
the answer while being forwarded to the client, whereas the model's reasoning summary arrives as
\texttt{thinking\_delta} events on a separate callback that the client renders in a distinct
panel. Second, a tool call is not delivered atomically: the tool's name and identifier arrive in a
\texttt{content\_block\_start}, its arguments stream in as a sequence of
\texttt{input\_json\_delta} fragments that the parser concatenates into a partial JSON string, and
only at \texttt{content\_block\_stop} is that string parsed into the structured argument object.
The loop must therefore wait for the block to close before a tool becomes executable. Third, every
forwarded event is flushed to the client immediately, so the user perceives output as it is
generated rather than in batches.

\subsection{The recursive tool-calling loop}
\label{subsec:loop}

The agentic loop is implemented as recursion. A single routine issues one streaming model call;
when the call resolves it either terminates---if the model requested no tools---or executes the
requested tools, appends their results to the conversation, and calls itself again.
Algorithm~\ref{alg:loop} states the procedure and Figure~\ref{fig:loop} depicts it.

\input{figures/loop}

\begin{algorithm}[htbp]
\caption{Recursive tool-calling loop (\textsc{ProcessResponse})}
\label{alg:loop}
\begin{algorithmic}[1]
\Require message list $M$ (mutated in place), model $m$, tool catalogue $\mathcal{T}$ ($|\mathcal{T}|=245$)
\State $M \gets \textsc{PrepareContext}(M, m)$ \Comment{Algorithm~\ref{alg:context}}
\State $(\textit{text}, R, \textit{usage}) \gets \textsc{StreamModel}(m, M, \mathcal{T})$ \Comment{Section~\ref{subsec:transport}}
\State accumulate \textit{usage}; stream \textit{text} and reasoning summary to the client
\If{$R = \varnothing$} \Comment{no tool calls $\Rightarrow$ final answer}
  \State \Return
\EndIf
\State append $\langle\text{assistant}: \text{tool\_use blocks of } R\rangle$ to $M$ \Comment{tool-use blocks only; no text or thinking blocks}
\ForAll{batch $B$ in $\textsc{Chunk}(R, 5)$} \Comment{Section~\ref{subsec:batching}}
  \State emit a \texttt{status} event naming the tools in $B$ if more than one batch
  \ForAll{tool call $r \in B$}
    \State $\textit{res} \gets \textsc{Execute}(r)$; stream a typed result event to the client
    \State $\textit{res} \gets \textsc{Truncate}(\textit{res}, c_1(r))$ \Comment{$c_1 = 3{,}000$ chars; $15{,}000$ for extraction and Python tools}
    \State collect $\textit{res}$ as a tool\_result block keyed to $r$'s identifier
  \EndFor
  \State \textsc{Wait}(500\,\text{ms}) if another batch follows
\EndFor
\State append $\langle\text{user}: \text{all tool\_result blocks}\rangle$ to $M$
\State \textsc{ProcessResponse}($M, m, \mathcal{T}$) \Comment{recurse with augmented context}
\end{algorithmic}
\end{algorithm}

The conversation is represented as a list of messages mutated as the loop proceeds. When the model
requests tools, the server first appends an \emph{assistant} message containing the model's
tool-use blocks, then---after execution---appends a \emph{user} message whose content is the array
of tool-result blocks, each keyed to its call identifier. This pairing of tool-use and tool-result
blocks is what the provider's protocol requires, and maintaining it across the recursion is the
loop's central invariant. Two consequences of the implementation should be noted. The assistant
message carries only the tool-use blocks: any answer text the model emitted in the same turn is
streamed to the user but not re-sent to the model, and reasoning blocks are not replayed, so the
model re-derives its plan from the tool results on the next turn rather than from its own prior
deliberation. And because the routine recurses without a depth bound, termination relies on the
model eventually returning no tool requests; the first version of this paper did not state this,
and a maximum recursion depth is a reasonable hardening (Section~\ref{subsec:threats}). Recursion
terminates when a call returns no tool requests, at which point the accumulated text is the
report. Multimodal input is handled at the entry point: an uploaded image is attached to the
first user message as a base64 block alongside the query.

\subsection{Batched tool execution}
\label{subsec:batching}

Within a single turn the model frequently requests many tools at once; a comprehensive report may
trigger a dozen or more. Executing all of them simultaneously would risk provider rate limits and
connection timeouts. VQuant therefore partitions the requested calls into batches of at most
five, executes the batches sequentially, and inserts a 500-millisecond pause between consecutive
batches. Within a batch the calls are executed one after another, not concurrently; the batch is
a unit of pacing rather than of parallelism. Results from all batches are accumulated and returned
to the model together in the single tool-result message of Algorithm~\ref{alg:loop}, so the model
sees a complete result set on the next turn. This matters for a second reason: the provider's
guidance is that parallel tool calls answered across several user messages lead the model to stop
issuing them in parallel, so the single-message return preserves the model's tendency to batch its
own requests. When more than one batch is required the server emits a \texttt{status} event naming
the tools in flight, so the throttling is visible to the user rather than appearing as a stall.

\subsection{System-prompt routing}
\label{subsec:routing}

The behaviour of the loop is shaped by a system prompt assembled on every call. Its first lines
inject the current date and time, in a human-readable and an ISO form, so that relative date
expressions resolve against live data rather than against the model's training cut-off; the
remainder is a static policy block whose structure is given in Appendix~\ref{app:prompt}. The
policy instructs the model to answer in the language of the request; to act without procedural
narration and, in particular, to perform an entire custom analysis in a single Python execution
(Section~\ref{subsec:onecall}); to calibrate effort to request complexity---one or two tools for a
simple lookup, three to five for a moderate request, ten or more for a comprehensive report; to
prefer the primary provider for United States equities and the second for international
instruments; and to follow output-formatting conventions for downloadable artifacts, including a
preference for live spreadsheet formulas over constants. A final block directs the model to
refuse, with a fixed bilingual notice, attempts to extract the platform's own implementation.

This prompt-level routing is the mechanism by which the 245-tool catalogue
(Table~\ref{tab:tools}) is navigated without an explicit planner or a tool retriever of the kind
proposed by \citet{patil2024gorilla} and \citet{qin2024toolllm}: the model is taught \emph{when
not to call} tools as much as when to call them, and the per-tool schema descriptions carry the
rest. The prompt names only about sixty tools explicitly; the others are discoverable solely from
their schemas. Two of the tools the prompt does name---a Monte Carlo simulator and a
volatility-surface generator---are not in the catalogue at the audited commit
(Section~\ref{subsec:numerical}), an inconsistency between prompt and catalogue that the model
would experience as a tool that does not exist.

\input{tables/tools}

\subsection{Adaptive reasoning}
\label{subsec:thinking}

For models that support it, the orchestration service enables adaptive extended thinking by
attaching a thinking directive to the request and asking for a summarised display. A short
capability list determines which models receive the directive, so that the same code path serves
the full model range. As noted in Section~\ref{subsec:transport}, the resulting reasoning summary
arrives on its own delta channel and is streamed to a dedicated client panel. Surfacing the
summary makes the agent's plan legible to the user and provides a partial audit trail of how a
report was assembled; it does not alter the loop, and nothing downstream parses or depends on the
summary's content.

\subsection{Context management under a token budget}
\label{subsec:context}

A persistent challenge for conversational agents is the bounded context window: accumulated
history---especially verbose tool results---eventually exceeds what the model can attend to, and
attention over long contexts degrades before the hard limit is reached \citep{liu2024lost}.
VQuant addresses this with a three-stage procedure. The first stage operates at execution time
inside Algorithm~\ref{alg:loop}; the other two are applied immediately before every model call
and are stated as Algorithm~\ref{alg:context}.

\paragraph{Stage 1: per-result truncation at execution.} When a tool result is serialised for the
model, it is cut at 3{,}000 characters, or 15{,}000 characters for the fourteen web-extraction
tools and the Python tool, whose outputs carry extracted text or figure references that must not
be lost; a marker recording the original length is appended. The full result is retained for
session persistence and for the client.

\paragraph{Stage 2: re-send cap.} Before each call the server walks every message and caps any
individual tool-result block at 5{,}000 characters. This second cap is reached only by results
that were admitted under the higher 15{,}000-character limit at Stage~1 and are now being re-sent
on a later turn; it removes bulk that the analysis no longer needs.

\paragraph{Stage 3: adaptive summarisation.} The server then estimates the token count of the
prepared messages with a deliberately cheap heuristic---one token per four characters, summed over
text and tool-result blocks---because the estimate is a trigger, not an accounting figure. If it
exceeds a threshold of $10^{5}$ tokens, the server retains the most recent five messages verbatim
and replaces the older history with a model-generated summary, reinserted as a single synthetic
user message. The threshold was chosen as a conservative fraction of a 200{,}000-token window; the
reference models now offer windows five times larger, so the constant is tunable and its effect
is one of the ablations proposed in Section~\ref{subsec:protocol}. Figure~\ref{fig:context}
illustrates the transformation.

\input{figures/context}

\begin{algorithm}[htbp]
\caption{Context preparation before each model call (\textsc{PrepareContext})}
\label{alg:context}
\begin{algorithmic}[1]
\Require message list $M$, model $m$, threshold $\tau = 10^{5}$ tokens, recency $k = 5$, cap $c_2 = 5{,}000$ chars, summary input cap $c_3 = 10{,}000$ chars
\State cap every tool-result block in $M$ at $c_2$ characters \Comment{Stage 2}
\State $t \gets \textsc{EstimateTokens}(M)$ \Comment{$\lceil \text{chars}/4 \rceil$ over text and tool-result blocks}
\If{$t > \tau$} \Comment{Stage 3}
  \State $M_{\text{recent}} \gets$ last $k$ messages of $M$;\quad $M_{\text{old}} \gets$ the rest
  \If{$M_{\text{old}} \neq \varnothing$}
    \State $T \gets \textsc{ExtractText}(M_{\text{old}})$ \Comment{natural-language turns only; last $c_3$ characters}
    \State $s \gets \textsc{Summarize}(T, m)$ \Comment{non-streaming call; 2{,}000-token cap; temperature 0.5}
    \If{the call fails} $s \gets$ first 500 characters of $T$ \Comment{graceful degradation}
    \EndIf
    \State $M \gets [\,\langle\text{user: ``[Previous conversation summary]: ''}+s\rangle\,] \mathbin{\Vert} M_{\text{recent}}$
  \EndIf
\EndIf
\State \Return $M$
\end{algorithmic}
\end{algorithm}

The summary is produced by a dedicated non-streaming call to the same model with its own system
prompt, a temperature of 0.5, and a 2{,}000-token output cap, instructed to preserve the
analytically relevant context---instruments under discussion, prior conclusions, and user
preferences. Before summarising, a helper extracts only the natural-language turns (discarding
tool-use and tool-result blocks) and keeps at most the last 10{,}000 characters, bounding the cost
of summarisation itself. Should the call fail, the mechanism degrades to a literal excerpt rather
than aborting the request. The net effect is that conversation length is, for practical purposes,
unbounded, at the cost of one additional model call whenever the threshold is crossed and of
whatever the summary omits. The procedure is lossy by construction, and the paper makes no claim
about how much analytically relevant information survives it; that is a measurable quantity and
is included in the evaluation protocol.

\subsection{Token accounting and its correction}
\label{subsec:accounting}

Token accounting matters for per-session cost reporting, and the streaming protocol does not
always report final output-token counts reliably. VQuant therefore applies a correction. Input
tokens are read from the provider's \texttt{message\_start} event; output tokens are taken from
streaming usage updates but reconciled, at \texttt{message\_stop}, against a local count obtained
by running the accumulated answer text through a tokenizer. The system records the \emph{maximum}
of the provider's figure and the local count, so that cost is never understated, and falls back to
the provider's figure if local encoding fails.

Two qualifications correct the first version of this paper. First, the local tokenizer is a
general-purpose byte-pair encoder for OpenAI models (the \texttt{gpt-tokenizer} package), not the
provider's tokenizer; its count is an approximation whose direction of error is not fixed, so the
``exact tokenizer for billing'' of the earlier description overstated what the mechanism
achieves. Second, the local count covers answer text only, not the reasoning tokens that the
provider bills, so the maximum rule guards against under-reporting of text but not of thinking.
Cost is then computed from a single linear schedule---\$3 per million input tokens and \$15 per
million output tokens---and accumulated across all calls in a session, including recursive turns
and summarisation calls. The schedule is the same for every selectable model, so the ledger
mis-states cost whenever a model with different list prices is selected; a per-model price table
is a straightforward correction. The separation of a cheap heuristic for the context trigger from a
tokenizer-based count for the ledger is sound; the earlier description's statement that the
tokenizer also drives the context trigger was incorrect.

\subsection{The single-call computation convention}
\label{subsec:onecall}

The final mechanism concerns \emph{how} numerical work is delegated. A naive agent decomposes a
quantitative task into a sequence of tool calls---fetch data, then in a separate turn analyse it,
then in a third turn plot it---each of which incurs a full recursion through
Algorithm~\ref{alg:loop} and therefore a full model round trip. VQuant instead instructs the
model, through the system prompt and the Python tool's own description, to perform an entire
analysis in a single execution of the Python tool: acquire data inside the sandbox, compute, and
render all figures in one call (Listing~\ref{lst:onecall}).

\begin{lstlisting}[language=Python,caption={Single-call execution: data acquisition,
computation, and visualisation are consolidated into one sandboxed run, so that what would
otherwise be three recursive model round trips becomes one. The \texttt{fmp} client,
\texttt{pd}, \texttt{plt}, and \texttt{save\_excel} are injected by the sandbox
(Section~\ref{subsec:sandbox}).},label={lst:onecall}]
# One tool call performs the whole analysis:
data = fmp.get_historical_prices("NVDA")      # live fetch inside the sandbox
df = pd.DataFrame(data["historical"]).sort_values("date")
df["ret"] = df["close"].pct_change()

plt.figure(figsize=(12, 6))                    # figure 1: price path
plt.plot(pd.to_datetime(df["date"]), df["close"]); plt.tight_layout()

plt.figure(figsize=(12, 6))                    # figure 2: return distribution
plt.hist(df["ret"].dropna(), bins=50); plt.tight_layout()

save_excel(df, "nvda.xlsx")                     # spreadsheet export in the same run
\end{lstlisting}

Three rules, enforced through the prompt, make the convention effective. First, the model must not
hard-code data arrays: inlining hundreds of values bloats the generated tokens, which must be
streamed and paid for, and defeats reproducibility, since the provenance of the numbers is lost;
data must be fetched programmatically through the injected client. Second, each visualisation
must be a separate figure rather than a subplot grid, which yields individually captionable
images under the capture sweep of Section~\ref{subsec:sandbox}. Third, all artifacts---figures
and spreadsheets---are produced within the same execution. The convention reduces the recursion
depth for an analytical task from three or more to one. We call it a convention rather than a
mechanism because it is enforced by instruction, not by code: the model can disobey it, and the
rate at which it does so is one of the quantities the evaluation protocol is designed to measure.
Its cost is a loss of granularity---a single failed execution forfeits the whole analysis, whereas
a decomposed agent can recover at a finer level---which Section~\ref{subsec:threats} discusses.