spb/wp13_uqo

1% ============================================================================2\section{Performance, Reliability, and Security}3\label{sec:evaluation}4% ============================================================================56This section discusses the engineering properties that make VQuant usable in production:7its performance characteristics, cost discipline, reliability and security measures, and the8limitations that bound the contribution. We frame these as the practical counterparts to the9methodology of Section~\ref{sec:methodology}.1011\subsection{Performance and the streaming experience}12\label{subsec:performance}1314The dominant cost in an agentic system is not raw computation but \emph{round trips}: every15model invocation and every external request adds latency. VQuant's design attacks this on three16fronts (Table~\ref{tab:robustness}). The single-call paradigm of Section~\ref{subsec:onecall}17collapses what would be three or more model round trips---fetch, analyze, plot---into one,18which is the platform's largest latency saving for analytically intensive queries. Execution19batching overlaps independent tool calls within a turn while bounding instantaneous provider20load. And end-to-end streaming decouples \emph{perceived} latency from \emph{total} latency: the21user sees reasoning, text, tool progress, and figures appear incrementally, so a report that22takes tens of seconds to complete begins responding within a second. This last property is why23the streaming architecture of Section~\ref{subsec:frontend} is a performance feature and not24merely a cosmetic one.2526\input{tables/robustness}2728\subsection{Cost discipline}29\label{subsec:cost}3031Because model usage is metered by token, cost control is a first-order concern. VQuant estimates32token counts locally with a tokenizer before each call, which both drives the33context-management trigger of Algorithm~\ref{alg:context} and yields accurate per-session34accounting. The adaptive-summarization mechanism directly limits cost growth in long sessions by35preventing the unbounded accumulation of history, and truncating oversized tool results removes36payload bloat that would otherwise be re-sent on every turn. The user-facing model selector37(Table~\ref{tab:models}) lets users match model capability to query difficulty, routing simple38lookups to a fast, inexpensive tier and reserving the flagship model for the hardest analyses.3940\subsection{Authentication, limits, and bring-your-own-keys}41\label{subsec:auth}4243VQuant supports both local accounts and third-party single sign-on, with sessions persisted in44the same store used in either deployment environment. Usage is governed by daily limits that45differ for anonymous and authenticated users, enforced through per-user counters in the data46layer. A notable feature is \emph{bring-your-own-keys}: a user may supply personal API47credentials for the model provider, the market-data provider, and the web-research services;48when present, these are used in place of the platform defaults and exempt the user from the49daily cap. This design cleanly separates the platform's orchestration value from the cost of the50underlying APIs, and it lets power users operate at their own expense and rate limits.5152\subsection{Reliability and security}53\label{subsec:security}5455Several measures harden the system. All request bodies are validated against schemas at the56boundary, so malformed or untyped input is rejected uniformly before reaching business logic.57Database access is mediated by an object-relational mapper that parameterizes queries,58mitigating injection. The front end escapes rendered content and parses Markdown through a safe59pipeline, limiting cross-site scripting. Secrets are injected at runtime rather than committed,60and the bring-your-own-keys path keeps user credentials out of shared configuration. At the61model layer, the system prompt includes a guardrail that responds to attempts to extract the62platform's own implementation with a refusal rather than disclosure, providing a measure of63protection for the orchestration logic itself. Finally, the platform records analytics64events---active users, query rates, tool-usage frequency, model selection, and65document-generation breakdowns---giving operators the observability needed to detect anomalies66and capacity-plan.6768\subsection{Limitations and threats to validity}69\label{subsec:limitations}7071We close with an honest accounting of what this paper does and does not establish. First, the72contribution is \emph{architectural and methodological}; we do not provide a controlled73empirical evaluation of report quality, latency, or cost against baseline agents, and the74operational figures in Table~\ref{tab:robustness} are indicative rather than benchmarked.75Second, the platform inherits the limitations of its dependencies: the accuracy of any report is76bounded by the correctness and timeliness of the upstream market-data provider, and the model's77synthesis can still contain interpretive errors even when every number is computed78deterministically. The strict separation of reasoning from computation contains numerical79hallucination but does not eliminate the risk of a flawed \emph{narrative} around correct80numbers. Third, the quantitative methods rest on classical modeling assumptions---log-normal81prices, constant parameters over the estimation window, normally distributed returns in the82parametric VaR---that may be violated in practice; the platform exposes these methods as tools83and does not, by itself, certify their appropriateness for a given asset. Fourth, the84single-call paradigm trades some robustness for speed: a single failed execution forfeits an85entire analysis, whereas a decomposed agent can recover at a finer granularity. We regard the86design as the right default for interactive use but note the trade-off. These limitations87delineate a clear agenda for evaluation, which we sketch in the conclusion.88