spb/wp13_uqo

1% ============================================================================2\section{Design Rationale, Threats to Validity, and a Proposed Evaluation}3\label{sec:evaluation}4% ============================================================================56This section replaces what the first version of this paper called ``Performance, Reliability,7and Security''. It states the rationale for each design decision as a hypothesis about the8property the decision is meant to secure and the measurement that would test it9(Table~\ref{tab:rationale}); it enumerates threats to the validity of the design and of this10description; it records what is and is not fixed for reproducibility; and it specifies the11evaluation that has not yet been executed.1213\subsection{Design rationale}14\label{subsec:rationale}1516The dominant cost in an agentic system is not raw computation but round trips: every model17invocation and every external request adds latency and, for model calls, tokens. Three decisions18address this. The single-call convention (Section~\ref{subsec:onecall}) is intended to collapse19the fetch--analyse--plot sequence into one recursion level; batching20(Section~\ref{subsec:batching}) bounds instantaneous provider load without changing what the21model observes; and end-to-end streaming decouples perceived from total latency, so that the user22sees reasoning, text, and progress while the report is still being assembled. Three further23decisions address correctness and auditability: the process-isolated sandbox, the rule that data24are fetched programmatically rather than inlined, and the live-formula spreadsheets. Two address25scale: the three-stage context procedure and the tokenizer-based ledger. Table~\ref{tab:rationale}26lists each with the property it targets and the measurement that would confirm it. None of these27measurements has been made; the table is a statement of intent, and the first version's28``indicative'' latency figures, which had no measurement behind them, have been removed.2930\input{tables/robustness}3132\subsection{Authentication, usage limits, and observability}33\label{subsec:auth}3435At the audited commit the platform supports local accounts with e-mail and password, e-mail36verification by one-time code, trusted devices that skip the code on subsequent logins, rate37limiting of authentication attempts per account and per connection, and a security event log.38Anonymous use is permitted and, for anonymous users only, each completed analysis is39automatically published as a shared report so that it remains retrievable. The platform records40request logs and aggregated analytics (request counts, failures, active and unique users, response41times) for operators. Three features described in the first version of this paper do not exist in42any committed version of the code: third-party single sign-on, daily usage caps that differ for43anonymous and authenticated users, and the ``bring-your-own-keys'' mechanism by which users44would supply personal provider credentials. They may have been planned or deployed through45configuration outside the repository, but they are not implemented in it, and we withdraw the46corresponding claims.4748\subsection{Security measures}49\label{subsec:security}5051Several measures harden the system. Request bodies are validated against schemas at the52boundary. Database access is mediated by the object-relational mapper, which parameterises53queries. The front end parses Markdown through a sanitising pipeline, limiting cross-site54scripting from model output. Secrets are injected from the environment rather than committed.55Model-authored code runs in a child process with a wall-clock timeout, so that an infinite loop or a56crash cannot take the server down with it. The system prompt includes a guardrail that answers57attempts to extract the platform's implementation with a fixed refusal notice; this is a58prompt-level measure and, like all such measures, provides no guarantee against a determined59adversary. We do not claim that the sandbox is a security boundary against malicious code: the60child process runs with the server's privileges and network access, and the model is not an61adversary in the threat model, but a user who can steer the model into writing arbitrary code62is. Containerisation or a seccomp-style policy would be the appropriate hardening.6364\subsection{Threats to validity}65\label{subsec:threats}6667We distinguish threats to the validity of the design from threats to the validity of this68description.6970\paragraph{Design.} First, the separation of reasoning from computation contains \emph{numerical}71hallucination but not \emph{interpretive} error: the model can still write a wrong narrative72around correct numbers, select an inappropriate method, or mis-specify the code it writes, and73nothing in the loop checks the narrative against the numbers. Second, the single-call convention74trades granularity for speed; a failed execution forfeits the analysis, and the model's only75recovery path is to rewrite the code on the next turn. Third, the recursion has no depth bound, so76termination depends on the model; a bound is cheap and should be added. Fourth, the context77procedure is lossy, and the threshold and recency constants were set by judgement rather than78measurement. Fifth, the accuracy of every report is bounded by the correctness and timeliness of79the upstream data providers, and the platform does not cross-check providers against each other.80Sixth, the classical methods rest on standard assumptions---log-normal prices, constant parameters81over the estimation window, normal returns for parametric VaR---that the platform exposes but82does not certify for a given asset. Seventh, PDF rendering depends on externally hosted renderer83assets, and the Beamer path on a local \TeX{} installation; both are deployment dependencies that84the code does not verify at start-up.8586\paragraph{Description.} The paper describes one commit of a system that continues to change;87the model identifiers, tool counts, and constants reported here are facts about that commit, and88three of them (the tool count, the tokenizer, and the reachability of the numerical routines)89already differed from the platform's own documentation at the time of the audit. Appendix~\ref{app:audit}90lists the source of every statement so that a reader with access to the repository can re-check91them, and the audit script that produced the counts is distributed with the paper.9293\subsection{Reproducibility}94\label{subsec:repro}9596Reproducibility of a system of this kind has two layers. The \emph{design} is reproducible from97this paper: Algorithms~\ref{alg:loop} and~\ref{alg:context}, the constants in98Table~\ref{tab:rationale}, the prompt structure in Appendix~\ref{app:prompt}, and the99inter-process contract of Section~\ref{subsec:sandbox} are sufficient to re-implement it. The100\emph{outputs} are not reproducible in the strict sense, for four reasons that we record rather101than conceal. The main model call sets the sampling temperature to~1 and the summarisation call102to~0.5; the reference models are not seedable through the provider interface; the Python code is103authored by the model and therefore differs across runs even for an identical request; and the104pre-built Monte Carlo routine draws pseudo-random numbers without a fixed seed. What \emph{is}105reproducible, given a report, is every number in it: the tool calls and the executed code are106persisted with the session and displayed to the user, so any figure can be recomputed from its107recorded inputs. The system prompt is reproduced structurally in Appendix~\ref{app:prompt}; its108full text is in the repository.109110\subsection{A proposed evaluation protocol (not executed)}111\label{subsec:protocol}112113The following protocol is the evaluation step that the design-science process requires114\citep{peffers2007design} and that this paper does not perform. It is stated in enough detail to115be pre-registered.116117\paragraph{Task suite.} Three task families, each with ground truth that does not depend on a118model. (a)~\emph{Retrieval tasks}: 200 questions whose answers are single values obtainable from119the market-data endpoints at a frozen date (``What was NVDA's trailing twelve-month revenue as120of 30~June~2026?''), with the endpoint's value as ground truth; FinanceBench121\citep{islam2023financebench} supplies a complementary set over public filings.122(b)~\emph{Computation tasks}: 100 requests for standard quantities---Black--Scholes prices and123Greeks, GARCH(1,1) parameters, historical and parametric VaR, Monte Carlo terminal-value124quantiles---on frozen price histories, with ground truth from independent reference125implementations and seeded simulation; the quantitative-reasoning tasks of FinBen126\citep{xie2024finben} can be added. (c)~\emph{Report tasks}: 50 comprehensive requests graded by127two human analysts on a fixed rubric (coverage, correctness of every stated number against the128tool results, soundness of narrative, absence of unsupported claims), with an LLM judge129\citep{zheng2023judge} calibrated against the human grades on a held-out subset before being used130at scale.131132\paragraph{Metrics.} For (a) and (b): exact-match or tolerance-based accuracy; the133\emph{unsupported-number rate}, defined as the share of numbers in the response that do not appear134in any tool result or executed-code output, which operationalises numerical hallucination; and135tool-selection precision and recall against a minimal correct tool set. For (c): rubric scores,136inter-rater agreement, and the unsupported-claim rate. For all families: end-to-end latency137(median and 95th percentile), time to first token, number of model round trips, input and output138tokens (reasoning tokens reported separately), dollar cost from per-model list prices, and the139sandbox failure rate with its causes.140141\paragraph{Ablations.} Each mechanism is removed or varied while the rest is held fixed:142(i)~tools disabled (the model answers from parameters), which measures what grounding adds;143(ii)~adaptive thinking disabled; (iii)~the single-call instruction removed, so the model is free144to decompose, which measures the round trips and the failure rate the convention trades off;145(iv)~the context procedure varied over thresholds of $5\times10^{4}$, $10^{5}$, and $2\times10^{5}$146tokens and recency of 3, 5, and 10 messages on 20-turn sessions, scoring the retention of147analytically relevant facts planted early in the session; (v)~batch size varied over 1, 5, and148unbounded, measuring rate-limit errors and latency; and (vi)~model tier varied across the selector149of Table~\ref{tab:models}.150151\paragraph{Baselines.} A plain model without tools; a ReAct-style loop \citep{yao2023react} with152one tool per turn; a code-interpreter-only agent in the spirit of program-aided language models153\citep{gao2023pal} with the same sandbox but no structured endpoints; and, as an upper reference154on the computation tasks, the reference implementations themselves.155156\paragraph{Controls.} All runs at a frozen data snapshot (recorded endpoint responses replayed157through a cache), three repetitions per task to estimate variance under temperature~1, and the158same prompts and constants as the audited commit unless the ablation varies them. Reported159outcomes would be accompanied by confidence intervals from the repetitions and by the full160transcripts.161162The protocol would answer the questions the first version of this paper left as assertions:163whether grounding reduces unsupported numbers to near zero, by how much the single-call164convention reduces round trips and at what cost in failures, how much the context procedure165loses, and what the architecture costs relative to simpler baselines.166