% ============================================================================
\section{Design Rationale, Threats to Validity, and a Proposed Evaluation}
\label{sec:evaluation}
% ============================================================================

This section replaces what the first version of this paper called ``Performance, Reliability,
and Security''. It states the rationale for each design decision as a hypothesis about the
property the decision is meant to secure and the measurement that would test it
(Table~\ref{tab:rationale}); it enumerates threats to the validity of the design and of this
description; it records what is and is not fixed for reproducibility; and it specifies the
evaluation that has not yet been executed.

\subsection{Design rationale}
\label{subsec:rationale}

The dominant cost in an agentic system is not raw computation but round trips: every model
invocation and every external request adds latency and, for model calls, tokens. Three decisions
address this. The single-call convention (Section~\ref{subsec:onecall}) is intended to collapse
the fetch--analyse--plot sequence into one recursion level; batching
(Section~\ref{subsec:batching}) bounds instantaneous provider load without changing what the
model observes; and end-to-end streaming decouples perceived from total latency, so that the user
sees reasoning, text, and progress while the report is still being assembled. Three further
decisions address correctness and auditability: the process-isolated sandbox, the rule that data
are fetched programmatically rather than inlined, and the live-formula spreadsheets. Two address
scale: the three-stage context procedure and the tokenizer-based ledger. Table~\ref{tab:rationale}
lists each with the property it targets and the measurement that would confirm it. None of these
measurements has been made; the table is a statement of intent, and the first version's
``indicative'' latency figures, which had no measurement behind them, have been removed.

\input{tables/robustness}

\subsection{Authentication, usage limits, and observability}
\label{subsec:auth}

At the audited commit the platform supports local accounts with e-mail and password, e-mail
verification by one-time code, trusted devices that skip the code on subsequent logins, rate
limiting of authentication attempts per account and per connection, and a security event log.
Anonymous use is permitted and, for anonymous users only, each completed analysis is
automatically published as a shared report so that it remains retrievable. The platform records
request logs and aggregated analytics (request counts, failures, active and unique users, response
times) for operators. Three features described in the first version of this paper do not exist in
any committed version of the code: third-party single sign-on, daily usage caps that differ for
anonymous and authenticated users, and the ``bring-your-own-keys'' mechanism by which users
would supply personal provider credentials. They may have been planned or deployed through
configuration outside the repository, but they are not implemented in it, and we withdraw the
corresponding claims.

\subsection{Security measures}
\label{subsec:security}

Several measures harden the system. Request bodies are validated against schemas at the
boundary. Database access is mediated by the object-relational mapper, which parameterises
queries. The front end parses Markdown through a sanitising pipeline, limiting cross-site
scripting from model output. Secrets are injected from the environment rather than committed.
Model-authored code runs in a child process with a wall-clock timeout, so that an infinite loop or a
crash cannot take the server down with it. The system prompt includes a guardrail that answers
attempts to extract the platform's implementation with a fixed refusal notice; this is a
prompt-level measure and, like all such measures, provides no guarantee against a determined
adversary. We do not claim that the sandbox is a security boundary against malicious code: the
child process runs with the server's privileges and network access, and the model is not an
adversary in the threat model, but a user who can steer the model into writing arbitrary code
is. Containerisation or a seccomp-style policy would be the appropriate hardening.

\subsection{Threats to validity}
\label{subsec:threats}

We distinguish threats to the validity of the design from threats to the validity of this
description.

\paragraph{Design.} First, the separation of reasoning from computation contains \emph{numerical}
hallucination but not \emph{interpretive} error: the model can still write a wrong narrative
around correct numbers, select an inappropriate method, or mis-specify the code it writes, and
nothing in the loop checks the narrative against the numbers. Second, the single-call convention
trades granularity for speed; a failed execution forfeits the analysis, and the model's only
recovery path is to rewrite the code on the next turn. Third, the recursion has no depth bound, so
termination depends on the model; a bound is cheap and should be added. Fourth, the context
procedure is lossy, and the threshold and recency constants were set by judgement rather than
measurement. Fifth, the accuracy of every report is bounded by the correctness and timeliness of
the upstream data providers, and the platform does not cross-check providers against each other.
Sixth, the classical methods rest on standard assumptions---log-normal prices, constant parameters
over the estimation window, normal returns for parametric VaR---that the platform exposes but
does not certify for a given asset. Seventh, PDF rendering depends on externally hosted renderer
assets, and the Beamer path on a local \TeX{} installation; both are deployment dependencies that
the code does not verify at start-up.

\paragraph{Description.} The paper describes one commit of a system that continues to change;
the model identifiers, tool counts, and constants reported here are facts about that commit, and
three of them (the tool count, the tokenizer, and the reachability of the numerical routines)
already differed from the platform's own documentation at the time of the audit. Appendix~\ref{app:audit}
lists the source of every statement so that a reader with access to the repository can re-check
them, and the audit script that produced the counts is distributed with the paper.

\subsection{Reproducibility}
\label{subsec:repro}

Reproducibility of a system of this kind has two layers. The \emph{design} is reproducible from
this paper: Algorithms~\ref{alg:loop} and~\ref{alg:context}, the constants in
Table~\ref{tab:rationale}, the prompt structure in Appendix~\ref{app:prompt}, and the
inter-process contract of Section~\ref{subsec:sandbox} are sufficient to re-implement it. The
\emph{outputs} are not reproducible in the strict sense, for four reasons that we record rather
than conceal. The main model call sets the sampling temperature to~1 and the summarisation call
to~0.5; the reference models are not seedable through the provider interface; the Python code is
authored by the model and therefore differs across runs even for an identical request; and the
pre-built Monte Carlo routine draws pseudo-random numbers without a fixed seed. What \emph{is}
reproducible, given a report, is every number in it: the tool calls and the executed code are
persisted with the session and displayed to the user, so any figure can be recomputed from its
recorded inputs. The system prompt is reproduced structurally in Appendix~\ref{app:prompt}; its
full text is in the repository.

\subsection{A proposed evaluation protocol (not executed)}
\label{subsec:protocol}

The following protocol is the evaluation step that the design-science process requires
\citep{peffers2007design} and that this paper does not perform. It is stated in enough detail to
be pre-registered.

\paragraph{Task suite.} Three task families, each with ground truth that does not depend on a
model. (a)~\emph{Retrieval tasks}: 200 questions whose answers are single values obtainable from
the market-data endpoints at a frozen date (``What was NVDA's trailing twelve-month revenue as
of 30~June~2026?''), with the endpoint's value as ground truth; FinanceBench
\citep{islam2023financebench} supplies a complementary set over public filings.
(b)~\emph{Computation tasks}: 100 requests for standard quantities---Black--Scholes prices and
Greeks, GARCH(1,1) parameters, historical and parametric VaR, Monte Carlo terminal-value
quantiles---on frozen price histories, with ground truth from independent reference
implementations and seeded simulation; the quantitative-reasoning tasks of FinBen
\citep{xie2024finben} can be added. (c)~\emph{Report tasks}: 50 comprehensive requests graded by
two human analysts on a fixed rubric (coverage, correctness of every stated number against the
tool results, soundness of narrative, absence of unsupported claims), with an LLM judge
\citep{zheng2023judge} calibrated against the human grades on a held-out subset before being used
at scale.

\paragraph{Metrics.} For (a) and (b): exact-match or tolerance-based accuracy; the
\emph{unsupported-number rate}, defined as the share of numbers in the response that do not appear
in any tool result or executed-code output, which operationalises numerical hallucination; and
tool-selection precision and recall against a minimal correct tool set. For (c): rubric scores,
inter-rater agreement, and the unsupported-claim rate. For all families: end-to-end latency
(median and 95th percentile), time to first token, number of model round trips, input and output
tokens (reasoning tokens reported separately), dollar cost from per-model list prices, and the
sandbox failure rate with its causes.

\paragraph{Ablations.} Each mechanism is removed or varied while the rest is held fixed:
(i)~tools disabled (the model answers from parameters), which measures what grounding adds;
(ii)~adaptive thinking disabled; (iii)~the single-call instruction removed, so the model is free
to decompose, which measures the round trips and the failure rate the convention trades off;
(iv)~the context procedure varied over thresholds of $5\times10^{4}$, $10^{5}$, and $2\times10^{5}$
tokens and recency of 3, 5, and 10 messages on 20-turn sessions, scoring the retention of
analytically relevant facts planted early in the session; (v)~batch size varied over 1, 5, and
unbounded, measuring rate-limit errors and latency; and (vi)~model tier varied across the selector
of Table~\ref{tab:models}.

\paragraph{Baselines.} A plain model without tools; a ReAct-style loop \citep{yao2023react} with
one tool per turn; a code-interpreter-only agent in the spirit of program-aided language models
\citep{gao2023pal} with the same sandbox but no structured endpoints; and, as an upper reference
on the computation tasks, the reference implementations themselves.

\paragraph{Controls.} All runs at a frozen data snapshot (recorded endpoint responses replayed
through a cache), three repetitions per task to estimate variance under temperature~1, and the
same prompts and constants as the audited commit unless the ablation varies them. Reported
outcomes would be accompanied by confidence intervals from the repetitions and by the full
transcripts.

The protocol would answer the questions the first version of this paper left as assertions:
whether grounding reduces unsupported numbers to near zero, by how much the single-call
convention reduces round trips and at what cost in failures, how much the context procedure
loses, and what the architecture costs relative to simpler baselines.