SPB Git

spb/wp27_uqo

Public
13.4 KB · 49 lines latex
1\section{Implementation and validation}2\label{sec:implementation}34\paragraph{Architecture.}5The simulator is a pure-Python package (\code{market\_agents}, about 3,900 lines, Python~3.12) organised in six layers: \code{assets/} (dividend process and the two value benchmarks), \code{signals/} (news feed and shock schedule), \code{exchange/} (typed order records, the limit order book, the integer ledger, the engine with latency queue, fees, risk checks and halts, and the call auction), \code{agents/} (the \code{Observation} interface and the classical policies), \code{llm/} (decision schema, prompt builder, policy with validation, budget tracker, SQLite cache, frozen library, and the provider adapters), and \code{experiments/} (pydantic configuration, population builder, single-run \code{Simulation}, and the design runner), with \code{metrics/outcomes.py} computing every run-level outcome from the saved frames and \code{analysis/} holding the estimators and plots. The exchange is \emph{event-sourced}: every state change (\code{submit}, \code{accept}, \code{reject}, \code{trade}, \code{cancel}, \code{expire}, \code{halt}, \code{resume}, \code{dividend}, \code{margin\_call}, \code{liquidate}, \code{bankrupt}, \code{auction}, \code{shock}, \code{tick}) is appended to a sequence-numbered log with a flat scalar payload, and the engine exposes a hashable \code{snapshot} of all balances, positions, resting orders and counters.\footnote{\code{exchange/orders.py}, \code{Event}; \code{exchange/engine.py}, \code{Exchange.snapshot}. The full event log is written to \code{events.parquet} when \code{output.events} is on; by default a run saves the per-tick, trade, agent and LLM-decision frames.} All randomness flows from one master seed through named \code{SeedSequence} children (fundamentals, news, slots, assignment, LLM assignment, activation, tie-break, mock LLM, within-tick shuffle) plus one generator per agent, so a run is a deterministic function of (configuration, seed) up to the provider's sampling; two executions of the same classical configuration produce identical event logs, tick frames and trade frames. Every run writes a \code{manifest.json} with the configuration hash (first 16 hex digits of the SHA-256 of the canonical JSON configuration), the package version, the git revision (suffixed \code{-dirty} when the working tree had uncommitted changes), the seed and stream names, the trader and realised LLM shares of traders, orders and volume, the number and validity rate of LLM decisions, token and USD usage by model, the provider and model identifiers, the prompt version, the full price table used by the budget tracker, the treatment labels, the shock log, the number of trades, halts and bankruptcies, and the wall-clock time.\footnote{\code{experiments/runner.py}, \code{Simulation.\_finish}; \code{utils/hashing.py}.}67\paragraph{Test suite.}8The package ships 24 tests in three groups.\footnote{\code{tests/unit}, \code{tests/property}, \code{tests/integration}; run with \code{make test}.} \emph{Unit tests} cover the book (price--time priority, limit orders never cross beyond their price, market orders walk the book and stop when it is empty, cancel and depth), the engine (a fill moves exactly the notional and fees, latency orders arrivals, each rejection reason, cancel releases the reservation, a market buy is capped by cash, dividends preserve conservation, a halt rejects new orders, a margin breach generates a liquidation), the LLM layer (lenient parsing versus strict schema, each validation rule, the budget cap halting calls) and the configuration override semantics. \emph{Property tests} use Hypothesis to generate random streams of up to 120 intents from six agents with partial activations, cancels, a maker rebate and a short time-to-live, and assert after every processed tick that the book is uncrossed and indexed consistently, that the ledger identities \eqref{eq:conservation} hold, that each account's reserved cash and lots equal the sums over its resting orders, and that no position is negative; a second property asserts that two executions of the same stream yield identical snapshots and identical event logs (determinism and replay). \emph{Integration tests} run the full simulation: an end-to-end classical run with 100 agents writing all artefacts, bit-for-bit determinism across runs, common random numbers across \code{llm\_share} (identical latent value paths and identical endowments and latencies of every non-replaced slot), a mock-LLM run whose realised shares and validity rate are within the expected bands, the \emph{zero-trade edge case} (a population of inactive market makers produces 30 ticks with a well-defined midpoint and no trades), and the logging and application of each shock kind.910\paragraph{Computational performance.}11Table~\ref{tab:scale} reports the wall-clock time of a 600-tick baseline run as the population grows from 100 to 10,000 traders with the per-type activity scaled by $100/N$ so that order flow per tick is constant.\footnote{\code{scripts/11\_scale\_benchmark.py}; numbers from \code{results/calibration/scale\_benchmark.json}, single process on an Apple-silicon (arm64) laptop, regenerated on the frozen code revision.} A run takes 0.6~s at $N=100$ (three seeds: 0.56, 0.6, 0.59~s), 2.6~s at $N=1{,}000$ and 23.7~s at $N=10{,}000$, i.e.\ roughly linear in $N$ at fixed order flow, with 12,770--14,679 orders and 9,483--11,506 trades per run. The 480-run classical confirmatory design (16 cells $\times$ 30 seeds) completes in about one minute on six worker processes. Live LLM cells are bound by provider latency: with 25 LLM agents in four staggered batches a run makes about 690 calls at 2.5--3.2~s each, so a $\varphi=0.25$ run takes 6--8 minutes with eight concurrent requests (Appendix~\ref{app:repro}).1213\begin{table}[t]14\centering15\caption{Computational performance of the simulator at constant order flow}16\label{tab:scale}17\small18\input{tables/tab_scale}19\begin{minipage}{0.95\linewidth}\footnotesize\vspace{2pt}20Baseline configuration, 600 ticks, activities scaled by $100/N$; generated by \code{scripts/50\_tables.py} from \code{results/calibration/scale\_benchmark.json}. Mean over seeds.21\end{minipage}22\end{table}2324\paragraph{Calibration.}25The classical baseline was calibrated once, before any LLM run, by a sweep over four behavioural parameters: fundamentalist aggressiveness $\{10,20,40\}$ lots per percent of mispricing, the combined trend-plus-leveraged share $\{0.10,0.15,0.25\}$ (split 60/40 between the two types and rebalanced against noise traders), the noise traders' market-order probability $\{0.3,0.6\}$ and the market makers' quote size $\{10,20\}$ lots---36 cells $\times$ 10 seeds $=360$ runs of 600 ticks.\footnote{\code{scripts/10\_calibration\_sweep.py}; cell means in \code{results/calibration/sweep\_summary.csv}.} The pre-stated criteria were those of the stylized-facts literature \citep{cont2001}: first-order return autocorrelation close to zero, excess kurtosis above~3, positive volatility clustering (positive sum of the first ten autocorrelations of absolute returns), a quoted spread of 10--30 basis points, and a tracking error $\overline{|P/V-1|}$ near 1\%. The retained cell (aggressiveness 20, trend-plus-leveraged share 0.10, market-order probability 0.6, quote size 20) has, under the final economy, ACF$_1=0.023$, variance ratio VR(5) $=1.23$, excess kurtosis 22.3, $\sum_{1}^{10}\mathrm{ACF}|r|=2.06$, tick volatility 18~bp, quoted spread 21.1~bp, depth 139 lots, price impact 11.1~bp per 100 lots, 0.0 bubble episodes and a maximum drawdown of 3.4\% on average over ten seeds. The cell was selected on the sweep run at the calibration stage; the sweep was regenerated on the frozen revision (after the dividend-timing revision) and the ranking is essentially unchanged: the aggressiveness-40 variant scores marginally better on the pre-stated criteria (tracking error 0.46\%, ACF$_1$ 0.042, kurtosis 11.8) and the retained cell is second; we kept the frozen cell rather than re-optimize after LLM runs had started. The alternatives with market-order probability 0.3 produce tracking errors of 0.8--1.9\% and ACF$_1$ of -0.10--0.23. The retained mix (25/6/10/50/5/4\%) and the three parameters are frozen in \code{configs/baseline.yaml} and were not revisited after the LLM pilot.2627\paragraph{Stylized facts at the frozen revision.}28Table~\ref{tab:stylized_facts} summarises the 30 control runs of the classical confirmatory design (seeds 101--130) at the frozen code revision. Returns of the midpoint are uncorrelated at lag one (mean ACF$_1=-0.007$) with a variance ratio near one (VR(5) $=1.18$), heavy-tailed (excess kurtosis 20.4) and volatility-clustered ($\sum_1^{10}\mathrm{ACF}|r|=2.02$); the quoted spread averages 20.7~bp and the effective spread 30.4~bp, depth within five levels is 139 lots, price impact is 10.8~bp per 100 lots, the tracking error is 0.63\% and no run contains a bubble episode under the pre-specified 5\%/10-tick rule. Market makers earn 4.9\% over the run and the wealth Gini coefficient stays at 0.285. These are the properties against which every treatment is compared.2930\begin{table}[t]31\centering32\caption{Stylized facts of the classical baseline (control cell, 30 seeds)}33\label{tab:stylized_facts}34\small35\input{tables/tab_stylized_facts}36\begin{minipage}{0.95\linewidth}\footnotesize\vspace{2pt}37Outcomes on ticks $100$--$599$ of the \code{control} cell of \code{configs/classical\_confirmatory.yaml}; generated by \code{scripts/50\_tables.py} from \code{results/runs/classical\_confirmatory/outcomes.parquet}.38\end{minipage}39\end{table}4041\paragraph{LLM adapters.}42Three provider adapters share one interface, \code{complete(system, user, model, \dots)}~$\to$~\code{ProviderResult} (text, input, cached-input, output and reasoning tokens, latency, model, request id, error).\footnote{\code{llm/providers/}.} The \emph{OpenAI} adapter calls the Responses API with the frozen system prompt as \code{instructions}, the snapshot as the single user input, \code{max\_output\_tokens} $=1{,}200$ (raised from 400 because reasoning tokens count against the cap), \code{text.format} $=$ \{\code{json\_schema}, name \code{trading\_decision}, \code{strict: true}\} with the schema of Appendix~\ref{app:prompts}, \code{store=False} so that no response is retained in the account's response store, \code{reasoning.effort} $=$ \code{low} for the gpt-5 and o-series models (temperature is never sent to them), and \code{prompt\_cache\_key} $=$ \code{ma-v1-<prompt\_id>} so that the provider can reuse the cached prefix across agents sharing a persona; the client is configured with a 60-second timeout and three retries, and rate-limit, status and connection errors are returned as error strings rather than raised. The \emph{Anthropic} adapter calls the Messages API with the system prompt marked \code{cache\_control: ephemeral}, \code{output\_config.format} $=$ \{\code{json\_schema}, schema\} and \code{output\_config.effort} $=$ \code{low}, no \code{thinking} parameter, \code{max\_tokens} $=\max(1{,}200, 256)$, and maps \code{stop\_reason = refusal} to an invalid decision; it is implemented and dry-run tested but, because the key available on 2026-10-27 is rejected (HTTP~401), no Claude decision enters the reported results. The \emph{mock} adapter parses the midpoint, public value, public estimate, private signal, holdings and free cash out of the user message and applies a seeded value rule (buy or sell when $|\hat V/P-1|>0.5\%$, 60\% limit and 40\% market orders) with a 5\% share of deliberately malformed answers (half non-JSON, half schema-invalid) and an optional common tilt (\code{herd\_bias}); its tokens are word counts priced at zero. It is used for dry runs, tests and the mock cells, and is not evidence about LLM behaviour.4344\paragraph{Budget and cost accounting.}45A \code{BudgetTracker} per run prices every call from the table of USD per million tokens stored in the configuration (input, cached input, output) and reproduced in the manifest; an unknown model id is priced at the most expensive known output price so that the cap is conservative. The cap is checked before each call; once reached, the policy is halted for the rest of the run and every subsequent decision becomes a logged hold with error \code{budget\_halt}. Cached hits are counted (\code{cached\_hits}) but contribute no tokens and no cost; failures and invalid decisions are counted separately. Optional caps on total tokens and on decision opportunities per agent (\code{decision\_cap}) implement the fairness designs. Before any live design, \code{ma budget} enumerates the calls of every cell $\times$ seed (number of LLM slots $\times\lfloor(T-\texttt{warmup}/2)/20\rfloor$) and prices them at a stated token profile.\footnote{\code{llm/budget.py}; \code{cli.py}, \code{budget}.}4647\paragraph{Decision log.}48Each decision appends one row to \code{llm\_decisions.parquet}: tick, agent id, provider, model, prompt id, validity flag and reason, the parsed action, side, order type, quantity, limit price, confidence and rationale, input, cached-input, output and reasoning tokens, latency in seconds, USD cost, whether it was a cache hit, the provider error string, the first 16 hex digits of the SHA-256 of the concatenated system and user prompts, the provider request id, the seven-dimensional feature vector of the observation used by the frozen library, and the limit price relative to the midpoint.\footnote{\code{llm/policy.py}, \code{LLMDecisionRecord}.} Raw prompts are not stored in the decision log (they are reconstructible from the saved run and are stored verbatim in the SQLite cache together with the raw response text and token usage).49
Files · d09ab0c71293588caf435dcaebf6e4bc03a129ff
Menu

Theme