Research plan — UQO Working Paper 27
The Invisible Hand of Machines: Price Discovery in Markets Populated by LLM Agents
Methodology document (causal inference and experimental design). Version 1.0, 2026-10-27. Companion documents: identification.md (pre-analysis plan), protocol.md (run protocol and budget), assumptions.md (assumption register), model_cards.md. The simulator is src/market_agents; what it implements is summarised in DESIGN-BRIEF.md.
1. Primary contribution
A randomized, pre-registered simulation experiment that identifies the causal effect of replacing a share of human-like heuristic traders by LLM-driven traders on price discovery, liquidity and stability in a continuous double auction. The identification strategy is experimental: treatment (which trader slots become LLM-controlled, which model/prompt they use, which market safeguards apply) is assigned by dedicated seed streams, while fundamentals, news, endowments and latencies are held fixed within matched seeds (common random numbers). The stylized model in the brief (shared model error η_g, effective number of models M_eff = 1/Σφ_g²) supplies the directional hypotheses; the simulator tests them.
What is new relative to the LLM-agent-market literature: (i) explicit control arms and matched pairs instead of single demonstrations; (ii) dose–response in the LLM share; (iii) diversity as a treatment (same model vs diverse prompts vs model mix) at a fixed share; (iv) safeguards as treatments; (v) a budget-explicit, power-explicit protocol with pre-specified stopping and negative-result rules.
2. Research questions and hypotheses
| RQ | Question | Hypothesis (brief) | Lead outcome (outcomes.py) |
|---|---|---|---|
| RQ1 | Does price informativeness fall as the share φ of traders sharing one model rises, holding information quality fixed? | H1: msd_log increases in φ; falls in M_eff |
msd_log |
| RQ2 | Do LLM traders produce correlated, synchronised order flow? | H2: ρ̄ and synchrony rise in φ | llm_pairwise_corr |
| RQ3 | Does the φ × leverage interaction raise bubble/crash frequency and liquidity spirals? | H3: n_crashes, max_dd, n_bubbles rise with φ × leverage |
n_crashes |
| RQ4 | Does prompt or model diversity restore part of the loss? | H4: diverse < single on msd_log at equal φ |
msd_log (diverse − single) |
| RQ5 | Does a communication channel raise synchrony? | H5: channel on > off on synchrony | llm_synchrony |
| RQ6 | Do circuit breakers and position limits cut crash depth at a cost in absorption time? | H6: max_dd falls, time_to_absorb rises |
max_dd |
Liquidity (spread, depth, Kyle λ, resiliency) is treated as a sub-family H1b of RQ1 with its own lead outcome spread_bp. Full outcome families, directions, thresholds and estimators: identification.md §3–4.
3. Phases, gates and status
| Phase | Content | Gate (must hold to proceed) | Status 2026-10-27 |
|---|---|---|---|
| 0 | Scoping: brief, economy, treatments, outcome list | Brief agreed by author, theory and methods | Done (DESIGN-BRIEF.md) |
| 1 | Engine: event-sourced LOB, accounts, fees, latency, margin, halts, dividends, Kalman public value, news | Unit + property tests pass; cash/share conservation checked after every fill | Done — 23 tests pass (tests/unit, tests/property, tests/integration) |
| 2 | Classical population (fundamentalist, trend, mean-reverter, noise, market maker, leveraged, stat-arb), stratified slot assignment, seed streams | Determinism across identical runs; matched-pair invariance of fundamentals and endowments (test_common_random_numbers_across_llm_share) |
Done |
| 3 | Calibration and scale | Stylized facts reproduced (fat tails, volatility clustering, spread/depth in plausible range); run time ≤ 1 min at N = 10,000 | Done — 360-run sweep (results/calibration/sweep.parquet): default cell kurtosis ≈ 27, Σ ACF(abs r) ≈ 3.1, spread ≈ 12 bp, mad ≈ 3.3 %; wall time 0.5 s / 2.4 s / 22 s at N = 100 / 1,000 / 10,000 |
| 4 | LLM layer (OpenAI, Anthropic, mock, library), cache, budget tracker, prompts v1, decision schema; pre-analysis plan (this folder) | Gate 4 (open): (a) information parity fix — an LLM slot must inherit the informed/uninformed status of the slot it replaces (population.py, l. 105; today 30 % of all LLM slots are informed vs 7.5 % of classical slots: in a mock run 44 % vs 6 %); (b) activation-stream fix — runner.py l. 204 shuffles the active list with the activation stream, so activation draws diverge across arms after tick 0 (96 % of non-replaced agents had a different order count across arms in a mock test); (c) confirmatory fundamentals with latent uncertainty — in the baseline the dividend reveals x_t exactly, so V^pub ≡ V and private signals are redundant (H1 not identified); (d) shock size — size: 40.0 moves V by +8.4 %, not the +2 % stated in the YAML comment (a lagged response then counts as a 5 % "bubble"); use ≈ 10 c (+2.1 %); (e) first git commit so manifests carry a git_revision; (f) documents of this folder reviewed |
In progress: code exists, Anthropic key rejected (HTTP 401) → Claude cells mock-only |
| 5 | Mock dry-run of every cell (configs/exploratory.yaml), then live OpenAI pilot (configs/pilot_openai.yaml, ≤ 5 USD) |
Validity rate ≥ 80 %, failure rate ≤ 10 %, cost per decision within ±30 % of 0.0005 USD, σ of paired differences estimated | Configs written, not run |
| 6 | Freeze: prompt v1 SHA, schema, config hashes, thresholds, estimators, seed lists, budget tiers (§identification.md §6) |
Freeze record committed before any confirmatory call | Pending pilot |
| 7 | Confirmatory runs by tier (protocol.md §5), reconciliation of spend after each tier |
Cumulative live spend ≤ 40 USD; balanced seeds within tier | Not started |
| 8 | Analysis: paired estimates, randomization inference, dose–response, diversity contrasts, local projections, sensitivity grid, mediation (descriptive), robustness (mock, library at N = 1,000, call auction) | All pre-registered tables produced from results/runs/*/outcomes.parquet by scripted code; no hand edits |
Not started |
| 9 | Paper, replication package, model cards, data (decision logs without raw prompts) | Clean-room re-run of one tier from the frozen library reproduces tables | Literature review and theory in progress |
4. What would make us abandon or revise each hypothesis
Criteria are evaluated on the confirmatory tier for the primary contrast (25 % vs 0 %, gpt-5.4-nano, single prompt) unless stated. MDE = minimal detectable effect at the achieved seed count (≈ d = 0.9 at n = 10 pairs; see identification.md §5).
| H | Revise if | Abandon (report as negative) if |
|---|---|---|
| H1 (informativeness ↓ in φ) | msd_log effect is positive but driven entirely by the llm_valid_rate or by the realized order share being < 2 % (dose too small to matter) → re-interpret as "activity" effect; or effect disappears under latent-uncertainty fundamentals but not under exact-revelation fundamentals → it is a microstructure, not an information, effect |
Paired 95 % CI of Δmsd_log excludes d ≥ 0.5 in the predicted direction at both 25 % and 50 %, and the dose–response (Page's L) is flat |
| H2 (ρ̄, synchrony ↑ in φ) | ρ̄ among LLM agents is high but not higher than among classical agents of the replaced types (classical analogue computed from trades.parquet; see identification.md §3.4) → the correlation is a property of rule-like trading, not of model sharing |
ρ̄ does not rise with φ (Page's L, p > 0.05) and is below the classical analogue |
| H3 (φ × leverage → crashes) | Crash counts are zero at the 10 % threshold in all cells (floor effect) → use the pre-registered 7 % sensitivity threshold and max_dd as lead |
Interaction term CI excludes the predicted sign at 7, 10 and 15 % thresholds |
| H4 (diversity restores) | Diverse prompts help but model mix does not (or vice versa) → theory must distinguish persona-level from model-level error η_g | Neither contrast (diverse − single; mix − average of singles) has the predicted sign with CI excluding d ≥ 0.5 |
| H5 (channel ↑ synchrony) | Synchrony rises but msd_log does not → channel moves coordination without moving prices |
Channel effect on llm_synchrony CI excludes d ≥ 0.5 |
| H6 (safeguards) | max_dd falls but time_to_absorb does not rise → no trade-off (report as such) |
Breaker and position limit both leave max_dd and n_crashes unchanged (CI excludes d ≥ 0.5) |
Cross-cutting abandonment: if the pilot validity rate of gpt-5.4-nano is < 80 % after one prompt revision (v2), the treatment is "an agent that mostly holds" and the study is reframed around activity rather than reasoning; if the live spend cannot deliver ≥ 8 matched seeds on the primary contrast, the confirmatory claim is downgraded to "exploratory, powered for d ≥ 1 only".
5. Risks
| Risk | Mitigation |
|---|---|
| Anthropic key invalid (confirmed 2026-10-27) | Claude cells are implemented and mock-tested; they enter the design only when a key is validated; budget tiers do not depend on them |
| Budget: 40 USD ≈ 70,000–80,000 nano decisions; a full factorial is impossible | Tiered design with a pre-specified order of sacrifice (protocol.md §5.3) |
| Provider non-determinism and drift | Every response cached (SQLite) with request id; replay reproduces runs exactly; model snapshot and date recorded in manifests |
| Realized LLM dose far below slot share (mock: 3.5 % of orders at 25 % of slots) | Report both assigned and realized dose; decision_interval sensitivity with the mock; interpret effects per unit of realized order share |
| Pilot σ from 3 pairs is imprecise | Classical control σ from 30 free seeds as floor; conservative seed rule |