1# Identification and pre-analysis plan — WP 2723Version 1.0, 2026-10-27, written before any live API call. Everything in §3–§6 is frozen after the pilot (§6.2); changes afterwards are logged as deviations in the paper.45## 1. Causal structure67### 1.1 Variables89- **Z — treatment (cell labels)**: `llm_share` φ ∈ {0, …, 1}; provider/model mix (`provider_mix`, `models`); `prompt_set` ∈ {single, diverse}; `memory` ∈ {0, 5}; `channel` ∈ {off, on}; leverage (share of leveraged slots, `leverage_cap`); safeguards (`halt_threshold`, `position_limit`); fairness regime (`decision_cap`, `usd_cap`). A cell is a vector of these.10- **U — exogenous randomness, by named seed stream** (`utils/rng.py`): `fundamentals` (dividend path, jumps), `news`, `slots` (wealth multipliers, latencies, informed draws), `assign` (classical policy of each slot), `llm_assign` (which eligible slots become LLM, provider allocation, decision offsets), `activation`, `tiebreak`, and per-agent generators `agent_rng(seed, id)` (classical policy randomness, private-signal noise).11- **ε — provider noise**: the LLM's sampling randomness, not seeded and not controllable (gpt-5 family rejects `temperature`; no seed parameter). It is part of the treatment.12- **M — within-run mediators** (`outcomes.py`): order-flow correlation ρ̄ = `llm_pairwise_corr`, synchrony = `llm_synchrony`, aggressiveness β̂ = `llm_market_share` (share of market orders) and mean order size, validity = `llm_valid_rate`, realized dose = `llm_share_orders`, `llm_share_volume`.13- **Y — run-level outcomes** (§3).1415### 1.2 DAG1617```18 U_fund, U_news ───────────────────────────────────────────┐19 U_slots, U_assign, U_agents ───────────────┐ │20 │ │21 cell label ─────► Z ───┐ ▼ ▼22 ├──► LLM slot set & policies ──► M (ρ̄, synchrony, β̂, validity, realized dose) ──► Y23 U_llm_assign ──────────┘ ▲ ▲ ▲24 │ │ │25 ε_provider ─────────────────────────┴─────────────────────┘ │26 U_activation, U_tiebreak ───────────────────────────────────────────────────────────────────────────────┘27 Z ──────────────────────────────────────────────────────────────────────────────────────────────────────┘ (direct: fewer heuristic agents, halts, limits)28```2930Z → Y is identified because Z is set by the design file and the only random input that depends on Z (`llm_assign`) is independent of every other stream by construction (`SeedSequence.spawn`). Fundamentals, news, endowments, latencies, information draws and policy assignment are *identical* across cells sharing a seed; the treatment replaces the policy in a stratified subset of slots and nothing else. Slot replacement is stratified by classical policy type (proportional quotas, market makers never replaced).3132Two leaks of common random numbers exist in the current code and must be closed before the pilot (gate 4 in `research_plan.md`): (i) the per-tick shuffle of active classical agents consumes the `activation` stream, whose length depends on the number of classical agents, so activation draws differ across arms after tick 0; (ii) an LLM slot draws its informed status from `llm.informed_share` (30 %) instead of inheriting it from the replaced slot, so the treatment arm holds more private information than the control (in a mock run, 44 % vs 6 % informed). Neither leak biases the ATE (Z is still randomized), but (ii) changes the *estimand* to "LLM policy + extra information" and (i) reduces the precision gain from pairing.3334### 1.3 Confounders ruled out by design3536| Potential confounder | Why it cannot confound Z → Y |37|---|---|38| Fundamental path, shocks, news | Same `fundamentals`/`news` streams within a seed; shocks at fixed ticks in every cell |39| Endowments, wealth dispersion, latencies | Same `slots` stream; LLM slots keep the replaced slot's cash, shares and latency |40| Composition of classical types | `assign` stream fixed; replacement stratified by type |41| Private information | After the gate-4 fix, inherited from the replaced slot |42| Market-maker behaviour | Market makers never replaced (`llm_keep_market_makers`) |43| Code/version drift | `config_hash`, `git_revision`, `prompt_version`, price table recorded in every manifest |4445### 1.4 What is *not* identified4647- **Mediation.** M is generated inside the run jointly with Y by the same path of U and ε. Sequential ignorability (M ⫫ Y(z, m) | Z, covariates) does not hold: e.g. a volatile fundamental path raises both ρ̄ and `msd_log`. We therefore report mediation only as a *descriptive decomposition* (§4.6) and treat each M as an **outcome** of Z (that is H2 and is identified).48- **Effects of the model's reasoning per se.** Z bundles the model, the prompt, the decision frequency and the invalid-decision rule. The estimand is intention-to-treat: "deploy this agent as configured".4950## 2. Estimands5152Unit = run (seed s × cell z). Y_s(z) is the potential outcome of seed s under cell z.5354| Label | Estimand | Contrast |55|---|---|---|56| E1 | ATE of cell z vs control: τ_z = E_s[Y_s(z) − Y_s(0)] | Each treated cell vs `share_0` of the same seed |57| E2 | Dose–response: τ(φ) for φ ∈ {0.01, 0.05, 0.10, 0.25, 0.50, 0.75, 1.00} with gpt-5.4-nano, single prompt; linear slope ∂τ/∂φ and ordered-trend test | Levels vs 0 and across levels |58| E3 | Diversity at equal share φ: (a) τ_div = E[Y(diverse, φ) − Y(single, φ)]; (b) τ_mix = E[Y(mix, φ) − ½(Y(nano, φ) + Y(mini, φ))] | (a) prompt diversity holding the model fixed; (b) model diversity net of composition (requires the mini-only cell) |59| E4 | Mechanism arms at φ = 0.25: memory 5 vs 0; channel on vs off; leverage high vs base; breaker vs none; position limit vs none | Each vs `share_25` |60| E5 | φ × leverage interaction: [Y(φ, lev_hi) − Y(0, lev_hi)] − [Y(φ, lev_base) − Y(0, lev_base)] | Difference of paired differences |61| E6 | Local projection of the mid–value gap around the t = 300 fundamental jump: β_h = E[g_s(z, h) − g_s(0, h)], g = log P − log V, h ∈ [−20, 100]; placebo at t = 450 | Treated vs control path |62| E7 | Realized-dose-scaled effect: τ_z / E[`llm_share_orders`] (descriptive, not causal per unit of order flow) | Reported alongside E1 |6364Primary confirmatory contrast: **E1 at φ = 0.25** (gpt-5.4-nano, single prompt, confirmatory fundamentals); secondary confirmatory: E1 at φ = 0.50 and E3(a). Everything else is exploratory unless the tiered budget allows the pre-specified seed count (`protocol.md` §5).6566## 3. Outcomes6768All variables are produced by `metrics/outcomes.py` on ticks t ≥ `warmup` = 100 (baseline.yaml), one row per run in `outcomes.parquet`. Multi-asset variants carry the suffix `_a1` for asset 1.6970### 3.1 Primary families and directional hypotheses7172| Family | Variable | Definition (short) | Direction under H | Hypothesis |73|---|---|---|---|---|74| Efficiency | `msd_log` **(lead H1, H4)** | mean (log P − log V)² | ↑ in φ; ↓ with diversity | H1, H4 |75| | `mad_pct` | mean 100·|log P − log V| | ↑ in φ | H1 |76| | `time_to_absorb` | first h after t = 300 with |P/V − 1| < 1 % for 5 ticks (censored at 100) | ↑ in φ; ↑ with safeguards | H1, H6 |77| | `response_ratio` | (P_{t+20} − P_{t−1}) / (V_{t+20} − V_{t−1}) | |RR − 1| ↑ in φ (two-sided on RR) | H1 |78| | `vr5` | VR(5) of log mid | |VR − 1| ↑ in φ (two-sided on VR) | H1 |79| Liquidity | `spread_bp` **(lead H1b)** | mean quoted spread / mid, bp | ↑ in φ | H1b |80| | `eff_spread_bp` | 2·|trade − pre-trade mid| / mid, bp | ↑ in φ | H1b |81| | `depth5` | mean lots within 5 levels, both sides | ↓ in φ | H1b |82| | `lambda_bp_100` | OLS of Δlog mid (bp) on signed volume / 100 | two-sided (theory: λ = Cov/Var falls as flow gets noisier) | H1b |83| | `resiliency` | half-life (ticks) of log-spread deviations | ↑ in φ | H1b |84| Instability | `n_crashes` **(lead H3)** | drops ≥ 10 % from the trailing 20-tick max | ↑ in φ × leverage | H3 |85| | `max_dd` **(lead H6)** | max drawdown of mid | ↑ in φ; ↓ with safeguards | H3, H6 |86| | `n_bubbles`, `bubble_amp` | episodes with |P/V − 1| > 5 % for ≥ 10 ticks; max amplitude | ↑ in φ × leverage | H3 |87| | `acf_absr_1_10` | Σ ACF(|r|) lags 1–10 | ↑ in φ | H3 |88| | `kurtosis` | excess kurtosis of tick returns | ↑ in φ | H3 |89| Coordination | `llm_pairwise_corr` **(lead H2)** | mean pairwise corr. of agents' net order sign per 20-tick window | ↑ in φ; ↓ with diversity | H2, H4 |90| | `llm_synchrony` **(lead H5)** | mean |mean sign| per decision round | ↑ with channel | H2, H5 |91| Contagion (K = 2) | `ret_corr`, `excess_comovement` **(lead)**, `spillover_dy` | return correlation; minus fundamental correlation; Diebold–Yilmaz total spillover (VAR(2), h = 10) | ↑ in φ | H2/H3 extension |9293Thresholds are fixed: bubble 5 % for ≥ 10 ticks; crash 10 % within 20 ticks. **Sensitivity grid** (reported for every instability result, no re-selection): bubble ∈ {3, 5, 8} %, crash ∈ {7, 10, 15} %. In the classical control (10 seeds) `n_crashes` = 0 at 10 %: a floor effect is expected and the 7 % variant is the pre-specified fallback for H3 (§`research_plan.md` §4).9495Transformations fixed in advance: `msd_log` is analysed on log10 scale for t-based inference (strongly right-skewed: control mean 0.0020, SD 0.0017); `time_to_absorb` is censored at 100 and analysed with rank-based inference; counts (`n_bubbles`, `n_crashes`, `n_halts`) by mean difference with randomization inference.9697### 3.2 Secondary outcomes9899`vol_bp`, `bias_pct`, `vr10`, `acf_r_1`, `bubble_dur`, `volume`, `turnover`, `cancel_ratio`, `fill_ratio`, `reject_ratio`, `n_halts`, `share_ticks_halted`, `pnl_<policy>` (incl. `pnl_llm`), `mm_pnl`, `mm_inventory_abs`, `gini_final`, `n_bankrupt`, `leadlag_01`, `fund_corr`, and the LLM diagnostics `llm_valid_rate`, `llm_hold_rate`, `llm_market_share`, `llm_buy_share`, `llm_usd`, `llm_tokens`, `llm_latency_s`, `llm_cached_hits`, plus the manifest fields `llm_share_orders`, `llm_share_volume`, `n_llm_decisions`.100101### 3.3 Known measurement caveats102103- `llm_synchrony` groups decisions by tick; with staggered offsets only agents sharing an offset are in the same round (≈ n_llm/20 per tick), so at φ ≤ 0.05 it rests on very few rounds. `llm_pairwise_corr` (20-tick windows) is the preferred coordination measure at low φ.104- Both coordination measures exist only for LLM agents, so they are undefined in the control; H2 is tested as a dose–response among φ > 0 plus a comparison with a classical analogue (§3.4).105- In multi-asset runs the prompt covers asset 0 only (`user_message(obs, 0)`): LLM agents never trade asset 1, so contagion is measured from the LLM-traded asset to a classical-only asset.106107### 3.4 Analysis-stage additions (to be coded in `analysis/`, not in `outcomes.py`)108109Classical analogue of ρ̄: the same `pairwise_sign_corr` applied to classical agents' aggressor-signed trades from `trades.parquet`, by policy type. Local-projection paths (E6) from `ticks.parquet`.110111## 4. Estimators112113### 4.1 Paired differences (primary)114115For each cell z and seed s: Δ_s = Y_s(z) − Y_s(0). Report mean Δ̄, SD, paired-t 95 % CI (df = n − 1) and the effect size d_z = Δ̄ / SD(Δ). Also Glass's Δ = Δ̄ / SD(Y(0)) using 30 free control seeds, for comparability across cells.116117### 4.2 Randomization inference (primary p-value)118119Under the sharp null Y_s(z) = Y_s(0), the sign of each Δ_s is exchangeable within its pair. P-value = share of the 2ⁿ sign-flip assignments with |mean Δ| ≥ |observed| (exact enumeration for n ≤ 14; 10⁵ Monte Carlo draws otherwise). This is distribution-free and valid for counts, censored and skewed outcomes at small n. The paired t (§4.1) and Wilcoxon signed-rank are reported as secondary.120121### 4.3 OLS with seed fixed effects (dose–response and multi-cell arms)122123Y_{s,z} = α_s + Σ_k τ_k 1[z = k] + ε_{s,z}. With ≤ 12 clusters, cluster-robust SE by seed are unreliable; inference uses the wild cluster bootstrap (Rademacher weights, 9,999 draws, cluster = seed) and, for ordered φ, **Page's L trend test** for randomized blocks (block = seed). The linear slope ∂τ/∂φ is reported with a bootstrap CI; the dose–response figure shows cell means with paired CIs.124125### 4.4 Diversity contrasts126127E3(a): paired difference diverse − single within seed. E3(b): τ_mix,s = Y_s(mix) − ½[Y_s(nano) + Y_s(mini)], then §4.1–4.2.128129### 4.5 Multiplicity, CIs, effect sizes130131Holm–Bonferroni within each hypothesis family of §3.1 at family-wise α = 0.05 (the lead outcome is reported adjusted and unadjusted). Two-sided tests; the sign of the estimate is compared with the predicted direction. 95 % CIs everywhere; no result is reported as a bare p-value. Secondary outcomes are unadjusted and labelled exploratory.132133### 4.6 Mediation (descriptive only)134135Within treated cells, regress Y on M with seed FE and cell dummies; report the coefficient and the share of τ_z "associated" with M (product-of-coefficients with the Z → M regression). Stated assumption: this is a decomposition under *no unmeasured mediator–outcome confounding*, which §1.4 argues is false; the numbers describe co-movement, not causal paths.136137### 4.7 Local projections (E6)138139For h ∈ [−20, 100]: g_s(z, h) = log P_{300+h} − log V_{300+h}. β_h = mean over seeds of g_s(z, h) − g_s(0, h), with pointwise paired-t bands and a sup-t band from the bootstrap. The same at the placebo tick 450 must show β_h ≈ 0 (falsification). Summary scalars: `time_to_absorb`, `response_ratio`.140141## 5. Power and seed counts142143Paired design, standardized paired effect d = Δ̄ / SD(Δ), two-sided α = 0.05, power 0.80:144145n ≈ (z_{0.975} + z_{0.80})² / d² = (1.960 + 0.842)² / d² = 7.85 / d²146147| d | n (pairs) |148|---|---|149| 0.5 | 32 |150| 0.8 | 13 |151| 1.0 | 8 |152| MDE at n = 10 | 0.89 |153| MDE at n = 8 | 0.99 |154155σ_d (the SD of paired differences) is **TBD from the pilot** (3 pairs per share; a poor estimate — the 30-seed classical control SD is used as a floor: e.g. SD(`msd_log`) = 0.0017, SD(`spread_bp`) = 6.4, SD(`max_dd`) = 0.042 in the default calibration cell, n = 10). Rule for the confirmatory seed count of the primary contrast: n_conf = max(8, ⌈7.85 σ̂_d² / Δ_min²⌉) with Δ_min = 0.5 σ̂_d (i.e. 32), **capped by the tier budget** (`protocol.md` §5: 10 seeds at 25 % and 50 % ≈ 10 USD). The achieved MDE is reported in the paper; detecting d = 0.5 on the primary contrast would require ≈ 32 USD for that contrast alone and is only done if extra budget is approved before unblinding. Classical-only cells cost nothing and use 30 seeds.156157## 6. Stopping rules, freeze, negative results158159### 6.1 Stopping rules160161- Pilot stops at 5 USD (hard `usd_cap`), or if provider failure rate > 10 %, or validity < 80 %; the prompt is then revised once (v2), re-dry-run with the mock, re-piloted on one seed.162- Confirmatory: tiers run in the fixed order of `protocol.md` §5; no interim efficacy looks; spend reconciled after each tier against the plan; a tier is complete only when every cell has the same seeds (unbalanced seeds are dropped from all cells of the tier). The 40 USD cap is enforced by per-run `usd_cap` and by the running total.163164### 6.2 Frozen after the pilot165166Prompt v1 text (`prompting.py` SHA) and schema; `configs/baseline.yaml` plus the confirmatory overrides (fundamentals with latent uncertainty, shock size) and their `config_hash`; cell list and seed lists (pilot seeds 1–3, confirmatory seeds from 101; pilot runs are *not* pooled into confirmatory estimates); outcome definitions (`outcomes.py` SHA), thresholds and sensitivity grid; estimators of §4; tier budgets; model snapshots and price table; the git revision. A freeze record is written to `results/manifests/freeze.json`.167168### 6.3 What counts as a negative result169170For a lead outcome with predicted sign: (i) *refuting*: CI of d excludes d ≥ 0.5 in the predicted direction; (ii) *no large effect*: CI contains 0 and excludes |d| ≥ MDE; (iii) *inconclusive*: CI contains both 0 and the MDE — reported as such, never as "no effect". Secondary outcomes cannot rescue a refuted lead.171172## 7. Threats to validity173174### 7.1 Internal (inside the simulator)175176| Threat | Consequence | Handling |177|---|---|---|178| Provider non-determinism (no seed; `temperature` unsupported on gpt-5) | Same seed ≠ same LLM decisions across repeated online runs | Every response cached with request id; replay from cache is exact; ε treated as part of the treatment variance; online-vs-replay flag `llm_cached_hits` reported |179| Cache replay across cells | Identical prompts (e.g. first decision at t = 50) hit the cache across cells — desirable CRN, but mixes online and replayed decisions | Report cache hit share per cell; cache key includes provider, model, prompt version and full prompt text |180| Invalid-decision handling (→ hold) | Validity differs by model; "hold" is a one-sided failure | ITT estimand; `llm_valid_rate` reported as outcome; cells with validity < 80 % flagged; no per-protocol rerun |181| Decision interval (20 ticks) vs classical activation (0.2–0.5 per tick) | Realized order share ≪ slot share (mock: 3.5 % of orders, 8 % of volume at φ = 0.25) | Report assigned and realized dose; E7; `decision_interval` = 10 sensitivity with the mock |182| Exact-revelation fundamentals in the baseline | V^pub ≡ V; private signals redundant; H1 unidentified | Confirmatory fundamentals: noisy dividends (`jump_prob` = 1, `jump_sigma` TBD ≈ 4 c → SD(V^pub/V − 1) ≈ 0.5 %) |183| Shock size (+8.4 % at 40 c) exceeds bubble threshold | Lagged adjustment counted as a bubble | Shock ≈ 10 c (+2.1 %) |184| Information-parity leak and activation-stream leak (§1.2) | Estimand drift; precision loss | Gate-4 fixes before the pilot |185| Budget halt mid-run (`halted_for_budget`) | Remaining decisions become hold | Per-run `usd_cap` set ≥ 2× expected spend; halted runs excluded and rerun |186| Model drift between pilot and confirmatory | Different behaviour over weeks | Confirmatory calls within a short window; model id and date in manifest |187| Frozen library (nearest-neighbour replay, k = 5) | Not an adaptive agent; distribution shift at N = 1,000 | Labelled provider `library`; used only for scale and cached-vs-online arms; hit distance diagnostics reported |188189### 7.2 External190191Simulated heuristic traders and prompted LLM agents are not institutions: no risk management, no regulation, no learning across runs (memory ≤ 5 own decisions), one asset at a time in the prompt, model time without calendar mapping, fees and latencies stylized. Effects are statements about *this* market mechanism and *these* model snapshots; magnitudes do not transfer, signs and dose–response shapes may.192