# Identification and pre-analysis plan — WP 27

Version 1.0, 2026-10-27, written before any live API call. Everything in §3–§6 is frozen after the pilot (§6.2); changes afterwards are logged as deviations in the paper.

# 1. Causal structure

# 1.1 Variables

• Z — treatment (cell labels): llm_share φ ∈ {0, …, 1}; provider/model mix (provider_mix, models); prompt_set ∈ {single, diverse}; memory ∈ {0, 5}; channel ∈ {off, on}; leverage (share of leveraged slots, leverage_cap); safeguards (halt_threshold, position_limit); fairness regime (decision_cap, usd_cap). A cell is a vector of these.
• U — exogenous randomness, by named seed stream (utils/rng.py): fundamentals (dividend path, jumps), news, slots (wealth multipliers, latencies, informed draws), assign (classical policy of each slot), llm_assign (which eligible slots become LLM, provider allocation, decision offsets), activation, tiebreak, and per-agent generators agent_rng(seed, id) (classical policy randomness, private-signal noise).
• ε — provider noise: the LLM's sampling randomness, not seeded and not controllable (gpt-5 family rejects temperature; no seed parameter). It is part of the treatment.
• M — within-run mediators (outcomes.py): order-flow correlation ρ̄ = llm_pairwise_corr, synchrony = llm_synchrony, aggressiveness β̂ = llm_market_share (share of market orders) and mean order size, validity = llm_valid_rate, realized dose = llm_share_orders, llm_share_volume.
• Y — run-level outcomes (§3).

# 1.2 DAG

  U_fund, U_news ───────────────────────────────────────────┐
  U_slots, U_assign, U_agents ───────────────┐              │
                                             │              │
  cell label ─────► Z ───┐                   ▼              ▼
                         ├──► LLM slot set & policies ──► M (ρ̄, synchrony, β̂, validity, realized dose) ──► Y
  U_llm_assign ──────────┘            ▲                     ▲                                             ▲
                                      │                     │                                             │
  ε_provider ─────────────────────────┴─────────────────────┘                                             │
  U_activation, U_tiebreak ───────────────────────────────────────────────────────────────────────────────┘
  Z ──────────────────────────────────────────────────────────────────────────────────────────────────────┘ (direct: fewer heuristic agents, halts, limits)

Z → Y is identified because Z is set by the design file and the only random input that depends on Z (llm_assign) is independent of every other stream by construction (SeedSequence.spawn). Fundamentals, news, endowments, latencies, information draws and policy assignment are identical across cells sharing a seed; the treatment replaces the policy in a stratified subset of slots and nothing else. Slot replacement is stratified by classical policy type (proportional quotas, market makers never replaced).

Two leaks of common random numbers exist in the current code and must be closed before the pilot (gate 4 in research_plan.md): (i) the per-tick shuffle of active classical agents consumes the activation stream, whose length depends on the number of classical agents, so activation draws differ across arms after tick 0; (ii) an LLM slot draws its informed status from llm.informed_share (30 %) instead of inheriting it from the replaced slot, so the treatment arm holds more private information than the control (in a mock run, 44 % vs 6 % informed). Neither leak biases the ATE (Z is still randomized), but (ii) changes the estimand to "LLM policy + extra information" and (i) reduces the precision gain from pairing.

# 1.3 Confounders ruled out by design

# 1.4 What is not identified

• Mediation. M is generated inside the run jointly with Y by the same path of U and ε. Sequential ignorability (M ⫫ Y(z, m) | Z, covariates) does not hold: e.g. a volatile fundamental path raises both ρ̄ and msd_log. We therefore report mediation only as a descriptive decomposition (§4.6) and treat each M as an outcome of Z (that is H2 and is identified).
• Effects of the model's reasoning per se. Z bundles the model, the prompt, the decision frequency and the invalid-decision rule. The estimand is intention-to-treat: "deploy this agent as configured".

# 2. Estimands

Unit = run (seed s × cell z). Y_s(z) is the potential outcome of seed s under cell z.

Primary confirmatory contrast: E1 at φ = 0.25 (gpt-5.4-nano, single prompt, confirmatory fundamentals); secondary confirmatory: E1 at φ = 0.50 and E3(a). Everything else is exploratory unless the tiered budget allows the pre-specified seed count (protocol.md §5).

# 3. Outcomes

All variables are produced by metrics/outcomes.py on ticks t ≥ warmup = 100 (baseline.yaml), one row per run in outcomes.parquet. Multi-asset variants carry the suffix _a1 for asset 1.

# 3.1 Primary families and directional hypotheses

Thresholds are fixed: bubble 5 % for ≥ 10 ticks; crash 10 % within 20 ticks. Sensitivity grid (reported for every instability result, no re-selection): bubble ∈ {3, 5, 8} %, crash ∈ {7, 10, 15} %. In the classical control (10 seeds) n_crashes = 0 at 10 %: a floor effect is expected and the 7 % variant is the pre-specified fallback for H3 (§research_plan.md §4).

Transformations fixed in advance: msd_log is analysed on log10 scale for t-based inference (strongly right-skewed: control mean 0.0020, SD 0.0017); time_to_absorb is censored at 100 and analysed with rank-based inference; counts (n_bubbles, n_crashes, n_halts) by mean difference with randomization inference.

# 3.2 Secondary outcomes

vol_bp, bias_pct, vr10, acf_r_1, bubble_dur, volume, turnover, cancel_ratio, fill_ratio, reject_ratio, n_halts, share_ticks_halted, pnl_<policy> (incl. pnl_llm), mm_pnl, mm_inventory_abs, gini_final, n_bankrupt, leadlag_01, fund_corr, and the LLM diagnostics llm_valid_rate, llm_hold_rate, llm_market_share, llm_buy_share, llm_usd, llm_tokens, llm_latency_s, llm_cached_hits, plus the manifest fields llm_share_orders, llm_share_volume, n_llm_decisions.

# 3.3 Known measurement caveats

• llm_synchrony groups decisions by tick; with staggered offsets only agents sharing an offset are in the same round (≈ n_llm/20 per tick), so at φ ≤ 0.05 it rests on very few rounds. llm_pairwise_corr (20-tick windows) is the preferred coordination measure at low φ.
• Both coordination measures exist only for LLM agents, so they are undefined in the control; H2 is tested as a dose–response among φ > 0 plus a comparison with a classical analogue (§3.4).
• In multi-asset runs the prompt covers asset 0 only (user_message(obs, 0)): LLM agents never trade asset 1, so contagion is measured from the LLM-traded asset to a classical-only asset.

# 3.4 Analysis-stage additions (to be coded in analysis/, not in outcomes.py)

Classical analogue of ρ̄: the same pairwise_sign_corr applied to classical agents' aggressor-signed trades from trades.parquet, by policy type. Local-projection paths (E6) from ticks.parquet.

# 4. Estimators

# 4.1 Paired differences (primary)

For each cell z and seed s: Δ_s = Y_s(z) − Y_s(0). Report mean Δ̄, SD, paired-t 95 % CI (df = n − 1) and the effect size d_z = Δ̄ / SD(Δ). Also Glass's Δ = Δ̄ / SD(Y(0)) using 30 free control seeds, for comparability across cells.

# 4.2 Randomization inference (primary p-value)

Under the sharp null Y_s(z) = Y_s(0), the sign of each Δ_s is exchangeable within its pair. P-value = share of the 2ⁿ sign-flip assignments with |mean Δ| ≥ |observed| (exact enumeration for n ≤ 14; 10⁵ Monte Carlo draws otherwise). This is distribution-free and valid for counts, censored and skewed outcomes at small n. The paired t (§4.1) and Wilcoxon signed-rank are reported as secondary.

# 4.3 OLS with seed fixed effects (dose–response and multi-cell arms)

Y_{s,z} = α_s + Σ_k τ_k 1[z = k] + ε_{s,z}. With ≤ 12 clusters, cluster-robust SE by seed are unreliable; inference uses the wild cluster bootstrap (Rademacher weights, 9,999 draws, cluster = seed) and, for ordered φ, Page's L trend test for randomized blocks (block = seed). The linear slope ∂τ/∂φ is reported with a bootstrap CI; the dose–response figure shows cell means with paired CIs.

# 4.4 Diversity contrasts

E3(a): paired difference diverse − single within seed. E3(b): τ_mix,s = Y_s(mix) − ½[Y_s(nano) + Y_s(mini)], then §4.1–4.2.

# 4.5 Multiplicity, CIs, effect sizes

Holm–Bonferroni within each hypothesis family of §3.1 at family-wise α = 0.05 (the lead outcome is reported adjusted and unadjusted). Two-sided tests; the sign of the estimate is compared with the predicted direction. 95 % CIs everywhere; no result is reported as a bare p-value. Secondary outcomes are unadjusted and labelled exploratory.

# 4.6 Mediation (descriptive only)

Within treated cells, regress Y on M with seed FE and cell dummies; report the coefficient and the share of τ_z "associated" with M (product-of-coefficients with the Z → M regression). Stated assumption: this is a decomposition under no unmeasured mediator–outcome confounding, which §1.4 argues is false; the numbers describe co-movement, not causal paths.

# 4.7 Local projections (E6)

For h ∈ [−20, 100]: g_s(z, h) = log P_{300+h} − log V_{300+h}. β_h = mean over seeds of g_s(z, h) − g_s(0, h), with pointwise paired-t bands and a sup-t band from the bootstrap. The same at the placebo tick 450 must show β_h ≈ 0 (falsification). Summary scalars: time_to_absorb, response_ratio.

# 5. Power and seed counts

Paired design, standardized paired effect d = Δ̄ / SD(Δ), two-sided α = 0.05, power 0.80:

n ≈ (z_{0.975} + z_{0.80})² / d² = (1.960 + 0.842)² / d² = 7.85 / d²

σ_d (the SD of paired differences) is TBD from the pilot (3 pairs per share; a poor estimate — the 30-seed classical control SD is used as a floor: e.g. SD(msd_log) = 0.0017, SD(spread_bp) = 6.4, SD(max_dd) = 0.042 in the default calibration cell, n = 10). Rule for the confirmatory seed count of the primary contrast: n_conf = max(8, ⌈7.85 σ̂_d² / Δ_min²⌉) with Δ_min = 0.5 σ̂_d (i.e. 32), capped by the tier budget (protocol.md §5: 10 seeds at 25 % and 50 % ≈ 10 USD). The achieved MDE is reported in the paper; detecting d = 0.5 on the primary contrast would require ≈ 32 USD for that contrast alone and is only done if extra budget is approved before unblinding. Classical-only cells cost nothing and use 30 seeds.

# 6. Stopping rules, freeze, negative results

# 6.1 Stopping rules

• Pilot stops at 5 USD (hard usd_cap), or if provider failure rate > 10 %, or validity < 80 %; the prompt is then revised once (v2), re-dry-run with the mock, re-piloted on one seed.
• Confirmatory: tiers run in the fixed order of protocol.md §5; no interim efficacy looks; spend reconciled after each tier against the plan; a tier is complete only when every cell has the same seeds (unbalanced seeds are dropped from all cells of the tier). The 40 USD cap is enforced by per-run usd_cap and by the running total.

# 6.2 Frozen after the pilot

Prompt v1 text (prompting.py SHA) and schema; configs/baseline.yaml plus the confirmatory overrides (fundamentals with latent uncertainty, shock size) and their config_hash; cell list and seed lists (pilot seeds 1–3, confirmatory seeds from 101; pilot runs are not pooled into confirmatory estimates); outcome definitions (outcomes.py SHA), thresholds and sensitivity grid; estimators of §4; tier budgets; model snapshots and price table; the git revision. A freeze record is written to results/manifests/freeze.json.

# 6.3 What counts as a negative result

For a lead outcome with predicted sign: (i) refuting: CI of d excludes d ≥ 0.5 in the predicted direction; (ii) no large effect: CI contains 0 and excludes |d| ≥ MDE; (iii) inconclusive: CI contains both 0 and the MDE — reported as such, never as "no effect". Secondary outcomes cannot rescue a refuted lead.

# 7. Threats to validity

# 7.1 Internal (inside the simulator)

# 7.2 External

Simulated heuristic traders and prompted LLM agents are not institutions: no risk management, no regulation, no learning across runs (memory ≤ 5 own decisions), one asset at a time in the prompt, model time without calendar mapping, fees and latencies stylized. Effects are statements about this market mechanism and these model snapshots; magnitudes do not transfer, signs and dose–response shapes may.