Run protocol — WP 27
Version 1.0, 2026-10-27. Operational companion to identification.md. Every number marked TBD is filled by the pilot and recorded in the freeze file.
1. Environment
- Project root:
~/Desktop/Recherche-UQO/finance/invisible-hand-of-machines; Python ≥ 3.12 in.venv(uv lockuv.lock); packagemarket_agentswith entry pointma(.venv/bin/ma, equivalently.venv/bin/python -m market_agents.cli). - Keys in
.envonly (OPENAI_API_KEY,ANTHROPIC_API_KEY), loaded bymaviapython-dotenv; never in configs or manifests. Status 2026-10-27: OpenAI key live; Anthropic key rejected (HTTP 401 invalid x-api-key) — Claude cells run with the mock provider only until a valid key is installed. - OpenAI calls use the Responses API with
store=False,text.format = json_schema (strict),reasoning.effort = low,prompt_cache_keyper prompt id. Anthropic calls (when enabled) use the Messages API withoutput_config.format = json_schema,effort = low, system prompt marked for prompt caching. - Hard limits: project-wide live spend ≤ 40 USD; pilot ≤ 5 USD; per-run
population.llm.usd_capset to ≥ 2× the expected run cost so that no confirmatory run halts for budget. - Results:
results/runs/<design>/<cell>/seed<NNNN>/(config.yaml, manifest.json, ticks/trades/agents/llm_decisions parquet, outcomes.json) andresults/runs/<design>/outcomes.parquet. Caches:results/cache/<design>.sqlite(one per design; the cache is also the recorded decision library).
2. Commands
.venv/bin/python -m pytest tests # 24 tests (unit, property, integration) must pass
.venv/bin/ma run configs/baseline.yaml --seed 1 --out results/runs/baseline/seed0001 # one classical run (~0.5 s)
.venv/bin/ma budget configs/<design>.yaml --tokens-in 760 --tokens-out 230 # calls and USD before any live design
.venv/bin/ma design configs/<design>.yaml --workers 8 --cache results/cache/<design>.sqlite
.venv/bin/ma design configs/<design>.yaml --cells share_25 --max-seeds 1 --cache ... # single cell / single seedma design runs every cell in a process pool (--workers; the Makefile uses 3 for the pilot and 5 for the confirmatory design) with one shared SQLite decision cache (results/cache/llm_cache.sqlite). An existing run directory is reused only if its manifest config_hash equals the requested configuration's hash (otherwise it is deleted and re-run), so an interrupted design resumes without re-calling the API and a changed configuration can never pick up stale runs. Before relaunching after a code change, delete the design directory and make sure no pool worker is still alive (pkill -f multiprocessing.spawn).
3. Phase 5a — mock dry-run (no cost)
- Apply the gate-4 fixes listed in
research_plan.md§3 (information parity, activation stream, confirmatory fundamentals, shock size, first commit). Re-run the test suite. ma design configs/exploratory.yaml --workers 8— all 12 cells × seeds 1–3 with the mock provider (usd_cap: 0). Check: every cell completes;llm_share_tradersequals the slot share (minus market makers at 100 %);llm_valid_rate≈ 0.95 (the mock injects 5 % invalid answers);n_llm_decisions= 27–28 per LLM agent;n_halts> 0 only inbreaker_25;reject_ratiorises inposlimit_25;ret_corrandspillover_dydefined ink2_25.- Dry-run the Anthropic cells with the mock provider mapped to the Claude model ids (
provider_mix: {mock: 1.0},models: {mock: claude-haiku-5-5}) to exercise the price table and manifests; labeltreatment.provider: anthropic-mock. - Free σ estimates: run
share_0for seeds 1–30 (classical, ≈ 15 s) to obtain the control SD of every outcome.
4. Phase 5b — live pilot (OpenAI, ≤ 5 USD)
| Cell | llm_share |
Model / prompt | Seeds | Decisions per run | Planned USD (0.0005/decision) |
|---|---|---|---|---|---|
| share_0 | 0 | — (control, free) | 1–3 | 0 | 0 |
| share_5 | 0.05 | gpt-5.4-nano, single | 1–3 | 135 | 0.20 |
| share_25 | 0.25 | gpt-5.4-nano, single | 1–3 | 675 | 1.01 |
| share_50 | 0.50 | gpt-5.4-nano, single | 1–3 | 1,350 | 2.03 |
| Total | 6,480 | 3.24 (measured rate 0.00044 → 2.85) |
Design file: configs/pilot_openai.yaml (per-run usd_cap 1.5; add the free share_0 cell so the parquet is self-contained). Steps: ma budget → confirm ≤ 5 USD → ma design configs/pilot_openai.yaml --cache results/cache/pilot_openai.sqlite → record in results/manifests/pilot.json: total USD, tokens in/out/reasoning per decision, latency, validity rate, failure rate, cache-hit share, realized order and volume shares, and the SD of paired differences (vs share_0) for every primary outcome.
Pilot pass criteria: validity ≥ 80 %, failures ≤ 10 %, cost per decision within ±30 % of 0.0005 USD, no budget halts. Failure → one prompt revision (v2), mock re-run, one-seed re-pilot (≤ 0.7 USD). Pilot seeds 1–3 are excluded from confirmatory estimates.
5. Phase 6–7 — freeze and confirmatory design
5.1 Freeze (before the first confirmatory call)
results/manifests/freeze.json records: git revision; SHA of prompting.py, schema.py, outcomes.py; config_hash of the confirmatory base; cell list; seed lists (confirmatory seeds 101–132, used in order); thresholds (bubble 5 %/10 ticks, crash 10 %/20 ticks; sensitivity grid 3/5/8 % and 7/10/15 %); the σ̂_d table from the pilot; the tier budget below; model ids and prices. Confirmatory base = configs/baseline.yaml with design-level overrides fundamentals.jump_prob: 1.0, fundamentals.jump_sigma: 4.0 (provisional — chosen so that SD(V^pub/V − 1) ≈ 0.5 %; confirm at freeze), shocks: [{time: 300, kind: fundamental_jump, size: 10.0}, {time: 450, kind: placebo}] (+2.1 % on V).
5.2 Cells and budget by tier (gpt-5.4-nano unless stated; 27 decisions per LLM agent per run; planning cost 0.0005 USD per nano decision, 0.0016 per gpt-5.4-mini decision — the mini token profile is TBD and assumed equal to nano's)
| Tier | Cell | Overrides (on the confirmatory base) | Seeds | Live decisions | USD |
|---|---|---|---|---|---|
| T1 primary | share_0 | population.llm_share: 0.0 |
101–130 | 0 | 0 |
| share_25 | llm_share: 0.25 |
101–110 | 6,750 | 3.38 | |
| share_50 | llm_share: 0.50 |
101–110 | 13,500 | 6.75 | |
| T2 diversity | diverse_25 | llm_share: 0.25, llm.prompt_set: diverse |
101–106 | 4,050 | 2.03 |
| mix_25 | llm_share: 0.25, llm.provider_mix: {openai: 1.0} with a second OpenAI model id for half the slots (see note) |
101–106 | 2,028 nano + 2,022 mini | 4.25 | |
| mini_25 | llm_share: 0.25, llm.models.openai: gpt-5.4-mini |
101–104 | 2,700 mini | 4.32 | |
| T3 dose shape | share_1, share_5, share_10 | llm_share: 0.01 / 0.05 / 0.10 |
101–104 | 1,728 | 0.86 |
| share_75, share_100 | llm_share: 0.75 / 1.00 (95 eligible slots at 100 %) |
101–102 | 9,180 | 4.59 | |
| T4 mechanisms (φ = 0.25) | memory_25 | llm.memory: 5 |
101–103 | 2,025 | 1.01 |
| channel_25 | llm.channel: true (cohorts of 5) |
101–103 | 2,025 | 1.01 | |
| leverage_25 | population.mix.leveraged: 0.19, trend: 0.06 (+ free leverage_0 control at φ = 0 for the E5 interaction) |
101–103 | 2,025 | 1.01 | |
| breaker_25 | exchange.halt_threshold: 0.05 (10-tick window, 20-tick halt) |
101–103 | 2,025 | 1.01 | |
| poslimit_25 | exchange.position_limit: 300 |
101–102 | 1,350 | 0.68 | |
| T5 fairness | mini_25_equalusd | mini_25 with llm.usd_cap = mean nano spend of share_25 (TBD ≈ 0.34) → decisions stop at equal USD |
101–103 | ≤ 630 mini | ≤ 1.01 |
| T6 extensions | k2_25 | two assets, factor_loading: [0.7, 0.7], statarb: 0.05 |
101–102 | 1,350 | 0.68 |
| library_1000 | population.n_traders: 1000, llm.frozen_library: results/library/gpt-5.4-nano_single.npz, provider library |
101–110 | 0 | 0 | |
| auction_25, interval_10 | call auction every 5 ticks; decision_interval: 10 — mock only |
1–10 | 0 | 0 | |
| Confirmatory total | ≈ 66,000 | ≈ 33.3 | |||
| Pilot (§4) | 6,480 | 3.24 | |||
| Contingency (retries, longer outputs) | ≈ 3.5 (≈ 9 %) | ||||
| Project cap | 40.00 |
Notes. (i) Equal-decisions fairness is the default (decision_cap unset: every LLM agent gets the same 27 opportunities); equal-USD uses the per-run usd_cap and therefore applies to single-provider cells only — the code has no per-provider cap. (ii) The two-model mix: provider_mix keys are providers, not models, and models maps one model per provider; mixing two OpenAI models in one run requires either a second provider alias (e.g. openai_mini) or a per-slot model list — a gate-4 engineering item; until then mix_25 is dry-run with {mock: 0.5, openai: 0.5}. (iii) Claude cells (haiku_25 at ≈ 0.19 USD per 1,000 decisions; sonnet at ≈ 3.8 USD per 1,000) are planned only if a valid key and additional budget are approved before the freeze; they are not in the 40 USD plan. (iv) ma budget reproduces this table with --tokens-in 760 --tokens-out 230.
5.3 Execution order and order of sacrifice
Tiers run T1 → T6; spend is reconciled after each tier (sum(llm_usd) over outcomes.parquet plus the provider dashboard). If the cumulative spend exceeds the plan by > 10 %, seeds are cut in this fixed order: T6 → T5 → T3 (share_75/100 first) → T4 poslimit → T2 mini_25 (E3(b) then unavailable) → T4 others. T1 is never cut below 8 seeds. Nothing is added after seeing confirmatory results.
5.4 Frozen-library replay (T6)
Build the library from the T1 nano runs: DecisionLibrary.from_runs([...T1 run dirs...]).save("results/library/gpt-5.4-nano_single.npz"). Replay at N = 1,000 with activity scaled by 100/N (as in scripts/11_scale_benchmark.py) so order flow per tick is constant. Report the nearest-neighbour distance distribution; if the median standardized distance at N = 1,000 exceeds the 90th percentile observed at N = 100, the arm is reported as out-of-support, not as evidence.
6. Replication checklist
git log -1matchesfreeze.json;pytestpasses (24 tests).ma designof each tier reproducesoutcomes.parquetfrom the cache (--cachepath,llm_cached_hits= 1.0, zero USD).- Every run directory has
manifest.jsonwithconfig_hash,git_revision,prompt_version,price_assumptions,llm_usage,shocks,final_conservation_ok. - Analysis scripts read only
outcomes.parquet,ticks.parquet,trades.parquet,llm_decisions.parquet; tables and figures are regenerated by one command (paper/Makefile). - The sensitivity grid (bubble 3/5/8 %, crash 7/10/15 %) is produced by re-running
run_outcomeswith the alternative thresholds on saved ticks, not by re-simulating. - Decision logs are published without raw prompts if they contain nothing private (they do not: the prompt is a canonical market snapshot); model ids, dates and prices are listed in
model_cards.md. - Deviations from this protocol are listed in the paper's appendix with the date and reason.