# OpenAI model benchmarks — Note SPBoucher AI

Generated 2026-11-01T07:18:30.319Z from `benchmarks/results/raw/bench-runs.jsonl` (528 real runs, 2 trials per model × task). Reproduce: `npx tsx benchmarks/scripts/bench.ts` then `npx tsx benchmarks/scripts/report.ts`.

**Protocol.** Streaming requests, provider default reasoning settings, `max_tokens` per task (1.5K–6K). TTFT = first text/tool delta; tok/s = output tokens ÷ (total − TTFT). Tokens are provider-reported; cost = tokens × published list prices (estimate). Deterministic graders for 8 tasks (regex, executed code tests, JSON-schema + field checks, tool-call checks); the three writing/synthesis tasks add a two-judge LLM rubric (claude-sonnet-5-5 + gpt-5.4, 1–5) reported separately and **never mixed into the pass rate**.

## Overall

| Model | Pass rate | Mean score | LLM-judged (1–5) | TTFT med. | Latency med. | tok/s med. | Cost / run | Total cost | Errors |
|---|---|---|---|---|---|---|---|---|---|
| `chat-latest` | 100% | 1.00 | 5.00 | 741 ms | 1.4 s | 157 | $0.019 | $0.426 | 0/22 |
| `gpt-5.3-codex` | 100% | 1.00 | 4.67 | 602 ms | 1.4 s | 94 | $0.012 | $0.253 | 0/22 |
| `gpt-4.1` | 100% | 1.00 | 4.17 | 450 ms | 1.7 s | 163 | $0.012 | $0.259 | 0/22 |
| `gpt-6-sol` | 100% | 1.00 | 5.00 | 1106 ms | 1.7 s | 149 | $0.0075 | $0.164 | 0/22 |
| `gpt-5.4` | 100% | 1.00 | 5.00 | 716 ms | 1.8 s | 86 | $0.016 | $0.347 | 0/22 |
| `gpt-5.2` | 100% | 1.00 | 4.33 | 676 ms | 2.1 s | 85 | $0.0074 | $0.162 | 0/22 |
| `gpt-5.1` | 100% | 1.00 | 4.50 | 650 ms | 2.2 s | 139 | $0.0057 | $0.125 | 0/22 |
| `gpt-6.1-sol` | 100% | 1.00 | 4.67 | 1444 ms | 2.2 s | 89 | $0.0070 | $0.155 | 0/22 |
| `gpt-5.6-sol` | 100% | 1.00 | 5.00 | 1108 ms | 2.5 s | 133 | $0.015 | $0.335 | 0/22 |
| `gpt-5.5` | 100% | 1.00 | 4.83 | 1755 ms | 2.5 s | 147 | $0.021 | $0.467 | 0/22 |
| `o3` | 100% | 1.00 | 4.33 | 1763 ms | 2.9 s | 533 | $0.015 | $0.321 | 0/22 |
| `gpt-6-astra` | 100% | 1.00 | 5.00 | 2376 ms | 3.1 s | 78 | $0.037 | $0.810 | 0/22 |
| `gpt-6.1-terra` | 100% | 1.00 | 4.67 | 1515 ms | 3.2 s | 93 | $0.0037 | $0.081 | 0/22 |
| `gpt-5` | 100% | 1.00 | 4.33 | 4121 ms | 5.0 s | 962 | $0.012 | $0.265 | 0/22 |
| `gpt-5-mini` | 100% | 1.00 | 4.83 | 4395 ms | 5.7 s | 375 | $0.0026 | $0.056 | 0/22 |
| `gpt-4o` | 95% | 1.00 | 3.83 | 656 ms | 2.4 s | 106 | $0.015 | $0.330 | 0/22 |
| `gpt-5.6-luna` | 95% | 0.98 | 4.67 | 1398 ms | 3.0 s | 136 | $0.0008 | $0.019 | 0/22 |
| `gpt-4.1-mini` | 91% | 0.97 | 3.83 | 449 ms | 1.7 s | 153 | $0.0024 | $0.053 | 0/22 |
| `gpt-5-nano` | 95% | 0.97 | 3.67 | 6133 ms | 6.8 s | 1362 | $0.0006 | $0.014 | 0/22 |
| `gpt-5.4-mini` | 95% | 0.95 | 4.67 | 624 ms | 1.2 s | 167 | $0.0029 | $0.064 | 0/22 |
| `gpt-5.6-terra` | 95% | 0.95 | 5.00 | 869 ms | 1.8 s | 126 | $0.0077 | $0.169 | 0/22 |
| `gpt-6-luna` | 95% | 0.95 | 4.67 | 2045 ms | 3.0 s | 194 | $0.0007 | $0.015 | 0/22 |
| `gpt-4o-mini` | 82% | 0.94 | 4.00 | 561 ms | 2.0 s | 116 | $0.0009 | $0.020 | 0/22 |
| `gpt-5.4-nano` | 86% | 0.92 | 4.00 | 549 ms | 1.7 s | 135 | $0.0003 | $0.0077 | 0/22 |

## Pass rate by task

| Model | Document analysis | JSON generation | Complex reasoning | Long-context retrieval | English writing | French writing | Programming | Simple reasoning | Knowledge synthesis | Tool calling | Mathematics |
|---|---|---|---|---|---|---|---|---|---|---|---|
| `chat-latest` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 5.0) | 100% | 100% |
| `gpt-5.3-codex` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 4.0) | 100% | 100% |
| `gpt-4.1` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 2.5) | 100% | 100% |
| `gpt-6-sol` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 5.0) | 100% | 100% |
| `gpt-5.4` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 5.0) | 100% | 100% |
| `gpt-5.2` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 3.0) | 100% | 100% |
| `gpt-5.1` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 3.5) | 100% | 100% |
| `gpt-6.1-sol` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 4.0) | 100% | 100% |
| `gpt-5.6-sol` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 5.0) | 100% | 100% |
| `gpt-5.5` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 4.5) | 100% | 100% |
| `o3` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 4.5) | 100% | 100% | 100% (j 3.5) | 100% | 100% |
| `gpt-6-astra` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 5.0) | 100% | 100% |
| `gpt-6.1-terra` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 4.0) | 100% | 100% |
| `gpt-5` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 3.0) | 100% | 100% |
| `gpt-5-mini` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 4.5) | 100% | 100% |
| `gpt-4o` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 50% (j 4.0) | 100% | 100% | 100% (j 2.5) | 100% | 100% |
| `gpt-5.6-luna` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 4.0) | 100% | 50% |
| `gpt-4.1-mini` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 50% (j 4.0) | 100% | 100% | 100% (j 2.5) | 100% | 50% |
| `gpt-5-nano` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 50% (j 3.0) | 100% | 100% | 100% (j 3.0) | 100% | 100% |
| `gpt-5.4-mini` | 100% | 100% | 50% | 100% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 4.0) | 100% | 100% |
| `gpt-5.6-terra` | 100% | 100% | 100% | 50% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 5.0) | 100% | 100% |
| `gpt-6-luna` | 100% | 100% | 100% | 50% | 100% (j 5.0) | 100% (j 5.0) | 100% | 100% | 100% (j 4.0) | 100% | 100% |
| `gpt-4o-mini` | 100% | 100% | 100% | 100% | 50% (j 4.0) | 100% (j 5.0) | 100% | 0% | 100% (j 3.0) | 100% | 50% |
| `gpt-5.4-nano` | 100% | 100% | 100% | 100% | 100% (j 5.0) | 100% (j 4.5) | 100% | 0% | 100% (j 2.5) | 100% | 50% |

## Median latency by task (seconds)

| Model | Document analysis | JSON generation | Complex reasoning | Long-context retrieval | English writing | French writing | Programming | Simple reasoning | Knowledge synthesis | Tool calling | Mathematics |
|---|---|---|---|---|---|---|---|---|---|---|---|
| `chat-latest` | 1.3 | 1.1 | 2.8 | 1.3 | 2.9 | 2.9 | 1.4 | 0.9 | 2.5 | 0.9 | 2.0 |
| `gpt-5.3-codex` | 1.3 | 1.3 | 4.9 | 1.0 | 3.9 | 4.1 | 1.4 | 0.8 | 3.4 | 1.1 | 3.3 |
| `gpt-4.1` | 1.1 | 0.9 | 4.3 | 3.1 | 1.8 | 1.9 | 1.1 | 0.6 | 1.8 | 0.8 | 3.0 |
| `gpt-6-sol` | 1.5 | 1.7 | 2.8 | 1.5 | 4.2 | 4.6 | 2.6 | 1.2 | 2.5 | 1.3 | 1.3 |
| `gpt-5.4` | 1.6 | 1.5 | 8.3 | 1.5 | 4.8 | 7.7 | 1.8 | 1.0 | 4.7 | 1.0 | 3.6 |
| `gpt-5.2` | 1.6 | 1.4 | 5.0 | 1.0 | 4.3 | 3.8 | 2.1 | 1.0 | 3.9 | 1.1 | 4.1 |
| `gpt-5.1` | 1.4 | 1.1 | 4.1 | 1.8 | 4.7 | 3.9 | 1.7 | 0.8 | 2.6 | 1.0 | 2.9 |
| `gpt-6.1-sol` | 1.9 | 1.7 | 4.9 | 1.7 | 7.6 | 8.0 | 4.4 | 1.4 | 4.1 | 1.2 | 2.2 |
| `gpt-5.6-sol` | 1.3 | 1.3 | 4.2 | 1.9 | 3.8 | 4.0 | 2.6 | 1.1 | 4.1 | 1.2 | 2.6 |
| `gpt-5.5` | 1.6 | 1.6 | 5.8 | 2.3 | 4.6 | 13.4 | 3.2 | 1.8 | 4.8 | 1.3 | 2.3 |
| `o3` | 2.2 | 1.7 | 5.7 | 3.5 | 4.2 | 7.5 | 2.6 | 1.7 | 3.0 | 1.0 | 4.7 |
| `gpt-6-astra` | 2.2 | 2.8 | 6.3 | 2.8 | 8.4 | 9.4 | 4.5 | 1.9 | 7.0 | 2.0 | 2.9 |
| `gpt-6.1-terra` | 2.0 | 2.0 | 4.4 | 1.2 | 5.4 | 4.9 | 3.2 | 1.4 | 5.1 | 1.4 | 3.4 |
| `gpt-5` | 5.7 | 3.4 | 5.8 | 3.4 | 10.4 | 9.3 | 4.6 | 1.6 | 7.2 | 2.4 | 7.6 |
| `gpt-5-mini` | 5.3 | 3.8 | 12.4 | 3.1 | 10.4 | 16.5 | 5.5 | 2.9 | 9.2 | 1.6 | 8.3 |
| `gpt-4o` | 1.2 | 1.0 | 3.9 | 3.4 | 2.3 | 3.3 | 2.2 | 0.8 | 2.5 | 1.0 | 8.2 |
| `gpt-5.6-luna` | 1.6 | 1.3 | 4.2 | 1.3 | 4.7 | 4.8 | 3.1 | 1.7 | 4.6 | 1.4 | 3.7 |
| `gpt-4.1-mini` | 0.9 | 0.8 | 5.6 | 2.2 | 1.7 | 2.0 | 1.2 | 1.2 | 2.1 | 0.8 | 4.1 |
| `gpt-5-nano` | 10.7 | 3.0 | 6.2 | 2.2 | 12.4 | 11.2 | 7.0 | 2.1 | 8.3 | 1.6 | 8.2 |
| `gpt-5.4-mini` | 1.1 | 1.1 | 2.0 | 1.1 | 2.0 | 2.1 | 1.1 | 1.0 | 2.0 | 0.9 | 1.8 |
| `gpt-5.6-terra` | 1.3 | 1.1 | 4.2 | 1.5 | 4.2 | 4.8 | 1.5 | 1.0 | 3.3 | 1.4 | 3.3 |
| `gpt-6-luna` | 2.4 | 2.2 | 5.1 | 2.9 | 7.1 | 6.3 | 3.1 | 1.4 | 2.8 | 1.1 | 20.4 |
| `gpt-4o-mini` | 2.8 | 1.3 | 5.7 | 2.1 | 1.9 | 2.5 | 1.6 | 1.0 | 2.8 | 0.9 | 7.3 |
| `gpt-5.4-nano` | 1.5 | 1.1 | 3.5 | 0.8 | 1.9 | 2.4 | 1.9 | 0.8 | 2.3 | 0.9 | 1.8 |

## Mean cost by task (USD)

| Model | Document analysis | JSON generation | Complex reasoning | Long-context retrieval | English writing | French writing | Programming | Simple reasoning | Knowledge synthesis | Tool calling | Mathematics |
|---|---|---|---|---|---|---|---|---|---|---|---|
| `chat-latest` | $0.0050 | $0.0028 | $0.011 | $0.152 | $0.0075 | $0.0095 | $0.0041 | $0.0012 | $0.0098 | $0.0020 | $0.0076 |
| `gpt-5.3-codex` | $0.0021 | $0.0012 | $0.0068 | $0.097 | $0.0033 | $0.0040 | $0.0018 | $0.0005 | $0.0040 | $0.0008 | $0.0053 |
| `gpt-4.1` | $0.0018 | $0.0008 | $0.0082 | $0.105 | $0.0019 | $0.0024 | $0.0012 | $0.0004 | $0.0027 | $0.0006 | $0.0052 |
| `gpt-6-sol` | $0.0021 | $0.0010 | $0.0023 | $0.061 | $0.0038 | $0.0044 | $0.0025 | $0.0006 | $0.0031 | $0.0007 | $0.0008 |
| `gpt-5.4` | $0.0025 | $0.0014 | $0.010 | $0.138 | $0.0036 | $0.0044 | $0.0018 | $0.0006 | $0.0049 | $0.0010 | $0.0051 |
| `gpt-5.2` | $0.0021 | $0.0012 | $0.0057 | $0.054 | $0.0031 | $0.0037 | $0.0021 | $0.0005 | $0.0038 | $0.0008 | $0.0045 |
| `gpt-5.1` | $0.0015 | $0.0009 | $0.0068 | $0.038 | $0.0023 | $0.0031 | $0.0016 | $0.0004 | $0.0030 | $0.0007 | $0.0041 |
| `gpt-6.1-sol` | $0.0021 | $0.0010 | $0.0030 | $0.058 | $0.0026 | $0.0033 | $0.0023 | $0.0005 | $0.0028 | $0.0007 | $0.0008 |
| `gpt-5.6-sol` | $0.0037 | $0.0019 | $0.0072 | $0.122 | $0.0061 | $0.0069 | $0.0044 | $0.0009 | $0.0077 | $0.0017 | $0.0052 |
| `gpt-5.5` | $0.0050 | $0.0034 | $0.012 | $0.155 | $0.0077 | $0.022 | $0.0067 | $0.0028 | $0.012 | $0.0020 | $0.0041 |
| `o3` | $0.0035 | $0.0019 | $0.0075 | $0.111 | $0.0056 | $0.012 | $0.0038 | $0.0025 | $0.0040 | $0.0011 | $0.0078 |
| `gpt-6-astra` | $0.010 | $0.0049 | $0.015 | $0.304 | $0.014 | $0.016 | $0.011 | $0.0031 | $0.018 | $0.0036 | $0.0049 |
| `gpt-6.1-terra` | $0.0009 | $0.0005 | $0.0017 | $0.029 | $0.0016 | $0.0017 | $0.0010 | $0.0003 | $0.0021 | $0.0004 | $0.0011 |
| `gpt-5` | $0.0093 | $0.0056 | $0.011 | $0.046 | $0.015 | $0.013 | $0.0088 | $0.0024 | $0.011 | $0.0026 | $0.0087 |
| `gpt-5-mini` | $0.0010 | $0.0005 | $0.0021 | $0.014 | $0.0017 | $0.0029 | $0.0011 | $0.0006 | $0.0018 | $0.0002 | $0.0023 |
| `gpt-4o` | $0.0020 | $0.0010 | $0.0047 | $0.138 | $0.0020 | $0.0030 | $0.0014 | $0.0005 | $0.0026 | $0.0007 | $0.0091 |
| `gpt-5.6-luna` | $0.0002 | $0.0001 | $0.0006 | $0.0061 | $0.0004 | $0.0005 | $0.0003 | $0.0001 | $0.0004 | $0.0001 | $0.0004 |
| `gpt-4.1-mini` | $0.0003 | $0.0002 | $0.0016 | $0.022 | $0.0004 | $0.0005 | $0.0002 | $0.0001 | $0.0005 | $0.0001 | $0.0010 |
| `gpt-5-nano` | $0.0004 | $0.0002 | $0.0005 | $0.0016 | $0.0010 | $0.0011 | $0.0007 | $0.0002 | $0.0007 | $0.0001 | $0.0004 |
| `gpt-5.4-mini` | $0.0008 | $0.0004 | $0.0015 | $0.023 | $0.0011 | $0.0012 | $0.0007 | $0.0002 | $0.0012 | $0.0003 | $0.0015 |
| `gpt-5.6-terra` | $0.0020 | $0.0011 | $0.0036 | $0.061 | $0.0030 | $0.0038 | $0.0014 | $0.0005 | $0.0035 | $0.0008 | $0.0035 |
| `gpt-6-luna` | $0.0001 | $0.0001 | $0.0002 | $0.0056 | $0.0003 | $0.0003 | $0.0001 | $0.0000 | $0.0002 | $0.0000 | $0.0002 |
| `gpt-4o-mini` | $0.0001 | $0.0001 | $0.0005 | $0.0083 | $0.0001 | $0.0002 | $0.0001 | $0.0000 | $0.0002 | $0.0000 | $0.0006 |
| `gpt-5.4-nano` | $0.0002 | $0.0001 | $0.0008 | $0.0011 | $0.0003 | $0.0003 | $0.0002 | $0.0000 | $0.0004 | $0.0001 | $0.0003 |

## Token usage

| Model | Input tokens | Output tokens | of which reasoning |
|---|---|---|---|
| `chat-latest` | 113620 | 3506 | 128 |
| `gpt-5.3-codex` | 113620 | 3883 | 0 |
| `gpt-4.1` | 113634 | 5479 | 0 |
| `gpt-6-sol` | 113620 | 3633 | 1358 |
| `gpt-5.4` | 113620 | 4200 | 0 |
| `gpt-5.2` | 113620 | 3541 | 0 |
| `gpt-5.1` | 113620 | 4483 | 0 |
| `gpt-6.1-sol` | 113620 | 3227 | 619 |
| `gpt-5.6-sol` | 113620 | 3964 | 1123 |
| `gpt-5.5` | 113620 | 4892 | 2460 |
| `o3` | 113614 | 11724 | 8512 |
| `gpt-6-astra` | 113620 | 3406 | 776 |
| `gpt-6.1-terra` | 113620 | 3953 | 1303 |
| `gpt-5` | 113620 | 17346 | 13696 |
| `gpt-5-mini` | 113620 | 14001 | 10176 |
| `gpt-4o` | 113634 | 4623 | 0 |
| `gpt-5.6-luna` | 113620 | 4829 | 1858 |
| `gpt-4.1-mini` | 113634 | 5228 | 0 |
| `gpt-5-nano` | 113620 | 27168 | 24384 |
| `gpt-5.4-mini` | 113620 | 3439 | 0 |
| `gpt-5.6-terra` | 113620 | 3436 | 881 |
| `gpt-6-luna` | 113620 | 6410 | 3720 |
| `gpt-4o-mini` | 113634 | 5370 | 0 |
| `gpt-5.4-nano` | 113620 | 3805 | 0 |