# OpenAI model benchmarks — Note SPBoucher AI
Generated 2026-11-01T07:18:30.319Z from benchmarks/results/raw/bench-runs.jsonl (528 real runs, 2 trials per model × task). Reproduce: npx tsx benchmarks/scripts/bench.ts then npx tsx benchmarks/scripts/report.ts.
Protocol. Streaming requests, provider default reasoning settings, max_tokens per task (1.5K–6K). TTFT = first text/tool delta; tok/s = output tokens ÷ (total − TTFT). Tokens are provider-reported; cost = tokens × published list prices (estimate). Deterministic graders for 8 tasks (regex, executed code tests, JSON-schema + field checks, tool-call checks); the three writing/synthesis tasks add a two-judge LLM rubric (claude-sonnet-5-5 + gpt-5.4, 1–5) reported separately and never mixed into the pass rate.
# Overall
| Model |
Pass rate |
Mean score |
LLM-judged (1–5) |
TTFT med. |
Latency med. |
tok/s med. |
Cost / run |
Total cost |
Errors |
chat-latest |
100% |
1.00 |
5.00 |
741 ms |
1.4 s |
157 |
$0.019 |
$0.426 |
0/22 |
gpt-5.3-codex |
100% |
1.00 |
4.67 |
602 ms |
1.4 s |
94 |
$0.012 |
$0.253 |
0/22 |
gpt-4.1 |
100% |
1.00 |
4.17 |
450 ms |
1.7 s |
163 |
$0.012 |
$0.259 |
0/22 |
gpt-6-sol |
100% |
1.00 |
5.00 |
1106 ms |
1.7 s |
149 |
$0.0075 |
$0.164 |
0/22 |
gpt-5.4 |
100% |
1.00 |
5.00 |
716 ms |
1.8 s |
86 |
$0.016 |
$0.347 |
0/22 |
gpt-5.2 |
100% |
1.00 |
4.33 |
676 ms |
2.1 s |
85 |
$0.0074 |
$0.162 |
0/22 |
gpt-5.1 |
100% |
1.00 |
4.50 |
650 ms |
2.2 s |
139 |
$0.0057 |
$0.125 |
0/22 |
gpt-6.1-sol |
100% |
1.00 |
4.67 |
1444 ms |
2.2 s |
89 |
$0.0070 |
$0.155 |
0/22 |
gpt-5.6-sol |
100% |
1.00 |
5.00 |
1108 ms |
2.5 s |
133 |
$0.015 |
$0.335 |
0/22 |
gpt-5.5 |
100% |
1.00 |
4.83 |
1755 ms |
2.5 s |
147 |
$0.021 |
$0.467 |
0/22 |
o3 |
100% |
1.00 |
4.33 |
1763 ms |
2.9 s |
533 |
$0.015 |
$0.321 |
0/22 |
gpt-6-astra |
100% |
1.00 |
5.00 |
2376 ms |
3.1 s |
78 |
$0.037 |
$0.810 |
0/22 |
gpt-6.1-terra |
100% |
1.00 |
4.67 |
1515 ms |
3.2 s |
93 |
$0.0037 |
$0.081 |
0/22 |
gpt-5 |
100% |
1.00 |
4.33 |
4121 ms |
5.0 s |
962 |
$0.012 |
$0.265 |
0/22 |
gpt-5-mini |
100% |
1.00 |
4.83 |
4395 ms |
5.7 s |
375 |
$0.0026 |
$0.056 |
0/22 |
gpt-4o |
95% |
1.00 |
3.83 |
656 ms |
2.4 s |
106 |
$0.015 |
$0.330 |
0/22 |
gpt-5.6-luna |
95% |
0.98 |
4.67 |
1398 ms |
3.0 s |
136 |
$0.0008 |
$0.019 |
0/22 |
gpt-4.1-mini |
91% |
0.97 |
3.83 |
449 ms |
1.7 s |
153 |
$0.0024 |
$0.053 |
0/22 |
gpt-5-nano |
95% |
0.97 |
3.67 |
6133 ms |
6.8 s |
1362 |
$0.0006 |
$0.014 |
0/22 |
gpt-5.4-mini |
95% |
0.95 |
4.67 |
624 ms |
1.2 s |
167 |
$0.0029 |
$0.064 |
0/22 |
gpt-5.6-terra |
95% |
0.95 |
5.00 |
869 ms |
1.8 s |
126 |
$0.0077 |
$0.169 |
0/22 |
gpt-6-luna |
95% |
0.95 |
4.67 |
2045 ms |
3.0 s |
194 |
$0.0007 |
$0.015 |
0/22 |
gpt-4o-mini |
82% |
0.94 |
4.00 |
561 ms |
2.0 s |
116 |
$0.0009 |
$0.020 |
0/22 |
gpt-5.4-nano |
86% |
0.92 |
4.00 |
549 ms |
1.7 s |
135 |
$0.0003 |
$0.0077 |
0/22 |
# Pass rate by task
| Model |
Document analysis |
JSON generation |
Complex reasoning |
Long-context retrieval |
English writing |
French writing |
Programming |
Simple reasoning |
Knowledge synthesis |
Tool calling |
Mathematics |
chat-latest |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 5.0) |
100% |
100% |
gpt-5.3-codex |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.0) |
100% |
100% |
gpt-4.1 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 2.5) |
100% |
100% |
gpt-6-sol |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 5.0) |
100% |
100% |
gpt-5.4 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 5.0) |
100% |
100% |
gpt-5.2 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 3.0) |
100% |
100% |
gpt-5.1 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 3.5) |
100% |
100% |
gpt-6.1-sol |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.0) |
100% |
100% |
gpt-5.6-sol |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 5.0) |
100% |
100% |
gpt-5.5 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.5) |
100% |
100% |
o3 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 4.5) |
100% |
100% |
100% (j 3.5) |
100% |
100% |
gpt-6-astra |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 5.0) |
100% |
100% |
gpt-6.1-terra |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.0) |
100% |
100% |
gpt-5 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 3.0) |
100% |
100% |
gpt-5-mini |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.5) |
100% |
100% |
gpt-4o |
100% |
100% |
100% |
100% |
100% (j 5.0) |
50% (j 4.0) |
100% |
100% |
100% (j 2.5) |
100% |
100% |
gpt-5.6-luna |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.0) |
100% |
50% |
gpt-4.1-mini |
100% |
100% |
100% |
100% |
100% (j 5.0) |
50% (j 4.0) |
100% |
100% |
100% (j 2.5) |
100% |
50% |
gpt-5-nano |
100% |
100% |
100% |
100% |
100% (j 5.0) |
50% (j 3.0) |
100% |
100% |
100% (j 3.0) |
100% |
100% |
gpt-5.4-mini |
100% |
100% |
50% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.0) |
100% |
100% |
gpt-5.6-terra |
100% |
100% |
100% |
50% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 5.0) |
100% |
100% |
gpt-6-luna |
100% |
100% |
100% |
50% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.0) |
100% |
100% |
gpt-4o-mini |
100% |
100% |
100% |
100% |
50% (j 4.0) |
100% (j 5.0) |
100% |
0% |
100% (j 3.0) |
100% |
50% |
gpt-5.4-nano |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 4.5) |
100% |
0% |
100% (j 2.5) |
100% |
50% |
| Model |
Document analysis |
JSON generation |
Complex reasoning |
Long-context retrieval |
English writing |
French writing |
Programming |
Simple reasoning |
Knowledge synthesis |
Tool calling |
Mathematics |
chat-latest |
1.3 |
1.1 |
2.8 |
1.3 |
2.9 |
2.9 |
1.4 |
0.9 |
2.5 |
0.9 |
2.0 |
gpt-5.3-codex |
1.3 |
1.3 |
4.9 |
1.0 |
3.9 |
4.1 |
1.4 |
0.8 |
3.4 |
1.1 |
3.3 |
gpt-4.1 |
1.1 |
0.9 |
4.3 |
3.1 |
1.8 |
1.9 |
1.1 |
0.6 |
1.8 |
0.8 |
3.0 |
gpt-6-sol |
1.5 |
1.7 |
2.8 |
1.5 |
4.2 |
4.6 |
2.6 |
1.2 |
2.5 |
1.3 |
1.3 |
gpt-5.4 |
1.6 |
1.5 |
8.3 |
1.5 |
4.8 |
7.7 |
1.8 |
1.0 |
4.7 |
1.0 |
3.6 |
gpt-5.2 |
1.6 |
1.4 |
5.0 |
1.0 |
4.3 |
3.8 |
2.1 |
1.0 |
3.9 |
1.1 |
4.1 |
gpt-5.1 |
1.4 |
1.1 |
4.1 |
1.8 |
4.7 |
3.9 |
1.7 |
0.8 |
2.6 |
1.0 |
2.9 |
gpt-6.1-sol |
1.9 |
1.7 |
4.9 |
1.7 |
7.6 |
8.0 |
4.4 |
1.4 |
4.1 |
1.2 |
2.2 |
gpt-5.6-sol |
1.3 |
1.3 |
4.2 |
1.9 |
3.8 |
4.0 |
2.6 |
1.1 |
4.1 |
1.2 |
2.6 |
gpt-5.5 |
1.6 |
1.6 |
5.8 |
2.3 |
4.6 |
13.4 |
3.2 |
1.8 |
4.8 |
1.3 |
2.3 |
o3 |
2.2 |
1.7 |
5.7 |
3.5 |
4.2 |
7.5 |
2.6 |
1.7 |
3.0 |
1.0 |
4.7 |
gpt-6-astra |
2.2 |
2.8 |
6.3 |
2.8 |
8.4 |
9.4 |
4.5 |
1.9 |
7.0 |
2.0 |
2.9 |
gpt-6.1-terra |
2.0 |
2.0 |
4.4 |
1.2 |
5.4 |
4.9 |
3.2 |
1.4 |
5.1 |
1.4 |
3.4 |
gpt-5 |
5.7 |
3.4 |
5.8 |
3.4 |
10.4 |
9.3 |
4.6 |
1.6 |
7.2 |
2.4 |
7.6 |
gpt-5-mini |
5.3 |
3.8 |
12.4 |
3.1 |
10.4 |
16.5 |
5.5 |
2.9 |
9.2 |
1.6 |
8.3 |
gpt-4o |
1.2 |
1.0 |
3.9 |
3.4 |
2.3 |
3.3 |
2.2 |
0.8 |
2.5 |
1.0 |
8.2 |
gpt-5.6-luna |
1.6 |
1.3 |
4.2 |
1.3 |
4.7 |
4.8 |
3.1 |
1.7 |
4.6 |
1.4 |
3.7 |
gpt-4.1-mini |
0.9 |
0.8 |
5.6 |
2.2 |
1.7 |
2.0 |
1.2 |
1.2 |
2.1 |
0.8 |
4.1 |
gpt-5-nano |
10.7 |
3.0 |
6.2 |
2.2 |
12.4 |
11.2 |
7.0 |
2.1 |
8.3 |
1.6 |
8.2 |
gpt-5.4-mini |
1.1 |
1.1 |
2.0 |
1.1 |
2.0 |
2.1 |
1.1 |
1.0 |
2.0 |
0.9 |
1.8 |
gpt-5.6-terra |
1.3 |
1.1 |
4.2 |
1.5 |
4.2 |
4.8 |
1.5 |
1.0 |
3.3 |
1.4 |
3.3 |
gpt-6-luna |
2.4 |
2.2 |
5.1 |
2.9 |
7.1 |
6.3 |
3.1 |
1.4 |
2.8 |
1.1 |
20.4 |
gpt-4o-mini |
2.8 |
1.3 |
5.7 |
2.1 |
1.9 |
2.5 |
1.6 |
1.0 |
2.8 |
0.9 |
7.3 |
gpt-5.4-nano |
1.5 |
1.1 |
3.5 |
0.8 |
1.9 |
2.4 |
1.9 |
0.8 |
2.3 |
0.9 |
1.8 |
# Mean cost by task (USD)
| Model |
Document analysis |
JSON generation |
Complex reasoning |
Long-context retrieval |
English writing |
French writing |
Programming |
Simple reasoning |
Knowledge synthesis |
Tool calling |
Mathematics |
chat-latest |
$0.0050 |
$0.0028 |
$0.011 |
$0.152 |
$0.0075 |
$0.0095 |
$0.0041 |
$0.0012 |
$0.0098 |
$0.0020 |
$0.0076 |
gpt-5.3-codex |
$0.0021 |
$0.0012 |
$0.0068 |
$0.097 |
$0.0033 |
$0.0040 |
$0.0018 |
$0.0005 |
$0.0040 |
$0.0008 |
$0.0053 |
gpt-4.1 |
$0.0018 |
$0.0008 |
$0.0082 |
$0.105 |
$0.0019 |
$0.0024 |
$0.0012 |
$0.0004 |
$0.0027 |
$0.0006 |
$0.0052 |
gpt-6-sol |
$0.0021 |
$0.0010 |
$0.0023 |
$0.061 |
$0.0038 |
$0.0044 |
$0.0025 |
$0.0006 |
$0.0031 |
$0.0007 |
$0.0008 |
gpt-5.4 |
$0.0025 |
$0.0014 |
$0.010 |
$0.138 |
$0.0036 |
$0.0044 |
$0.0018 |
$0.0006 |
$0.0049 |
$0.0010 |
$0.0051 |
gpt-5.2 |
$0.0021 |
$0.0012 |
$0.0057 |
$0.054 |
$0.0031 |
$0.0037 |
$0.0021 |
$0.0005 |
$0.0038 |
$0.0008 |
$0.0045 |
gpt-5.1 |
$0.0015 |
$0.0009 |
$0.0068 |
$0.038 |
$0.0023 |
$0.0031 |
$0.0016 |
$0.0004 |
$0.0030 |
$0.0007 |
$0.0041 |
gpt-6.1-sol |
$0.0021 |
$0.0010 |
$0.0030 |
$0.058 |
$0.0026 |
$0.0033 |
$0.0023 |
$0.0005 |
$0.0028 |
$0.0007 |
$0.0008 |
gpt-5.6-sol |
$0.0037 |
$0.0019 |
$0.0072 |
$0.122 |
$0.0061 |
$0.0069 |
$0.0044 |
$0.0009 |
$0.0077 |
$0.0017 |
$0.0052 |
gpt-5.5 |
$0.0050 |
$0.0034 |
$0.012 |
$0.155 |
$0.0077 |
$0.022 |
$0.0067 |
$0.0028 |
$0.012 |
$0.0020 |
$0.0041 |
o3 |
$0.0035 |
$0.0019 |
$0.0075 |
$0.111 |
$0.0056 |
$0.012 |
$0.0038 |
$0.0025 |
$0.0040 |
$0.0011 |
$0.0078 |
gpt-6-astra |
$0.010 |
$0.0049 |
$0.015 |
$0.304 |
$0.014 |
$0.016 |
$0.011 |
$0.0031 |
$0.018 |
$0.0036 |
$0.0049 |
gpt-6.1-terra |
$0.0009 |
$0.0005 |
$0.0017 |
$0.029 |
$0.0016 |
$0.0017 |
$0.0010 |
$0.0003 |
$0.0021 |
$0.0004 |
$0.0011 |
gpt-5 |
$0.0093 |
$0.0056 |
$0.011 |
$0.046 |
$0.015 |
$0.013 |
$0.0088 |
$0.0024 |
$0.011 |
$0.0026 |
$0.0087 |
gpt-5-mini |
$0.0010 |
$0.0005 |
$0.0021 |
$0.014 |
$0.0017 |
$0.0029 |
$0.0011 |
$0.0006 |
$0.0018 |
$0.0002 |
$0.0023 |
gpt-4o |
$0.0020 |
$0.0010 |
$0.0047 |
$0.138 |
$0.0020 |
$0.0030 |
$0.0014 |
$0.0005 |
$0.0026 |
$0.0007 |
$0.0091 |
gpt-5.6-luna |
$0.0002 |
$0.0001 |
$0.0006 |
$0.0061 |
$0.0004 |
$0.0005 |
$0.0003 |
$0.0001 |
$0.0004 |
$0.0001 |
$0.0004 |
gpt-4.1-mini |
$0.0003 |
$0.0002 |
$0.0016 |
$0.022 |
$0.0004 |
$0.0005 |
$0.0002 |
$0.0001 |
$0.0005 |
$0.0001 |
$0.0010 |
gpt-5-nano |
$0.0004 |
$0.0002 |
$0.0005 |
$0.0016 |
$0.0010 |
$0.0011 |
$0.0007 |
$0.0002 |
$0.0007 |
$0.0001 |
$0.0004 |
gpt-5.4-mini |
$0.0008 |
$0.0004 |
$0.0015 |
$0.023 |
$0.0011 |
$0.0012 |
$0.0007 |
$0.0002 |
$0.0012 |
$0.0003 |
$0.0015 |
gpt-5.6-terra |
$0.0020 |
$0.0011 |
$0.0036 |
$0.061 |
$0.0030 |
$0.0038 |
$0.0014 |
$0.0005 |
$0.0035 |
$0.0008 |
$0.0035 |
gpt-6-luna |
$0.0001 |
$0.0001 |
$0.0002 |
$0.0056 |
$0.0003 |
$0.0003 |
$0.0001 |
$0.0000 |
$0.0002 |
$0.0000 |
$0.0002 |
gpt-4o-mini |
$0.0001 |
$0.0001 |
$0.0005 |
$0.0083 |
$0.0001 |
$0.0002 |
$0.0001 |
$0.0000 |
$0.0002 |
$0.0000 |
$0.0006 |
gpt-5.4-nano |
$0.0002 |
$0.0001 |
$0.0008 |
$0.0011 |
$0.0003 |
$0.0003 |
$0.0002 |
$0.0000 |
$0.0004 |
$0.0001 |
$0.0003 |
# Token usage
| Model |
Input tokens |
Output tokens |
of which reasoning |
chat-latest |
113620 |
3506 |
128 |
gpt-5.3-codex |
113620 |
3883 |
0 |
gpt-4.1 |
113634 |
5479 |
0 |
gpt-6-sol |
113620 |
3633 |
1358 |
gpt-5.4 |
113620 |
4200 |
0 |
gpt-5.2 |
113620 |
3541 |
0 |
gpt-5.1 |
113620 |
4483 |
0 |
gpt-6.1-sol |
113620 |
3227 |
619 |
gpt-5.6-sol |
113620 |
3964 |
1123 |
gpt-5.5 |
113620 |
4892 |
2460 |
o3 |
113614 |
11724 |
8512 |
gpt-6-astra |
113620 |
3406 |
776 |
gpt-6.1-terra |
113620 |
3953 |
1303 |
gpt-5 |
113620 |
17346 |
13696 |
gpt-5-mini |
113620 |
14001 |
10176 |
gpt-4o |
113634 |
4623 |
0 |
gpt-5.6-luna |
113620 |
4829 |
1858 |
gpt-4.1-mini |
113634 |
5228 |
0 |
gpt-5-nano |
113620 |
27168 |
24384 |
gpt-5.4-mini |
113620 |
3439 |
0 |
gpt-5.6-terra |
113620 |
3436 |
881 |
gpt-6-luna |
113620 |
6410 |
3720 |
gpt-4o-mini |
113634 |
5370 |
0 |
gpt-5.4-nano |
113620 |
3805 |
0 |