# AI model comparison — Note SPBoucher AI
Generated 2026-11-01T07:18:30.320Z. 836 real runs across 38 models; total estimated spend for Phase Zero ≈ $17.24.
# Ranking (deterministic mean score, then latency)
| # |
Model |
Provider |
Pass |
Score |
Judged |
TTFT |
Latency |
tok/s |
$/run |
| 1 |
chat-latest |
openai |
100% |
1.00 |
5.00 |
741 ms |
1.4 s |
157 |
$0.019 |
| 2 |
gpt-5.3-codex |
openai |
100% |
1.00 |
4.67 |
602 ms |
1.4 s |
94 |
$0.012 |
| 3 |
gpt-4.1 |
openai |
100% |
1.00 |
4.17 |
450 ms |
1.7 s |
163 |
$0.012 |
| 4 |
gpt-6-sol |
openai |
100% |
1.00 |
5.00 |
1106 ms |
1.7 s |
149 |
$0.0075 |
| 5 |
gpt-5.4 |
openai |
100% |
1.00 |
5.00 |
716 ms |
1.8 s |
86 |
$0.016 |
| 6 |
claude-fable-5-1 |
anthropic |
100% |
1.00 |
5.00 |
389 ms |
2.1 s |
118 |
$0.090 |
| 7 |
gpt-5.2 |
openai |
100% |
1.00 |
4.33 |
676 ms |
2.1 s |
85 |
$0.0074 |
| 8 |
claude-sonnet-5 |
anthropic |
100% |
1.00 |
4.33 |
950 ms |
2.2 s |
145 |
$0.019 |
| 9 |
gpt-5.1 |
openai |
100% |
1.00 |
4.50 |
650 ms |
2.2 s |
139 |
$0.0057 |
| 10 |
gpt-6.1-sol |
openai |
100% |
1.00 |
4.67 |
1444 ms |
2.2 s |
89 |
$0.0070 |
| 11 |
claude-sonnet-4-5-20250929 |
anthropic |
100% |
1.00 |
4.33 |
775 ms |
2.3 s |
90 |
$0.022 |
| 12 |
claude-haiku-5-5 |
anthropic |
100% |
1.00 |
4.83 |
1301 ms |
2.4 s |
487 |
$0.0011 |
| 13 |
gpt-5.6-sol |
openai |
100% |
1.00 |
5.00 |
1108 ms |
2.5 s |
133 |
$0.015 |
| 14 |
gpt-5.5 |
openai |
100% |
1.00 |
4.83 |
1755 ms |
2.5 s |
147 |
$0.021 |
| 15 |
claude-opus-4-5-20251101 |
anthropic |
100% |
1.00 |
4.67 |
862 ms |
2.5 s |
98 |
$0.036 |
| 16 |
claude-opus-5-5 |
anthropic |
100% |
1.00 |
4.83 |
1703 ms |
2.8 s |
161 |
$0.043 |
| 17 |
claude-sonnet-5-5 |
anthropic |
100% |
1.00 |
4.67 |
1020 ms |
2.9 s |
202 |
$0.020 |
| 18 |
claude-opus-4-8 |
anthropic |
100% |
1.00 |
4.67 |
488 ms |
2.9 s |
104 |
$0.045 |
| 19 |
o3 |
openai |
100% |
1.00 |
4.33 |
1763 ms |
2.9 s |
533 |
$0.015 |
| 20 |
gpt-6-astra |
openai |
100% |
1.00 |
5.00 |
2376 ms |
3.1 s |
78 |
$0.037 |
| 21 |
gpt-6.1-terra |
openai |
100% |
1.00 |
4.67 |
1515 ms |
3.2 s |
93 |
$0.0037 |
| 22 |
claude-opus-4-7 |
anthropic |
100% |
1.00 |
4.33 |
687 ms |
3.3 s |
91 |
$0.045 |
| 23 |
claude-opus-5 |
anthropic |
100% |
1.00 |
4.67 |
2530 ms |
4.0 s |
161 |
$0.051 |
| 24 |
claude-opus-4-6 |
anthropic |
100% |
1.00 |
4.83 |
1622 ms |
4.2 s |
75 |
$0.035 |
| 25 |
claude-sonnet-4-6 |
anthropic |
100% |
1.00 |
5.00 |
735 ms |
4.7 s |
90 |
$0.021 |
| 26 |
gpt-5 |
openai |
100% |
1.00 |
4.33 |
4121 ms |
5.0 s |
962 |
$0.012 |
| 27 |
gpt-5-mini |
openai |
100% |
1.00 |
4.83 |
4395 ms |
5.7 s |
375 |
$0.0026 |
| 28 |
gpt-4o |
openai |
95% |
1.00 |
3.83 |
656 ms |
2.4 s |
106 |
$0.015 |
| 29 |
gpt-5.6-luna |
openai |
95% |
0.98 |
4.67 |
1398 ms |
3.0 s |
136 |
$0.0008 |
| 30 |
gpt-4.1-mini |
openai |
91% |
0.97 |
3.83 |
449 ms |
1.7 s |
153 |
$0.0024 |
| 31 |
claude-haiku-4-5-20251001 |
anthropic |
91% |
0.97 |
3.33 |
502 ms |
2.0 s |
135 |
$0.0072 |
| 32 |
gpt-5-nano |
openai |
95% |
0.97 |
3.67 |
6133 ms |
6.8 s |
1362 |
$0.0006 |
| 33 |
gpt-5.4-mini |
openai |
95% |
0.95 |
4.67 |
624 ms |
1.2 s |
167 |
$0.0029 |
| 34 |
gpt-5.6-terra |
openai |
95% |
0.95 |
5.00 |
869 ms |
1.8 s |
126 |
$0.0077 |
| 35 |
gpt-6-luna |
openai |
95% |
0.95 |
4.67 |
2045 ms |
3.0 s |
194 |
$0.0007 |
| 36 |
gpt-4o-mini |
openai |
82% |
0.94 |
4.00 |
561 ms |
2.0 s |
116 |
$0.0009 |
| 37 |
gpt-5.4-nano |
openai |
86% |
0.92 |
4.00 |
549 ms |
1.7 s |
135 |
$0.0003 |
| 38 |
claude-fable-5 |
anthropic |
91% |
0.91 |
5.00 |
1764 ms |
2.8 s |
112 |
$0.092 |
# Best model per task (deterministic score, ties → cheaper)
| Task |
Best |
Score |
Latency |
$/run |
Cheapest model with a perfect score |
| Document analysis |
gpt-6-luna |
1.00 |
2.4 s |
$0.0001 |
gpt-6-luna ($0.0001) |
| JSON generation |
gpt-4o-mini |
1.00 |
1.3 s |
$0.0001 |
gpt-4o-mini ($0.0001) |
| Complex reasoning |
gpt-6-luna |
1.00 |
5.1 s |
$0.0002 |
gpt-6-luna ($0.0002) |
| Long-context retrieval |
gpt-5.4-nano |
1.00 |
0.8 s |
$0.0011 |
gpt-5.4-nano ($0.0011) |
| English writing |
claude-haiku-5-5 |
1.00 |
2.7 s |
$0.0002 |
claude-haiku-5-5 ($0.0002) |
| French writing |
gpt-4o-mini |
1.00 |
2.5 s |
$0.0002 |
gpt-4o-mini ($0.0002) |
| Programming |
gpt-4o-mini |
1.00 |
1.6 s |
$0.0001 |
gpt-4o-mini ($0.0001) |
| Simple reasoning |
gpt-6-luna |
1.00 |
1.4 s |
$0.0000 |
gpt-6-luna ($0.0000) |
| Knowledge synthesis |
gpt-4o-mini |
1.00 |
2.8 s |
$0.0002 |
gpt-4o-mini ($0.0002) |
| Tool calling |
gpt-6-luna |
1.00 |
1.1 s |
$0.0000 |
gpt-6-luna ($0.0000) |
| Mathematics |
gpt-6-luna |
1.00 |
20.4 s |
$0.0002 |
gpt-6-luna ($0.0002) |
# Writing quality (two-judge LLM rubric, 1–5; experimental, not used for pass rates)
| Model |
French writing |
English writing |
Knowledge synthesis |
chat-latest |
5.00 |
5.00 |
5.00 |
gpt-5.3-codex |
5.00 |
5.00 |
4.00 |
gpt-4.1 |
5.00 |
5.00 |
2.50 |
gpt-6-sol |
5.00 |
5.00 |
5.00 |
gpt-5.4 |
5.00 |
5.00 |
5.00 |
claude-fable-5-1 |
5.00 |
5.00 |
5.00 |
gpt-5.2 |
5.00 |
5.00 |
3.00 |
claude-sonnet-5 |
5.00 |
5.00 |
3.00 |
gpt-5.1 |
5.00 |
5.00 |
3.50 |
gpt-6.1-sol |
5.00 |
5.00 |
4.00 |
claude-sonnet-4-5-20250929 |
5.00 |
5.00 |
3.00 |
claude-haiku-5-5 |
4.50 |
5.00 |
5.00 |
gpt-5.6-sol |
5.00 |
5.00 |
5.00 |
gpt-5.5 |
5.00 |
5.00 |
4.50 |
claude-opus-4-5-20251101 |
5.00 |
5.00 |
4.00 |
claude-opus-5-5 |
5.00 |
5.00 |
4.50 |
claude-sonnet-5-5 |
5.00 |
5.00 |
4.00 |
claude-opus-4-8 |
5.00 |
5.00 |
4.00 |
o3 |
4.50 |
5.00 |
3.50 |
gpt-6-astra |
5.00 |
5.00 |
5.00 |
gpt-6.1-terra |
5.00 |
5.00 |
4.00 |
claude-opus-4-7 |
5.00 |
5.00 |
3.00 |
claude-opus-5 |
5.00 |
5.00 |
4.00 |
claude-opus-4-6 |
5.00 |
5.00 |
4.50 |
claude-sonnet-4-6 |
5.00 |
5.00 |
5.00 |
gpt-5 |
5.00 |
5.00 |
3.00 |
gpt-5-mini |
5.00 |
5.00 |
4.50 |
gpt-4o |
4.00 |
5.00 |
2.50 |
gpt-5.6-luna |
5.00 |
5.00 |
4.00 |
gpt-4.1-mini |
4.00 |
5.00 |
2.50 |
claude-haiku-4-5-20251001 |
3.50 |
4.00 |
2.50 |
gpt-5-nano |
3.00 |
5.00 |
3.00 |
gpt-5.4-mini |
5.00 |
5.00 |
4.00 |
gpt-5.6-terra |
5.00 |
5.00 |
5.00 |
gpt-6-luna |
5.00 |
5.00 |
4.00 |
gpt-4o-mini |
5.00 |
4.00 |
3.00 |
gpt-5.4-nano |
4.50 |
5.00 |
2.50 |
claude-fable-5 |
5.00 |
5.00 |
5.00 |
# Speed
| Model |
TTFT median |
Latency median |
Output tok/s median |
gpt-5.4-mini |
624 ms |
1.2 s |
167 |
chat-latest |
741 ms |
1.4 s |
157 |
gpt-5.3-codex |
602 ms |
1.4 s |
94 |
gpt-4.1-mini |
449 ms |
1.7 s |
153 |
gpt-4.1 |
450 ms |
1.7 s |
163 |
gpt-5.4-nano |
549 ms |
1.7 s |
135 |
gpt-6-sol |
1106 ms |
1.7 s |
149 |
gpt-5.6-terra |
869 ms |
1.8 s |
126 |
gpt-5.4 |
716 ms |
1.8 s |
86 |
gpt-4o-mini |
561 ms |
2.0 s |
116 |
claude-haiku-4-5-20251001 |
502 ms |
2.0 s |
135 |
claude-fable-5-1 |
389 ms |
2.1 s |
118 |
gpt-5.2 |
676 ms |
2.1 s |
85 |
claude-sonnet-5 |
950 ms |
2.2 s |
145 |
gpt-5.1 |
650 ms |
2.2 s |
139 |
gpt-6.1-sol |
1444 ms |
2.2 s |
89 |
claude-sonnet-4-5-20250929 |
775 ms |
2.3 s |
90 |
gpt-4o |
656 ms |
2.4 s |
106 |
claude-haiku-5-5 |
1301 ms |
2.4 s |
487 |
gpt-5.6-sol |
1108 ms |
2.5 s |
133 |
gpt-5.5 |
1755 ms |
2.5 s |
147 |
claude-opus-4-5-20251101 |
862 ms |
2.5 s |
98 |
claude-opus-5-5 |
1703 ms |
2.8 s |
161 |
claude-fable-5 |
1764 ms |
2.8 s |
112 |
claude-sonnet-5-5 |
1020 ms |
2.9 s |
202 |
claude-opus-4-8 |
488 ms |
2.9 s |
104 |
o3 |
1763 ms |
2.9 s |
533 |
gpt-6-luna |
2045 ms |
3.0 s |
194 |
gpt-5.6-luna |
1398 ms |
3.0 s |
136 |
gpt-6-astra |
2376 ms |
3.1 s |
78 |
gpt-6.1-terra |
1515 ms |
3.2 s |
93 |
claude-opus-4-7 |
687 ms |
3.3 s |
91 |
claude-opus-5 |
2530 ms |
4.0 s |
161 |
claude-opus-4-6 |
1622 ms |
4.2 s |
75 |
claude-sonnet-4-6 |
735 ms |
4.7 s |
90 |
gpt-5 |
4121 ms |
5.0 s |
962 |
gpt-5-mini |
4395 ms |
5.7 s |
375 |
gpt-5-nano |
6133 ms |
6.8 s |
1362 |
# Caveats
- Two trials per cell: differences of a few points are noise. Latency depends on provider load at run time (2026-11-01).
- Costs are estimates (provider-reported tokens × list price); provider invoices are authoritative.
- Writing scores come from LLM judges and are shown separately from deterministic results.
- Pro-tier models (gpt-5.5-pro, gpt-5.4-pro, gpt-5.2-pro, gpt-5-pro, o3-pro) passed discovery probes but were excluded from the benchmark for cost ($20–$30 per M input tokens); they are selectable manually in the app.