# Chat SPBoucher AI — Benchmark report
Generated 2026-11-01T07:22:05.282Z. Every figure below comes from a real API response recorded in openai-results.json, anthropic-results.json or modality-results.json. Prices come from the official pricing pages (see pricing.json). Nothing is estimated from memory.
# Summary
|
OpenAI |
Anthropic |
| Models listed by the API |
140 |
14 |
| Chat models benchmarked (verified) |
30 |
14 |
| Listed but unavailable (404/deprecated) |
23 |
0 |
| Specialized models verified (image/tts/stt/embedding/moderation) |
18 |
0 |
| Not exercised (realtime/video/legacy/deep-research) |
20 |
0 |
Total benchmark spend (estimated from usage × verified prices): $2.41.
# Test suite
| Test |
What is measured |
Grading |
| basic |
capital of Canada |
contains "Ottawa" |
| reasoning |
sisters/brothers riddle |
final answer 3 |
| math |
17·23+144/12 and Σ first 50 odd |
a=403;b=2500 |
| code |
isPrime(n) in JS |
code executed with node against 8 cases |
| french |
2 phrases sur la méthode du coût |
vocabulaire FR, aucun mot anglais courant |
| english |
hedonic pricing in 2 sentences |
key terms, ≤3 sentences |
| json |
strict JSON object about Gatineau |
JSON.parse + typed keys |
| tools |
get_weather function call |
tool call emitted with location=Gatineau |
| vision |
PNG with 3 red circles + blue square |
circles=3 and square named |
| longctx |
~12k-token document, needle at 42% |
code 7431-ALPHA returned |
| streaming |
haiku, streamed |
time-to-first-token measured |
Settings: provider defaults (OpenAI Responses API; Anthropic adaptive thinking where supported), max 8 000 output tokens (6 000 for *-pro models, which ran a reduced 4-test suite), one run per test. Single-shot results are indicative, not statistically robust.
# Verified chat models — ranked by quality score
Quality score = share of the 9 graded tests passed (reasoning, math, code, french, english, json, tools, vision, longctx). Latency = median over text tests. Prices in $/1M tokens (input / output).
| Model |
Provider |
Quality |
p50 latency |
p50 TTFT |
Price in/out |
Blended $/1M* |
Failures / errors |
| claude-haiku-5-5 |
anthropic |
100% |
883 ms |
637 ms |
$0.1 / $0.5 |
$0.20 |
— |
| claude-haiku-4-5-20251001 |
anthropic |
100% |
1119 ms |
405 ms |
$1 / $5 |
$2.00 |
— |
| claude-sonnet-5 |
anthropic |
100% |
1168 ms |
528 ms |
$2 / $10 |
$4.00 |
— |
| chat-latest |
openai |
100% |
1215 ms |
547 ms |
$5 / $30 |
$11.25 |
— |
| claude-fable-5-1 |
anthropic |
100% |
1321 ms |
364 ms |
$10 / $50 |
$20.00 |
— |
| claude-sonnet-5-5 |
anthropic |
100% |
1496 ms |
586 ms |
$2 / $10 |
$4.00 |
— |
| gpt-6-sol |
openai |
100% |
1516 ms |
1608 ms |
$2 / $10 |
$4.00 |
— |
| gpt-5.6-sol |
openai |
100% |
1527 ms |
2268 ms |
$4 / $20 |
$8.00 |
— |
| gpt-5.6-luna |
openai |
100% |
1581 ms |
3271 ms |
$0.2 / $1.2 |
$0.45 |
— |
| o3 |
openai |
100% |
1643 ms |
660 ms |
$2 / $8 |
$3.50 |
— |
| claude-opus-4-7 |
anthropic |
100% |
1659 ms |
537 ms |
$5 / $25 |
$10.00 |
— |
| claude-opus-5-5 |
anthropic |
100% |
1742 ms |
954 ms |
$4 / $20 |
$8.00 |
— |
| claude-opus-4-8 |
anthropic |
100% |
1887 ms |
622 ms |
$5 / $25 |
$10.00 |
— |
| claude-sonnet-4-5-20250929 |
anthropic |
100% |
1934 ms |
785 ms |
$3 / $15 |
$6.00 |
— |
| gpt-6-luna |
openai |
100% |
2005 ms |
2952 ms |
$0.1 / $0.5 |
$0.20 |
— |
| gpt-6.1-sol |
openai |
100% |
2020 ms |
3051 ms |
$2 / $10 |
$4.00 |
— |
| gpt-5.5 |
openai |
100% |
2089 ms |
730 ms |
$5 / $30 |
$11.25 |
— |
| gpt-6-astra |
openai |
100% |
2190 ms |
3799 ms |
$10 / $50 |
$20.00 |
— |
| gpt-6.1-terra |
openai |
100% |
2208 ms |
1841 ms |
$1 / $5 |
$2.00 |
— |
| claude-opus-5 |
anthropic |
100% |
2242 ms |
1136 ms |
$5 / $25 |
$10.00 |
— |
| claude-fable-5 |
anthropic |
100% |
2691 ms |
1862 ms |
$10 / $50 |
$20.00 |
— |
| claude-sonnet-4-6 |
anthropic |
100% |
2909 ms |
1732 ms |
$3 / $15 |
$6.00 |
— |
| gpt-5-nano |
openai |
100% |
3034 ms |
10356 ms |
$0.05 / $0.4 |
$0.14 |
— |
| claude-opus-4-6 |
anthropic |
100% |
3613 ms |
2309 ms |
$5 / $25 |
$10.00 |
— |
| gpt-5-mini |
openai |
100% |
3711 ms |
10075 ms |
$0.25 / $2 |
$0.69 |
— |
| gpt-5 |
openai |
100% |
3814 ms |
7922 ms |
$1.25 / $10 |
$3.44 |
— |
| gpt-5.5-pro |
openai |
100% |
5531 ms |
— ms |
$30 / $180 |
$67.50 |
— |
| gpt-5.2-pro |
openai |
100% |
6297 ms |
— ms |
$21 / $168 |
$57.75 |
— |
| o3-pro |
openai |
100% |
6627 ms |
— ms |
$20 / $80 |
$35.00 |
— |
| gpt-5.4-pro |
openai |
100% |
14838 ms |
— ms |
$30 / $180 |
$67.50 |
— |
| gpt-5-pro |
openai |
100% |
47132 ms |
— ms |
$15 / $120 |
$41.25 |
— |
| gpt-4.1 |
openai |
89% |
704 ms |
413 ms |
$2 / $8 |
$3.50 |
math |
| gpt-5.4-mini |
openai |
89% |
888 ms |
458 ms |
$0.75 / $4.5 |
$1.69 |
math |
| gpt-4.1-mini |
openai |
89% |
897 ms |
343 ms |
$0.4 / $1.6 |
$0.70 |
math |
| gpt-5.1 |
openai |
89% |
1084 ms |
610 ms |
$1.25 / $10 |
$3.44 |
math |
| gpt-5.3-codex |
openai |
89% |
1209 ms |
545 ms |
$1.75 / $14 |
$4.81 |
reasoning |
| gpt-5.6-terra |
openai |
89% |
1214 ms |
611 ms |
$2 / $12 |
$4.50 |
longctx |
| gpt-5.4 |
openai |
89% |
1466 ms |
651 ms |
$2.5 / $15 |
$5.63 |
reasoning |
| claude-opus-4-5-20251101 |
anthropic |
89% |
1940 ms |
806 ms |
$5 / $25 |
$10.00 |
longctx |
| gpt-4o-mini |
openai |
78% |
893 ms |
482 ms |
$0.15 / $0.6 |
$0.26 |
reasoning, math |
| gpt-5.4-nano |
openai |
78% |
969 ms |
541 ms |
$0.2 / $1.25 |
$0.46 |
reasoning, math |
| gpt-4o |
openai |
78% |
1049 ms |
457 ms |
$2.5 / $10 |
$4.38 |
reasoning, math |
| gpt-5.2 |
openai |
78% |
1506 ms |
529 ms |
$1.75 / $14 |
$4.81 |
reasoning, longctx |
| gpt-5-search-api |
openai |
67% |
1707 ms |
1073 ms |
$1.25 / $10 |
$3.44 |
math, tools⚠, vision |
* Blended = (3 × input + output) / 4, a simple comparison heuristic, not a billing figure.
# Per-test results
| Model |
reasoning |
math |
code |
french |
english |
json |
tools |
vision |
longctx |
basic |
streaming |
| claude-fable-5 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-fable-5-1 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-haiku-4-5-20251001 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-haiku-5-5 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-opus-4-6 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-opus-4-7 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-opus-4-8 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-opus-5 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-opus-5-5 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-sonnet-4-5-20250929 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-sonnet-4-6 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-sonnet-5 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-sonnet-5-5 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| claude-opus-4-5-20251101 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
❌ |
✅ |
✅ |
| chat-latest |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5-mini |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5-nano |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5-pro |
✅ |
· |
· |
· |
· |
✅ |
✅ |
· |
· |
✅ |
· |
| gpt-5.2-pro |
✅ |
· |
· |
· |
· |
✅ |
✅ |
· |
· |
✅ |
· |
| gpt-5.4-pro |
✅ |
· |
· |
· |
· |
✅ |
✅ |
· |
· |
✅ |
· |
| gpt-5.5 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5.5-pro |
✅ |
· |
· |
· |
· |
✅ |
✅ |
· |
· |
✅ |
· |
| gpt-5.6-luna |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5.6-sol |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-6-astra |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-6-luna |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-6-sol |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-6.1-sol |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-6.1-terra |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| o3 |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| o3-pro |
✅ |
· |
· |
· |
· |
✅ |
✅ |
· |
· |
✅ |
· |
| gpt-4.1 |
✅ |
❌ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-4.1-mini |
✅ |
❌ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5.1 |
✅ |
❌ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5.3-codex |
❌ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5.4 |
❌ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5.4-mini |
✅ |
❌ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5.6-terra |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
❌ |
✅ |
✅ |
| gpt-4o |
❌ |
❌ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-4o-mini |
❌ |
❌ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5.2 |
❌ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
❌ |
✅ |
✅ |
| gpt-5.4-nano |
❌ |
❌ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
| gpt-5-search-api |
✅ |
❌ |
✅ |
✅ |
✅ |
✅ |
⚠️ |
❌ |
✅ |
✅ |
✅ |
# Listed but unavailable
These ids are returned by the provider's model listing but reject inference. The app marks them unavailable and hides them by default.
| Model |
Provider |
Reason |
| gpt-3.5-turbo |
openai |
The model gpt-3.5-turbo has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-4 |
openai |
The model gpt-4 has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-4-turbo |
openai |
The model gpt-4-turbo has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-4.1-nano |
openai |
The model gpt-4.1-nano has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-4o-mini-search-preview |
openai |
The model gpt-4o-mini-search-preview has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-4o-search-preview |
openai |
The model gpt-4o-search-preview has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5-chat-latest |
openai |
The model gpt-5-chat-latest has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5-codex |
openai |
The model gpt-5-codex has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.1-chat-latest |
openai |
The model gpt-5.1-chat-latest has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.1-codex |
openai |
The model gpt-5.1-codex has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.1-codex-max |
openai |
The model gpt-5.1-codex-max has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.1-codex-mini |
openai |
The model gpt-5.1-codex-mini has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.2-chat-latest |
openai |
The model gpt-5.2-chat-latest has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.2-codex |
openai |
The model gpt-5.2-codex has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.3-chat-latest |
openai |
The model gpt-5.3-chat-latest has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-audio |
openai |
400 400 This model requires that either input content or output modality contain audio. |
| gpt-audio-1.5 |
openai |
400 400 This model requires that either input content or output modality contain audio. |
| gpt-audio-mini |
openai |
400 400 This model requires that either input content or output modality contain audio. |
| gpt-image-1 |
openai |
400 400 The model 'gpt-image-1' does not exist. |
| o1 |
openai |
The model o1 has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| o1-pro |
openai |
The model o1-pro has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| o3-mini |
openai |
The model o3-mini has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| o4-mini |
openai |
The model o4-mini has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
# Specialized models (modality tests)
| Model |
Kind |
Result |
Latency |
Notes |
| chatgpt-image-latest |
image |
✅ verified |
5382 ms |
tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-4o-mini-transcribe |
stt |
✅ verified |
412 ms |
"Bonjour, ceci est un test de synthèse vocale pour Chat SP Bo" |
| gpt-4o-mini-tts |
tts |
✅ verified |
542 ms |
|
| gpt-4o-transcribe |
stt |
✅ verified |
625 ms |
"Bonjour, ceci est un test de synthèse vocale pour Chat SP Bo" |
| gpt-4o-transcribe-diarize |
stt |
✅ verified |
2799 ms |
"Bonjour, ceci est un test de synthèse vocale pour Chat spbou" |
| gpt-audio |
audio-chat |
❌ 400 400 This model requires that either input content or output modality contain audio. |
122 ms |
|
| gpt-audio-1.5 |
audio-chat |
❌ 400 400 This model requires that either input content or output modality contain audio. |
103 ms |
|
| gpt-audio-mini |
audio-chat |
❌ 400 400 This model requires that either input content or output modality contain audio. |
132 ms |
|
| gpt-image-1 |
image |
❌ 400 400 The model 'gpt-image-1' does not exist. |
174 ms |
|
| gpt-image-1-mini |
image |
✅ verified |
5797 ms |
tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-image-1.5 |
image |
✅ verified |
4864 ms |
tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-image-2 |
image |
✅ verified |
6375 ms |
tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-image-2.5-flare |
image |
✅ verified |
4765 ms |
tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-image-2.5-sunburst |
image |
✅ verified |
6665 ms |
tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-transcribe |
stt |
✅ verified |
459 ms |
"Bonjour, ceci est un test de synthèse vocale pour Chat SPBoc" |
| omni-moderation-latest |
moderation |
✅ verified |
521 ms |
|
| text-embedding-3-large |
embedding |
✅ verified |
215 ms |
3072 dims |
| text-embedding-3-small |
embedding |
✅ verified |
319 ms |
1536 dims |
| text-embedding-ada-002 |
embedding |
✅ verified |
162 ms |
1536 dims |
| tts-1 |
tts |
✅ verified |
2195 ms |
|
| tts-1-hd |
tts |
✅ verified |
1689 ms |
|
| whisper-1 |
stt |
✅ verified |
642 ms |
"Bonjour, ceci est un test de synthèse vocale pour Tchat, SP " |
# Not exercised
| Model |
Kind |
Why |
| babbage-002 |
legacy-completion |
Legacy completions endpoint — not used by this app |
| davinci-002 |
legacy-completion |
Legacy completions endpoint — not used by this app |
| gpt-3.5-turbo-instruct |
legacy-completion |
Legacy completions endpoint — not used by this app |
| gpt-3.5-turbo-instruct-0914 |
legacy-completion |
Legacy completions endpoint — not used by this app |
| gpt-live-1 |
realtime |
WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-live-transcribe |
realtime |
WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime |
realtime |
WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-1.5 |
realtime |
WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-2 |
realtime |
WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-2.1 |
realtime |
WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-2.1-mini |
realtime |
WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-mini |
realtime |
WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-translate |
realtime |
WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-whisper |
realtime |
WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| o3-deep-research |
deep-research |
Requires hosted web-search tools and long runs — not benchmarked without explicit budget approval |
| o4-mini-deep-research |
deep-research |
Requires hosted web-search tools and long runs — not benchmarked without explicit budget approval |
| sora-2 |
video |
Video generation — not tested (cost) |
| sora-2-pro |
video |
Video generation — not tested (cost) |
| tts-1-1106 |
tts |
not tested |
| tts-1-hd-1106 |
tts |
not tested |
# Errors encountered on verified models
# Limitations
- One run per test: latency and pass/fail are single observations.
- Reasoning models spend hidden reasoning tokens; their latency and cost reflect default effort.
- Pricing was read from the public pricing pages on 2026-11-01; models missing from those pages have no cost estimate.
- The long-context test uses ~12k tokens, far below the advertised windows; it checks retrieval, not window size.