# Chat SPBoucher AI — Benchmark report

Generated 2026-11-01T07:22:05.282Z. Every figure below comes from a real API response recorded in `openai-results.json`, `anthropic-results.json` or `modality-results.json`. Prices come from the official pricing pages (see `pricing.json`). Nothing is estimated from memory.

## Summary

| | OpenAI | Anthropic |
|---|---|---|
| Models listed by the API | 140 | 14 |
| Chat models benchmarked (verified) | 30 | 14 |
| Listed but unavailable (404/deprecated) | 23 | 0 |
| Specialized models verified (image/tts/stt/embedding/moderation) | 18 | 0 |
| Not exercised (realtime/video/legacy/deep-research) | 20 | 0 |

Total benchmark spend (estimated from usage × verified prices): **$2.41**.

## Test suite

| Test | What is measured | Grading |
|---|---|---|
| basic | capital of Canada | contains "Ottawa" |
| reasoning | sisters/brothers riddle | final answer 3 |
| math | 17·23+144/12 and Σ first 50 odd | a=403;b=2500 |
| code | isPrime(n) in JS | code executed with node against 8 cases |
| french | 2 phrases sur la méthode du coût | vocabulaire FR, aucun mot anglais courant |
| english | hedonic pricing in 2 sentences | key terms, ≤3 sentences |
| json | strict JSON object about Gatineau | JSON.parse + typed keys |
| tools | get_weather function call | tool call emitted with location=Gatineau |
| vision | PNG with 3 red circles + blue square | circles=3 and square named |
| longctx | ~12k-token document, needle at 42% | code 7431-ALPHA returned |
| streaming | haiku, streamed | time-to-first-token measured |

Settings: provider defaults (OpenAI Responses API; Anthropic adaptive thinking where supported), max 8 000 output tokens (6 000 for *-pro models, which ran a reduced 4-test suite), one run per test. Single-shot results are indicative, not statistically robust.

## Verified chat models — ranked by quality score

Quality score = share of the 9 graded tests passed (reasoning, math, code, french, english, json, tools, vision, longctx). Latency = median over text tests. Prices in $/1M tokens (input / output).

| Model | Provider | Quality | p50 latency | p50 TTFT | Price in/out | Blended $/1M* | Failures / errors |
|---|---|---|---|---|---|---|---|
| claude-haiku-5-5 | anthropic | 100% | 883 ms | 637 ms | $0.1 / $0.5 | $0.20 | — |
| claude-haiku-4-5-20251001 | anthropic | 100% | 1119 ms | 405 ms | $1 / $5 | $2.00 | — |
| claude-sonnet-5 | anthropic | 100% | 1168 ms | 528 ms | $2 / $10 | $4.00 | — |
| chat-latest | openai | 100% | 1215 ms | 547 ms | $5 / $30 | $11.25 | — |
| claude-fable-5-1 | anthropic | 100% | 1321 ms | 364 ms | $10 / $50 | $20.00 | — |
| claude-sonnet-5-5 | anthropic | 100% | 1496 ms | 586 ms | $2 / $10 | $4.00 | — |
| gpt-6-sol | openai | 100% | 1516 ms | 1608 ms | $2 / $10 | $4.00 | — |
| gpt-5.6-sol | openai | 100% | 1527 ms | 2268 ms | $4 / $20 | $8.00 | — |
| gpt-5.6-luna | openai | 100% | 1581 ms | 3271 ms | $0.2 / $1.2 | $0.45 | — |
| o3 | openai | 100% | 1643 ms | 660 ms | $2 / $8 | $3.50 | — |
| claude-opus-4-7 | anthropic | 100% | 1659 ms | 537 ms | $5 / $25 | $10.00 | — |
| claude-opus-5-5 | anthropic | 100% | 1742 ms | 954 ms | $4 / $20 | $8.00 | — |
| claude-opus-4-8 | anthropic | 100% | 1887 ms | 622 ms | $5 / $25 | $10.00 | — |
| claude-sonnet-4-5-20250929 | anthropic | 100% | 1934 ms | 785 ms | $3 / $15 | $6.00 | — |
| gpt-6-luna | openai | 100% | 2005 ms | 2952 ms | $0.1 / $0.5 | $0.20 | — |
| gpt-6.1-sol | openai | 100% | 2020 ms | 3051 ms | $2 / $10 | $4.00 | — |
| gpt-5.5 | openai | 100% | 2089 ms | 730 ms | $5 / $30 | $11.25 | — |
| gpt-6-astra | openai | 100% | 2190 ms | 3799 ms | $10 / $50 | $20.00 | — |
| gpt-6.1-terra | openai | 100% | 2208 ms | 1841 ms | $1 / $5 | $2.00 | — |
| claude-opus-5 | anthropic | 100% | 2242 ms | 1136 ms | $5 / $25 | $10.00 | — |
| claude-fable-5 | anthropic | 100% | 2691 ms | 1862 ms | $10 / $50 | $20.00 | — |
| claude-sonnet-4-6 | anthropic | 100% | 2909 ms | 1732 ms | $3 / $15 | $6.00 | — |
| gpt-5-nano | openai | 100% | 3034 ms | 10356 ms | $0.05 / $0.4 | $0.14 | — |
| claude-opus-4-6 | anthropic | 100% | 3613 ms | 2309 ms | $5 / $25 | $10.00 | — |
| gpt-5-mini | openai | 100% | 3711 ms | 10075 ms | $0.25 / $2 | $0.69 | — |
| gpt-5 | openai | 100% | 3814 ms | 7922 ms | $1.25 / $10 | $3.44 | — |
| gpt-5.5-pro | openai | 100% | 5531 ms | — ms | $30 / $180 | $67.50 | — |
| gpt-5.2-pro | openai | 100% | 6297 ms | — ms | $21 / $168 | $57.75 | — |
| o3-pro | openai | 100% | 6627 ms | — ms | $20 / $80 | $35.00 | — |
| gpt-5.4-pro | openai | 100% | 14838 ms | — ms | $30 / $180 | $67.50 | — |
| gpt-5-pro | openai | 100% | 47132 ms | — ms | $15 / $120 | $41.25 | — |
| gpt-4.1 | openai | 89% | 704 ms | 413 ms | $2 / $8 | $3.50 | math |
| gpt-5.4-mini | openai | 89% | 888 ms | 458 ms | $0.75 / $4.5 | $1.69 | math |
| gpt-4.1-mini | openai | 89% | 897 ms | 343 ms | $0.4 / $1.6 | $0.70 | math |
| gpt-5.1 | openai | 89% | 1084 ms | 610 ms | $1.25 / $10 | $3.44 | math |
| gpt-5.3-codex | openai | 89% | 1209 ms | 545 ms | $1.75 / $14 | $4.81 | reasoning |
| gpt-5.6-terra | openai | 89% | 1214 ms | 611 ms | $2 / $12 | $4.50 | longctx |
| gpt-5.4 | openai | 89% | 1466 ms | 651 ms | $2.5 / $15 | $5.63 | reasoning |
| claude-opus-4-5-20251101 | anthropic | 89% | 1940 ms | 806 ms | $5 / $25 | $10.00 | longctx |
| gpt-4o-mini | openai | 78% | 893 ms | 482 ms | $0.15 / $0.6 | $0.26 | reasoning, math |
| gpt-5.4-nano | openai | 78% | 969 ms | 541 ms | $0.2 / $1.25 | $0.46 | reasoning, math |
| gpt-4o | openai | 78% | 1049 ms | 457 ms | $2.5 / $10 | $4.38 | reasoning, math |
| gpt-5.2 | openai | 78% | 1506 ms | 529 ms | $1.75 / $14 | $4.81 | reasoning, longctx |
| gpt-5-search-api | openai | 67% | 1707 ms | 1073 ms | $1.25 / $10 | $3.44 | math, tools⚠, vision |

\* Blended = (3 × input + output) / 4, a simple comparison heuristic, not a billing figure.

## Per-test results

| Model | reasoning | math | code | french | english | json | tools | vision | longctx | basic | streaming |
|---|---|---|---|---|---|---|---|---|---|---|---|
| claude-fable-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-fable-5-1 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-haiku-4-5-20251001 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-haiku-5-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-opus-4-6 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-opus-4-7 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-opus-4-8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-opus-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-opus-5-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-sonnet-4-5-20250929 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-sonnet-4-6 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-sonnet-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-sonnet-5-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| claude-opus-4-5-20251101 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
| chat-latest | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5-mini | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5-nano | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5-pro | ✅ | · | · | · | · | ✅ | ✅ | · | · | ✅ | · |
| gpt-5.2-pro | ✅ | · | · | · | · | ✅ | ✅ | · | · | ✅ | · |
| gpt-5.4-pro | ✅ | · | · | · | · | ✅ | ✅ | · | · | ✅ | · |
| gpt-5.5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5.5-pro | ✅ | · | · | · | · | ✅ | ✅ | · | · | ✅ | · |
| gpt-5.6-luna | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5.6-sol | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-6-astra | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-6-luna | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-6-sol | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-6.1-sol | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-6.1-terra | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| o3 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| o3-pro | ✅ | · | · | · | · | ✅ | ✅ | · | · | ✅ | · |
| gpt-4.1 | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-4.1-mini | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5.1 | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5.3-codex | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5.4 | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5.4-mini | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5.6-terra | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
| gpt-4o | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-4o-mini | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5.2 | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
| gpt-5.4-nano | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| gpt-5-search-api | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ⚠️ | ❌ | ✅ | ✅ | ✅ |

## Listed but unavailable

These ids are returned by the provider's model listing but reject inference. The app marks them **unavailable** and hides them by default.

| Model | Provider | Reason |
|---|---|---|
| gpt-3.5-turbo | openai | The model `gpt-3.5-turbo` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-4 | openai | The model `gpt-4` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-4-turbo | openai | The model `gpt-4-turbo` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-4.1-nano | openai | The model `gpt-4.1-nano` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-4o-mini-search-preview | openai | The model `gpt-4o-mini-search-preview` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-4o-search-preview | openai | The model `gpt-4o-search-preview` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5-chat-latest | openai | The model `gpt-5-chat-latest` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5-codex | openai | The model `gpt-5-codex` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.1-chat-latest | openai | The model `gpt-5.1-chat-latest` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.1-codex | openai | The model `gpt-5.1-codex` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.1-codex-max | openai | The model `gpt-5.1-codex-max` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.1-codex-mini | openai | The model `gpt-5.1-codex-mini` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.2-chat-latest | openai | The model `gpt-5.2-chat-latest` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.2-codex | openai | The model `gpt-5.2-codex` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-5.3-chat-latest | openai | The model `gpt-5.3-chat-latest` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| gpt-audio | openai | 400 400 This model requires that either input content or output modality contain audio. |
| gpt-audio-1.5 | openai | 400 400 This model requires that either input content or output modality contain audio. |
| gpt-audio-mini | openai | 400 400 This model requires that either input content or output modality contain audio. |
| gpt-image-1 | openai | 400 400 The model 'gpt-image-1' does not exist. |
| o1 | openai | The model `o1` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| o1-pro | openai | The model `o1-pro` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| o3-mini | openai | The model `o3-mini` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |
| o4-mini | openai | The model `o4-mini` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |

## Specialized models (modality tests)

| Model | Kind | Result | Latency | Notes |
|---|---|---|---|---|
| chatgpt-image-latest | image | ✅ verified | 5382 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-4o-mini-transcribe | stt | ✅ verified | 412 ms | "Bonjour, ceci est un test de synthèse vocale pour Chat SP Bo" |
| gpt-4o-mini-tts | tts | ✅ verified | 542 ms |  |
| gpt-4o-transcribe | stt | ✅ verified | 625 ms | "Bonjour, ceci est un test de synthèse vocale pour Chat SP Bo" |
| gpt-4o-transcribe-diarize | stt | ✅ verified | 2799 ms | "Bonjour, ceci est un test de synthèse vocale pour Chat spbou" |
| gpt-audio | audio-chat | ❌ 400 400 This model requires that either input content or output modality contain audio. | 122 ms |  |
| gpt-audio-1.5 | audio-chat | ❌ 400 400 This model requires that either input content or output modality contain audio. | 103 ms |  |
| gpt-audio-mini | audio-chat | ❌ 400 400 This model requires that either input content or output modality contain audio. | 132 ms |  |
| gpt-image-1 | image | ❌ 400 400 The model 'gpt-image-1' does not exist. | 174 ms |  |
| gpt-image-1-mini | image | ✅ verified | 5797 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-image-1.5 | image | ✅ verified | 4864 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-image-2 | image | ✅ verified | 6375 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-image-2.5-flare | image | ✅ verified | 4765 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-image-2.5-sunburst | image | ✅ verified | 6665 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |
| gpt-transcribe | stt | ✅ verified | 459 ms | "Bonjour, ceci est un test de synthèse vocale pour Chat SPBoc" |
| omni-moderation-latest | moderation | ✅ verified | 521 ms |  |
| text-embedding-3-large | embedding | ✅ verified | 215 ms | 3072 dims |
| text-embedding-3-small | embedding | ✅ verified | 319 ms | 1536 dims |
| text-embedding-ada-002 | embedding | ✅ verified | 162 ms | 1536 dims |
| tts-1 | tts | ✅ verified | 2195 ms |  |
| tts-1-hd | tts | ✅ verified | 1689 ms |  |
| whisper-1 | stt | ✅ verified | 642 ms | "Bonjour, ceci est un test de synthèse vocale pour Tchat, SP " |

## Not exercised

| Model | Kind | Why |
|---|---|---|
| babbage-002 | legacy-completion | Legacy completions endpoint — not used by this app |
| davinci-002 | legacy-completion | Legacy completions endpoint — not used by this app |
| gpt-3.5-turbo-instruct | legacy-completion | Legacy completions endpoint — not used by this app |
| gpt-3.5-turbo-instruct-0914 | legacy-completion | Legacy completions endpoint — not used by this app |
| gpt-live-1 | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-live-transcribe | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-1.5 | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-2 | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-2.1 | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-2.1-mini | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-mini | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-translate | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| gpt-realtime-whisper | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |
| o3-deep-research | deep-research | Requires hosted web-search tools and long runs — not benchmarked without explicit budget approval |
| o4-mini-deep-research | deep-research | Requires hosted web-search tools and long runs — not benchmarked without explicit budget approval |
| sora-2 | video | Video generation — not tested (cost) |
| sora-2-pro | video | Video generation — not tested (cost) |
| tts-1-1106 | tts | not tested |
| tts-1-hd-1106 | tts | not tested |

## Errors encountered on verified models

- **gpt-5-search-api**: tools: 404 404 tools is not supported in this model. For a list of supported models, refer to https://platform.openai.com/docs/guides/function-call

## Limitations

- One run per test: latency and pass/fail are single observations.
- Reasoning models spend hidden reasoning tokens; their latency and cost reflect default effort.
- Pricing was read from the public pricing pages on 2026-11-01; models missing from those pages have no cost estimate.
- The long-context test uses ~12k tokens, far below the advertised windows; it checks retrieval, not window size.