spb/chat-spboucher-ai

1# Chat SPBoucher AI — Benchmark report23Generated 2026-11-01T07:22:05.282Z. Every figure below comes from a real API response recorded in `openai-results.json`, `anthropic-results.json` or `modality-results.json`. Prices come from the official pricing pages (see `pricing.json`). Nothing is estimated from memory.45## Summary67| | OpenAI | Anthropic |8|---|---|---|9| Models listed by the API | 140 | 14 |10| Chat models benchmarked (verified) | 30 | 14 |11| Listed but unavailable (404/deprecated) | 23 | 0 |12| Specialized models verified (image/tts/stt/embedding/moderation) | 18 | 0 |13| Not exercised (realtime/video/legacy/deep-research) | 20 | 0 |1415Total benchmark spend (estimated from usage × verified prices): **$2.41**.1617## Test suite1819| Test | What is measured | Grading |20|---|---|---|21| basic | capital of Canada | contains "Ottawa" |22| reasoning | sisters/brothers riddle | final answer 3 |23| math | 17·23+144/12 and Σ first 50 odd | a=403;b=2500 |24| code | isPrime(n) in JS | code executed with node against 8 cases |25| french | 2 phrases sur la méthode du coût | vocabulaire FR, aucun mot anglais courant |26| english | hedonic pricing in 2 sentences | key terms, ≤3 sentences |27| json | strict JSON object about Gatineau | JSON.parse + typed keys |28| tools | get_weather function call | tool call emitted with location=Gatineau |29| vision | PNG with 3 red circles + blue square | circles=3 and square named |30| longctx | ~12k-token document, needle at 42% | code 7431-ALPHA returned |31| streaming | haiku, streamed | time-to-first-token measured |3233Settings: provider defaults (OpenAI Responses API; Anthropic adaptive thinking where supported), max 8 000 output tokens (6 000 for *-pro models, which ran a reduced 4-test suite), one run per test. Single-shot results are indicative, not statistically robust.3435## Verified chat models — ranked by quality score3637Quality score = share of the 9 graded tests passed (reasoning, math, code, french, english, json, tools, vision, longctx). Latency = median over text tests. Prices in $/1M tokens (input / output).3839| Model | Provider | Quality | p50 latency | p50 TTFT | Price in/out | Blended $/1M* | Failures / errors |40|---|---|---|---|---|---|---|---|41| claude-haiku-5-5 | anthropic | 100% | 883 ms | 637 ms | $0.1 / $0.5 | $0.20 | — |42| claude-haiku-4-5-20251001 | anthropic | 100% | 1119 ms | 405 ms | $1 / $5 | $2.00 | — |43| claude-sonnet-5 | anthropic | 100% | 1168 ms | 528 ms | $2 / $10 | $4.00 | — |44| chat-latest | openai | 100% | 1215 ms | 547 ms | $5 / $30 | $11.25 | — |45| claude-fable-5-1 | anthropic | 100% | 1321 ms | 364 ms | $10 / $50 | $20.00 | — |46| claude-sonnet-5-5 | anthropic | 100% | 1496 ms | 586 ms | $2 / $10 | $4.00 | — |47| gpt-6-sol | openai | 100% | 1516 ms | 1608 ms | $2 / $10 | $4.00 | — |48| gpt-5.6-sol | openai | 100% | 1527 ms | 2268 ms | $4 / $20 | $8.00 | — |49| gpt-5.6-luna | openai | 100% | 1581 ms | 3271 ms | $0.2 / $1.2 | $0.45 | — |50| o3 | openai | 100% | 1643 ms | 660 ms | $2 / $8 | $3.50 | — |51| claude-opus-4-7 | anthropic | 100% | 1659 ms | 537 ms | $5 / $25 | $10.00 | — |52| claude-opus-5-5 | anthropic | 100% | 1742 ms | 954 ms | $4 / $20 | $8.00 | — |53| claude-opus-4-8 | anthropic | 100% | 1887 ms | 622 ms | $5 / $25 | $10.00 | — |54| claude-sonnet-4-5-20250929 | anthropic | 100% | 1934 ms | 785 ms | $3 / $15 | $6.00 | — |55| gpt-6-luna | openai | 100% | 2005 ms | 2952 ms | $0.1 / $0.5 | $0.20 | — |56| gpt-6.1-sol | openai | 100% | 2020 ms | 3051 ms | $2 / $10 | $4.00 | — |57| gpt-5.5 | openai | 100% | 2089 ms | 730 ms | $5 / $30 | $11.25 | — |58| gpt-6-astra | openai | 100% | 2190 ms | 3799 ms | $10 / $50 | $20.00 | — |59| gpt-6.1-terra | openai | 100% | 2208 ms | 1841 ms | $1 / $5 | $2.00 | — |60| claude-opus-5 | anthropic | 100% | 2242 ms | 1136 ms | $5 / $25 | $10.00 | — |61| claude-fable-5 | anthropic | 100% | 2691 ms | 1862 ms | $10 / $50 | $20.00 | — |62| claude-sonnet-4-6 | anthropic | 100% | 2909 ms | 1732 ms | $3 / $15 | $6.00 | — |63| gpt-5-nano | openai | 100% | 3034 ms | 10356 ms | $0.05 / $0.4 | $0.14 | — |64| claude-opus-4-6 | anthropic | 100% | 3613 ms | 2309 ms | $5 / $25 | $10.00 | — |65| gpt-5-mini | openai | 100% | 3711 ms | 10075 ms | $0.25 / $2 | $0.69 | — |66| gpt-5 | openai | 100% | 3814 ms | 7922 ms | $1.25 / $10 | $3.44 | — |67| gpt-5.5-pro | openai | 100% | 5531 ms | — ms | $30 / $180 | $67.50 | — |68| gpt-5.2-pro | openai | 100% | 6297 ms | — ms | $21 / $168 | $57.75 | — |69| o3-pro | openai | 100% | 6627 ms | — ms | $20 / $80 | $35.00 | — |70| gpt-5.4-pro | openai | 100% | 14838 ms | — ms | $30 / $180 | $67.50 | — |71| gpt-5-pro | openai | 100% | 47132 ms | — ms | $15 / $120 | $41.25 | — |72| gpt-4.1 | openai | 89% | 704 ms | 413 ms | $2 / $8 | $3.50 | math |73| gpt-5.4-mini | openai | 89% | 888 ms | 458 ms | $0.75 / $4.5 | $1.69 | math |74| gpt-4.1-mini | openai | 89% | 897 ms | 343 ms | $0.4 / $1.6 | $0.70 | math |75| gpt-5.1 | openai | 89% | 1084 ms | 610 ms | $1.25 / $10 | $3.44 | math |76| gpt-5.3-codex | openai | 89% | 1209 ms | 545 ms | $1.75 / $14 | $4.81 | reasoning |77| gpt-5.6-terra | openai | 89% | 1214 ms | 611 ms | $2 / $12 | $4.50 | longctx |78| gpt-5.4 | openai | 89% | 1466 ms | 651 ms | $2.5 / $15 | $5.63 | reasoning |79| claude-opus-4-5-20251101 | anthropic | 89% | 1940 ms | 806 ms | $5 / $25 | $10.00 | longctx |80| gpt-4o-mini | openai | 78% | 893 ms | 482 ms | $0.15 / $0.6 | $0.26 | reasoning, math |81| gpt-5.4-nano | openai | 78% | 969 ms | 541 ms | $0.2 / $1.25 | $0.46 | reasoning, math |82| gpt-4o | openai | 78% | 1049 ms | 457 ms | $2.5 / $10 | $4.38 | reasoning, math |83| gpt-5.2 | openai | 78% | 1506 ms | 529 ms | $1.75 / $14 | $4.81 | reasoning, longctx |84| gpt-5-search-api | openai | 67% | 1707 ms | 1073 ms | $1.25 / $10 | $3.44 | math, tools⚠, vision |8586\* Blended = (3 × input + output) / 4, a simple comparison heuristic, not a billing figure.8788## Per-test results8990| Model | reasoning | math | code | french | english | json | tools | vision | longctx | basic | streaming |91|---|---|---|---|---|---|---|---|---|---|---|---|92| claude-fable-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |93| claude-fable-5-1 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |94| claude-haiku-4-5-20251001 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |95| claude-haiku-5-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |96| claude-opus-4-6 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |97| claude-opus-4-7 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |98| claude-opus-4-8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |99| claude-opus-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |100| claude-opus-5-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |101| claude-sonnet-4-5-20250929 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |102| claude-sonnet-4-6 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |103| claude-sonnet-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |104| claude-sonnet-5-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |105| claude-opus-4-5-20251101 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |106| chat-latest | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |107| gpt-5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |108| gpt-5-mini | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |109| gpt-5-nano | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |110| gpt-5-pro | ✅ | · | · | · | · | ✅ | ✅ | · | · | ✅ | · |111| gpt-5.2-pro | ✅ | · | · | · | · | ✅ | ✅ | · | · | ✅ | · |112| gpt-5.4-pro | ✅ | · | · | · | · | ✅ | ✅ | · | · | ✅ | · |113| gpt-5.5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |114| gpt-5.5-pro | ✅ | · | · | · | · | ✅ | ✅ | · | · | ✅ | · |115| gpt-5.6-luna | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |116| gpt-5.6-sol | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |117| gpt-6-astra | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |118| gpt-6-luna | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |119| gpt-6-sol | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |120| gpt-6.1-sol | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |121| gpt-6.1-terra | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |122| o3 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |123| o3-pro | ✅ | · | · | · | · | ✅ | ✅ | · | · | ✅ | · |124| gpt-4.1 | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |125| gpt-4.1-mini | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |126| gpt-5.1 | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |127| gpt-5.3-codex | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |128| gpt-5.4 | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |129| gpt-5.4-mini | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |130| gpt-5.6-terra | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |131| gpt-4o | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |132| gpt-4o-mini | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |133| gpt-5.2 | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |134| gpt-5.4-nano | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |135| gpt-5-search-api | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ⚠️ | ❌ | ✅ | ✅ | ✅ |136137## Listed but unavailable138139These ids are returned by the provider's model listing but reject inference. The app marks them **unavailable** and hides them by default.140141| Model | Provider | Reason |142|---|---|---|143| gpt-3.5-turbo | openai | The model `gpt-3.5-turbo` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |144| gpt-4 | openai | The model `gpt-4` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |145| gpt-4-turbo | openai | The model `gpt-4-turbo` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |146| gpt-4.1-nano | openai | The model `gpt-4.1-nano` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |147| gpt-4o-mini-search-preview | openai | The model `gpt-4o-mini-search-preview` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |148| gpt-4o-search-preview | openai | The model `gpt-4o-search-preview` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |149| gpt-5-chat-latest | openai | The model `gpt-5-chat-latest` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |150| gpt-5-codex | openai | The model `gpt-5-codex` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |151| gpt-5.1-chat-latest | openai | The model `gpt-5.1-chat-latest` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |152| gpt-5.1-codex | openai | The model `gpt-5.1-codex` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |153| gpt-5.1-codex-max | openai | The model `gpt-5.1-codex-max` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |154| gpt-5.1-codex-mini | openai | The model `gpt-5.1-codex-mini` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |155| gpt-5.2-chat-latest | openai | The model `gpt-5.2-chat-latest` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |156| gpt-5.2-codex | openai | The model `gpt-5.2-codex` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |157| gpt-5.3-chat-latest | openai | The model `gpt-5.3-chat-latest` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |158| gpt-audio | openai | 400 400 This model requires that either input content or output modality contain audio. |159| gpt-audio-1.5 | openai | 400 400 This model requires that either input content or output modality contain audio. |160| gpt-audio-mini | openai | 400 400 This model requires that either input content or output modality contain audio. |161| gpt-image-1 | openai | 400 400 The model 'gpt-image-1' does not exist. |162| o1 | openai | The model `o1` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |163| o1-pro | openai | The model `o1-pro` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |164| o3-mini | openai | The model `o3-mini` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |165| o4-mini | openai | The model `o4-mini` has been deprecated, learn more here: https://platform.openai.com/docs/deprecations |166167## Specialized models (modality tests)168169| Model | Kind | Result | Latency | Notes |170|---|---|---|---|---|171| chatgpt-image-latest | image | ✅ verified | 5382 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |172| gpt-4o-mini-transcribe | stt | ✅ verified | 412 ms | "Bonjour, ceci est un test de synthèse vocale pour Chat SP Bo" |173| gpt-4o-mini-tts | tts | ✅ verified | 542 ms |  |174| gpt-4o-transcribe | stt | ✅ verified | 625 ms | "Bonjour, ceci est un test de synthèse vocale pour Chat SP Bo" |175| gpt-4o-transcribe-diarize | stt | ✅ verified | 2799 ms | "Bonjour, ceci est un test de synthèse vocale pour Chat spbou" |176| gpt-audio | audio-chat | ❌ 400 400 This model requires that either input content or output modality contain audio. | 122 ms |  |177| gpt-audio-1.5 | audio-chat | ❌ 400 400 This model requires that either input content or output modality contain audio. | 103 ms |  |178| gpt-audio-mini | audio-chat | ❌ 400 400 This model requires that either input content or output modality contain audio. | 132 ms |  |179| gpt-image-1 | image | ❌ 400 400 The model 'gpt-image-1' does not exist. | 174 ms |  |180| gpt-image-1-mini | image | ✅ verified | 5797 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |181| gpt-image-1.5 | image | ✅ verified | 4864 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |182| gpt-image-2 | image | ✅ verified | 6375 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |183| gpt-image-2.5-flare | image | ✅ verified | 4765 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |184| gpt-image-2.5-sunburst | image | ✅ verified | 6665 ms | tokens {"input_tokens":17,"input_tokens_details":{"image_tokens":0,"text_tokens":17},"o |185| gpt-transcribe | stt | ✅ verified | 459 ms | "Bonjour, ceci est un test de synthèse vocale pour Chat SPBoc" |186| omni-moderation-latest | moderation | ✅ verified | 521 ms |  |187| text-embedding-3-large | embedding | ✅ verified | 215 ms | 3072 dims |188| text-embedding-3-small | embedding | ✅ verified | 319 ms | 1536 dims |189| text-embedding-ada-002 | embedding | ✅ verified | 162 ms | 1536 dims |190| tts-1 | tts | ✅ verified | 2195 ms |  |191| tts-1-hd | tts | ✅ verified | 1689 ms |  |192| whisper-1 | stt | ✅ verified | 642 ms | "Bonjour, ceci est un test de synthèse vocale pour Tchat, SP " |193194## Not exercised195196| Model | Kind | Why |197|---|---|---|198| babbage-002 | legacy-completion | Legacy completions endpoint — not used by this app |199| davinci-002 | legacy-completion | Legacy completions endpoint — not used by this app |200| gpt-3.5-turbo-instruct | legacy-completion | Legacy completions endpoint — not used by this app |201| gpt-3.5-turbo-instruct-0914 | legacy-completion | Legacy completions endpoint — not used by this app |202| gpt-live-1 | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |203| gpt-live-transcribe | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |204| gpt-realtime | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |205| gpt-realtime-1.5 | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |206| gpt-realtime-2 | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |207| gpt-realtime-2.1 | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |208| gpt-realtime-2.1-mini | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |209| gpt-realtime-mini | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |210| gpt-realtime-translate | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |211| gpt-realtime-whisper | realtime | WebSocket/WebRTC realtime model — not a chat-completion model; not exercised |212| o3-deep-research | deep-research | Requires hosted web-search tools and long runs — not benchmarked without explicit budget approval |213| o4-mini-deep-research | deep-research | Requires hosted web-search tools and long runs — not benchmarked without explicit budget approval |214| sora-2 | video | Video generation — not tested (cost) |215| sora-2-pro | video | Video generation — not tested (cost) |216| tts-1-1106 | tts | not tested |217| tts-1-hd-1106 | tts | not tested |218219## Errors encountered on verified models220221- **gpt-5-search-api**: tools: 404 404 tools is not supported in this model. For a list of supported models, refer to https://platform.openai.com/docs/guides/function-call222223## Limitations224225- One run per test: latency and pass/fail are single observations.226- Reasoning models spend hidden reasoning tokens; their latency and cost reflect default effort.227- Pricing was read from the public pricing pages on 2026-11-01; models missing from those pages have no cost estimate.228- The long-context test uses ~12k tokens, far below the advertised windows; it checks retrieval, not window size.229