spb/note-spboucher-ai

1# AI model comparison — Note SPBoucher AI23Generated 2026-11-01T07:18:30.320Z. 836 real runs across 38 models; total estimated spend for Phase Zero ≈ $17.24.45## Ranking (deterministic mean score, then latency)67| # | Model | Provider | Pass | Score | Judged | TTFT | Latency | tok/s | $/run |8|---|---|---|---|---|---|---|---|---|---|9| 1 | `chat-latest` | openai | 100% | 1.00 | 5.00 | 741 ms | 1.4 s | 157 | $0.019 |10| 2 | `gpt-5.3-codex` | openai | 100% | 1.00 | 4.67 | 602 ms | 1.4 s | 94 | $0.012 |11| 3 | `gpt-4.1` | openai | 100% | 1.00 | 4.17 | 450 ms | 1.7 s | 163 | $0.012 |12| 4 | `gpt-6-sol` | openai | 100% | 1.00 | 5.00 | 1106 ms | 1.7 s | 149 | $0.0075 |13| 5 | `gpt-5.4` | openai | 100% | 1.00 | 5.00 | 716 ms | 1.8 s | 86 | $0.016 |14| 6 | `claude-fable-5-1` | anthropic | 100% | 1.00 | 5.00 | 389 ms | 2.1 s | 118 | $0.090 |15| 7 | `gpt-5.2` | openai | 100% | 1.00 | 4.33 | 676 ms | 2.1 s | 85 | $0.0074 |16| 8 | `claude-sonnet-5` | anthropic | 100% | 1.00 | 4.33 | 950 ms | 2.2 s | 145 | $0.019 |17| 9 | `gpt-5.1` | openai | 100% | 1.00 | 4.50 | 650 ms | 2.2 s | 139 | $0.0057 |18| 10 | `gpt-6.1-sol` | openai | 100% | 1.00 | 4.67 | 1444 ms | 2.2 s | 89 | $0.0070 |19| 11 | `claude-sonnet-4-5-20250929` | anthropic | 100% | 1.00 | 4.33 | 775 ms | 2.3 s | 90 | $0.022 |20| 12 | `claude-haiku-5-5` | anthropic | 100% | 1.00 | 4.83 | 1301 ms | 2.4 s | 487 | $0.0011 |21| 13 | `gpt-5.6-sol` | openai | 100% | 1.00 | 5.00 | 1108 ms | 2.5 s | 133 | $0.015 |22| 14 | `gpt-5.5` | openai | 100% | 1.00 | 4.83 | 1755 ms | 2.5 s | 147 | $0.021 |23| 15 | `claude-opus-4-5-20251101` | anthropic | 100% | 1.00 | 4.67 | 862 ms | 2.5 s | 98 | $0.036 |24| 16 | `claude-opus-5-5` | anthropic | 100% | 1.00 | 4.83 | 1703 ms | 2.8 s | 161 | $0.043 |25| 17 | `claude-sonnet-5-5` | anthropic | 100% | 1.00 | 4.67 | 1020 ms | 2.9 s | 202 | $0.020 |26| 18 | `claude-opus-4-8` | anthropic | 100% | 1.00 | 4.67 | 488 ms | 2.9 s | 104 | $0.045 |27| 19 | `o3` | openai | 100% | 1.00 | 4.33 | 1763 ms | 2.9 s | 533 | $0.015 |28| 20 | `gpt-6-astra` | openai | 100% | 1.00 | 5.00 | 2376 ms | 3.1 s | 78 | $0.037 |29| 21 | `gpt-6.1-terra` | openai | 100% | 1.00 | 4.67 | 1515 ms | 3.2 s | 93 | $0.0037 |30| 22 | `claude-opus-4-7` | anthropic | 100% | 1.00 | 4.33 | 687 ms | 3.3 s | 91 | $0.045 |31| 23 | `claude-opus-5` | anthropic | 100% | 1.00 | 4.67 | 2530 ms | 4.0 s | 161 | $0.051 |32| 24 | `claude-opus-4-6` | anthropic | 100% | 1.00 | 4.83 | 1622 ms | 4.2 s | 75 | $0.035 |33| 25 | `claude-sonnet-4-6` | anthropic | 100% | 1.00 | 5.00 | 735 ms | 4.7 s | 90 | $0.021 |34| 26 | `gpt-5` | openai | 100% | 1.00 | 4.33 | 4121 ms | 5.0 s | 962 | $0.012 |35| 27 | `gpt-5-mini` | openai | 100% | 1.00 | 4.83 | 4395 ms | 5.7 s | 375 | $0.0026 |36| 28 | `gpt-4o` | openai | 95% | 1.00 | 3.83 | 656 ms | 2.4 s | 106 | $0.015 |37| 29 | `gpt-5.6-luna` | openai | 95% | 0.98 | 4.67 | 1398 ms | 3.0 s | 136 | $0.0008 |38| 30 | `gpt-4.1-mini` | openai | 91% | 0.97 | 3.83 | 449 ms | 1.7 s | 153 | $0.0024 |39| 31 | `claude-haiku-4-5-20251001` | anthropic | 91% | 0.97 | 3.33 | 502 ms | 2.0 s | 135 | $0.0072 |40| 32 | `gpt-5-nano` | openai | 95% | 0.97 | 3.67 | 6133 ms | 6.8 s | 1362 | $0.0006 |41| 33 | `gpt-5.4-mini` | openai | 95% | 0.95 | 4.67 | 624 ms | 1.2 s | 167 | $0.0029 |42| 34 | `gpt-5.6-terra` | openai | 95% | 0.95 | 5.00 | 869 ms | 1.8 s | 126 | $0.0077 |43| 35 | `gpt-6-luna` | openai | 95% | 0.95 | 4.67 | 2045 ms | 3.0 s | 194 | $0.0007 |44| 36 | `gpt-4o-mini` | openai | 82% | 0.94 | 4.00 | 561 ms | 2.0 s | 116 | $0.0009 |45| 37 | `gpt-5.4-nano` | openai | 86% | 0.92 | 4.00 | 549 ms | 1.7 s | 135 | $0.0003 |46| 38 | `claude-fable-5` | anthropic | 91% | 0.91 | 5.00 | 1764 ms | 2.8 s | 112 | $0.092 |4748## Best model per task (deterministic score, ties → cheaper)4950| Task | Best | Score | Latency | $/run | Cheapest model with a perfect score |51|---|---|---|---|---|---|52| Document analysis | `gpt-6-luna` | 1.00 | 2.4 s | $0.0001 | `gpt-6-luna` ($0.0001) |53| JSON generation | `gpt-4o-mini` | 1.00 | 1.3 s | $0.0001 | `gpt-4o-mini` ($0.0001) |54| Complex reasoning | `gpt-6-luna` | 1.00 | 5.1 s | $0.0002 | `gpt-6-luna` ($0.0002) |55| Long-context retrieval | `gpt-5.4-nano` | 1.00 | 0.8 s | $0.0011 | `gpt-5.4-nano` ($0.0011) |56| English writing | `claude-haiku-5-5` | 1.00 | 2.7 s | $0.0002 | `claude-haiku-5-5` ($0.0002) |57| French writing | `gpt-4o-mini` | 1.00 | 2.5 s | $0.0002 | `gpt-4o-mini` ($0.0002) |58| Programming | `gpt-4o-mini` | 1.00 | 1.6 s | $0.0001 | `gpt-4o-mini` ($0.0001) |59| Simple reasoning | `gpt-6-luna` | 1.00 | 1.4 s | $0.0000 | `gpt-6-luna` ($0.0000) |60| Knowledge synthesis | `gpt-4o-mini` | 1.00 | 2.8 s | $0.0002 | `gpt-4o-mini` ($0.0002) |61| Tool calling | `gpt-6-luna` | 1.00 | 1.1 s | $0.0000 | `gpt-6-luna` ($0.0000) |62| Mathematics | `gpt-6-luna` | 1.00 | 20.4 s | $0.0002 | `gpt-6-luna` ($0.0002) |6364## Writing quality (two-judge LLM rubric, 1–5; experimental, not used for pass rates)6566| Model | French writing | English writing | Knowledge synthesis |67|---|---|---|---|68| `chat-latest` | 5.00 | 5.00 | 5.00 |69| `gpt-5.3-codex` | 5.00 | 5.00 | 4.00 |70| `gpt-4.1` | 5.00 | 5.00 | 2.50 |71| `gpt-6-sol` | 5.00 | 5.00 | 5.00 |72| `gpt-5.4` | 5.00 | 5.00 | 5.00 |73| `claude-fable-5-1` | 5.00 | 5.00 | 5.00 |74| `gpt-5.2` | 5.00 | 5.00 | 3.00 |75| `claude-sonnet-5` | 5.00 | 5.00 | 3.00 |76| `gpt-5.1` | 5.00 | 5.00 | 3.50 |77| `gpt-6.1-sol` | 5.00 | 5.00 | 4.00 |78| `claude-sonnet-4-5-20250929` | 5.00 | 5.00 | 3.00 |79| `claude-haiku-5-5` | 4.50 | 5.00 | 5.00 |80| `gpt-5.6-sol` | 5.00 | 5.00 | 5.00 |81| `gpt-5.5` | 5.00 | 5.00 | 4.50 |82| `claude-opus-4-5-20251101` | 5.00 | 5.00 | 4.00 |83| `claude-opus-5-5` | 5.00 | 5.00 | 4.50 |84| `claude-sonnet-5-5` | 5.00 | 5.00 | 4.00 |85| `claude-opus-4-8` | 5.00 | 5.00 | 4.00 |86| `o3` | 4.50 | 5.00 | 3.50 |87| `gpt-6-astra` | 5.00 | 5.00 | 5.00 |88| `gpt-6.1-terra` | 5.00 | 5.00 | 4.00 |89| `claude-opus-4-7` | 5.00 | 5.00 | 3.00 |90| `claude-opus-5` | 5.00 | 5.00 | 4.00 |91| `claude-opus-4-6` | 5.00 | 5.00 | 4.50 |92| `claude-sonnet-4-6` | 5.00 | 5.00 | 5.00 |93| `gpt-5` | 5.00 | 5.00 | 3.00 |94| `gpt-5-mini` | 5.00 | 5.00 | 4.50 |95| `gpt-4o` | 4.00 | 5.00 | 2.50 |96| `gpt-5.6-luna` | 5.00 | 5.00 | 4.00 |97| `gpt-4.1-mini` | 4.00 | 5.00 | 2.50 |98| `claude-haiku-4-5-20251001` | 3.50 | 4.00 | 2.50 |99| `gpt-5-nano` | 3.00 | 5.00 | 3.00 |100| `gpt-5.4-mini` | 5.00 | 5.00 | 4.00 |101| `gpt-5.6-terra` | 5.00 | 5.00 | 5.00 |102| `gpt-6-luna` | 5.00 | 5.00 | 4.00 |103| `gpt-4o-mini` | 5.00 | 4.00 | 3.00 |104| `gpt-5.4-nano` | 4.50 | 5.00 | 2.50 |105| `claude-fable-5` | 5.00 | 5.00 | 5.00 |106107## Speed108109| Model | TTFT median | Latency median | Output tok/s median |110|---|---|---|---|111| `gpt-5.4-mini` | 624 ms | 1.2 s | 167 |112| `chat-latest` | 741 ms | 1.4 s | 157 |113| `gpt-5.3-codex` | 602 ms | 1.4 s | 94 |114| `gpt-4.1-mini` | 449 ms | 1.7 s | 153 |115| `gpt-4.1` | 450 ms | 1.7 s | 163 |116| `gpt-5.4-nano` | 549 ms | 1.7 s | 135 |117| `gpt-6-sol` | 1106 ms | 1.7 s | 149 |118| `gpt-5.6-terra` | 869 ms | 1.8 s | 126 |119| `gpt-5.4` | 716 ms | 1.8 s | 86 |120| `gpt-4o-mini` | 561 ms | 2.0 s | 116 |121| `claude-haiku-4-5-20251001` | 502 ms | 2.0 s | 135 |122| `claude-fable-5-1` | 389 ms | 2.1 s | 118 |123| `gpt-5.2` | 676 ms | 2.1 s | 85 |124| `claude-sonnet-5` | 950 ms | 2.2 s | 145 |125| `gpt-5.1` | 650 ms | 2.2 s | 139 |126| `gpt-6.1-sol` | 1444 ms | 2.2 s | 89 |127| `claude-sonnet-4-5-20250929` | 775 ms | 2.3 s | 90 |128| `gpt-4o` | 656 ms | 2.4 s | 106 |129| `claude-haiku-5-5` | 1301 ms | 2.4 s | 487 |130| `gpt-5.6-sol` | 1108 ms | 2.5 s | 133 |131| `gpt-5.5` | 1755 ms | 2.5 s | 147 |132| `claude-opus-4-5-20251101` | 862 ms | 2.5 s | 98 |133| `claude-opus-5-5` | 1703 ms | 2.8 s | 161 |134| `claude-fable-5` | 1764 ms | 2.8 s | 112 |135| `claude-sonnet-5-5` | 1020 ms | 2.9 s | 202 |136| `claude-opus-4-8` | 488 ms | 2.9 s | 104 |137| `o3` | 1763 ms | 2.9 s | 533 |138| `gpt-6-luna` | 2045 ms | 3.0 s | 194 |139| `gpt-5.6-luna` | 1398 ms | 3.0 s | 136 |140| `gpt-6-astra` | 2376 ms | 3.1 s | 78 |141| `gpt-6.1-terra` | 1515 ms | 3.2 s | 93 |142| `claude-opus-4-7` | 687 ms | 3.3 s | 91 |143| `claude-opus-5` | 2530 ms | 4.0 s | 161 |144| `claude-opus-4-6` | 1622 ms | 4.2 s | 75 |145| `claude-sonnet-4-6` | 735 ms | 4.7 s | 90 |146| `gpt-5` | 4121 ms | 5.0 s | 962 |147| `gpt-5-mini` | 4395 ms | 5.7 s | 375 |148| `gpt-5-nano` | 6133 ms | 6.8 s | 1362 |149150## Caveats151152- Two trials per cell: differences of a few points are noise. Latency depends on provider load at run time (2026-11-01).153- Costs are estimates (provider-reported tokens × list price); provider invoices are authoritative.154- Writing scores come from LLM judges and are shown separately from deterministic results.155- Pro-tier models (gpt-5.5-pro, gpt-5.4-pro, gpt-5.2-pro, gpt-5-pro, o3-pro) passed discovery probes but were excluded from the benchmark for cost ($20–$30 per M input tokens); they are selectable manually in the app.156