# Anthropic model benchmarks — Note SPBoucher AI
Generated 2026-11-01T07:18:30.320Z from benchmarks/results/raw/bench-runs.jsonl (308 real runs, 2 trials per model × task). Reproduce: npx tsx benchmarks/scripts/bench.ts then npx tsx benchmarks/scripts/report.ts.
Protocol. Streaming requests, provider default reasoning settings, max_tokens per task (1.5K–6K). TTFT = first text/tool delta; tok/s = output tokens ÷ (total − TTFT). Tokens are provider-reported; cost = tokens × published list prices (estimate). Deterministic graders for 8 tasks (regex, executed code tests, JSON-schema + field checks, tool-call checks); the three writing/synthesis tasks add a two-judge LLM rubric (claude-sonnet-5-5 + gpt-5.4, 1–5) reported separately and never mixed into the pass rate.
# Overall
| Model |
Pass rate |
Mean score |
LLM-judged (1–5) |
TTFT med. |
Latency med. |
tok/s med. |
Cost / run |
Total cost |
Errors |
claude-fable-5-1 |
100% |
1.00 |
5.00 |
389 ms |
2.1 s |
118 |
$0.090 |
$1.979 |
0/22 |
claude-sonnet-5 |
100% |
1.00 |
4.33 |
950 ms |
2.2 s |
145 |
$0.019 |
$0.416 |
0/22 |
claude-sonnet-4-5-20250929 |
100% |
1.00 |
4.33 |
775 ms |
2.3 s |
90 |
$0.022 |
$0.474 |
0/22 |
claude-haiku-5-5 |
100% |
1.00 |
4.83 |
1301 ms |
2.4 s |
487 |
$0.0011 |
$0.023 |
0/22 |
claude-opus-4-5-20251101 |
100% |
1.00 |
4.67 |
862 ms |
2.5 s |
98 |
$0.036 |
$0.783 |
0/22 |
claude-opus-5-5 |
100% |
1.00 |
4.83 |
1703 ms |
2.8 s |
161 |
$0.043 |
$0.943 |
0/22 |
claude-sonnet-5-5 |
100% |
1.00 |
4.67 |
1020 ms |
2.9 s |
202 |
$0.020 |
$0.432 |
0/22 |
claude-opus-4-8 |
100% |
1.00 |
4.67 |
488 ms |
2.9 s |
104 |
$0.045 |
$0.982 |
0/22 |
claude-opus-4-7 |
100% |
1.00 |
4.33 |
687 ms |
3.3 s |
91 |
$0.045 |
$0.982 |
0/22 |
claude-opus-5 |
100% |
1.00 |
4.67 |
2530 ms |
4.0 s |
161 |
$0.051 |
$1.116 |
0/22 |
claude-opus-4-6 |
100% |
1.00 |
4.83 |
1622 ms |
4.2 s |
75 |
$0.035 |
$0.776 |
0/22 |
claude-sonnet-4-6 |
100% |
1.00 |
5.00 |
735 ms |
4.7 s |
90 |
$0.021 |
$0.466 |
0/22 |
claude-haiku-4-5-20251001 |
91% |
0.97 |
3.33 |
502 ms |
2.0 s |
135 |
$0.0072 |
$0.159 |
0/22 |
claude-fable-5 |
91% |
0.91 |
5.00 |
1764 ms |
2.8 s |
112 |
$0.092 |
$2.022 |
0/22 |
# Pass rate by task
| Model |
Document analysis |
JSON generation |
Complex reasoning |
Long-context retrieval |
English writing |
French writing |
Programming |
Simple reasoning |
Knowledge synthesis |
Tool calling |
Mathematics |
claude-fable-5-1 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 5.0) |
100% |
100% |
claude-sonnet-5 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 3.0) |
100% |
100% |
claude-sonnet-4-5-20250929 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 3.0) |
100% |
100% |
claude-haiku-5-5 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 4.5) |
100% |
100% |
100% (j 5.0) |
100% |
100% |
claude-opus-4-5-20251101 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.0) |
100% |
100% |
claude-opus-5-5 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.5) |
100% |
100% |
claude-sonnet-5-5 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.0) |
100% |
100% |
claude-opus-4-8 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.0) |
100% |
100% |
claude-opus-4-7 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 3.0) |
100% |
100% |
claude-opus-5 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.0) |
100% |
100% |
claude-opus-4-6 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 4.5) |
100% |
100% |
claude-sonnet-4-6 |
100% |
100% |
100% |
100% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 5.0) |
100% |
100% |
claude-haiku-4-5-20251001 |
100% |
100% |
100% |
100% |
100% (j 4.0) |
100% (j 3.5) |
100% |
0% |
100% (j 2.5) |
100% |
100% |
claude-fable-5 |
100% |
100% |
100% |
0% |
100% (j 5.0) |
100% (j 5.0) |
100% |
100% |
100% (j 5.0) |
100% |
100% |
| Model |
Document analysis |
JSON generation |
Complex reasoning |
Long-context retrieval |
English writing |
French writing |
Programming |
Simple reasoning |
Knowledge synthesis |
Tool calling |
Mathematics |
claude-fable-5-1 |
1.6 |
1.6 |
5.2 |
1.4 |
7.0 |
7.6 |
2.1 |
0.7 |
8.2 |
1.6 |
2.5 |
claude-sonnet-5 |
1.7 |
1.9 |
11.1 |
1.6 |
6.3 |
7.2 |
1.8 |
1.2 |
4.9 |
1.5 |
4.1 |
claude-sonnet-4-5-20250929 |
1.9 |
1.8 |
10.6 |
2.1 |
6.4 |
8.1 |
2.3 |
1.1 |
7.2 |
1.7 |
7.7 |
claude-haiku-5-5 |
3.3 |
1.3 |
4.9 |
2.4 |
2.7 |
7.8 |
0.9 |
0.7 |
5.3 |
0.9 |
2.1 |
claude-opus-4-5-20251101 |
1.7 |
1.4 |
9.2 |
2.1 |
6.9 |
7.3 |
2.5 |
1.2 |
7.5 |
1.7 |
6.3 |
claude-opus-5-5 |
1.8 |
2.4 |
6.1 |
2.3 |
15.5 |
18.4 |
2.6 |
1.3 |
5.9 |
1.9 |
3.8 |
claude-sonnet-5-5 |
2.0 |
2.2 |
5.1 |
2.3 |
4.6 |
10.0 |
4.4 |
1.2 |
3.7 |
1.6 |
2.9 |
claude-opus-4-8 |
3.2 |
2.0 |
5.2 |
2.2 |
7.5 |
9.6 |
1.9 |
0.7 |
5.6 |
1.7 |
3.1 |
claude-opus-4-7 |
3.3 |
2.6 |
4.5 |
3.0 |
8.5 |
11.9 |
1.8 |
0.9 |
6.5 |
1.9 |
4.0 |
claude-opus-5 |
3.3 |
2.3 |
8.8 |
2.7 |
14.5 |
22.2 |
7.1 |
2.5 |
9.9 |
2.4 |
4.0 |
claude-opus-4-6 |
3.7 |
4.0 |
12.8 |
2.8 |
8.6 |
9.2 |
3.7 |
1.7 |
8.8 |
2.9 |
6.3 |
claude-sonnet-4-6 |
4.5 |
3.4 |
10.3 |
2.7 |
7.5 |
8.0 |
2.6 |
1.2 |
7.6 |
1.8 |
5.5 |
claude-haiku-4-5-20251001 |
1.9 |
1.5 |
7.5 |
1.0 |
3.2 |
3.7 |
1.9 |
0.6 |
3.1 |
1.2 |
4.6 |
claude-fable-5 |
2.0 |
2.8 |
8.8 |
1.7 |
8.4 |
9.3 |
2.8 |
2.1 |
8.0 |
2.7 |
4.9 |
# Mean cost by task (USD)
| Model |
Document analysis |
JSON generation |
Complex reasoning |
Long-context retrieval |
English writing |
French writing |
Programming |
Simple reasoning |
Knowledge synthesis |
Tool calling |
Mathematics |
claude-fable-5-1 |
$0.016 |
$0.010 |
$0.026 |
$0.810 |
$0.022 |
$0.028 |
$0.011 |
$0.0028 |
$0.037 |
$0.014 |
$0.012 |
claude-sonnet-5 |
$0.0034 |
$0.0020 |
$0.013 |
$0.162 |
$0.0049 |
$0.0067 |
$0.0022 |
$0.0011 |
$0.0052 |
$0.0029 |
$0.0046 |
claude-sonnet-4-5-20250929 |
$0.0040 |
$0.0024 |
$0.013 |
$0.185 |
$0.0039 |
$0.0060 |
$0.0029 |
$0.0007 |
$0.0049 |
$0.0042 |
$0.0096 |
claude-haiku-5-5 |
$0.0002 |
$0.0001 |
$0.0006 |
$0.0082 |
$0.0002 |
$0.0011 |
$0.0001 |
$0.0001 |
$0.0006 |
$0.0001 |
$0.0002 |
claude-opus-4-5-20251101 |
$0.0067 |
$0.0037 |
$0.018 |
$0.309 |
$0.0069 |
$0.010 |
$0.0047 |
$0.0012 |
$0.0091 |
$0.0071 |
$0.014 |
claude-opus-5-5 |
$0.0064 |
$0.0041 |
$0.015 |
$0.324 |
$0.039 |
$0.049 |
$0.0059 |
$0.0019 |
$0.012 |
$0.0053 |
$0.0092 |
claude-sonnet-5-5 |
$0.0032 |
$0.0020 |
$0.0080 |
$0.162 |
$0.0050 |
$0.017 |
$0.0076 |
$0.0006 |
$0.0049 |
$0.0027 |
$0.0034 |
claude-opus-4-8 |
$0.0080 |
$0.0051 |
$0.013 |
$0.405 |
$0.012 |
$0.014 |
$0.0049 |
$0.0014 |
$0.013 |
$0.0070 |
$0.0081 |
claude-opus-4-7 |
$0.0081 |
$0.0052 |
$0.010 |
$0.405 |
$0.012 |
$0.015 |
$0.0047 |
$0.0015 |
$0.012 |
$0.0090 |
$0.0076 |
claude-opus-5 |
$0.0080 |
$0.0051 |
$0.018 |
$0.405 |
$0.026 |
$0.047 |
$0.012 |
$0.0027 |
$0.018 |
$0.0074 |
$0.0087 |
claude-opus-4-6 |
$0.0067 |
$0.0037 |
$0.018 |
$0.309 |
$0.0069 |
$0.010 |
$0.0045 |
$0.0012 |
$0.0088 |
$0.0074 |
$0.011 |
claude-sonnet-4-6 |
$0.0040 |
$0.0022 |
$0.011 |
$0.185 |
$0.0043 |
$0.0059 |
$0.0029 |
$0.0007 |
$0.0054 |
$0.0044 |
$0.0070 |
claude-haiku-4-5-20251001 |
$0.0014 |
$0.0007 |
$0.0048 |
$0.062 |
$0.0014 |
$0.0018 |
$0.0011 |
$0.0002 |
$0.0015 |
$0.0014 |
$0.0029 |
claude-fable-5 |
$0.017 |
$0.011 |
$0.041 |
$0.809 |
$0.026 |
$0.030 |
$0.0092 |
$0.0027 |
$0.032 |
$0.014 |
$0.020 |
# Token usage
| Model |
Input tokens |
Output tokens |
of which reasoning |
claude-fable-5-1 |
168774 |
5817 |
709 |
claude-sonnet-5 |
168866 |
7802 |
2116 |
claude-sonnet-4-5-20250929 |
129532 |
5724 |
0 |
claude-haiku-5-5 |
168774 |
12487 |
6979 |
claude-opus-4-5-20251101 |
129532 |
5424 |
0 |
claude-opus-5-5 |
168774 |
13381 |
7584 |
claude-sonnet-5-5 |
168774 |
9494 |
2881 |
claude-opus-4-8 |
168738 |
5530 |
0 |
claude-opus-4-7 |
169618 |
5347 |
0 |
claude-opus-5 |
168730 |
10889 |
5458 |
claude-opus-4-6 |
129538 |
5116 |
0 |
claude-sonnet-4-6 |
129538 |
5170 |
0 |
claude-haiku-4-5-20251001 |
129532 |
5798 |
0 |
claude-fable-5 |
168738 |
6695 |
1105 |