Provider test matrix
Generated by pnpm providers:matrix against the REAL provider APIs on 2026-09-08. A cell is ✅ only when the live request succeeded and the assertion passed. “—” = not run (no key).
Test models: openai: gpt-5.4-mini / reasoning gpt-5.4-mini; anthropic: claude-haiku-4-5 / reasoning claude-sonnet-5; gemini: gemini-3.5-flash-lite / reasoning gemini-3.5-flash-lite; xai: grok-4.20-0309-non-reasoning / reasoning grok-4.3.
| Capability | OpenAI | Anthropic | Gemini | xAI |
|---|---|---|---|---|
| Auth | ✅ | ✅ | ✅ | ✅ |
| Model list | ✅ | ✅ | ✅ | ✅ |
| Text | ✅ | ✅ | ✅ | ✅ |
| Streaming | ✅ | ✅ | ✅ | ✅ |
| System | ✅ | ✅ | ✅ | ✅ |
| Vision | ✅ | ✅ | ✅ | ✅ |
| Tools | ✅ | ✅ | ✅ | ✅ |
| Structured output | ✅ | ✅ | ✅ | ✅ |
| Reasoning | ✅ | ✅ | ✅ | ✅ |
| Token usage | ✅ | ✅ | ✅ | ✅ |
| Invalid key | ✅ | ✅ | ✅ | ✅ |
| Error normalization | ✅ | ✅ | ✅ | ✅ |
| Long response | ✅ | ✅ | ✅ | ✅ |
Notes
- gemini: models: 18 (gemini-2.5-flash, gemini-2.5-pro, gemma-4-26b-a4b-it, gemma-4-31b-it, gemini-flash-latest…)
- gemini: text(gemini-3.5-flash-lite): "OK" deltas=1 usage={"inputTokens":6,"outputTokens":1,"totalTokens":7} finish=stop
- gemini: system: "Arr, hello there, me hearty!"
- gemini: vision: "Red"
- gemini: tools: [{"n":"calculator","a":{"expression":"1234 * 5678"}}] finish=tool-calls
- gemini: structured: { "city": "Ottawa", "country": "Canada" }
- gemini: reasoning(gemini-3.5-flash-lite): text="16" reasoningChars=2115 reasoningTokens=3907
- gemini: invalid key → INVALID_API_KEY (400)
- gemini: unknown model → MODEL_NOT_FOUND: { "error": { "code": 404, "message": "models/definitely-not-a-model-xyz is not f
- gemini: long: 471 words, 25 deltas