PolyLLM v0.1.0 — multi-provider BYOK AI workspace (OpenAI, Anthropic, Gemini, xAI)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
44 changed files +20,001 −0
added
.env.example
+35 −0
@@ -0,0 +1,35 @@ | |||||
| 1 | +# --- Runtime ------------------------------------------------------------- | ||||
| 2 | +NODE_ENV=production | ||||
| 3 | +PORT=3000 | ||||
| 4 | +# Public origin used in emails, auth callbacks and canonical links. No trailing slash. | ||||
| 5 | +PUBLIC_APP_URL=https://www.polyllm.io | ||||
| 6 | + | ||||
| 7 | +# --- Database ------------------------------------------------------------ | ||||
| 8 | +DATABASE_URL=postgres://localhost:5432/polyllm | ||||
| 9 | +DB_POOL_MAX=10 | ||||
| 10 | + | ||||
| 11 | +# --- Auth ---------------------------------------------------------------- | ||||
| 12 | +# 32+ random bytes, hex or base64. `openssl rand -hex 32` | ||||
| 13 | +AUTH_SECRET= | ||||
| 14 | + | ||||
| 15 | +# --- Email (Resend) ------------------------------------------------------ | ||||
| 16 | +RESEND_API_KEY= | ||||
| 17 | +RESEND_FROM_EMAIL="PolyLLM <mail@spboucher.ai>" | ||||
| 18 | +# Set to 1 to log emails instead of sending (development). | ||||
| 19 | +EMAIL_DRY_RUN=0 | ||||
| 20 | + | ||||
| 21 | +# --- Encryption of user provider keys (AES-256-GCM, server-side only) ---- | ||||
| 22 | +# 32+ random bytes. Rotating this secret invalidates every stored key. | ||||
| 23 | +API_KEY_ENCRYPTION_SECRET= | ||||
| 24 | + | ||||
| 25 | +# --- Owner / development provider keys (TEST ONLY — never used for users) - | ||||
| 26 | +OPENAI_API_KEY= | ||||
| 27 | +ANTHROPIC_API_KEY= | ||||
| 28 | +GOOGLE_GEMINI_API_KEY= | ||||
| 29 | +XAI_API_KEY= | ||||
| 30 | + | ||||
| 31 | +# --- Admin --------------------------------------------------------------- | ||||
| 32 | +# Comma-separated emails with access to /admin. | ||||
| 33 | +ADMIN_EMAILS= | ||||
| 34 | +# Model catalog refresh interval (minutes). 0 disables the scheduler. | ||||
| 35 | +MODEL_SYNC_INTERVAL_MINUTES=360 | ||||
added
.gitignore
+33 −0
@@ -0,0 +1,33 @@ | |||||
| 1 | +# dependencies | ||||
| 2 | +node_modules/ | ||||
| 3 | +.pnpm-store/ | ||||
| 4 | + | ||||
| 5 | +# next | ||||
| 6 | +.next/ | ||||
| 7 | +out/ | ||||
| 8 | +next-env.d.ts | ||||
| 9 | +*.tsbuildinfo | ||||
| 10 | + | ||||
| 11 | +# env & secrets — never commit | ||||
| 12 | +.env | ||||
| 13 | +.env.* | ||||
| 14 | +!.env.example | ||||
| 15 | +deploy/*.mld.json | ||||
| 16 | + | ||||
| 17 | +# research scratch (installed SDKs, probe outputs) | ||||
| 18 | +research/*/node_modules/ | ||||
| 19 | +research/*/out/ | ||||
| 20 | +research/*/*.log | ||||
| 21 | + | ||||
| 22 | +# tests | ||||
| 23 | +coverage/ | ||||
| 24 | +playwright-report/ | ||||
| 25 | +test-results/ | ||||
| 26 | +qa/out/ | ||||
| 27 | + | ||||
| 28 | +# misc | ||||
| 29 | +.DS_Store | ||||
| 30 | +logs/ | ||||
| 31 | +tmp/ | ||||
| 32 | +.turbo/ | ||||
| 33 | +.claude/ | ||||
added
CLAUDE.md
+38 −0
@@ -0,0 +1,38 @@ | |||||
| 1 | +# PolyLLM — repository guide | ||||
| 2 | + | ||||
| 3 | +PolyLLM (www.polyllm.io) is a BYOK multi-provider AI workspace: OpenAI, Anthropic, Gemini, xAI behind one normalized | ||||
| 4 | +adapter layer, real streaming, capability-driven configuration, Arena comparisons, usage/cost tracking. | ||||
| 5 | +The original product brief lives in `docs/SPEC-original.md`; the UI conventions in `docs/UI-BRIEF.md`. | ||||
| 6 | + | ||||
| 7 | +## Non-negotiables | ||||
| 8 | +- **Research before provider changes.** Read `docs/provider-research/<provider>.md`, re-fetch the official docs, probe the | ||||
| 9 | + real API (`pnpm providers:matrix`), then update `src/lib/ai/providers/<provider>/catalog*.ts`. Never assume model ids, | ||||
| 10 | + parameters, event names or limits from memory. Bump `Last documentation audit`. | ||||
| 11 | +- **Provider quirks stay inside `src/lib/ai/providers/*`.** UI and services only see `PolyModel`, `UnifiedChatRequest`, | ||||
| 12 | + `UnifiedStreamEvent`, `PolyProviderError`. | ||||
| 13 | +- **Capabilities drive the UI.** Never branch on a model id in components; use `model.capabilities` / `model.parameters`. | ||||
| 14 | + Never send a parameter the model does not accept (`filterSettings()` + adapter translation). | ||||
| 15 | +- **Keys.** User keys are encrypted (`src/lib/crypto/keys.ts`) and decrypted only in `getDecryptedKey()` right before a | ||||
| 16 | + request. Never log, return or serialize them; `PublicConnection` exposes `keyHint` only. Owner keys in `.env` are for the | ||||
| 17 | + registry sync, admin diagnostics and tests — never for user requests. | ||||
| 18 | +- **No fake integrations.** A capability is ✅ only after a real request passed (`docs/provider-test-matrix.md`). | ||||
| 19 | +- Secrets only in `.env` (gitignored) and the mld manifest on M1M32; `.env.example` documents the variables. | ||||
| 20 | + | ||||
| 21 | +## Commands | ||||
| 22 | +`pnpm dev` · `pnpm typecheck` · `pnpm lint` · `pnpm test` (vitest) · `pnpm test:e2e` (Playwright; dev server with | ||||
| 23 | +`EMAIL_DRY_RUN=1 EMAIL_DRY_RUN_PRINT=1`) · `pnpm db:generate` / `pnpm db:migrate` · `pnpm models:sync [provider]` · | ||||
| 24 | +`pnpm providers:matrix [provider]`. | ||||
| 25 | + | ||||
| 26 | +## Deploy | ||||
| 27 | +`mld` from the laptop (see `docs/deployment-cluster.md`): `mld stage . polyllm && mld deploy polyllm --node M3U96a`. | ||||
| 28 | +Manifest with secrets: `M1M32:~/dispatch/apps/polyllm.json` (local gitignored copy `deploy/polyllm.mld.json`). | ||||
| 29 | + | ||||
| 30 | +## Gotchas | ||||
| 31 | +- React Compiler lint: no `Date.now()`/impure calls in render, no setState synchronously in effects without an explicit | ||||
| 32 | + eslint-disable, memoize derived lists. | ||||
| 33 | +- `import "server-only"` modules are used by CLI scripts through `scripts/hooks.mjs` (tsx `--import`) and by vitest via an alias. | ||||
| 34 | +- Anthropic: adaptive thinking may legitimately produce zero thinking tokens on easy prompts; Haiku 4.5 needs `budget_tokens`. | ||||
| 35 | +- Gemini: free-tier keys have tiny per-model daily quotas (429 `limit: 20`) and some models are paid-tier only (429 `limit: 0`); | ||||
| 36 | + thought signatures must be replayed on function-call parts. | ||||
| 37 | +- xAI: bad keys return HTTP **400** (`Incorrect API key`), `frequency/presence_penalty` always 400, web search only via `/v1/responses`. | ||||
| 38 | +- OpenAI: Responses API has no `seed`/`stop`; sampling params only when `reasoning.effort === "none"` on reasoning models. | ||||
added
README.md
+46 −0
@@ -0,0 +1,46 @@ | |||||
| 1 | +# PolyLLM | ||||
| 2 | + | ||||
| 3 | +**One interface. Every model. Bring your own keys.** — https://www.polyllm.io | ||||
| 4 | + | ||||
| 5 | +PolyLLM is a universal control center for AI models: connect your own OpenAI, Anthropic, Google Gemini and xAI API keys, | ||||
| 6 | +discover the models your keys can reach, configure each one with only the parameters it supports, chat with real | ||||
| 7 | +token-by-token streaming, compare up to four models side by side in the Arena, and track tokens, latency and estimated | ||||
| 8 | +cost. Keys are encrypted at rest (AES-256-GCM, HKDF-derived data key, AAD-bound to the user) and only decrypted on the | ||||
| 9 | +server immediately before a provider request. | ||||
| 10 | + | ||||
| 11 | +## Stack | ||||
| 12 | +Next.js 16 (App Router, Route Handlers with SSE streaming) · React 19 · TypeScript · Tailwind v4 · PostgreSQL 17 + Drizzle · | ||||
| 13 | +Better Auth 1.7 (Argon2id, email verification, sessions, reset, change email, delete account) · Resend · Zod · | ||||
| 14 | +official provider SDKs (`openai`, `@anthropic-ai/sdk`, `@google/genai`; xAI through the OpenAI SDK at `api.x.ai`). | ||||
| 15 | + | ||||
| 16 | +## Layout | ||||
| 17 | +``` | ||||
| 18 | +src/app routes: (marketing) landing/legal · (auth) login/signup/verify/reset · app/* workspace · admin/* · share/[id] · api/* | ||||
| 19 | +src/lib/ai/core provider-neutral contract: PolyModel, UnifiedChatRequest, UnifiedStreamEvent, errors, retry, pricing, normalization | ||||
| 20 | +src/lib/ai/providers adapters (openai, anthropic, gemini, xai) + catalogs — provider quirks live ONLY here | ||||
| 21 | +src/lib/ai/registry model registry: live listing → catalog merge → Postgres; scheduled + manual sync | ||||
| 22 | +src/lib/chat turn orchestration (edit/regenerate/continue/branch), built-in tools loop, SSE | ||||
| 23 | +src/lib/providers encrypted key storage & validation | ||||
| 24 | +src/components ui kit · app shell · chat · arena · settings · marketing | ||||
| 25 | +docs/ provider-research (per provider, dated), provider-test-matrix (generated), deployment-cluster, final-audit, UI brief | ||||
| 26 | +scripts/ sync-models, provider-matrix (real-API tests) | ||||
| 27 | +e2e/, tests/ Playwright workflows, vitest unit tests | ||||
| 28 | +``` | ||||
| 29 | + | ||||
| 30 | +## Development | ||||
| 31 | +```bash | ||||
| 32 | +cp .env.example .env # fill AUTH_SECRET, API_KEY_ENCRYPTION_SECRET, RESEND_API_KEY, owner provider keys (optional) | ||||
| 33 | +createdb polyllm && pnpm install && pnpm db:migrate | ||||
| 34 | +pnpm models:sync # populate the registry with the owner keys | ||||
| 35 | +pnpm dev # http://localhost:3000 | ||||
| 36 | +pnpm typecheck && pnpm lint && pnpm test | ||||
| 37 | +pnpm providers:matrix # real-API capability matrix → docs/provider-test-matrix.md | ||||
| 38 | +EMAIL_DRY_RUN=1 EMAIL_DRY_RUN_PRINT=1 pnpm dev && pnpm test:e2e # full browser workflow (links read from the dev log) | ||||
| 39 | +``` | ||||
| 40 | + | ||||
| 41 | +## Deployment | ||||
| 42 | +Runs on the private MacLustr cluster behind ngrok — see `docs/deployment-cluster.md`. | ||||
| 43 | + | ||||
| 44 | +## Security notes | ||||
| 45 | +No provider key ever reaches the browser, logs or analytics (structured logger redacts key-shaped strings and sensitive | ||||
| 46 | +field names). CSP, HSTS, HttpOnly/SameSite cookies, rate limits on auth and provider endpoints, per-user audit log. | ||||
added
deploy/backup/install.sh
+28 −0
@@ -0,0 +1,28 @@ | |||||
| 1 | +#!/bin/bash | ||||
| 2 | +# Installs the daily PostgreSQL backup for PolyLLM on the deployment node (run ON the node). | ||||
| 3 | +set -euo pipefail | ||||
| 4 | +mkdir -p ~/backups/polyllm ~/Library/LaunchAgents | ||||
| 5 | +cat > ~/backups/polyllm/backup.sh <<'SH' | ||||
| 6 | +#!/bin/bash | ||||
| 7 | +set -euo pipefail | ||||
| 8 | +export PATH="/opt/homebrew/opt/postgresql@17/bin:/opt/homebrew/bin:$PATH" | ||||
| 9 | +out=~/backups/polyllm/polyllm-$(date +%Y%m%d).sql.gz | ||||
| 10 | +pg_dump polyllm | gzip -9 > "$out.tmp" && mv "$out.tmp" "$out" | ||||
| 11 | +find ~/backups/polyllm -name 'polyllm-*.sql.gz' -mtime +14 -delete | ||||
| 12 | +echo "$(date -Iseconds) ok $(du -h "$out" | cut -f1)" >> ~/backups/polyllm/backup.log | ||||
| 13 | +SH | ||||
| 14 | +chmod +x ~/backups/polyllm/backup.sh | ||||
| 15 | +cat > ~/Library/LaunchAgents/dev.polyllm.backup.plist <<PLIST | ||||
| 16 | +<?xml version="1.0" encoding="UTF-8"?> | ||||
| 17 | +<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> | ||||
| 18 | +<plist version="1.0"><dict> | ||||
| 19 | + <key>Label</key><string>dev.polyllm.backup</string> | ||||
| 20 | + <key>ProgramArguments</key><array><string>/bin/bash</string><string>$HOME/backups/polyllm/backup.sh</string></array> | ||||
| 21 | + <key>StartCalendarInterval</key><dict><key>Hour</key><integer>3</integer><key>Minute</key><integer>45</integer></dict> | ||||
| 22 | + <key>StandardOutPath</key><string>$HOME/backups/polyllm/launchd.log</string> | ||||
| 23 | + <key>StandardErrorPath</key><string>$HOME/backups/polyllm/launchd.log</string> | ||||
| 24 | +</dict></plist> | ||||
| 25 | +PLIST | ||||
| 26 | +launchctl bootout gui/$(id -u)/dev.polyllm.backup 2>/dev/null || true | ||||
| 27 | +launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/dev.polyllm.backup.plist | ||||
| 28 | +~/backups/polyllm/backup.sh && echo "backup installed + first dump done" | ||||
added
docs/SPEC-original.md
+3 −0
@@ -0,0 +1,3 @@ | |||||
| 1 | +# PolyLLM — original product brief (CLAUDE.md provided by the owner, 2026-09-08) | ||||
| 2 | + | ||||
| 3 | +The full brief was supplied in the build session; its normative rules are reproduced in the repository `CLAUDE.md` and `docs/UI-BRIEF.md`. Key sections: research-before-implementation, live model catalog, adapter architecture, mandatory email verification, encrypted BYOK keys, unified streaming events, capability-driven configuration, Arena, usage/cost tracking, security/privacy/logging rules, cluster deployment through ngrok at https://www.polyllm.io, real-API test matrix, final audit. | ||||
added
docs/UI-BRIEF.md
+64 −0
@@ -0,0 +1,64 @@ | |||||
| 1 | +# PolyLLM — UI brief for contributors (humans and agents) | ||||
| 2 | + | ||||
| 3 | +PolyLLM (www.polyllm.io) is a **universal control center for AI models**: users bring their own API keys | ||||
| 4 | +(OpenAI, Anthropic, Gemini, xAI), discover current models, configure them precisely, chat with real streaming, | ||||
| 5 | +compare models in the Arena and track usage/costs. Tagline: **"Your models. Your keys. One workspace."** | ||||
| 6 | + | ||||
| 7 | +## Stack & conventions | ||||
| 8 | +- Next.js 16 App Router, React 19, TypeScript strict, Tailwind v4 (tokens in `src/app/globals.css`), shadcn-style | ||||
| 9 | + primitives in `src/components/ui/*` (button, input/textarea/label/field, badge, dialog, dropdown-menu, popover, | ||||
| 10 | + select, slider, switch, tabs, tooltip, toast (`toast.success/error/info`), misc: Card/CardHeader/CardBody, Stat, | ||||
| 11 | + PageHeader, EmptyState, Skeleton, CopyButton, Kbd, Spinner, Divider). Icons: `lucide-react`. Motion: `motion/react` (sparingly). | ||||
| 12 | +- Client data: `useApi<T>(url)` (SWR) and `api<T>(url, { method, json })` from `src/lib/client/api.ts`; | ||||
| 13 | + `streamEvents(url, body, onEvent, signal)` for SSE. Types: `src/lib/client/types.ts`. Provider metadata: | ||||
| 14 | + `src/lib/client/providers.ts` (`PROVIDERS`, `PROVIDER_ORDER`, `providerName`). Brand: `src/components/brand/logo.tsx` | ||||
| 15 | + (`Logo`, `LogoMark`), `src/components/brand/provider-icon.tsx` (`ProviderIcon`). | ||||
| 16 | +- App state: `useApp()` from `src/components/app/store.tsx` → `{ user, preferences, updatePreferences, models, modelsByKey, | ||||
| 17 | + favorites, recents, connectedProviders, connections, folders, toggleFavorite, refreshModels, refreshConnections, | ||||
| 18 | + selectedModelKey, setSelectedModelKey, setPaletteOpen }`. Everything under `/app/*` is wrapped by the shell | ||||
| 19 | + (sidebar + command palette) in `src/app/app/layout.tsx`; pages render inside `<div className="relative flex min-w-0 flex-1 flex-col">`. | ||||
| 20 | + A page should typically be `<main className="min-h-0 flex-1 overflow-y-auto scrollbar-thin"><div className="mx-auto w-full max-w-5xl px-4 py-6 sm:px-6 lg:px-8">…</div></main>`. | ||||
| 21 | +- Auth client: `authClient`, `signIn`, `signUp`, `signOut`, `useSession` from `src/lib/auth-client.ts` (Better Auth 1.7). | ||||
| 22 | + Server: `requireUser()` / `getSession()` in `src/lib/session.ts`. | ||||
| 23 | +- Formatting helpers: `formatUsd`, `formatTokens`, `formatMs`, `formatRelative`, `formatNumber`, `cn` in `src/lib/utils.ts`. | ||||
| 24 | +- Never render key material. Connections only expose `keyHint` like `sk-••••••••9A2K`. | ||||
| 25 | +- React Compiler lint rules apply: no setState synchronously in effects without the eslint-disable, no components defined | ||||
| 26 | + inside render, derive state with useMemo. Run `pnpm lint` and `pnpm typecheck`. | ||||
| 27 | + | ||||
| 28 | +## Design language (must feel premium, unique, not a template) | ||||
| 29 | +- Minimal, dense-when-useful, excellent typography (Geist Sans/Mono via `font-sans`/`font-mono`), subtle borders | ||||
| 30 | + (`border-border`), surfaces `bg-bg / bg-bg-subtle / bg-bg-muted / bg-bg-elevated`, text `text-fg / text-fg-muted / text-fg-subtle`, | ||||
| 31 | + accent `text-accent / bg-accent / bg-accent-soft`, semantic `success/warning/danger/info` (+ `-soft`). Radii `rounded-md/lg/xl`. | ||||
| 32 | + Provider colors: `text-openai`, `text-anthropic`, `text-gemini`, `text-xai` or `style={{color: PROVIDERS[p].colorVar}}`. | ||||
| 33 | +- Dark and light must both look great. No cheap gradients, no childish AI imagery, no huge empty whitespace, no boxed-card overload. | ||||
| 34 | + A single restrained gradient (`.text-gradient`) is allowed for hero words. Use `.dot-grid` / `.glass` utilities sparingly. | ||||
| 35 | +- Mobile first: no horizontal overflow, 16px inputs on mobile (`text-[15px] sm:text-sm` is already in Input), safe-area aware. | ||||
| 36 | +- Keyboard-first: ⌘K palette, ⌘N new chat, ⌘Enter send, Esc stop, ⌘/ model selector, ⌘B sidebar. | ||||
| 37 | +- Accessibility: semantic HTML, labels, visible focus, aria on icon buttons, reduced motion respected. | ||||
| 38 | + | ||||
| 39 | +## API routes (all JSON unless noted; all under a verified session) | ||||
| 40 | +- `GET /api/models` → `{ models: PolyModel[], favorites: string[], recents: string[], connectedProviders }`; | ||||
| 41 | + `POST /api/models {action:"toggle-favorite", modelKey}`; `POST /api/models/sync {provider?}` → `{results: SyncResult[]}`. | ||||
| 42 | +- `GET /api/providers` → `{connections: PublicConnection[]}`; `PUT /api/providers {provider, apiKey}` → `{ok, error?, modelsAvailable?, connections}`; | ||||
| 43 | + `POST /api/providers {action:"validate", provider}`; `DELETE /api/providers?provider=`. | ||||
| 44 | +- `GET /api/conversations?q&provider&model&folder&archived=1&pinned=1&since&until&limit&cursor`; `POST /api/conversations`; | ||||
| 45 | + `GET|PATCH|DELETE /api/conversations/:id`; `POST /api/conversations/:id/actions {action: duplicate|branch|export|share|unshare|share-status|delete-message}`. | ||||
| 46 | +- `POST /api/chat` (SSE) body `ChatRequestInput` → events `ChatStreamEvent` (see `src/lib/client/types.ts`). | ||||
| 47 | +- `GET /api/folders`, `POST {name,color}`, `PATCH {id,…}`, `DELETE ?id=`. | ||||
| 48 | +- `POST /api/attachments` (multipart `file`) → `{attachment}`; `GET /api/attachments?id=` streams the file. | ||||
| 49 | +- `GET /api/usage?range=today|7d|30d|all` → `{ totals, series[{bucket,requests,inputTokens,outputTokens,costUsd}], byProvider[], byModel[], recent[] }`. | ||||
| 50 | +- `GET /api/presets` → `{modelPresets, promptPresets}`; `POST /api/presets?kind=model|prompt`; `PATCH ?kind&id`; `DELETE ?kind&id`. | ||||
| 51 | +- `GET /api/preferences` / `PATCH /api/preferences` (theme, defaultModelKey, defaultSystemPrompt, enterToSend, streaming, codeWrap, showReasoning, showCosts, autoTitle, onboardingCompleted). | ||||
| 52 | +- `GET /api/arena` → `{sessions}`; `POST /api/arena {prompt, systemPrompt?, modelKeys[1..4], settings?, attachmentIds?}` → `{session}`; | ||||
| 53 | + `POST /api/arena/stream {sessionId, modelKey}` (SSE: meta → text-delta/reasoning-delta/citation → done|error); `PATCH /api/arena {responseId, ratings}`. | ||||
| 54 | +- `GET /api/account` → `{audit}`; `GET /api/account?export=1` downloads the full account export. | ||||
| 55 | +- `GET /api/search?q=`. | ||||
| 56 | +- Admin: `GET /api/admin/providers`, `POST /api/admin/providers {action:"sync", provider?}` (owner only, `user.isAdmin`). | ||||
| 57 | +- Better Auth: `/api/auth/*` (client SDK). Email verification is mandatory: unverified users are redirected to `/verify-email`. | ||||
| 58 | + | ||||
| 59 | +## Model capability sheet (drives every configuration UI) | ||||
| 60 | +`PolyModel.capabilities` {text, vision, audioInput, audioOutput, imageGeneration, video, reasoning, tools, structuredOutput, | ||||
| 61 | +streaming, files, webSearch}; `parameters` {temperature, topP, topK, maxTokens, reasoningEffort, reasoningEffortLevels[], | ||||
| 62 | +thinkingBudget, thinkingBudgetRange, stop, seed, frequencyPenalty, presencePenalty, verbosity, temperatureRange}; | ||||
| 63 | +`limits` {contextTokens, maxOutputTokens}; `pricing` {inputPerMillion, cachedInputPerMillion, outputPerMillion}; | ||||
| 64 | +`status` active|preview|deprecated|unknown; `metadata.shutdownDate`, `metadata.aliases`, etc. **Never show a control the model does not support.** | ||||
added
docs/deployment-cluster.md
+91 −0
@@ -0,0 +1,91 @@ | |||||
| 1 | +# PolyLLM — cluster deployment (MacLustr) | ||||
| 2 | + | ||||
| 3 | +Production runs on the owner's private Apple Silicon cluster and is exposed through ngrok at | ||||
| 4 | +**https://www.polyllm.io**. Everything is orchestrated by `mld` (maclustr-dispatch) from the gateway node M1M32. | ||||
| 5 | + | ||||
| 6 | +``` | ||||
| 7 | +Internet → www.polyllm.io (CNAME → ngrok) → ngrok agent (PM2 polyllm-ngrok) → Next.js `next start` 0.0.0.0:8240 → PostgreSQL 17 (localhost) | ||||
| 8 | +``` | ||||
| 9 | + | ||||
| 10 | +| Item | Value | | ||||
| 11 | +| --- | --- | | ||||
| 12 | +| Node | M3U96a (pinned: local PostgreSQL 17, same host as fetcha / spinza / rareindex) | | ||||
| 13 | +| Directory | `~/apps/polyllm` (rsync'd by mld from the laptop staging area) | | ||||
| 14 | +| Processes | PM2 `polyllm-web` (Next 16, port 8240, binds 0.0.0.0), PM2 `polyllm-ngrok` (`--domain www.polyllm.io`) | | ||||
| 15 | +| Database | `postgres://localhost:5432/polyllm` (created by the post-sync hook if missing) | | ||||
| 16 | +| Manifest | `M1M32:~/dispatch/apps/polyllm.json` — **the only place secrets live** (gitignored copy `deploy/polyllm.mld.json`) | | ||||
| 17 | +| Health | `GET /api/health` (liveness), `GET /api/health/ready` (DB ping; used by mld) | | ||||
| 18 | +| Registry | `mld status` / `mld status --live` | | ||||
| 19 | + | ||||
| 20 | +## Environment variables (production) | ||||
| 21 | + | ||||
| 22 | +See `.env.example`. Required: `NODE_ENV=production`, `PORT=8240`, `PUBLIC_APP_URL=https://www.polyllm.io`, | ||||
| 23 | +`DATABASE_URL`, `AUTH_SECRET` (32+ bytes), `API_KEY_ENCRYPTION_SECRET` (32+ bytes — **rotating it invalidates every | ||||
| 24 | +stored user key**), `RESEND_API_KEY`, `RESEND_FROM_EMAIL`, `ADMIN_EMAILS`. Optional owner keys | ||||
| 25 | +(`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_GEMINI_API_KEY`, `XAI_API_KEY`) are used **only** for the scheduled | ||||
| 26 | +model-registry refresh and the admin diagnostics; user requests always use the user's own encrypted key. | ||||
| 27 | + | ||||
| 28 | +## Build & deploy | ||||
| 29 | + | ||||
| 30 | +```bash | ||||
| 31 | +# laptop | ||||
| 32 | +cd ~/Desktop/Projets/apps-web/polyllm | ||||
| 33 | +pnpm typecheck && pnpm lint && pnpm test # gates | ||||
| 34 | +scp deploy/polyllm.mld.json M1M32:~/dispatch/apps/polyllm.json # first time / when secrets change | ||||
| 35 | +~/Desktop/cluster-skill/mld stage . polyllm # laptop → M1M32 staging | ||||
| 36 | +~/Desktop/cluster-skill/mld deploy polyllm --node M3U96a | ||||
| 37 | +``` | ||||
| 38 | + | ||||
| 39 | +`mld deploy` runs the manifest hooks on the node: `pnpm install --frozen-lockfile` → `createdb polyllm` (if needed) + | ||||
| 40 | +`pnpm db:migrate` (Drizzle migrations in `drizzle/`, version-controlled) → `pnpm build` → PM2 start/reload of | ||||
| 41 | +`polyllm-web` and `polyllm-ngrok` → local health check (`/api/health/ready`) → public health check (`https://www.polyllm.io/api/health/ready`) → registry update. | ||||
| 42 | + | ||||
| 43 | +Restart / logs on the node: | ||||
| 44 | + | ||||
| 45 | +```bash | ||||
| 46 | +ssh M3U96a 'pm2 restart polyllm-web; pm2 logs polyllm-web --lines 100 --nostream' | ||||
| 47 | +ssh M3U96a 'pm2 logs polyllm-ngrok --lines 30 --nostream' | ||||
| 48 | +``` | ||||
| 49 | + | ||||
| 50 | +Migrations are applied by the hook before the new build starts; they must stay backward compatible with the running | ||||
| 51 | +version (add columns, never drop in the same release). | ||||
| 52 | + | ||||
| 53 | +## ngrok / domain / TLS | ||||
| 54 | + | ||||
| 55 | +- DNS (GoDaddy): `www.polyllm.io` CNAME → the ngrok-cname target shown in the ngrok dashboard for the reserved domain. | ||||
| 56 | + The apex `polyllm.io` should redirect to `https://www.polyllm.io` (GoDaddy forwarding). | ||||
| 57 | +- TLS terminates at ngrok (automatic certificate for the reserved domain). The app sees `X-Forwarded-For` / | ||||
| 58 | + `X-Forwarded-Proto`; Better Auth is configured with `baseURL = PUBLIC_APP_URL` and secure cookies in production. | ||||
| 59 | +- The PM2 process `polyllm-ngrok` runs `ngrok http 8240 --domain www.polyllm.io`. Only port 8240 is exposed; no admin port. | ||||
| 60 | +- Streaming: SSE responses send a `: ping` heartbeat every 15 s so ngrok never closes idle long reasoning turns. | ||||
| 61 | + | ||||
| 62 | +## Process management | ||||
| 63 | + | ||||
| 64 | +PM2 with `pm2 startup` (LaunchAgent) on the node → processes survive terminal closure and reboots; `autorestart: true`, | ||||
| 65 | +`max_memory_restart: 2G`. The model-registry scheduler runs inside the web process (`src/instrumentation.ts`, every | ||||
| 66 | +`MODEL_SYNC_INTERVAL_MINUTES`, default 360) plus a startup sync. | ||||
| 67 | + | ||||
| 68 | +## Backups | ||||
| 69 | + | ||||
| 70 | +- Daily `pg_dump` at 03:45 via launchd `dev.polyllm.backup` on M3U96a → `~/backups/polyllm/polyllm-YYYYMMDD.sql.gz`, | ||||
| 71 | + 14-day retention (`find -mtime +14 -delete`). Install with `deploy/backup/install.sh` (see that file). | ||||
| 72 | +- Restore: `gunzip -c ~/backups/polyllm/polyllm-YYYYMMDD.sql.gz | psql polyllm_restore` after `createdb polyllm_restore`; | ||||
| 73 | + swap with `ALTER DATABASE … RENAME` during a maintenance window. Test restore on the 1st of each month into a scratch DB | ||||
| 74 | + and run `select count(*) from users;` to validate. | ||||
| 75 | +- Encrypted user keys are only decryptable with the same `API_KEY_ENCRYPTION_SECRET`: back up the manifest too | ||||
| 76 | + (`M1M32:~/dispatch/apps/polyllm.json` is included in the spbgit gateway backup). | ||||
| 77 | + | ||||
| 78 | +## Rollback | ||||
| 79 | + | ||||
| 80 | +```bash | ||||
| 81 | +ssh M3U96a 'cd ~/apps/polyllm && git log --oneline -5' # deployed commit | ||||
| 82 | +# redeploy the previous tag from the laptop: | ||||
| 83 | +git checkout <previous-tag> && ~/Desktop/cluster-skill/mld stage . polyllm && ~/Desktop/cluster-skill/mld deploy polyllm --node M3U96a | ||||
| 84 | +``` | ||||
| 85 | + | ||||
| 86 | +If a migration must be reverted, restore last night's dump (see Backups) — Drizzle migrations are forward-only. | ||||
| 87 | + | ||||
| 88 | +## Moving to another node | ||||
| 89 | + | ||||
| 90 | +`mld move polyllm --to <node>` handles rsync, PM2 and ngrok, but the database is local: dump on M3U96a, restore on the | ||||
| 91 | +target (PostgreSQL 17 via Homebrew), then update `DATABASE_URL` in the manifest before the move. | ||||
added
docs/provider-research.md
+30 −0
@@ -0,0 +1,30 @@ | |||||
| 1 | +# Provider research — index | ||||
| 2 | + | ||||
| 3 | +**Last documentation audit: 2026-09-08** | ||||
| 4 | + | ||||
| 5 | +PolyLLM never relies on remembered API shapes. Before each adapter was written, the CURRENT official | ||||
| 6 | +documentation was fetched and every claim that matters for the adapter was **probed against the real API** | ||||
| 7 | +with the owner's keys. The per-provider reports below contain the base URLs, auth, SDKs and versions, streaming | ||||
| 8 | +protocols and event names, tool/structured-output formats, reasoning controls, modalities, limits, parameter | ||||
| 9 | +support matrices with exact rejection strings, error schemas, rate limits, pricing, lifecycle, and the probe | ||||
| 10 | +scripts that produced the evidence. | ||||
| 11 | + | ||||
| 12 | +| Provider | Report | Model data | SDK used | Primary API | Probe scripts | | ||||
| 13 | +| --- | --- | --- | --- | --- | --- | | ||||
| 14 | +| OpenAI | [openai.md](provider-research/openai.md) | [openai.models.json](provider-research/openai.models.json) (84) | `openai` 7.10 | Responses API (`store:false`, encrypted reasoning replay) | `research/openai/` | | ||||
| 15 | +| Anthropic | [anthropic.md](provider-research/anthropic.md) | Models API (live capabilities) + `src/lib/ai/providers/anthropic/catalog.ts` (pricing) | `@anthropic-ai/sdk` 0.124 | Messages API (adaptive thinking, `output_config`) | `scripts/provider-matrix.ts` | | ||||
| 16 | +| Google Gemini | [gemini.md](provider-research/gemini.md) | [gemini.models.json](provider-research/gemini.models.json) (19) | `@google/genai` 2.21 | `generateContentStream` (thinkingLevel/budget per family, thought signatures) | `research/gemini/` | | ||||
| 17 | +| xAI | [xai.md](provider-research/xai.md) | [xai.models.json](provider-research/xai.models.json) (7) | `openai` 7.10 with `baseURL https://api.x.ai/v1` | Chat Completions (+ `/v1/responses` for server-side web search) | `research/xai/` | | ||||
| 18 | + | ||||
| 19 | +The verified behaviour is encoded in code, not prose: | ||||
| 20 | + | ||||
| 21 | +- `src/lib/ai/providers/<provider>/catalog*.ts` — capability sheets, parameter support, pricing, lifecycle. | ||||
| 22 | +- `src/lib/ai/providers/<provider>/index.ts` — request translation that only sends what the model accepts. | ||||
| 23 | +- `docs/provider-test-matrix.md` — generated by `pnpm providers:matrix` from live requests. | ||||
| 24 | + | ||||
| 25 | +## Re-audit procedure (before every release) | ||||
| 26 | + | ||||
| 27 | +1. `pnpm models:sync` — refresh the registry from the live listing endpoints; check `model_sync_runs` for removed/added ids. | ||||
| 28 | +2. Re-fetch the docs pages linked at the bottom of each report; update the catalog data when parameters, limits or prices moved. | ||||
| 29 | +3. `pnpm providers:matrix` — must be all ✅; investigate any ❌ before shipping. | ||||
| 30 | +4. Bump `Last documentation audit` in this file and in each report. | ||||
added
docs/provider-research/anthropic.md
+41 −0
@@ -0,0 +1,41 @@ | |||||
| 1 | +# Anthropic — provider research | ||||
| 2 | + | ||||
| 3 | +**Last documentation audit: 2026-09-08** — sources: Claude Platform docs (Messages API, Models API, extended thinking, | ||||
| 4 | +structured outputs, tool use, prompt caching, pricing, errors), the bundled `claude-api` reference (models table cached | ||||
| 5 | +2026-06-24), SDK type definitions of `@anthropic-ai/sdk` 0.124.0, and live probes with the owner key (`scripts/provider-matrix.ts` | ||||
| 6 | +and `GET /v1/models`). | ||||
| 7 | + | ||||
| 8 | +## Endpoint, auth, SDK | ||||
| 9 | +- Base URL `https://api.anthropic.com`; headers `x-api-key`, `anthropic-version: 2023-06-01`. Invalid key → **401** `authentication_error`. | ||||
| 10 | +- SDK `@anthropic-ai/sdk` 0.124.0 (`client.messages.create({stream:true})` returns an async iterable of raw events; `client.models.list()` auto-paginates; `client.messages.countTokens`). | ||||
| 11 | +- Timeouts: SDK default 10 min; PolyLLM uses 10 min for chats, 20 s for validation/listing; SDK retries 1 (PolyLLM never retries mid-stream). | ||||
| 12 | + | ||||
| 13 | +## Models API (live capability metadata — verified) | ||||
| 14 | +`GET /v1/models` returns `id, display_name, created_at, max_input_tokens, max_tokens, capabilities` with | ||||
| 15 | +`thinking.types.{adaptive,enabled}`, `effort.{low,medium,high,xhigh,max}`, `image_input`, `pdf_input`, `structured_outputs`, | ||||
| 16 | +`code_execution`, `citations`, `batch`, `context_management`. 11 models for this key: `claude-fable-5-1`, `claude-opus-5`, | ||||
| 17 | +`claude-sonnet-5`, `claude-fable-5`, `claude-opus-4-8`, `claude-opus-4-7`, `claude-sonnet-4-6`, `claude-opus-4-6`, | ||||
| 18 | +`claude-opus-4-5-20251101`, `claude-haiku-4-5-20251001`, `claude-sonnet-4-5-20250929`. Aliases (`claude-haiku-4-5`) resolve to dated snapshots. | ||||
| 19 | +All current models: 1M context (Haiku 4.5: 200k), 128k max output (Haiku 4.5: 64k). | ||||
| 20 | + | ||||
| 21 | +## Verified rules encoded in the adapter | ||||
| 22 | +- **Adaptive-only generations** (`thinking.types.enabled.supported === false`: Fable 5/5.1, Opus 5/4.8/4.7, Sonnet 5) reject | ||||
| 23 | + `temperature`/`top_p`/`top_k` (400) and `budget_tokens`; the adapter exposes none of them. Effort via `output_config.effort`. | ||||
| 24 | + Sending `thinking:{type:"adaptive"}` to Haiku 4.5 → 400 `adaptive thinking is not supported on this model` (probed) → Haiku uses `{type:"enabled", budget_tokens}` (≥1024, < max_tokens). | ||||
| 25 | +- Fable 5.x: thinking always on, `{type:"disabled"}` → 400; forced `tool_choice` `any`/`tool` → 400 on Fable 5.1 (adapter downgrades to `auto`). | ||||
| 26 | +- Adaptive thinking decides when to think: trivial prompts produce **no thinking block and 0 thinking tokens** even at effort `high` (probed on Sonnet 5); a harder prompt produced 3.5k thinking tokens with `display:"summarized"`. | ||||
| 27 | +- Streaming events: `message_start` (input usage incl. cache), `content_block_start/delta/stop` with `text_delta`, `thinking_delta`, | ||||
| 28 | + `signature_delta`, `input_json_delta`, `citations_delta`; `message_delta` carries `stop_reason` (`end_turn|max_tokens|stop_sequence|tool_use|pause_turn|refusal|model_context_window_exceeded`), | ||||
| 29 | + `usage.output_tokens` and `usage.output_tokens_details.thinking_tokens`; `stop_details` on refusal. | ||||
| 30 | +- Tools: `tools[{name, description, input_schema, strict}]`; tool results as `tool_result` blocks in a **user** turn; thinking blocks replayed with signature only on the same model. | ||||
| 31 | +- Server tools: `web_search_20260209` (Opus 5/4.8/4.7/4.6, Sonnet 5/4.6; not combinable with `code_execution_20260521`), `web_search_20250305` for older models. | ||||
| 32 | +- Structured output: `output_config.format = { type: "json_schema", schema }` (Opus 4.7+, Sonnet 5, Fable — `structured_outputs.supported`). | ||||
| 33 | +- Files: PDF as `document` (base64), text files as `document` with `text/plain` source; images base64 `image` blocks. | ||||
| 34 | +- Errors: `{type:"error", error:{type, message}}`: `authentication_error`, `permission_error`, `not_found_error`, `rate_limit_error`, `overloaded_error` (529, retryable), `billing_error`, `request_too_large`, `invalid_request_error`. | ||||
| 35 | +- Pricing (USD/M, first-party): Fable 5.1 10/50 (cache read 0.25), Opus 5 5/25, Sonnet 5 2/10, Sonnet 4.6 3/15, Haiku 4.5 1/5 — see `catalog.ts`; cache reads 10 % of input. | ||||
| 36 | +- Prompt caching: `cache_control` (not used by PolyLLM v0.1; usage reports `cache_read_input_tokens`, folded into cached tokens). | ||||
| 37 | + | ||||
| 38 | +## Probe results (2026-09-08, `pnpm providers:matrix`) | ||||
| 39 | +Auth ✅ · Model list ✅ (11) · Text ✅ · Streaming ✅ · System ✅ · Vision ✅ (64×64 PNG → "Red") · Tools ✅ (streamed `input_json_delta`, args rebuilt) · | ||||
| 40 | +Structured output ✅ · Reasoning ✅ (Sonnet 5, effort high, 3 558 thinking tokens, summary streamed) · Token usage ✅ · Invalid key ✅ (401) · | ||||
| 41 | +Error normalization ✅ (unknown model → `not_found_error` → MODEL_NOT_FOUND) · Long response ✅ (407 words). | ||||
added
docs/provider-research/gemini.md
+302 −0
@@ -0,0 +1,302 @@ | |||||
| 1 | +# Google Gemini (Gemini Developer API) — provider research for PolyLLM | ||||
| 2 | + | ||||
| 3 | +Last documentation audit: **2026-09-08**. SDK probed: `@google/genai` **2.21.0** (Node 25.9, tsx 4.23). Probe scripts and raw outputs: `research/gemini/*.ts`, `research/gemini/out/*.json` (keys redacted). Key used: free-tier `AQ.`-style ("auth") key. | ||||
| 4 | + | ||||
| 5 | +> Headline for the adapter: `generateContent` / `streamGenerateContent` remain **fully supported** but Google now labels them legacy and documents everything on the new **Interactions API** (`ai.interactions.create`, GA June 2026, "recommended for all new projects", all new features land there first). This audit targets `generateContent` (what the SDK's `ai.models.*` uses) and notes Interactions where relevant. | ||||
| 6 | + | ||||
| 7 | +--- | ||||
| 8 | + | ||||
| 9 | +## 1. Endpoint, auth, headers | ||||
| 10 | + | ||||
| 11 | +| Item | Value (verified by probe unless marked docs) | | ||||
| 12 | +|---|---| | ||||
| 13 | +| Base URL | `https://generativelanguage.googleapis.com` | | ||||
| 14 | +| API version | `v1beta` (SDK default; needed for computer use / MCP / preview features). `v1` also works: probed `gemini-3.8-flash` + `thinkingLevel` on `v1` -> OK. Select via `new GoogleGenAI({ apiKey, httpOptions: { apiVersion: "v1" } })`. | | ||||
| 15 | +| Auth header | `x-goog-api-key: <key>` (works with `AQ.` keys). `?key=` query param also works (200). **`Authorization: Bearer <api key>` does NOT work**: 401 `UNAUTHENTICATED`, reason `ACCESS_TOKEN_TYPE_UNSUPPORTED` (Bearer is only for the OpenAI-compat endpoint). | | ||||
| 16 | +| Other headers | `Content-Type: application/json`. No rate-limit headers are returned (only `server-timing`, `vary`, `alt-svc`). | | ||||
| 17 | +| Key formats | New keys created in AI Studio are "auth keys" bound to a service account (`AQ.` prefix). Docs: **standard (legacy) keys will be rejected after September 2026** — expect BYOK users with old keys to break. Leaked keys: `"Your API key was reported as leaked. Please use another API key."` | | ||||
| 18 | +| SDK env vars | `GEMINI_API_KEY` or `GOOGLE_API_KEY` (`GOOGLE_API_KEY` wins if both). Pass `apiKey` explicitly in a BYOK app. | | ||||
| 19 | +| OpenAI-compat | `https://generativelanguage.googleapis.com/v1beta/openai/` with `Authorization: Bearer <key>`; `reasoning_effort` maps to thinking levels (3.x) or budgets 1024/8192/24576 (2.5); Gemini extras via `extra_body.google.*`. Beta. Not probed. | | ||||
| 20 | + | ||||
| 21 | +Docs: https://ai.google.dev/gemini-api/docs/api-key , https://ai.google.dev/gemini-api/docs/api-versions , https://ai.google.dev/gemini-api/docs/openai | ||||
| 22 | + | ||||
| 23 | +## 2. SDK | ||||
| 24 | + | ||||
| 25 | +```bash | ||||
| 26 | +pnpm add @google/genai@latest # 2.21.0 on 2026-09-08; Node >= 20 (3.x will require Node 22+) | ||||
| 27 | +``` | ||||
| 28 | +```ts | ||||
| 29 | +import { GoogleGenAI } from "@google/genai"; | ||||
| 30 | +const ai = new GoogleGenAI({ apiKey, httpOptions: { timeout: 120_000 /* ms */, apiVersion: "v1beta" } }); | ||||
| 31 | +``` | ||||
| 32 | +- `ai.models.generateContent / generateContentStream / countTokens / list / get`, `ai.chats.create` (client-side history helper), `ai.caches`, `ai.files`, `ai.batches`, `ai.live`, `ai.interactions` (new), `mcpToTool()`. | ||||
| 33 | +- Errors: `ApiError { name: "ApiError", status: <http>, message: <JSON string of the body> }` — message is the raw `{"error":{code,message,status,details}}` JSON; parse it. Retries: `httpOptions.retryOptions` exists (defaults not documented for JS). | ||||
| 34 | +- `config.abortSignal` supported on generate calls. | ||||
| 35 | +- Legacy `@google/generative-ai` is deprecated (since 2025-11-30) — do not use. | ||||
| 36 | +- SDK-side guard: `toolConfig.functionCallingConfig.streamFunctionCallArguments` throws `"streamFunctionCallArguments parameter is only supported in Gemini Enterprise Agent Platform mode, not in Gemini Developer API mode."` (no partial-args streaming on the Developer API). | ||||
| 37 | + | ||||
| 38 | +Docs: https://github.com/googleapis/js-genai , https://ai.google.dev/gemini-api/docs/libraries | ||||
| 39 | + | ||||
| 40 | +## 3. generateContent / streamGenerateContent | ||||
| 41 | + | ||||
| 42 | +REST: `POST /v1beta/models/{model}:generateContent` and `POST /v1beta/models/{model}:streamGenerateContent?alt=sse`. | ||||
| 43 | + | ||||
| 44 | +Request body: `contents[]` (`{role:"user"|"model", parts:[...]}`), `systemInstruction` (`{parts:[{text}]}`), `tools[]`, `toolConfig`, `safetySettings[]`, `generationConfig`, `cachedContent`, `serviceTier`, `store`. | ||||
| 45 | +Response: `candidates[0].content.parts[]`, `candidates[0].finishReason`, `usageMetadata`, `modelVersion`, `responseId`, `promptFeedback`, `modelStatus`. | ||||
| 46 | + | ||||
| 47 | +Part fields seen in probes: `text`, `thought: true` (thought summary), `thoughtSignature` (base64 string, 300–700 chars), `functionCall {name, args, id}`, `functionResponse`, `inlineData {mimeType, data}`, `fileData {fileUri, mimeType}`, `executableCode {language:"PYTHON", code, id}`, `codeExecutionResult {outcome:"OUTCOME_OK", output, id}`. | ||||
| 48 | + | ||||
| 49 | +FinishReason enum (SDK 2.21): `STOP, MAX_TOKENS, SAFETY, RECITATION, LANGUAGE, OTHER, BLOCKLIST, PROHIBITED_CONTENT, SPII, MALFORMED_FUNCTION_CALL, IMAGE_SAFETY, UNEXPECTED_TOOL_CALL, TOO_MANY_TOOL_CALLS, IMAGE_PROHIBITED_CONTENT, NO_IMAGE, IMAGE_RECITATION, IMAGE_OTHER`. | ||||
| 50 | + | ||||
| 51 | +Docs: https://ai.google.dev/api/generate-content , https://ai.google.dev/gemini-api/docs/text-generation | ||||
| 52 | + | ||||
| 53 | +### 3.1 Streaming protocol (verified) | ||||
| 54 | + | ||||
| 55 | +- REST with `alt=sse`: `Content-Type: text/event-stream`; each event is `data: {GenerateContentResponse JSON}` terminated by **CRLF CRLF** (`\r\n\r\n`); no `event:` lines, no `[DONE]` sentinel — the stream simply ends. Errors before the first token come back as a normal HTTP error status (400 etc.) but still with `text/event-stream` content type and a JSON `{error}` body — check `res.ok` before parsing SSE. | ||||
| 56 | +- REST without `alt=sse`: `application/json` **JSON array** of responses (`[{...},{...}]`) streamed incrementally — avoid. | ||||
| 57 | +- SDK: `for await (const chunk of await ai.models.generateContentStream({...}))`, each chunk is a full `GenerateContentResponse` (`chunk.text` getter concatenates non-thought text). | ||||
| 58 | +- Chunk shape observed (Gemini 3.x Flash): 2–4 chunks for short answers. With `includeThoughts: true` the **first chunk** carries `parts:[{text:"…", thought:true}]`; then text chunks; the **last chunk** has `finishReason` and typically an **empty text part carrying `thoughtSignature`** (`{text:"", thoughtSignature:"…"}`) — do not render it as text, but keep it if you replay history. | ||||
| 59 | +- `usageMetadata` is present on **every** chunk. On 3.5/3.6 Flash the first chunk has only `promptTokenCount/totalTokenCount/promptTokensDetails/serviceTier`; subsequent chunks have the full set (`candidatesTokenCount`, `thoughtsTokenCount`). On 3.7 Flash all chunks were full. Rule: **take usage from the last chunk**. | ||||
| 60 | +- Function calls stream as a single chunk: `parts:[{functionCall:{name,args,id:"call_…"}, thoughtSignature:"…"}]`, followed by a final chunk with an empty `text` part and `finishReason:"STOP"` (not `FUNCTION_CALL`). Args are complete (never partial). | ||||
| 61 | + | ||||
| 62 | +### 3.2 usageMetadata fields | ||||
| 63 | +`promptTokenCount`, `candidatesTokenCount`, `thoughtsTokenCount` (absent when 0), `cachedContentTokenCount` (absent when 0), `toolUsePromptTokenCount`, `totalTokenCount` (= prompt + candidates + thoughts), `promptTokensDetails[{modality:"TEXT"|"IMAGE"|…, tokenCount}]`, `cacheTokensDetails[]`, `candidatesTokensDetails[]`, `serviceTier:"standard"`. Thinking tokens are billed as output. | ||||
| 64 | + | ||||
| 65 | +### 3.3 Exact SDK code that worked | ||||
| 66 | + | ||||
| 67 | +```ts | ||||
| 68 | +const stream = await ai.models.generateContentStream({ | ||||
| 69 | + model: "gemini-3.5-flash-lite", | ||||
| 70 | + contents: [{ role: "user", parts: [{ text: "Count from 1 to 12, then say DONE." }] }], | ||||
| 71 | + config: { | ||||
| 72 | + systemInstruction: "You are terse.", | ||||
| 73 | + maxOutputTokens: 2000, // includes thinking tokens! | ||||
| 74 | + thinkingConfig: { includeThoughts: true, thinkingLevel: "LOW" }, // never combine with thinkingBudget | ||||
| 75 | + }, | ||||
| 76 | +}); | ||||
| 77 | +let usage; | ||||
| 78 | +for await (const chunk of stream) { | ||||
| 79 | + for (const part of chunk.candidates?.[0]?.content?.parts ?? []) { | ||||
| 80 | + if (part.thought) emitReasoning(part.text); | ||||
| 81 | + else if (part.text) emitText(part.text); // may be "" on the final signature-only part | ||||
| 82 | + if (part.functionCall) emitToolCall(part.functionCall.id, part.functionCall.name, part.functionCall.args, part.thoughtSignature); | ||||
| 83 | + } | ||||
| 84 | + if (chunk.candidates?.[0]?.finishReason) finish = chunk.candidates[0].finishReason; | ||||
| 85 | + usage = chunk.usageMetadata ?? usage; // last chunk wins | ||||
| 86 | +} | ||||
| 87 | +``` | ||||
| 88 | + | ||||
| 89 | +## 4. generationConfig parameter support (probed matrix) | ||||
| 90 | + | ||||
| 91 | +Probed with `maxOutputTokens: 1500` on the models the free-tier key can call. Legend: OK / 400 "…" = exact server message / — = not probed (quota). | ||||
| 92 | + | ||||
| 93 | +| param | 3.8-flash | 3.5-flash | 3.5-flash-lite | 3.1-flash-lite | 3-flash-preview | gemma-4-31b-it | | ||||
| 94 | +|---|---|---|---|---|---|---| | ||||
| 95 | +| temperature 0 / 1.5 / 2 | OK | OK | OK | OK | OK | 2 OK; 0,1.5 → 500 INTERNAL (flaky) | | ||||
| 96 | +| temperature 2.5 | 400 `* GenerateContentRequest.generation_config.temperature: temperature must be in the range [0.0, 2.0].` | same | same | same | same | same | | ||||
| 97 | +| topP 0.9 | OK | OK | OK | OK | OK | 500 INTERNAL (flaky) | | ||||
| 98 | +| topK 40 | — | OK | OK | OK | OK | OK | | ||||
| 99 | +| seed 42 | — | OK | OK | OK | OK | OK | | ||||
| 100 | +| stopSequences ["DONE"] | — | OK (stopped before DONE) | OK | OK | OK | 500 INTERNAL (flaky) | | ||||
| 101 | +| frequencyPenalty 0.5 | — | 400 `Penalty is not enabled for this model` | same | same | same | same | | ||||
| 102 | +| presencePenalty 0.5 | — | 400 `Penalty is not enabled for this model` | same | same | same | same | | ||||
| 103 | +| candidateCount 2 | 400 `Multiple candidates is not enabled for this model` | same | same | same | same | same | | ||||
| 104 | +| responseMimeType application/json | — | OK | OK | OK | OK | 500 (flaky) | | ||||
| 105 | +| responseSchema (OpenAPI, `OBJECT/STRING`) | — | OK | OK | OK | OK | OK | | ||||
| 106 | +| responseJsonSchema (JSON Schema) | — | OK | OK | OK | OK | OK | | ||||
| 107 | +| responseMimeType text/x.enum + enum schema | — | OK (`blue`) | OK | OK | OK | OK | | ||||
| 108 | +| thinkingConfig.thinkingBudget 0 | — | OK (0 thoughts) | 400 `Request contains an invalid argument.` | OK | OK | 400 `Thinking budget is not supported for this model.` | | ||||
| 109 | +| thinkingBudget 1024 / -1 | — | — | OK / OK | OK / OK | OK / OK (-1 → 1436 thought tokens) | 400 same | | ||||
| 110 | +| includeThoughts true | — | OK (thought parts) | OK | OK | OK | 500 (flaky) | | ||||
| 111 | +| thinkingLevel MINIMAL | docs: **error** | — | OK (0 thoughts) | OK (0) | OK (0) | 500 (flaky) | | ||||
| 112 | +| thinkingLevel LOW / MEDIUM / HIGH | — | — | OK | OK | HIGH, MEDIUM OK; LOW — | 400 `Thinking level is not supported for this model.` (HIGH → 500) | | ||||
| 113 | +| thinkingLevel + thinkingBudget | — | — | 400 `You can only set only one of thinking budget and thinking level.` | same | same | same | | ||||
| 114 | +| responseLogprobs/logprobs | 400 `Logprobs is not enabled for this model` | — | same | same | — | same | | ||||
| 115 | + | ||||
| 116 | +Notes: | ||||
| 117 | +- **Sampling deprecation**: changelog 2026-07-21 — "The sampling parameters `temperature`, `top_p` and `top_k` are now deprecated" (for 3.6 Flash / 3.5 Flash-Lite onward; the 3.8 guide says "Strip `temperature`, `top_p`, and `top_k` from generation configs"). They are still **accepted** by `generateContent` (probed) but Google recommends not sending them; Gemini 3 guide: "strongly recommend keeping the temperature parameter at its default value of 1.0" (lower values can cause looping). Adapter: default to not sending temperature/topP/topK for Gemini 3.x unless the user overrides. | ||||
| 118 | +- Default temperature/topP/topK reported by the models endpoint: 1 / 0.95 / 64 (maxTemperature 2) for all Gemini text models. | ||||
| 119 | +- `maxOutputTokens` **includes thinking tokens**: `gemini-3.8-flash` with `maxOutputTokens: 100` returned empty text, `finishReason: MAX_TOKENS`, `thoughtsTokenCount: 97`. Give thinking models a generous budget (>= 1024 + expected answer) or disable thinking. | ||||
| 120 | +- `stopSequences` are honoured but the stop string itself is stripped. | ||||
| 121 | + | ||||
| 122 | +## 5. Reasoning controls per family | ||||
| 123 | + | ||||
| 124 | +| Family | Default | Controls | Probe evidence | | ||||
| 125 | +|---|---|---|---| | ||||
| 126 | +| Gemini 3.8 / 3.7 Flash | `thinkingLevel` **medium** | `low, medium, high`; `minimal` → error (docs). `thinkingBudget` still accepted "for backward compatibility" per Gemini 3 guide but 3.8 guide says replace it. Cannot be disabled. | 3.8: thoughts 199–330 on trivial prompts. | | ||||
| 127 | +| Gemini 3.6 Flash | medium | `minimal, low, medium, high` | stream has thought part first. | | ||||
| 128 | +| Gemini 3.5 Flash | medium (docs) | all four levels (docs); `thinkingBudget: 0` **does** disable (probed). | thoughts 136–515 by default. | | ||||
| 129 | +| Gemini 3.5 Flash-Lite / 3.1 Flash-Lite | **minimal (off)** | all four levels; `thinkingBudget` 1024 / -1 OK; `thinkingBudget: 0` rejected on 3.5-lite (use MINIMAL) but OK on 3.1-lite. | thoughts 0 unless requested. | | ||||
| 130 | +| Gemini 3 Flash Preview / 3.1 Pro Preview | high (dynamic) | `minimal, low, medium, high`; budgets 0/1024/-1 OK on 3-flash-preview. | | | ||||
| 131 | +| Gemini 2.5 Pro / Flash / Flash-Lite | on / on / off | `thinkingBudget` (Pro 128–32768 cannot disable; Flash 0–24576; Lite 512–24576; -1 dynamic) + `thinkingLevel` low/medium/high (docs). | **Not callable with new keys (404)** — docs only. | | ||||
| 132 | +| Gemma 4 | thinks by default | **no control**: both `thinkingBudget` and `thinkingLevel` → 400. | thoughts 47–430. | | ||||
| 133 | + | ||||
| 134 | +- `includeThoughts: true` → thought **summaries** as parts with `thought: true` (streamed first). Raw reasoning is never returned. | ||||
| 135 | +- Thinking tokens: `usageMetadata.thoughtsTokenCount`, billed at the output rate. | ||||
| 136 | +- SDK enum: `ThinkingLevel.MINIMAL|LOW|MEDIUM|HIGH` (strings `"MINIMAL"`… accepted; Interactions API uses lowercase). | ||||
| 137 | + | ||||
| 138 | +Docs: https://ai.google.dev/gemini-api/docs/thinking , https://ai.google.dev/gemini-api/docs/gemini-3 , https://ai.google.dev/gemini-api/docs/latest-model | ||||
| 139 | + | ||||
| 140 | +### 5.1 Thought signatures (critical for tool calling) | ||||
| 141 | +- Gemini 3.x attaches `thoughtSignature` to the **functionCall part** (and to the last text/empty part of a text answer, and to `executableCode` parts). | ||||
| 142 | +- **Function calling is strict**: replaying the model turn without the signature → 400 `INVALID_ARGUMENT`: `Function call is missing a thought_signature in functionCall parts. This is required for tools to work correctly, and missing thought_signature may lead to degraded model performance. Additional data, function call `default_api:get_weather` , position 2. Please refer to https://ai.google.dev/gemini-api/docs/thought-signatures for more details.` (probed on gemini-3.7-flash). | ||||
| 143 | +- Echoing the real signature → OK. The documented escape hatch `thoughtSignature: "skip_thought_signature_validator"` on the functionCall part → accepted (probed) — useful when importing history from another provider. | ||||
| 144 | +- Text-only multi-turn **without** signatures works (probed): signatures on text parts are optional. | ||||
| 145 | +- Adapter rule: persist `thoughtSignature` alongside each tool call (and ideally each assistant part) in the conversation store and replay it verbatim in `contents`. | ||||
| 146 | + | ||||
| 147 | +## 6. Tool calling (function declarations) | ||||
| 148 | + | ||||
| 149 | +```ts | ||||
| 150 | +const tools = [{ functionDeclarations: [{ | ||||
| 151 | + name: "get_weather", description: "Get weather for a city.", | ||||
| 152 | + parametersJsonSchema: { type: "object", properties: { city: { type: "string" } }, required: ["city"] }, // JSON Schema (lowercase types) | ||||
| 153 | + // or legacy: parameters: { type: "OBJECT", properties: { city: { type: "STRING" } }, required: ["city"] } | ||||
| 154 | +}]}]; | ||||
| 155 | +// model turn → parts: [{ functionCall: { name, args, id: "call_88210" }, thoughtSignature }] | ||||
| 156 | +// reply: | ||||
| 157 | +contents.push({ role: "model", parts: modelPartsVerbatim }); // keep thoughtSignature! | ||||
| 158 | +contents.push({ role: "user", parts: [{ functionResponse: { name, id, response: { /* any JSON object */ } } }] }); | ||||
| 159 | +``` | ||||
| 160 | +- `toolConfig.functionCallingConfig`: `mode: AUTO | ANY | NONE | VALIDATED`, `allowedFunctionNames[]`. `ANY` + `allowedFunctionNames` probed OK on 3.6 Flash (forced call). | ||||
| 161 | +- Parallel calls arrive as multiple `functionCall` parts in one turn; send all `functionResponse` parts in one user turn (same order). | ||||
| 162 | +- `functionCall.id` is present on 3.x (`call_NNNNN`); echo it in `functionResponse.id`. | ||||
| 163 | +- Built-in tools **can be combined with function declarations on Gemini 3** (not on 2.5). Structured output + tools also allowed on Gemini 3. | ||||
| 164 | +- SDK automatic function calling exists (`config.automaticFunctionCalling`) — disable it in a UI adapter (`{ disable: true }`) to keep control of the loop. | ||||
| 165 | +- MCP: `mcpToTool(client)` in the SDK (experimental) and server-side `tools:[{ mcpServers: [...] }]` (v1beta, docs: HTTP transport only). | ||||
| 166 | + | ||||
| 167 | +Docs: https://ai.google.dev/gemini-api/docs/function-calling (now Interactions-only; legacy shapes verified by probe and SDK types) | ||||
| 168 | + | ||||
| 169 | +## 7. Structured output / JSON schema | ||||
| 170 | +- `responseMimeType: "application/json"` alone → valid JSON of free shape (one model returned an array). | ||||
| 171 | +- `responseJsonSchema` (standard JSON Schema: lowercase types, `additionalProperties`, `anyOf`, `$ref:"#"`, `enum`, `format` date/time, `minimum/maximum`, `items/prefixItems/minItems/maxItems`) → **preferred**; probed OK on all callable models incl. Gemma 4. | ||||
| 172 | +- `responseSchema` (legacy OpenAPI `Schema` with uppercase `TYPE`s, `propertyOrdering`) → still OK. Don't send both. | ||||
| 173 | +- `responseMimeType: "text/x.enum"` + `{type:"STRING", enum:[…]}` → bare enum value (probed). | ||||
| 174 | +- Works together with thinking (JSON is in non-thought parts) and, on Gemini 3, with tools. | ||||
| 175 | + | ||||
| 176 | +Docs: https://ai.google.dev/gemini-api/docs/structured-output | ||||
| 177 | + | ||||
| 178 | +## 8. Modalities | ||||
| 179 | +| Modality | How | Limits / notes | | ||||
| 180 | +|---|---|---| | ||||
| 181 | +| Image in | `inlineData {mimeType, data(base64)}` or `fileData {fileUri}`; PNG/JPEG/WebP/HEIC/HEIF; inline request total <= 20 MB; up to 3,600 images | Probed 2x2 PNG → `promptTokensDetails IMAGE 1089` tokens on 3.8/3.5 Flash (Gemini 3 default media resolution is high; docs' "258 tokens" applies to older models / low res). Use `mediaResolution` to cut cost. | | ||||
| 182 | +| PDF | `inlineData application/pdf` or Files API; <= 50 MB, <= 1000 pages, ~258 tokens/page | Not probed. | | ||||
| 183 | +| Audio in | wav/mp3/aiff/aac/ogg/flac/m4a/opus/webm…; 32 tokens/s; <= 9.5 h/prompt; >20 MB via Files API | Not probed. | | ||||
| 184 | +| Video in | inline <100 MB, Files API up to 2 GB (free) / 20 GB (paid), YouTube URLs via `fileData.fileUri`; ~100 tok/s low / 300 tok/s high res; 3.5+ have "agentic" video mode | Not probed. | | ||||
| 185 | +| Audio out | **Not via chat models.** TTS models (`gemini-3.1-flash-tts-preview`, `gemini-2.5-*-preview-tts`) with `responseModalities:["AUDIO"]` + `speechConfig` → `inlineData audio/L16 24 kHz PCM`; 32k context; Live API (`bidiGenerateContent`, WebSocket) for realtime voice. | Not probed. | | ||||
| 186 | +| Image out | Image models only (`gemini-3.1-flash-image`, `gemini-3.1-flash-lite-image`, `gemini-3-pro-image`, `gemini-2.5-flash-image`), `responseModalities:["TEXT","IMAGE"]`, `imageConfig {aspectRatio, imageSize "512px"/"1K"/"2K"/"4K"}` → `inlineData image/png`; SynthID watermark; priced per image (~$0.045–0.151). | Not probed. | | ||||
| 187 | +| Video out | `gemini-omni-1.1-flash` / `gemini-omni-flash-preview` (paid tier only), Veo 3.1 via `predictLongRunning`. | Not chat models. | | ||||
| 188 | + | ||||
| 189 | +Docs: image-understanding, document-processing, audio, video-understanding, speech-generation, image-generation, files pages under https://ai.google.dev/gemini-api/docs/ | ||||
| 190 | + | ||||
| 191 | +## 9. Files API | ||||
| 192 | +`ai.files.upload({file, config:{mimeType}})` → `{uri, mimeType, state}`; poll until `ACTIVE`; reference with `fileData:{fileUri, mimeType}`. 48 h retention, 2 GB/file, 20 GB/project. Use when the request exceeds 20 MB (100 MB for video per newer docs). Not probed. | ||||
| 193 | + | ||||
| 194 | +## 10. System instruction | ||||
| 195 | +`config.systemInstruction` (string or `Content`) — probed OK on 3.5/3.6/3.7/3.8 Flash ("BLUE" uppercase obeyed). Not probed on Gemma 4 (historically Gemma rejected developer instructions on this API; Gemma 4 docs say the system role is now supported — verify before enabling). | ||||
| 196 | + | ||||
| 197 | +## 11. Conversation state | ||||
| 198 | +- `generateContent` is **stateless**: send the full `contents` history each turn (`ai.chats` is a client-side helper only). | ||||
| 199 | +- Server-side state exists only in the **Interactions API** (`previous_interaction_id`, `store` default true; paid tier retains 55 days, free 1 day). Probed `ai.interactions.create({model, input, generation_config:{thinking_level:"low"}, store:false})` → works with this key; response has `steps[]`, `output_text`, `usage{total_input_tokens,total_output_tokens,total_thought_tokens,total_cached_tokens,total_tool_use_tokens}`. Consider it for a later "Gemini v2" adapter; it lacks Batch, explicit caching and custom safety settings today. | ||||
| 200 | + | ||||
| 201 | +## 12. Context caching | ||||
| 202 | +- **Implicit** caching is on by default for 2.5+ (min 4,096 tokens on 3.x Flash / 3.1 Pro, 2,048 on 2.5); savings show as `usageMetadata.cachedContentTokenCount`; cached input billed at ~10 % (e.g. 3.8 Flash $0.075/M). Put static content first. | ||||
| 203 | +- **Explicit**: `ai.caches.create({model, config:{contents, systemInstruction, tools, ttl:"300s"|expireTime, displayName}})` → pass `config.cachedContent = cache.name`. Models with `createCachedContent` in `supportedGenerationMethods` (all Gemini text models; not Gemma/omni). Storage $0.50–4.50 /M tokens/hour. Probe on 2.5-flash hit the 404 (model gone); not re-probed. | ||||
| 204 | + | ||||
| 205 | +Docs: https://ai.google.dev/gemini-api/docs/caching , https://ai.google.dev/api/caching | ||||
| 206 | + | ||||
| 207 | +## 13. Built-in tools | ||||
| 208 | +- **Google Search grounding**: `tools:[{googleSearch:{}}]`. Response `candidates[0].groundingMetadata { webSearchQueries[], searchEntryPoint{renderedContent: HTML chip — must be displayed per ToS}, groundingChunks[{web:{uri,title}}], groundingSupports[{segment{startIndex,endIndex,text}, groundingChunkIndices[], confidenceScores[]}] }` (docs + SDK types; probe hit 429 on both attempts — **unverified**). Pricing: 3.x — 5,000 free requests/month shared, then $14/1,000; 2.5 — 1,500 free RPD then $35/1,000. Legacy `googleSearchRetrieval` only for 1.5. | ||||
| 209 | +- **URL context**: `tools:[{urlContext:{}}]`, <= 20 URLs, 34 MB each, `candidates[0].urlContextMetadata`; billed as input tokens. | ||||
| 210 | +- **Code execution**: `tools:[{codeExecution:{}}]` probed OK on 3.6 Flash → parts `executableCode{language:"PYTHON", code, id}` + `codeExecutionResult{outcome:"OUTCOME_OK", output, id}` (both also carry `thoughtSignature`), then text. 30 s runtime, matplotlib only for plots, billed as tokens. | ||||
| 211 | +- **File Search** (RAG): stores + `tools:[{fileSearch:{fileSearchStoreNames:[…]}}]`, citations with `media_id`/page numbers; indexing $0.15/M embedding tokens, storage free. Not probed. | ||||
| 212 | +- **Computer use**: `tools:[{computerUse:{environment:"browser"|"mobile"|"desktop"}}]` on 3.8/3.7/3.5 Flash(-Lite) (preview, documented for Interactions). Not probed. | ||||
| 213 | +- **Google Maps grounding**, **MCP servers** tool: v1beta. Not probed. | ||||
| 214 | + | ||||
| 215 | +## 14. Safety settings | ||||
| 216 | +`safetySettings:[{category, threshold}]`; categories `HARM_CATEGORY_HARASSMENT | HATE_SPEECH | SEXUALLY_EXPLICIT | DANGEROUS_CONTENT` (+ `CIVIC_INTEGRITY` in SDK enum); thresholds `OFF | BLOCK_NONE | BLOCK_ONLY_HIGH | BLOCK_MEDIUM_AND_ABOVE | BLOCK_LOW_AND_ABOVE`. Default for 2.5/3 models is **OFF**. Blocked prompt → `promptFeedback.blockReason` (`SAFETY|OTHER|BLOCKLIST|PROHIBITED_CONTENT|IMAGE_SAFETY`) with no candidates; blocked answer → `finishReason: SAFETY` + `safetyRatings`. Probe was routed to 2.5-flash-lite (404) — not re-probed. Not supported on the Interactions API. | ||||
| 217 | + | ||||
| 218 | +## 15. Rate limits and tiers | ||||
| 219 | +- Tiers: Free, Tier 1 (billing linked), Tier 2 ($100 spent + 3 days), Tier 3 ($1,000 + 30 days). Dimensions: RPM, TPM (input), RPD (resets midnight Pacific) + rolling 10-minute spend caps ($10/$50/$200). Per-model tables are only shown in AI Studio (`https://aistudio.google.com/rate-limit`), not in docs. | ||||
| 220 | +- Observed free-tier quotas (from 429 `google.rpc.QuotaFailure` details): `GenerateRequestsPerDayPerProjectPerModel-FreeTier` **quotaValue 20** for `gemini-3.8-flash`; per-minute quota trips after ~5–10 requests; `gemini-3.1-pro*`, `gemini-pro-latest`, `gemini-omni-*` → **limit 0** (paid only). Quotas are **per model**, so parallelising across models is fine. | ||||
| 221 | +- 429 body includes `details[]`: `google.rpc.Help`, `google.rpc.QuotaFailure{violations[{quotaMetric, quotaId, quotaDimensions{model,location}, quotaValue}]}`, `google.rpc.RetryInfo{retryDelay:"24s"}` → honour `retryDelay`. | ||||
| 222 | +- 503 `UNAVAILABLE` "This model is currently experiencing high demand…" was frequent on 3.8-flash and gemini-flash-latest (transient; retry with backoff). Gemma 4 returned 500 `INTERNAL` on ~50 % of calls (retry). | ||||
| 223 | + | ||||
| 224 | +Docs: https://ai.google.dev/gemini-api/docs/rate-limits | ||||
| 225 | + | ||||
| 226 | +## 16. Errors | ||||
| 227 | +Body: `{"error":{"code":<http>,"message":"…","status":"<grpc status>","details":[…]}}`. Seen: | ||||
| 228 | + | ||||
| 229 | +| HTTP | status | Example | | ||||
| 230 | +|---|---|---| | ||||
| 231 | +| 400 | INVALID_ARGUMENT | temperature range, `Penalty is not enabled for this model`, `Multiple candidates is not enabled for this model`, `Logprobs is not enabled for this model`, `You can only set only one of thinking budget and thinking level.`, missing thought_signature, `Request contains an invalid argument.` | | ||||
| 232 | +| 401 | UNAUTHENTICATED | invalid key: `Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential…`, `details[0].reason: "ACCESS_TOKEN_TYPE_UNSUPPORTED"`. (Docs list 400 `API_KEY_INVALID` for malformed keys — with `AQ.`-style bogus key we got 401.) | | ||||
| 233 | +| 403 | PERMISSION_DENIED | key lacks permission / wrong project (docs) | | ||||
| 234 | +| 404 | NOT_FOUND | unknown model: `models/gemini-9-ultra is not found for API version v1beta, or is not supported for generateContent. Call ModelService.ListModels…`; **retired-for-new-users**: `This model models/gemini-2.5-flash is no longer available to new users. Please update your code to use models/gemini-3.6-flash for the latest features and improvements. We recommend you to use the Interactions API.` | | ||||
| 235 | +| 429 | RESOURCE_EXHAUSTED | quota (see §15) | | ||||
| 236 | +| 500 | INTERNAL | `Internal error encountered.` (Gemma 4, transient) | | ||||
| 237 | +| 503 | UNAVAILABLE | high demand | | ||||
| 238 | +| 504 | DEADLINE_EXCEEDED | docs | | ||||
| 239 | + | ||||
| 240 | +Retry: exponential backoff with jitter on 429/500/503/504 (and network), honour `RetryInfo.retryDelay`; never retry 400/401/403/404. Streaming errors mid-stream: SDK throws from the iterator. Timeouts: set `httpOptions.timeout` (ms); long outputs with thinking can take >60 s — use 120 s+ for streaming. | ||||
| 241 | + | ||||
| 242 | +Docs: https://ai.google.dev/gemini-api/docs/troubleshooting , https://ai.google.dev/gemini-api/docs/api-errors (now describes Interactions snake_case codes) | ||||
| 243 | + | ||||
| 244 | +## 17. Token counting | ||||
| 245 | +`ai.models.countTokens({model, contents})` → `{totalTokens}` (free, no quota cost documented). REST `:countTokens` with `generateContentRequest{contents, systemInstruction, tools}` counts everything (probed: 11 tokens text-only → 51 with system + tool) and returns `promptTokensDetails`. Rules of thumb: ~4 chars/token; image 258 tokens (<=384 px) or per 768 px tile — but Gemini 3 high res gave 1,089 for a tiny PNG; audio 32 tok/s; video ~100–300 tok/s. | ||||
| 246 | + | ||||
| 247 | +## 18. Model listing | ||||
| 248 | +`GET /v1beta/models?pageSize=1000` (paginated via `nextPageToken`; SDK `ai.models.list()` / `ai.models.get({model})`). 54 models on 2026-09-08. Fields: `name`, `version`, `displayName`, `description`, `inputTokenLimit`, `outputTokenLimit`, `supportedGenerationMethods[]` (SDK renames to `supportedActions`), `temperature`, `maxTemperature`, `topP`, `topK`, `thinking` (boolean). Filter chat models with `supportedGenerationMethods.includes("generateContent")` and exclude by name pattern (`-image`, `-tts`, `lyria`, `veo`, `transcribe`, `live`, `native-audio`, `robotics`, `computer-use`, `deep-research`, `antigravity`, `embedding`, `aqa`). **The list includes models the key cannot call** (2.5 family → 404 for new users; paid-only → 429 limit 0), so the picker must tolerate per-model failures. | ||||
| 249 | + | ||||
| 250 | +## 19. Pricing (paid tier, USD per 1M tokens; pricing page 2026-09-08) | ||||
| 251 | +| Model | Input | Cached | Output (incl. thinking) | | ||||
| 252 | +|---|---|---|---| | ||||
| 253 | +| 3.8 / 3.7 / 3.6 Flash | 0.75 (1.50 from 2027-01-01) | 0.075 (0.15) | 3.75 (7.50) | | ||||
| 254 | +| 3.5 Flash | 1.50 | 0.15 | 9.00 | | ||||
| 255 | +| 3.5 Flash-Lite | 0.30 | 0.03 | 2.50 | | ||||
| 256 | +| 3.1 Flash-Lite | 0.25 (audio 0.50) | n/a | 1.50 | | ||||
| 257 | +| 3 Flash Preview | 0.50 (audio 1.00) | n/a | 3.00 | | ||||
| 258 | +| 3.1 Pro Preview | 2.00 / 4.00 (>200k) | 0.20 / 0.40 | 12.00 / 18.00 | | ||||
| 259 | +| 2.5 Pro | 1.25 / 2.50 | 0.125 / 0.25 | 10.00 / 15.00 | | ||||
| 260 | +| 2.5 Flash | 0.30 (audio 1.00) | 0.03 | 2.50 | | ||||
| 261 | +| 2.5 Flash-Lite | 0.10 (audio 0.30) | 0.01 | 0.40 | | ||||
| 262 | +| Omni Flash | 1.50 | n/a | 9.00 text / 17.50 video | | ||||
| 263 | +| Gemma 4 | free (free tier only; "Not available" on paid) | | | | ||||
| 264 | +Batch = 50 % off; priority tier = 1.8x; cache storage $0.50–4.50 /M/h; Search grounding see §13. Free tier: prompts may be used for product improvement. | ||||
| 265 | + | ||||
| 266 | +Docs: https://ai.google.dev/gemini-api/docs/pricing | ||||
| 267 | + | ||||
| 268 | +## 20. Lifecycle, aliases, deprecations | ||||
| 269 | +- Stable ids don't change (`gemini-3.6-flash`); previews get >= 2 weeks' notice; `-latest` aliases are hot-swapped with a 2-week email notice: `gemini-flash-latest` → 3.5 Flash (changelog), `gemini-flash-lite-latest` → 3.5 Flash-Lite (probed), `gemini-pro-latest` → 3.1 Pro (quota dimension). | ||||
| 270 | +- Shut down: all `gemini-2.0-*` (2026-06-01), `gemini-3.1-flash-lite-preview` (2026-05-25; the id still answers as 3.1-flash-lite), 2.5 previews. **Gemini 2.5 Pro/Flash/Flash-Lite are closed to new users** (404) though still listed and priced. | ||||
| 271 | +- Preview shutdown windows are 3–9 months. Legacy JS SDK deprecated 2025-11-30. Standard API keys rejected after Sept 2026. | ||||
| 272 | + | ||||
| 273 | +Docs: https://ai.google.dev/gemini-api/docs/deprecations , https://ai.google.dev/gemini-api/docs/changelog , https://ai.google.dev/gemini-api/docs/models | ||||
| 274 | + | ||||
| 275 | +## 21. Probe results summary | ||||
| 276 | + | ||||
| 277 | +| Probe | 3.8-flash | 3.7-flash | 3.6-flash | 3.5-flash | 3.5-flash-lite | 3.1-flash-lite | 3-flash-preview | gemma-4-31b-it | 3.1-pro-preview | 2.5-flash / 2.5-pro | omni-1.1-flash | | ||||
| 278 | +|---|---|---|---|---|---|---|---|---|---|---|---| | ||||
| 279 | +| generateContent + systemInstruction | OK | OK | OK | OK | OK (no sysinstr) | OK (no sysinstr) | OK (no sysinstr) | OK (no sysinstr) | 429 limit 0 | 404 gone | 429 limit 0 | | ||||
| 280 | +| stream (chunk shape, usage) | 503 then quota | OK 3 chunks, usage full each | OK 4 chunks | OK 3 chunks | — | — | — | — | — | 404 | — | | ||||
| 281 | +| params matrix | partial (RPD 20) | — | — | most | full | full | most | full (flaky 500s) | — | 404 | — | | ||||
| 282 | +| function call stream + round trip | 503/429 | OK (+signature tests) | OK | OK | OK | OK | — | — | — | 404 | — | | ||||
| 283 | +| responseJsonSchema | — | — | OK | OK | OK | OK | OK | OK | — | 404 | — | | ||||
| 284 | +| vision (inline PNG) | OK 1089 img tokens | — | — | OK | — | — | — | — | 429 | 404 | 429 | | ||||
| 285 | +| googleSearch grounding | 429 | — | 429 | — | — | — | — | — | — | 404 | — | | ||||
| 286 | +| codeExecution | 429 | — | OK | — | — | — | — | — | — | — | — | | ||||
| 287 | +| invalid key | 401 UNAUTHENTICATED (REST + SDK) | | | | | | | | | | | | ||||
| 288 | +| countTokens / models.get / v1 apiVersion / interactions.create | OK / OK / OK / OK | | | | | | | | | | | | ||||
| 289 | + | ||||
| 290 | +Not probed (docs only): audio/video/PDF input, Files API, explicit caching, safety settings behaviour, TTS/image output, Live API, File Search, computer use, MCP, Batch, 3.1 Pro & omni anything (paid tier), 2.5 anything (closed to new users). | ||||
| 291 | + | ||||
| 292 | +## 22. Gotchas for the adapter (checklist) | ||||
| 293 | +1. Use `x-goog-api-key`, never Bearer. Expect 401 (not 400/403) for bad keys. | ||||
| 294 | +2. Filter the model list by `generateContent` + name patterns, and **probe-or-handle 404 "no longer available to new users"** (2.5 family) and 429 "limit: 0" (paid-only) gracefully — hide or badge those models. | ||||
| 295 | +3. Gemini 3.x: prefer `thinkingLevel` (uppercase enum in SDK), never send both level and budget, `MINIMAL` errors on 3.7/3.8, Gemma rejects both. Map PolyLLM "reasoning effort" → level; "off" → `MINIMAL` on lite/3.5-flash-lite, `thinkingBudget: 0` on 3.5 Flash / 3.1-lite / 3-flash-preview, and "not disableable" on 3.6+/Pro. | ||||
| 296 | +4. `maxOutputTokens` includes thoughts — floor it (e.g. >= 2048) when thinking is on, or you get empty text with `MAX_TOKENS`. | ||||
| 297 | +5. Store and replay `thoughtSignature` on tool-call parts (400 otherwise); `skip_thought_signature_validator` is the documented escape hatch. Strip empty signature-only text parts from the UI but keep them in history. | ||||
| 298 | +6. Usage: read `usageMetadata` from the last chunk; thoughts = `thoughtsTokenCount`, cache hits = `cachedContentTokenCount`. | ||||
| 299 | +7. Penalties, `candidateCount > 1`, logprobs → 400 on every model: don't expose them for Gemini. Temperature/topP/topK are accepted but deprecated for 3.6+; don't send by default. | ||||
| 300 | +8. SSE events are CRLF-delimited `data:` lines with no terminator; HTTP error status can arrive with `text/event-stream` content type. | ||||
| 301 | +9. Quotas are per model; free tier is tiny (20 RPD on 3.8-flash) — surface `RetryInfo.retryDelay` to the user. | ||||
| 302 | +10. Consider a future switch to the Interactions API (server-side state, unified steps); today `generateContent` is stable and complete for chat. | ||||
added
docs/provider-research/gemini.models.json
+231 −0
@@ -0,0 +1,231 @@ | |||||
| 1 | +[ | ||||
| 2 | + { | ||||
| 3 | + "id": "gemini-3.8-flash", | ||||
| 4 | + "displayName": "Gemini 3.8 Flash", | ||||
| 5 | + "family": "gemini-3.x-flash", | ||||
| 6 | + "contextTokens": 1048576, | ||||
| 7 | + "maxOutputTokens": 65536, | ||||
| 8 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 9 | + "parameters": { "temperature": "0-2 (accepted; Google recommends leaving 1.0; deprecated for 3.6+ per changelog 2026-07-21)", "topP": true, "topK": true, "maxTokens": true, "reasoningEffort": ["low", "medium", "high"], "thinkingBudget": "docs: legacy, replace with thinkingLevel; not probed (free-tier RPD=20 exhausted)", "stop": true, "seed": true, "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 10 | + "pricing": { "inputPerMillion": 0.75, "cachedInputPerMillion": 0.075, "outputPerMillion": 3.75, "note": "introductory through 2026-12-31; from 2027-01-01: 1.50 / 0.15 / 7.50. >200k-token prompts not tiered." }, | ||||
| 11 | + "status": "active", | ||||
| 12 | + "notes": "Stable, released 2026-09-02. Probed: generateContent+systemInstruction OK, vision OK (2x2 PNG = 1089 image tokens), temperature 0/1.5/2 OK, candidateCount=2 -> 400, logprobs -> 400. Default thinkingLevel medium; 'minimal' returns an error (docs). Free tier: 20 requests/day (quotaValue=20 seen in 429). Thinking tokens count against maxOutputTokens (maxOutputTokens=100 gave empty text, finishReason MAX_TOKENS, 97 thought tokens)." | ||||
| 13 | + }, | ||||
| 14 | + { | ||||
| 15 | + "id": "gemini-3.7-flash", | ||||
| 16 | + "displayName": "Gemini 3.7 Flash", | ||||
| 17 | + "family": "gemini-3.x-flash", | ||||
| 18 | + "contextTokens": 1048576, | ||||
| 19 | + "maxOutputTokens": 65536, | ||||
| 20 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 21 | + "parameters": { "temperature": "0-2 (deprecated, keep 1.0)", "topP": true, "topK": true, "maxTokens": true, "reasoningEffort": ["low", "medium", "high"], "thinkingBudget": "docs-only: legacy", "stop": "docs-only", "seed": "docs-only", "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 22 | + "pricing": { "inputPerMillion": 0.75, "cachedInputPerMillion": 0.075, "outputPerMillion": 3.75, "note": "same as 3.8 Flash: introductory through 2026-12-31, then 1.50 / 0.15 / 7.50" }, | ||||
| 23 | + "status": "active", | ||||
| 24 | + "notes": "Stable, Aug 2026. Probed: generateContent+systemInstruction OK; stream 3 chunks with full usageMetadata on every chunk; function call round trip OK; thought signature on functionCall part is MANDATORY (400 'Function call is missing a thought_signature in functionCall parts...' when omitted; dummy 'skip_thought_signature_validator' accepted); text-only multi-turn without signatures OK. Param matrix not probed (family assumed = 3.8)." | ||||
| 25 | + }, | ||||
| 26 | + { | ||||
| 27 | + "id": "gemini-3.6-flash", | ||||
| 28 | + "displayName": "Gemini 3.6 Flash", | ||||
| 29 | + "family": "gemini-3.x-flash", | ||||
| 30 | + "contextTokens": 1048576, | ||||
| 31 | + "maxOutputTokens": 65536, | ||||
| 32 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 33 | + "parameters": { "temperature": "0-2 (deprecated per changelog 2026-07-21, keep 1.0)", "topP": "deprecated", "topK": "deprecated", "maxTokens": true, "reasoningEffort": ["minimal", "low", "medium", "high"], "thinkingBudget": "docs-only: legacy", "stop": "docs-only", "seed": "docs-only", "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 34 | + "pricing": { "inputPerMillion": 0.75, "cachedInputPerMillion": 0.075, "outputPerMillion": 3.75, "note": "same as 3.8 Flash: introductory through 2026-12-31, then 1.50 / 0.15 / 7.50" }, | ||||
| 35 | + "status": "active", | ||||
| 36 | + "status_note": "Google's recommended replacement for gemini-2.5-flash (404 message).", | ||||
| 37 | + "notes": "Stable, July 2026. Probed: generateContent OK; stream 4 chunks (thought summary chunk first when includeThoughts, last chunk = empty text part carrying thoughtSignature + finishReason STOP); function call streaming (1 chunk with functionCall+thoughtSignature+id, then final empty chunk) and round trip OK; responseJsonSchema OK; codeExecution tool OK (executableCode{language:'PYTHON',code,id} + codeExecutionResult{outcome:'OUTCOME_OK',output,id}); googleSearch probe hit 429 (unverified). Param matrix not probed." | ||||
| 38 | + }, | ||||
| 39 | + { | ||||
| 40 | + "id": "gemini-3.5-flash", | ||||
| 41 | + "displayName": "Gemini 3.5 Flash", | ||||
| 42 | + "family": "gemini-3.x-flash", | ||||
| 43 | + "contextTokens": 1048576, | ||||
| 44 | + "maxOutputTokens": 65536, | ||||
| 45 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 46 | + "parameters": { "temperature": "0-2 probed OK (2.5 -> 400)", "topP": true, "topK": true, "maxTokens": true, "reasoningEffort": "docs: minimal/low/medium/high (default medium); levels not probed (429)", "thinkingBudget": "0 probed OK (disables thinking); 1024/-1 not probed (429)", "stop": true, "seed": true, "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 47 | + "pricing": { "inputPerMillion": 1.5, "cachedInputPerMillion": 0.15, "outputPerMillion": 9.0 }, | ||||
| 48 | + "status": "active", | ||||
| 49 | + "notes": "GA May 2026; current target of gemini-flash-latest per changelog. Probed: systemInstruction OK, stream 3 chunks (first chunk usage has only promptTokenCount; later chunks full), includeThoughts -> parts with thought:true, function call streaming + round trip OK (functionCall has id 'call_...'), responseMimeType json / responseSchema / responseJsonSchema / text/x.enum OK, penalties -> 400 'Penalty is not enabled for this model', candidateCount=2 -> 400, vision OK. Free tier RPD low (~20)." | ||||
| 50 | + }, | ||||
| 51 | + { | ||||
| 52 | + "id": "gemini-3.5-flash-lite", | ||||
| 53 | + "displayName": "Gemini 3.5 Flash-Lite", | ||||
| 54 | + "family": "gemini-3.x-flash-lite", | ||||
| 55 | + "contextTokens": 1048576, | ||||
| 56 | + "maxOutputTokens": 65536, | ||||
| 57 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 58 | + "parameters": { "temperature": "0-2 probed OK (deprecated per changelog 2026-07-21)", "topP": true, "topK": true, "maxTokens": true, "reasoningEffort": ["minimal", "low", "medium", "high"], "thinkingBudget": "1024 and -1 OK; 0 -> 400 'Request contains an invalid argument.' (use thinkingLevel MINIMAL to disable)", "stop": true, "seed": true, "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 59 | + "pricing": { "inputPerMillion": 0.3, "cachedInputPerMillion": 0.03, "outputPerMillion": 2.5 }, | ||||
| 60 | + "status": "active", | ||||
| 61 | + "notes": "Stable, July 2026; default thinking OFF (thoughtsTokenCount 0 unless thinkingLevel/thinkingBudget set). Full param matrix probed: all thinkingLevel values OK (minimal=0 thoughts), thinkingLevel+thinkingBudget together -> 400 'You can only set only one of thinking budget and thinking level.', penalties/candidateCount/logprobs -> 400. Function calling + responseJsonSchema OK. gemini-flash-lite-latest currently resolves here (modelVersion)." | ||||
| 62 | + }, | ||||
| 63 | + { | ||||
| 64 | + "id": "gemini-3.1-flash-lite", | ||||
| 65 | + "displayName": "Gemini 3.1 Flash-Lite", | ||||
| 66 | + "family": "gemini-3.x-flash-lite", | ||||
| 67 | + "contextTokens": 1048576, | ||||
| 68 | + "maxOutputTokens": 65536, | ||||
| 69 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 70 | + "parameters": { "temperature": "0-2 probed OK", "topP": true, "topK": true, "maxTokens": true, "reasoningEffort": ["minimal", "low", "medium", "high"], "thinkingBudget": "0 / 1024 / -1 all OK", "stop": true, "seed": true, "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 71 | + "pricing": { "inputPerMillion": 0.25, "cachedInputPerMillion": null, "outputPerMillion": 1.5, "note": "audio input 0.50/M; cached price not listed on pricing page" }, | ||||
| 72 | + "status": "active", | ||||
| 73 | + "notes": "GA May 2026; default thinkingLevel minimal (thinking off). Full param matrix probed: same acceptance pattern as 3.5-flash-lite except thinkingBudget 0 is accepted. Function calling + responseJsonSchema OK. gemini-3.1-flash-lite-preview alias still answers and reports modelVersion gemini-3.1-flash-lite (changelog says preview shut down 2026-05-25)." | ||||
| 74 | + }, | ||||
| 75 | + { | ||||
| 76 | + "id": "gemini-3-flash-preview", | ||||
| 77 | + "displayName": "Gemini 3 Flash Preview", | ||||
| 78 | + "family": "gemini-3.x-flash", | ||||
| 79 | + "contextTokens": 1048576, | ||||
| 80 | + "maxOutputTokens": 65536, | ||||
| 81 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 82 | + "parameters": { "temperature": "0-2 probed OK", "topP": true, "topK": true, "maxTokens": true, "reasoningEffort": ["minimal", "low", "medium", "high"], "thinkingBudget": "0 / 1024 / -1 all OK (-1 produced 1436 thought tokens)", "stop": true, "seed": true, "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 83 | + "pricing": { "inputPerMillion": 0.5, "cachedInputPerMillion": null, "outputPerMillion": 3.0, "note": "audio input 1.00/M" }, | ||||
| 84 | + "status": "preview", | ||||
| 85 | + "notes": "Dec 2025 preview; superseded by gemini-3.5-flash (stable). Default thinking high (dynamic). Param matrix probed except thinkingLevel low / logprobs (429). No shutdown date published." | ||||
| 86 | + }, | ||||
| 87 | + { | ||||
| 88 | + "id": "gemini-3.1-pro-preview", | ||||
| 89 | + "displayName": "Gemini 3.1 Pro Preview", | ||||
| 90 | + "family": "gemini-3.x-pro", | ||||
| 91 | + "contextTokens": 1048576, | ||||
| 92 | + "maxOutputTokens": 65536, | ||||
| 93 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 94 | + "parameters": { "temperature": "0-2 (2.5 -> 400 probed; keep 1.0)", "topP": "docs-only", "topK": "docs-only", "maxTokens": true, "reasoningEffort": ["minimal", "low", "medium", "high"], "thinkingBudget": "docs-only: legacy", "stop": "docs-only", "seed": "docs-only", "frequencyPenalty": "docs-only (family pattern: rejected)", "presencePenalty": "docs-only (family pattern: rejected)" }, | ||||
| 95 | + "pricing": { "inputPerMillion": 2.0, "cachedInputPerMillion": 0.2, "outputPerMillion": 12.0, "note": ">200k prompt: 4.00 in / 0.40 cached / 18.00 out; storage 4.50/M/hr" }, | ||||
| 96 | + "status": "preview", | ||||
| 97 | + "notes": "PAID TIER ONLY: free-tier quota is 0 (429 RESOURCE_EXHAUSTED 'limit: 0', quotaId GenerateRequestsPerDayPerProjectPerModel-FreeTier). Not probed beyond that. Default thinkingLevel high. gemini-pro-latest currently resolves here (also 429 limit 0 on free tier). Google's recommended replacement for gemini-2.5-pro." | ||||
| 98 | + }, | ||||
| 99 | + { | ||||
| 100 | + "id": "gemini-3.1-pro-preview-customtools", | ||||
| 101 | + "displayName": "Gemini 3.1 Pro Preview (custom tools)", | ||||
| 102 | + "family": "gemini-3.x-pro", | ||||
| 103 | + "contextTokens": 1048576, | ||||
| 104 | + "maxOutputTokens": 65536, | ||||
| 105 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 106 | + "parameters": { "temperature": "0-2", "topP": "docs-only", "topK": "docs-only", "maxTokens": true, "reasoningEffort": ["low", "medium", "high"], "thinkingBudget": "docs-only", "stop": "docs-only", "seed": "docs-only", "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 107 | + "pricing": { "inputPerMillion": 2.0, "cachedInputPerMillion": 0.2, "outputPerMillion": 12.0, "note": "assumed same as gemini-3.1-pro-preview (not separately listed)" }, | ||||
| 108 | + "status": "preview", | ||||
| 109 | + "notes": "Variant that prioritises custom (function) tools over built-ins in agentic/bash workflows; 'may see quality fluctuations'. Paid tier only (429 limit 0 on free tier). Not probed." | ||||
| 110 | + }, | ||||
| 111 | + { | ||||
| 112 | + "id": "gemini-2.5-pro", | ||||
| 113 | + "displayName": "Gemini 2.5 Pro", | ||||
| 114 | + "family": "gemini-2.5", | ||||
| 115 | + "contextTokens": 1048576, | ||||
| 116 | + "maxOutputTokens": 65536, | ||||
| 117 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 118 | + "parameters": { "temperature": "0-2 (2.5 -> 400 probed)", "topP": "docs-only", "topK": "docs-only", "maxTokens": true, "reasoningEffort": "docs: thinkingLevel low/medium/high accepted; OpenAI-compat maps minimal/low=1024, medium=8192, high=24576 budget", "thinkingBudget": "docs-only: 128-32768, cannot be disabled (-1 dynamic)", "stop": "docs-only", "seed": "docs-only", "frequencyPenalty": "docs-only", "presencePenalty": "docs-only" }, | ||||
| 119 | + "pricing": { "inputPerMillion": 1.25, "cachedInputPerMillion": 0.125, "outputPerMillion": 10.0, "note": ">200k prompt: 2.50 / 0.25 / 15.00; paid tier only" }, | ||||
| 120 | + "status": "deprecated", | ||||
| 121 | + "notes": "LISTED by /v1beta/models but generateContent returns 404 NOT_FOUND: 'This model models/gemini-2.5-pro is no longer available to new users. Please update your code to use models/gemini-3.1-pro-preview ... We recommend you to use the Interactions API.' Only pre-existing users can call it. Knowledge cutoff Jan 2025. Treat as legacy/hidden in a BYOK app unless the user's key is grandfathered (detect via 404 message)." | ||||
| 122 | + }, | ||||
| 123 | + { | ||||
| 124 | + "id": "gemini-2.5-flash", | ||||
| 125 | + "displayName": "Gemini 2.5 Flash", | ||||
| 126 | + "family": "gemini-2.5", | ||||
| 127 | + "contextTokens": 1048576, | ||||
| 128 | + "maxOutputTokens": 65536, | ||||
| 129 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 130 | + "parameters": { "temperature": "0-2 (2.5 -> 400 probed)", "topP": "docs-only", "topK": "docs-only", "maxTokens": true, "reasoningEffort": "docs: low/medium/high", "thinkingBudget": "docs-only: 0 (off) to 24576, -1 dynamic", "stop": "docs-only", "seed": "docs-only", "frequencyPenalty": "docs-only", "presencePenalty": "docs-only" }, | ||||
| 131 | + "pricing": { "inputPerMillion": 0.3, "cachedInputPerMillion": 0.03, "outputPerMillion": 2.5, "note": "audio input 1.00/M, cached audio 0.10/M" }, | ||||
| 132 | + "status": "deprecated", | ||||
| 133 | + "notes": "LISTED by /v1beta/models but generateContent, streamGenerateContent, cachedContents.create all return 404 'no longer available to new users' -> use gemini-3.6-flash. Only grandfathered keys can use it. Knowledge cutoff Jan 2025." | ||||
| 134 | + }, | ||||
| 135 | + { | ||||
| 136 | + "id": "gemini-2.5-flash-lite", | ||||
| 137 | + "displayName": "Gemini 2.5 Flash-Lite", | ||||
| 138 | + "family": "gemini-2.5", | ||||
| 139 | + "contextTokens": 1048576, | ||||
| 140 | + "maxOutputTokens": 65536, | ||||
| 141 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 142 | + "parameters": { "temperature": "0-2", "topP": "docs-only", "topK": "docs-only", "maxTokens": true, "reasoningEffort": "docs: low/medium/high (default off)", "thinkingBudget": "docs-only: 512-24576, 0 off", "stop": "docs-only", "seed": "docs-only", "frequencyPenalty": "docs-only", "presencePenalty": "docs-only" }, | ||||
| 143 | + "pricing": { "inputPerMillion": 0.1, "cachedInputPerMillion": 0.01, "outputPerMillion": 0.4, "note": "audio input 0.30/M" }, | ||||
| 144 | + "status": "deprecated", | ||||
| 145 | + "notes": "LISTED but 404 'no longer available to new users' -> use gemini-3.5-flash-lite (probed via chats + safetySettings probes)." | ||||
| 146 | + }, | ||||
| 147 | + { | ||||
| 148 | + "id": "gemini-flash-latest", | ||||
| 149 | + "displayName": "Gemini Flash (latest alias)", | ||||
| 150 | + "family": "alias", | ||||
| 151 | + "contextTokens": 1048576, | ||||
| 152 | + "maxOutputTokens": 65536, | ||||
| 153 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 154 | + "parameters": { "temperature": "0-2", "topP": true, "topK": true, "maxTokens": true, "reasoningEffort": ["minimal", "low", "medium", "high"], "thinkingBudget": "legacy", "stop": true, "seed": true, "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 155 | + "pricing": { "inputPerMillion": 1.5, "cachedInputPerMillion": 0.15, "outputPerMillion": 9.0, "note": "billed as the model it resolves to (currently gemini-3.5-flash per changelog May 2026); not listed on pricing page" }, | ||||
| 156 | + "status": "active", | ||||
| 157 | + "notes": "Hot-swapped alias; Google gives 2 weeks' email notice before re-pointing. Probe returned 503 'high demand' (transient) so resolution not confirmed live; changelog says gemini-3.5-flash. Avoid as default in a registry: pin explicit ids and show alias as convenience." | ||||
| 158 | + }, | ||||
| 159 | + { | ||||
| 160 | + "id": "gemini-flash-lite-latest", | ||||
| 161 | + "displayName": "Gemini Flash-Lite (latest alias)", | ||||
| 162 | + "family": "alias", | ||||
| 163 | + "contextTokens": 1048576, | ||||
| 164 | + "maxOutputTokens": 65536, | ||||
| 165 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 166 | + "parameters": { "temperature": "0-2", "topP": true, "topK": true, "maxTokens": true, "reasoningEffort": ["minimal", "low", "medium", "high"], "thinkingBudget": "1024/-1 OK, 0 rejected (as 3.5-flash-lite)", "stop": true, "seed": true, "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 167 | + "pricing": { "inputPerMillion": 0.3, "cachedInputPerMillion": 0.03, "outputPerMillion": 2.5, "note": "billed as gemini-3.5-flash-lite (probe modelVersion = gemini-3.5-flash-lite)" }, | ||||
| 168 | + "status": "active", | ||||
| 169 | + "notes": "Probed: resolves to gemini-3.5-flash-lite (modelVersion). Used for the raw REST SSE probe (works)." | ||||
| 170 | + }, | ||||
| 171 | + { | ||||
| 172 | + "id": "gemini-pro-latest", | ||||
| 173 | + "displayName": "Gemini Pro (latest alias)", | ||||
| 174 | + "family": "alias", | ||||
| 175 | + "contextTokens": 1048576, | ||||
| 176 | + "maxOutputTokens": 65536, | ||||
| 177 | + "capabilities": { "text": true, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": true, "structuredOutput": true, "streaming": true, "files": true, "webSearch": true }, | ||||
| 178 | + "parameters": { "temperature": "0-2", "topP": "docs-only", "topK": "docs-only", "maxTokens": true, "reasoningEffort": ["low", "medium", "high"], "thinkingBudget": "legacy", "stop": "docs-only", "seed": "docs-only", "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 179 | + "pricing": { "inputPerMillion": 2.0, "cachedInputPerMillion": 0.2, "outputPerMillion": 12.0, "note": "billed as the resolved model (429 quota dimension showed model=gemini-3.1-pro -> resolves to 3.1 Pro preview)" }, | ||||
| 180 | + "status": "active", | ||||
| 181 | + "notes": "Paid tier only (free-tier 429 limit 0, quota dimension model=gemini-3.1-pro)." | ||||
| 182 | + }, | ||||
| 183 | + { | ||||
| 184 | + "id": "gemma-4-31b-it", | ||||
| 185 | + "displayName": "Gemma 4 31B (instruction-tuned)", | ||||
| 186 | + "family": "gemma-4", | ||||
| 187 | + "contextTokens": 262144, | ||||
| 188 | + "maxOutputTokens": 32768, | ||||
| 189 | + "capabilities": { "text": true, "vision": "docs: image/video/audio input supported by Gemma 4; not probed on the API", "audioInput": null, "audioOutput": false, "imageGeneration": false, "video": null, "reasoning": true, "tools": "docs: yes; not probed", "structuredOutput": true, "streaming": true, "files": null, "webSearch": false }, | ||||
| 190 | + "parameters": { "temperature": "0-2 accepted (2 probed OK; 0/1.5 hit transient 500s)", "topP": "probed: 500 INTERNAL x2 (transient?)", "topK": true, "maxTokens": true, "reasoningEffort": false, "thinkingBudget": false, "stop": "probed: 500 INTERNAL x2 (transient?)", "seed": true, "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 191 | + "pricing": { "inputPerMillion": 0, "cachedInputPerMillion": null, "outputPerMillion": 0, "note": "Free tier: 'Free of charge' (data used for product improvement). Paid tier: 'Not available'." }, | ||||
| 192 | + "status": "active", | ||||
| 193 | + "notes": "Open model served on Gemini API. Thinks by default (thoughtsTokenCount 47-430) but thinkingConfig is rejected: thinkingBudget -> 400 'Thinking budget is not supported for this model.'; thinkingLevel -> 400 'Thinking level is not supported for this model.' responseSchema / responseJsonSchema / text/x.enum OK. ~50% of probe requests returned 500 INTERNAL 'Internal error encountered.' (flaky; retry). systemInstruction not probed. No createCachedContent / batch support (supportedGenerationMethods = generateContent, countTokens)." | ||||
| 194 | + }, | ||||
| 195 | + { | ||||
| 196 | + "id": "gemma-4-26b-a4b-it", | ||||
| 197 | + "displayName": "Gemma 4 26B A4B (MoE, instruction-tuned)", | ||||
| 198 | + "family": "gemma-4", | ||||
| 199 | + "contextTokens": 262144, | ||||
| 200 | + "maxOutputTokens": 32768, | ||||
| 201 | + "capabilities": { "text": true, "vision": null, "audioInput": null, "audioOutput": false, "imageGeneration": false, "video": null, "reasoning": true, "tools": null, "structuredOutput": null, "streaming": true, "files": null, "webSearch": false }, | ||||
| 202 | + "parameters": { "temperature": "0-2", "topP": null, "topK": null, "maxTokens": true, "reasoningEffort": false, "thinkingBudget": false, "stop": null, "seed": null, "frequencyPenalty": false, "presencePenalty": false }, | ||||
| 203 | + "pricing": { "inputPerMillion": 0, "cachedInputPerMillion": null, "outputPerMillion": 0, "note": "free tier only" }, | ||||
| 204 | + "status": "active", | ||||
| 205 | + "notes": "Probed once: 'pong' OK with 58 thought tokens. Assume same behaviour as gemma-4-31b-it (unknowns = null)." | ||||
| 206 | + }, | ||||
| 207 | + { | ||||
| 208 | + "id": "gemini-omni-1.1-flash", | ||||
| 209 | + "displayName": "Gemini Omni 1.1 Flash", | ||||
| 210 | + "family": "gemini-omni", | ||||
| 211 | + "contextTokens": 131072, | ||||
| 212 | + "maxOutputTokens": 65536, | ||||
| 213 | + "capabilities": { "text": "docs: text output priced (9.00/M) but model card says output = video only", "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": null, "structuredOutput": null, "streaming": null, "files": null, "webSearch": false }, | ||||
| 214 | + "parameters": { "temperature": "0-2 (2.5 -> 400 probed)", "topP": null, "topK": null, "maxTokens": true, "reasoningEffort": null, "thinkingBudget": null, "stop": null, "seed": null, "frequencyPenalty": null, "presencePenalty": null }, | ||||
| 215 | + "pricing": { "inputPerMillion": 1.5, "cachedInputPerMillion": null, "outputPerMillion": 9.0, "note": "video output 17.50/M (~0.10/s); paid tier only" }, | ||||
| 216 | + "status": "active", | ||||
| 217 | + "notes": "Video-generation model (GA Aug 2026; 3-10 s clips, 360p-4K). Models endpoint says inputTokenLimit 131072 (model card says 1,048,576 — endpoint value preferred). PAID TIER ONLY: free tier 429 'limit: 0'. Not usable for chat; exclude from the chat model picker or gate behind paid-tier detection." | ||||
| 218 | + }, | ||||
| 219 | + { | ||||
| 220 | + "id": "gemini-omni-flash-preview", | ||||
| 221 | + "displayName": "Gemini Omni Flash Preview", | ||||
| 222 | + "family": "gemini-omni", | ||||
| 223 | + "contextTokens": 131072, | ||||
| 224 | + "maxOutputTokens": 65536, | ||||
| 225 | + "capabilities": { "text": null, "vision": true, "audioInput": true, "audioOutput": false, "imageGeneration": false, "video": true, "reasoning": true, "tools": null, "structuredOutput": null, "streaming": null, "files": null, "webSearch": false }, | ||||
| 226 | + "parameters": { "temperature": "0-2", "topP": null, "topK": null, "maxTokens": true, "reasoningEffort": null, "thinkingBudget": null, "stop": null, "seed": null, "frequencyPenalty": null, "presencePenalty": null }, | ||||
| 227 | + "pricing": { "inputPerMillion": 1.5, "cachedInputPerMillion": null, "outputPerMillion": 9.0, "note": "video output 17.50/M; paid tier only" }, | ||||
| 228 | + "status": "preview", | ||||
| 229 | + "notes": "Preview predecessor of gemini-omni-1.1-flash (June 2026). Free tier 429 'limit: 0'. Not a chat model." | ||||
| 230 | + } | ||||
| 231 | +] | ||||
added
docs/provider-research/openai.md
+390 −0
@@ -0,0 +1,390 @@ | |||||
| 1 | +# OpenAI provider audit for PolyLLM | ||||
| 2 | + | ||||
| 3 | +Last documentation audit: 2026-09-08 | ||||
| 4 | +SDK probed: `openai@7.10.0` (npm), Node 25.9.0, `tsx` 4.23.13. Probe scripts and raw results: `research/openai/` (`results/*.json`, no secrets). | ||||
| 5 | +Model registry produced from this audit: `docs/provider-research/openai.models.json` (84 entries). | ||||
| 6 | + | ||||
| 7 | +> Docs moved: every `platform.openai.com/docs/...` URL now 301-redirects to `https://developers.openai.com/api/docs/...`. The API reference pages there render only partially through a headless fetch (the "create response" and "streaming events" pages surfaced only the cancel endpoint), so event names, enums and parameter lists below were cross-checked against the SDK type definitions (`node_modules/openai/resources/responses/responses.d.ts`, `shared.d.ts`) and against live probes. Where docs and probes disagree, the probe wins and the disagreement is called out. | ||||
| 8 | + | ||||
| 9 | +--- | ||||
| 10 | + | ||||
| 11 | +## 1. Endpoint, auth, headers | ||||
| 12 | + | ||||
| 13 | +| Item | Value | | ||||
| 14 | +|---|---| | ||||
| 15 | +| Base URL | `https://api.openai.com/v1/` (docs also mention data-residency endpoints in the SDK 7.6 release notes; not needed for BYOK) | | ||||
| 16 | +| Auth | `Authorization: Bearer <OPENAI_API_KEY>` | | ||||
| 17 | +| Optional headers | `OpenAI-Organization: org_...`, `OpenAI-Project: proj_...` (billing scope), `X-Client-Request-Id` (ASCII, <= 512 chars), `OpenAI-Safety-Identifier` (Realtime only; for HTTP use the `safety_identifier` body param) | | ||||
| 18 | +| Response headers seen | `x-request-id`, `openai-processing-ms`, `openai-version: 2020-10-01`, `openai-organization`, `openai-project`, `x-ratelimit-*` (see section 13) | | ||||
| 19 | +| Header size limits | total < 64 KiB, single custom header value < 60 KiB | | ||||
| 20 | +| `OpenAI-Beta` | Not required for Responses, Chat Completions, tools or structured outputs (none sent by SDK 7.x for these). | | ||||
| 21 | + | ||||
| 22 | +Source: https://developers.openai.com/api/docs/api-reference/introduction | ||||
| 23 | + | ||||
| 24 | +## 2. Which API: Responses vs Chat Completions | ||||
| 25 | + | ||||
| 26 | +- **Use the Responses API (`POST /v1/responses`) as the primary adapter.** Docs: "While Chat Completions remains supported, Responses is recommended for all new projects." Chat Completions is *not* labelled legacy/deprecated and has no shutdown date, but: | ||||
| 27 | + - Pro models (`gpt-5.5-pro`, `gpt-5.4-pro`, `gpt-5.2-pro`, `gpt-5-pro`, `o3-pro`, `o1-pro`), `gpt-5.3-codex` and `gpt-5.6-cyber` are **Responses-only**. | ||||
| 28 | + - Built-in tools (web_search, file_search, code_interpreter, MCP, image_generation, computer use) exist only in Responses (Chat Completions has only `web_search_options` on the `*-search-*` models). | ||||
| 29 | + - Reasoning docs: "Reasoning models work better with the Responses API"; GPT-6 Astra page: "Requires Responses API for tool calling (Chat Completions available but limited)". | ||||
| 30 | + - Reasoning summaries, encrypted reasoning, `previous_response_id`, `conversation`, background mode, `reasoning.context` are Responses-only. | ||||
| 31 | +- Chat Completions still works (verified 2026-09-08 on gpt-4.1-mini, gpt-5.5, o4-mini): chunk shape `chat.completion.chunk` with `choices[0].delta`, `usage` only in the final chunk when `stream_options.include_usage: true`; usage fields `prompt_tokens`/`completion_tokens`/`prompt_tokens_details.cached_tokens`/`completion_tokens_details.reasoning_tokens`. `max_tokens` is rejected on new models (`gpt-6-astra`: `400 unsupported_parameter "'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead."`). | ||||
| 32 | +- The Chat-Completions-only web search models (`gpt-5-search-api`, `gpt-4o-search-preview`, `gpt-4o-mini-search-preview`) return `400 model_not_found "The requested model 'gpt-5-search-api' is not supported with the Responses API."` on Responses. Do not expose them; use the `web_search` tool on a normal model. | ||||
| 33 | +- Assistants API shut down 2026-08-26 (replaced by Responses + Conversations). `v1/prompts` (reusable prompts), Evals, Agent Builder shut down 2026-11-30. | ||||
| 34 | + | ||||
| 35 | +Sources: https://developers.openai.com/api/docs/api-reference/chat , https://developers.openai.com/api/docs/guides/migrate-to-responses , https://developers.openai.com/api/docs/deprecations | ||||
| 36 | + | ||||
| 37 | +### Chat Completions -> Responses mapping (for the adapter) | ||||
| 38 | + | ||||
| 39 | +| Chat Completions | Responses | | ||||
| 40 | +|---|---| | ||||
| 41 | +| `messages` | `input` (string, or array of items: messages `{role, content}` / `function_call` / `function_call_output` / `reasoning` ...) | | ||||
| 42 | +| `system` message | `instructions` (top-level) or a `developer` (or `system`) role message; all three verified to work | | ||||
| 43 | +| `max_tokens` / `max_completion_tokens` | `max_output_tokens` (min 16; includes reasoning tokens) | | ||||
| 44 | +| `response_format` | `text.format` (`{type:"text"}` / `{type:"json_schema", name, schema, strict}` / `{type:"json_object"}`) | | ||||
| 45 | +| `tools[].function.{name,parameters}` | flattened `tools[] = {type:"function", name, description, parameters, strict}` | | ||||
| 46 | +| `tool_calls[]` in message | separate `function_call` output items (`call_id`, `name`, `arguments`) | | ||||
| 47 | +| `role:"tool"` message | `{type:"function_call_output", call_id, output}` input item | | ||||
| 48 | +| `reasoning_effort`, `verbosity` | `reasoning.effort`, `text.verbosity` | | ||||
| 49 | +| `usage.prompt_tokens` / `completion_tokens` | `usage.input_tokens` / `output_tokens` (+ `input_tokens_details.cached_tokens`, `cache_write_tokens`, `output_tokens_details.reasoning_tokens`, `total_tokens`) | | ||||
| 50 | +| `n`, `seed`, `stop`, `logprobs` (bool) | `n` removed; **`seed` and `stop` do not exist** (`400 unknown_parameter`); logprobs via `top_logprobs` + `include:["message.output_text.logprobs"]` | | ||||
| 51 | +| `stream_options.include_usage` | not needed: `response.completed` carries the full `response.usage` | | ||||
| 52 | + | ||||
| 53 | +## 3. Request shape (Responses) - parameters that matter | ||||
| 54 | + | ||||
| 55 | +From SDK `ResponseCreateParams` (7.10.0) and probes: | ||||
| 56 | + | ||||
| 57 | +`model`, `input`, `instructions`, `max_output_tokens` (>= 16), `reasoning {effort, summary, mode, context}`, `text {format, verbosity}`, `tools`, `tool_choice` (`"auto"|"none"|"required"|{type:"function",name}|{type:"allowed_tools",...}`), `parallel_tool_calls`, `temperature`, `top_p`, `top_logprobs`, `include[]`, `store` (default **true**), `previous_response_id`, `conversation`, `stream`, `stream_options {include_obfuscation}`, `background`, `truncation` (`"auto"|"disabled"`, default disabled), `metadata`, `service_tier` (`auto|default|flex|priority|fast`), `prompt_cache_key`, `prompt_cache_retention` (`in_memory|24h`, pre-5.6), `prompt_cache_options {mode, ttl:"30m"}` (5.6+), `safety_identifier`, `user`, `context_management` (compaction), `moderation`, `prompt`. | ||||
| 58 | + | ||||
| 59 | +`include` values: `file_search_call.results`, `web_search_call.results`, `web_search_call.action.sources`, `message.input_image.image_url`, `computer_call_output.output.image_url`, `code_interpreter_call.outputs`, `reasoning.encrypted_content`, `message.output_text.logprobs`. | ||||
| 60 | + | ||||
| 61 | +Response object: `id` (`resp_...`), `object:"response"`, `status` (`completed|failed|in_progress|cancelled|queued|incomplete`), `incomplete_details.reason` (`max_output_tokens|max_messages|content_filter|steered`), `error`, `model` (resolved snapshot, e.g. `gpt-5.5-2026-04-23`), `output[]` items, `usage`, plus echoes of `reasoning`, `text`, `temperature`, `top_p`, `truncation`, `store`, `service_tier`, `previous_response_id`, `tools`. The SDK adds a convenience `output_text` getter **only on non-streaming `create()` results** (the `response` object inside a `response.completed` event does not have it - accumulate deltas or read `output[].content[].text`). | ||||
| 62 | + | ||||
| 63 | +## 4. Streaming protocol | ||||
| 64 | + | ||||
| 65 | +SSE; every event is JSON with `type` and a monotonically increasing `sequence_number` (verified: strictly +1 per event). Text deltas also carry an `obfuscation` padding string (disable via `stream_options.include_obfuscation:false`). | ||||
| 66 | + | ||||
| 67 | +Event names observed in probes (exact strings): | ||||
| 68 | + | ||||
| 69 | +``` | ||||
| 70 | +response.created > response.in_progress | ||||
| 71 | + > response.output_item.added {item:{type:"reasoning"...}} (reasoning models, effort >= low) | ||||
| 72 | + > response.reasoning_summary_part.added | ||||
| 73 | + > response.reasoning_summary_text.delta {delta} (x N) (only when reasoning.summary is set AND the model emits one) | ||||
| 74 | + > response.reasoning_summary_text.done {text} | ||||
| 75 | + > response.reasoning_summary_part.done | ||||
| 76 | + > response.output_item.done | ||||
| 77 | + > response.output_item.added {item:{type:"message", role:"assistant"}} | ||||
| 78 | + > response.content_part.added {part:{type:"output_text", text:"", annotations:[]}} | ||||
| 79 | + > response.output_text.delta {delta, item_id, output_index, content_index, logprobs, obfuscation} (x N) | ||||
| 80 | + > response.output_text.annotation.added {annotation:{type:"url_citation",...}, annotation_index} (web search) | ||||
| 81 | + > response.output_text.done {text} | ||||
| 82 | + > response.content_part.done | ||||
| 83 | + > response.output_item.done | ||||
| 84 | +> response.completed {response:{...usage}} | response.incomplete | response.failed | error | ||||
| 85 | +``` | ||||
| 86 | + | ||||
| 87 | +Function calls (verified on all six models): | ||||
| 88 | + | ||||
| 89 | +``` | ||||
| 90 | +response.output_item.added {item:{type:"function_call", id:"fc_...", call_id:"call_...", name, arguments:"", status:"in_progress"}, output_index} | ||||
| 91 | +response.function_call_arguments.delta {delta, item_id, output_index} | ||||
| 92 | +response.function_call_arguments.done {arguments:"{...}", item_id} | ||||
| 93 | +response.output_item.done {item:{type:"function_call", ..., arguments:"{...}", status:"completed"}} | ||||
| 94 | +response.completed | ||||
| 95 | +``` | ||||
| 96 | + | ||||
| 97 | +Web search: `response.web_search_call.in_progress` > `.searching` > `.completed` (item type `web_search_call`). | ||||
| 98 | + | ||||
| 99 | +Full list of event types in the SDK (for exhaustive switch statements): `response.created, response.in_progress, response.queued, response.completed, response.incomplete, response.failed, error, response.output_item.added/done, response.content_part.added/done, response.output_text.delta/done, response.output_text.annotation.added, response.refusal.delta/done, response.reasoning_text.delta/done, response.reasoning_summary_part.added/done, response.reasoning_summary_text.delta/done, response.function_call_arguments.delta/done, response.custom_tool_call_input.delta/done, response.web_search_call.in_progress/searching/completed, response.file_search_call.in_progress/searching/completed, response.code_interpreter_call.in_progress/interpreting/completed, response.code_interpreter_call_code.delta/done, response.image_generation_call.in_progress/generating/partial_image/completed, response.mcp_call.in_progress/completed/failed, response.mcp_call_arguments.delta/done, response.mcp_list_tools.in_progress/completed/failed, response.shell_call_command.added/delta/done, response.shell_call_output_content.delta/done, response.audio.delta/done, response.audio.transcript.delta/done, response.steer.*`. | ||||
| 100 | + | ||||
| 101 | +Resumable streams: with `background:true, stream:true`, reconnect with `GET /v1/responses/{id}?stream=true&starting_after={sequence_number}`. | ||||
| 102 | + | ||||
| 103 | +Sources: https://developers.openai.com/api/docs/guides/streaming-responses , https://developers.openai.com/api/docs/guides/background , SDK types. | ||||
| 104 | + | ||||
| 105 | +### Exact SDK streaming code that worked (research/openai/03-stream.ts, 05-tools-stream.ts) | ||||
| 106 | + | ||||
| 107 | +```ts | ||||
| 108 | +import OpenAI from "openai"; | ||||
| 109 | +const client = new OpenAI({ maxRetries: 1, timeout: 180_000 }); // key from OPENAI_API_KEY | ||||
| 110 | + | ||||
| 111 | +const stream = await client.responses.create({ | ||||
| 112 | + model: "gpt-5.5", | ||||
| 113 | + input: [{ role: "user", content: "Say hello in five words." }], | ||||
| 114 | + reasoning: { effort: "low", summary: "auto" }, | ||||
| 115 | + max_output_tokens: 200, | ||||
| 116 | + stream: true, | ||||
| 117 | +}); | ||||
| 118 | +let text = "", final: OpenAI.Responses.Response | null = null; | ||||
| 119 | +for await (const ev of stream) { | ||||
| 120 | + switch (ev.type) { | ||||
| 121 | + case "response.output_text.delta": text += ev.delta; break; | ||||
| 122 | + case "response.reasoning_summary_text.delta": /* show thinking */ break; | ||||
| 123 | + case "response.function_call_arguments.delta": /* accumulate by ev.item_id */ break; | ||||
| 124 | + case "response.output_item.done": if (ev.item.type === "function_call") { /* ev.item.call_id, name, arguments */ } break; | ||||
| 125 | + case "response.completed": case "response.incomplete": final = ev.response; break; // final.usage, final.output | ||||
| 126 | + case "error": throw new Error(ev.message); | ||||
| 127 | + } | ||||
| 128 | +} | ||||
| 129 | +``` | ||||
| 130 | + | ||||
| 131 | +Tool round trip (stateless): replay `[...previousInput, ...final.output, { type: "function_call_output", call_id, output: JSON.stringify(result) }]`. Replaying the *entire* `output` array (including `reasoning` items) is what the docs require for reasoning models; verified on gpt-5.5, gpt-5.4-mini, gpt-5.6-sol, gpt-6-astra, gpt-4.1-mini. Stateful alternative verified on gpt-5.4-mini: `previous_response_id: final.id, input: [function_call_output]`. | ||||
| 132 | + | ||||
| 133 | +## 5. Tool calling format | ||||
| 134 | + | ||||
| 135 | +- Definition: `{ type: "function", name, description, parameters: <JSON Schema>, strict: true }` (flattened; `strict` recommended; strict requires `additionalProperties:false` and all properties in `required`). | ||||
| 136 | +- Output item: `{ id: "fc_...", type: "function_call", status: "completed", call_id: "call_...", name, arguments: "<json string>" }` (verified shape). `call_id` is what you echo back; `id` is the item id. | ||||
| 137 | +- Result item: `{ type: "function_call_output", call_id, output: string | [content parts for images/files] }` (SDK 7.7 made the `id` optional). | ||||
| 138 | +- `tool_choice`: `"auto"` (default) | `"none"` | `"required"` | `{type:"function", name}` | `{type:"allowed_tools", mode:"auto"|"required", tools:[...]}`. | ||||
| 139 | +- `parallel_tool_calls` default true. | ||||
| 140 | +- Other tool types accepted in `tools[]`: `web_search` (and legacy `web_search_preview`), `file_search`, `code_interpreter`, `image_generation`, `mcp`, `computer` / `computer_use_preview`, `shell`, `local_shell`, `apply_patch`, `custom` (free-form text tool), `namespace` (grouping, `defer_loading`), `tool_search`, `programmatic_tool_calling`. | ||||
| 141 | + | ||||
| 142 | +Sources: https://developers.openai.com/api/docs/guides/function-calling , https://developers.openai.com/api/docs/guides/tools | ||||
| 143 | + | ||||
| 144 | +## 6. Structured output / JSON schema | ||||
| 145 | + | ||||
| 146 | +- `text: { format: { type: "json_schema", name, schema, strict: true } }` - verified on gpt-5.5, gpt-5.4-mini, gpt-5.6-sol, gpt-6-astra, gpt-4.1-mini (parsed `{name:"Marie Curie",age:66,city:"Paris"}`); o4-mini accepted it but ran out of the 200-token budget on reasoning. | ||||
| 147 | +- Schema subset: root must be `object`; every property in `required`; `additionalProperties:false`; optional fields via `type:["string","null"]`; supports `enum`, `anyOf`, `$ref`/`$defs`, recursion; no `minLength`/`pattern`/`format` etc. Numeric limits (depth/property count/enum size) exist but the page did not expose them - keep schemas modest. | ||||
| 148 | +- Refusals arrive as a `refusal` content part (`response.refusal.delta/done`) instead of JSON. | ||||
| 149 | +- `{type:"json_object"}` (JSON mode) is the fallback for gpt-4 / gpt-3.5-turbo which lack structured outputs. | ||||
| 150 | +- SDK helpers: `zodTextFormat(schema, "name")` + `client.responses.parse(...)` -> `response.output_parsed`. | ||||
| 151 | + | ||||
| 152 | +Source: https://developers.openai.com/api/docs/guides/structured-outputs | ||||
| 153 | + | ||||
| 154 | +## 7. Reasoning controls | ||||
| 155 | + | ||||
| 156 | +- `reasoning.effort` values in the SDK enum: `none | minimal | low | medium | high | xhigh | max`. **Per-model acceptance (verified, exact error strings in section 20):** | ||||
| 157 | + - gpt-6-astra: `low, medium, high, xhigh, max` (default `medium`; `none`/`minimal` -> 400) | ||||
| 158 | + - gpt-5.6-sol/terra/luna: `none, low, medium, high, xhigh, max` (default `medium`) | ||||
| 159 | + - gpt-5.5: `none, low, medium, high, xhigh` (default `medium`) | ||||
| 160 | + - gpt-5.4 / 5.4-mini / 5.4-nano, gpt-5.2: `none, low, medium, high, xhigh` (default **`none`** - verified echo) | ||||
| 161 | + - gpt-5.1: `none, low, medium, high` (default `none`) | ||||
| 162 | + - gpt-5 / 5-mini / 5-nano: `minimal, low, medium, high` (default `medium`; no `none`) | ||||
| 163 | + - o3, o4-mini, o3-mini, o1: `low, medium, high` (default `medium`) | ||||
| 164 | + - pro models: 5.5-pro `medium|high|xhigh` (default high), 5.4-pro / 5.2-pro `medium|high|xhigh`, gpt-5-pro `high` only | ||||
| 165 | + - gpt-4.1 / gpt-4o / chat-latest: `reasoning.effort` -> `400 unsupported_parameter` | ||||
| 166 | +- `reasoning.summary`: `auto | concise | detailed` (server echoes `detailed` for `auto`). Summary text arrives in `reasoning` items' `summary[]` and via `response.reasoning_summary_*` events; gpt-5.5 and gpt-6-astra returned no summary on a trivial prompt while gpt-5.6-sol and o4-mini did. Never assume a summary will exist. Non-reasoning models silently ignore `summary`. | ||||
| 167 | +- `reasoning.mode`: `standard | pro` (GPT-5.6; pro replaces the `*-pro` models). Not probed (cost). | ||||
| 168 | +- `reasoning.context`: `current_turn | all_turns` (`all_turns` is the default on 5.6 and 6; earlier models default `current_turn`). Reasoning persists only within a family. | ||||
| 169 | +- Reasoning tokens are billed as output and count against `max_output_tokens`; `usage.output_tokens_details.reasoning_tokens` reports them. Docs recommend reserving >= 25k tokens; with tiny caps, o4-mini / gpt-5-nano return `status:"incomplete", incomplete_details.reason:"max_output_tokens"` with an empty message (verified). **Adapter rule: never cap reasoning models below ~1-2k output tokens; treat `incomplete` + empty text as "reasoning exhausted the budget".** | ||||
| 170 | +- Encrypted reasoning for stateless (store:false) multi-turn: `store:false, include:["reasoning.encrypted_content"]` -> `reasoning` items carry `encrypted_content` (~1.2 kB for a trivial turn); replaying them in the next `input` works (verified on gpt-5.5). `previous_response_id` on a `store:false` response fails: `400 previous_response_not_found`. | ||||
| 171 | +- GPT-6 Astra: mid-conversation `configuration_update` items to change effort; async tool calling (`async:true`); no `none` effort. | ||||
| 172 | + | ||||
| 173 | +Sources: https://developers.openai.com/api/docs/guides/reasoning , https://developers.openai.com/api/docs/guides/latest-model | ||||
| 174 | + | ||||
| 175 | +## 8. Sampling parameter support matrix (Responses API, verified 2026-09-08) | ||||
| 176 | + | ||||
| 177 | +| Param | gpt-6-astra | gpt-5.6-sol | gpt-5.5 | gpt-5.4-mini (default effort none) | gpt-4.1-mini | o4-mini | | ||||
| 178 | +|---|---|---|---|---|---|---| | ||||
| 179 | +| `temperature` | rejected | rejected unless `effort:"none"` (then accepted) | rejected unless `effort:"none"` | accepted | accepted | rejected | | ||||
| 180 | +| `top_p` | rejected | same as temperature | same | accepted | accepted | rejected | | ||||
| 181 | +| `frequency_penalty` | rejected | rejected (effort medium) | rejected | accepted | accepted | rejected | | ||||
| 182 | +| `presence_penalty` | rejected | rejected | rejected | accepted | accepted | rejected | | ||||
| 183 | +| `seed` | `unknown_parameter` on every model (does not exist in Responses) | | | | | | | ||||
| 184 | +| `stop` | `unknown_parameter` on every model ("Did you mean 'store'?") | | | | | | | ||||
| 185 | +| logprobs (`top_logprobs`+include) | rejected (reasoning) | rejected | rejected | accepted (`logprobs[]` on content part) | accepted | rejected | | ||||
| 186 | +| `reasoning.effort` | low..max | none..max | none..xhigh | none..xhigh | rejected | low..high | | ||||
| 187 | +| `reasoning.summary` | accepted | accepted | accepted | accepted | ignored (no error) | accepted | | ||||
| 188 | +| `text.verbosity: "low"` | accepted | accepted | accepted | accepted | rejected (`Supported values are: 'medium'`) | rejected | | ||||
| 189 | +| `max_output_tokens` | min 16 on all (`integer_below_min_value`) | | | | | | | ||||
| 190 | +| `truncation:"auto"` | accepted on all | | | | | | | ||||
| 191 | +| `store:false` + `include:["reasoning.encrypted_content"]` | accepted on all (non-reasoning models just have no reasoning item) | | | | | | | ||||
| 192 | +| `prompt_cache_key` | accepted on all | | | | | | | ||||
| 193 | + | ||||
| 194 | +Default `top_p` echoed by GPT-5.x/6 is `0.98`, by GPT-4.1 `1`. `temperature` default 1. `top_k` does not exist. There is no thinking *budget* parameter (effort only). Chat Completions on gpt-5.5: `temperature:0.3` -> `400 unsupported_value "'temperature' does not support 0.3 with this model. Only the default (1) value is supported."` | ||||
| 195 | + | ||||
| 196 | +Practical rule for the adapter: send sampling params only when (model is non-reasoning) or (reasoning-capable model with `reasoning.effort === "none"`); otherwise drop them client-side and show "not supported" in the UI. Never send `seed`/`stop` to Responses. | ||||
| 197 | + | ||||
| 198 | +## 9. Roles and instructions | ||||
| 199 | + | ||||
| 200 | +Input roles accepted: `user`, `assistant`, `developer`, `system` (both `system` and `developer` verified on gpt-5.5 and gpt-4.1-mini; docs describe `developer` as the canonical name and treat `instructions` as equivalent to a developer message). `instructions` is not inherited through `previous_response_id`; resend it each call. Content parts: `input_text`, `input_image`, `input_file`, (`input_audio` rejected, see 10); assistant output parts: `output_text`, `refusal`. | ||||
| 201 | + | ||||
| 202 | +Source: https://developers.openai.com/api/docs/guides/text | ||||
| 203 | + | ||||
| 204 | +## 10. Modalities | ||||
| 205 | + | ||||
| 206 | +| Modality | Responses API support | Notes | | ||||
| 207 | +|---|---|---| | ||||
| 208 | +| Text in/out | yes | | | ||||
| 209 | +| Image in | yes on every current model except `o3-mini`, `gpt-4`, `gpt-3.5-turbo` | `{type:"input_image", image_url:"data:image/png;base64,..." \| https URL \| file_id, detail:"low"\|"high"\|"auto"\|"original"}`. PNG/JPEG/WEBP/non-animated GIF; <= 512 MB per request, <= 1,500 images, <= 30,000 patches after resize. Tokens ~ `ceil(w/32)*ceil(h/32)` x model multiplier (1.2-2.46). Verified with a 2x2 PNG on all six probe models (31-33 input tokens). | | ||||
| 210 | +| PDF / file in | yes (vision models) | `{type:"input_file", filename, file_data:"data:application/pdf;base64,..." \| file_id \| file_url}`; <= 50 MB per file and per request; both text and page images are sent (costly). Verified on gpt-5.4-mini (answered `42` from the PDF). | | ||||
| 211 | +| Image out | via tool only | `tools:[{type:"image_generation"}]` uses `gpt-image-2`; returns base64 in an `image_generation_call` item; events `response.image_generation_call.partial_image`. Not probed. Older gpt-image-1* shut down 2026-10-23 / 2026-12-01. | | ||||
| 212 | +| Audio in/out | **no** in Responses | probe: `400 invalid_request_error param=input "Audio input is not available."`. Audio chat = Chat Completions with `gpt-audio-1.5` (`input_audio` parts, `modalities:["text","audio"]`) or Realtime (`gpt-realtime-2.1`). TTS: `gpt-4o-mini-tts`; STT: `gpt-transcribe`, `gpt-live-transcribe` (whisper-1/gpt-4o-transcribe shut down 2027-02-26). | | ||||
| 213 | +| Video | no | Sora 2 / Videos API shut down 2026-09-24. | | ||||
| 214 | + | ||||
| 215 | +Sources: https://developers.openai.com/api/docs/guides/images-vision , https://developers.openai.com/api/docs/guides/pdf-files , https://developers.openai.com/api/docs/guides/audio | ||||
| 216 | + | ||||
| 217 | +## 11. Context windows, max output, pricing (per 1M tokens, USD, standard tier) | ||||
| 218 | + | ||||
| 219 | +| Model | Context | Max output | Input | Cached | Output | Notes | | ||||
| 220 | +|---|---|---|---|---|---|---| | ||||
| 221 | +| gpt-6-astra | 1,050,000 | 128,000 | 10 | 1 | 50 | cache write 12.50; >272K input: 2x in / 1.5x out | | ||||
| 222 | +| gpt-5.6-sol (`gpt-5.6`) | 1,050,000 | 128,000 | 4 | 0.40 | 20 | | | ||||
| 223 | +| gpt-5.6-terra | 1,050,000 (922K input) | 128,000 | 2 | 0.20 | 12 | | | ||||
| 224 | +| gpt-5.6-luna | 1,050,000 | 128,000 | 0.20 | 0.02 | 1.20 | | | ||||
| 225 | +| gpt-5.5 | 1,050,000 | 128,000 | 5 | 0.50 | 30 | >272K: 2x/1.5x | | ||||
| 226 | +| gpt-5.5-pro | 1,050,000 | 128,000 | 30 | - | 180 | Responses only | | ||||
| 227 | +| gpt-5.4 | 1,050,000 | 128,000 | 2.50 | 0.25 | 15 | | | ||||
| 228 | +| gpt-5.4-mini | 400,000 | 128,000 | 0.75 | 0.075 | 4.50 | | | ||||
| 229 | +| gpt-5.4-nano | 400,000 | 128,000 | 0.20 | 0.02 | 1.25 | | | ||||
| 230 | +| gpt-5.4-pro | 1,050,000 | 128,000 | 30 | - | 180 | Responses only | | ||||
| 231 | +| gpt-5.3-codex | 400,000 | 128,000 | 1.75 | 0.175 | 14 | Responses only, coding | | ||||
| 232 | +| gpt-5.2 | 400,000 | 128,000 | 1.75 | 0.175 | 14 | | | ||||
| 233 | +| gpt-5.2-pro | 400,000 | 128,000 | 21 | - | 168 | Responses only | | ||||
| 234 | +| gpt-5.1 | 400,000 | 128,000 | 1.25 | 0.125 | 10 | | | ||||
| 235 | +| gpt-5 | 400,000 | 128,000 | 1.25 | 0.125 | 10 | snapshot shuts down 2026-12-11 | | ||||
| 236 | +| gpt-5-mini | 400,000 | 128,000 | 0.25 | 0.025 | 2 | snapshot shuts down 2026-12-11 | | ||||
| 237 | +| gpt-5-nano | 400,000 | 128,000 | 0.05 | 0.005 | 0.40 | snapshot shuts down 2026-12-11 | | ||||
| 238 | +| gpt-5-pro | 400,000 | 272,000 | 15 | - | 120 | Responses only | | ||||
| 239 | +| chat-latest | 400,000 (272K input) | 128,000 | 5 | 0.50 | 30 | ChatGPT Instant, non-reasoning | | ||||
| 240 | +| gpt-4.1 | 1,047,576 | 32,768 | 2 | 0.50 | 8 | | | ||||
| 241 | +| gpt-4.1-mini | 1,047,576 | 32,768 | 0.40 | 0.10 | 1.60 | | | ||||
| 242 | +| gpt-4.1-nano | 1,047,576 | 32,768 | 0.10 | 0.025 | 0.40 | shuts down 2026-10-23 | | ||||
| 243 | +| gpt-4o | 128,000 | 16,384 | 2.50 | 1.25 | 10 | alias -> 2024-08-06 | | ||||
| 244 | +| gpt-4o-mini | 128,000 | 16,384 | 0.15 | 0.075 | 0.60 | | | ||||
| 245 | +| o3 | 200,000 | 100,000 | 2 | 0.50 | 8 | snapshot shuts down 2026-12-11 | | ||||
| 246 | +| o3-pro | 200,000 | 100,000 | 20 | - | 80 | Responses only | | ||||
| 247 | +| o4-mini | 200,000 | 100,000 | 1.10 | 0.275 | 4.40 | shuts down 2026-10-23 | | ||||
| 248 | +| o3-mini | 200,000 | 100,000 | 1.10 | 0.55 | 4.40 | text only; shuts down 2026-10-23 | | ||||
| 249 | +| o1 | 200,000 | 100,000 | 15 | 7.50 | 60 | shuts down 2026-10-23 | | ||||
| 250 | +| o1-pro | 200,000 | 100,000 | 150 | - | 600 | shuts down 2026-10-23 | | ||||
| 251 | +| gpt-4-turbo | 128,000 | 4,096 | 10 | - | 30 | shuts down 2026-10-23 | | ||||
| 252 | +| gpt-4 | 8,192 | 8,192 | 30 | - | 60 | shuts down 2026-10-23 | | ||||
| 253 | +| gpt-3.5-turbo | 16,385 | 4,096 | 0.50 | - | 1.50 | shuts down 2026-10-23 | | ||||
| 254 | + | ||||
| 255 | +Tiers: Batch and Flex = ~50% of standard; Priority higher; Fast = 2x (not for gpt-6-astra with EU residency). Regional data residency +10% for models released after 2026-03-05. Tools: web search $10 / 1k calls (+ tokens), file search $2.50 / 1k calls + $0.10/GB/day storage (1 GB free), code interpreter $0.03-$1.92 per 20-min session by memory tier, MCP = tokens only. Unknown for this key: `gpt-5-search-api`, `gpt-4o-mini-search-preview`, the dead codex snapshots (set to `null` in the JSON). | ||||
| 256 | + | ||||
| 257 | +Sources: per-model pages under https://developers.openai.com/api/docs/models/<id> , https://developers.openai.com/api/docs/pricing | ||||
| 258 | + | ||||
| 259 | +## 12. Prompt caching | ||||
| 260 | + | ||||
| 261 | +Automatic prefix caching; minimum cacheable prefix 1,024 tokens (GPT-5.6+) / 2,048 (earlier). Reads at the "cached" price (0.1x on 5.6+, 0.5x on gpt-4o/o3-mini/o1); GPT-5.6+ also bill **cache writes at 1.25x** (reported in `usage.input_tokens_details.cache_write_tokens`, present in every usage object we saw). `prompt_cache_key` (routing hint, accepted on every model) ; `prompt_cache_retention: "in_memory" | "24h"` for pre-5.6 models (accepted on gpt-5.5 and even gpt-6-astra, but docs say it is deprecated for 6 -> use `prompt_cache_options: { ttl: "30m" }`, verified accepted on gpt-5.6-sol, echoed `{mode:"implicit", ttl:"30m"}`). `cached_tokens` is exact on 5.6+, rounded to 128 earlier. Cache lifetime 5-10 min (up to 1 h) or 24 h; not shared across orgs. | ||||
| 262 | + | ||||
| 263 | +Source: https://developers.openai.com/api/docs/guides/prompt-caching | ||||
| 264 | + | ||||
| 265 | +## 13. Rate limits, headers, retries, timeouts | ||||
| 266 | + | ||||
| 267 | +- Dimensions: RPM, TPM, RPD, TPD, IPM (+ audio minutes). Tiers Free..Tier 5 by cumulative spend; this key is Tier 5 (headers show 15,000 RPM / 40M TPM on gpt-5.5, 30,000 RPM / 150M TPM on gpt-4.1-mini). | ||||
| 268 | +- Headers (verified on every 200): `x-ratelimit-limit-requests`, `x-ratelimit-remaining-requests`, `x-ratelimit-reset-requests` (e.g. `4ms`, `6m0s`), `x-ratelimit-limit-tokens`, `x-ratelimit-remaining-tokens`, `x-ratelimit-reset-tokens`; docs also list `x-ratelimit-limit-project-tokens` / `x-ratelimit-remaining-project-tokens` and `Retry-After` (seconds) on 429/503. Also `x-request-id` and `openai-processing-ms`. | ||||
| 269 | +- 429 codes: `rate_limit_exceeded`, `slow_down` (traffic ramped too fast), `insufficient_quota` / `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded`, `organization_usage_limit_exceeded`. 503 `server_is_overloaded`. Billing/quota 429s must **not** be retried. | ||||
| 270 | +- Retry guidance: honour `Retry-After` when present, else exponential backoff with jitter; ramp traffic <= +50% per 15 min above 1M TPM. SDK default: `maxRetries: 2`, retries 408/409/429/>=500 and any response with `x-should-retry: true`, honours `retry-after-ms` / `retry-after`, backoff 0.5 s x 2^n capped at 8 s with up to 25% jitter. For a BYOK chat UI use `maxRetries: 1-2` for non-streaming, and do not retry once streaming has begun. | ||||
| 271 | +- Timeouts: SDK default 10 minutes (`OpenAI.DEFAULT_TIMEOUT = 600000`). Recommend: 60 s connect/first-byte for chat, overall 5-10 min for high/xhigh/max effort, and `background: true` (+ polling / resumable stream) for pro models and anything expected to run > 1-2 min. Docs: streaming is "the single most effective approach" to perceived latency; `service_tier: "priority" | "fast"` buys lower latency at higher price (`flex` accepted on gpt-5.4-mini: slower, cheaper). | ||||
| 272 | + | ||||
| 273 | +Sources: https://developers.openai.com/api/docs/guides/rate-limits , https://developers.openai.com/api/docs/guides/error-codes , https://developers.openai.com/api/docs/guides/latency-optimization , SDK `client.js` | ||||
| 274 | + | ||||
| 275 | +## 14. Error schema | ||||
| 276 | + | ||||
| 277 | +HTTP body: `{ "error": { "message", "type", "param", "code" } }` (raw fetch with a bad key also returned a top-level `"status": 401`). SDK maps to `BadRequestError` (400), `AuthenticationError` (401), `PermissionDeniedError` (403), `NotFoundError` (404), `RateLimitError` (429), `InternalServerError` (>=500), `APIConnectionError`, `APITimeoutError`; each exposes `.status`, `.code`, `.type`, `.param`, `.requestID`, `.headers`, `.error` (raw body). | ||||
| 278 | + | ||||
| 279 | +Verified examples: | ||||
| 280 | + | ||||
| 281 | +| Case | Status | `type` | `code` | `param` | `message` | | ||||
| 282 | +|---|---|---|---|---|---| | ||||
| 283 | +| invalid key | 401 | invalid_request_error | `invalid_api_key` | null | `Incorrect API key provided: sk-inva*****key. You can find your API key at https://platform.openai.com/account/api-keys.` | | ||||
| 284 | +| unknown model | 404 | invalid_request_error | `model_not_found` | null | ``The model `gpt-does-not-exist` does not exist or you do not have access to it.`` | | ||||
| 285 | +| CC-only model on Responses | 400 | invalid_request_error | `model_not_found` | model | `The requested model 'gpt-5-search-api' is not supported with the Responses API.` | | ||||
| 286 | +| wrong type | 400 | invalid_request_error | `invalid_type` | max_output_tokens | `Invalid type for 'max_output_tokens': expected an integer, but got a string instead.` | | ||||
| 287 | +| below min | 400 | invalid_request_error | `integer_below_min_value` | max_output_tokens | `Invalid 'max_output_tokens': integer below minimum value. Expected a value >= 16, but got 5 instead.` | | ||||
| 288 | +| unsupported param | 400 | invalid_request_error | **null** | temperature | `Unsupported parameter: 'temperature' is not supported with this model.` | | ||||
| 289 | +| unsupported param | 400 | invalid_request_error | `unsupported_parameter` | reasoning.effort | `Unsupported parameter: 'reasoning.effort' is not supported with this model.` | | ||||
| 290 | +| unsupported value | 400 | invalid_request_error | `unsupported_value` | reasoning.effort | `Unsupported value: 'minimal' is not supported with the 'gpt-5.5' model. Supported values are: 'none', 'low', 'medium', 'high', and 'xhigh'.` | | ||||
| 291 | +| unknown param | 400 | invalid_request_error | `unknown_parameter` | stop | `Unknown parameter: 'stop'. Did you mean 'store'?` | | ||||
| 292 | +| logprobs on reasoning | 400 | invalid_request_error | `unsupported_parameter` | - | `logprobs are not supported with reasoning models.` | | ||||
| 293 | +| prev response gone | 400 | invalid_request_error | `previous_response_not_found` | previous_response_id | `Previous response with id 'resp_...' not found.` | | ||||
| 294 | +| audio in Responses | 400 | invalid_request_error | null | input | `Audio input is not available.` | | ||||
| 295 | + | ||||
| 296 | +Note `code` is sometimes null for "Unsupported parameter" errors; match on `param` + message prefix as well. Other documented codes: `context_length_exceeded`, `invalid_service_tier`, `insufficient_quota`, `rate_limit_exceeded`, `slow_down`, `server_is_overloaded`. | ||||
| 297 | + | ||||
| 298 | +## 15. Usage reporting and token counting | ||||
| 299 | + | ||||
| 300 | +- Every response (and `response.completed`) has `usage: { input_tokens, input_tokens_details: { cached_tokens, cache_write_tokens }, output_tokens, output_tokens_details: { reasoning_tokens }, total_tokens }`. Web search inflates `input_tokens` heavily (4.7k-8.8k for a one-line question) because search results are injected as context and billed as input. | ||||
| 301 | +- Chat Completions usage: `prompt_tokens`, `completion_tokens`, `total_tokens`, `prompt_tokens_details {cached_tokens, audio_tokens}`, `completion_tokens_details {reasoning_tokens, audio_tokens, accepted_prediction_tokens, rejected_prediction_tokens}`; streaming needs `stream_options: {include_usage: true}` (usage arrives in a final chunk with empty `choices`). | ||||
| 302 | +- Pre-flight counting: `POST /v1/responses/input_tokens` (`{model, input, instructions?, tools?, conversation?, previous_response_id?}` -> `{object:"response.input_tokens", input_tokens}`); SDK: `client.responses.inputTokens.count(...)`. Client-side: `tiktoken` / `js-tiktoken` with `o200k_base` for gpt-4o and newer (approximation only for images/PDF). Rule of thumb 1 token ~ 4 chars English. | ||||
| 303 | +- SDK 7.8 added `compute_units` tracking on usage for some tiers (not observed in our responses). | ||||
| 304 | + | ||||
| 305 | +Source: https://developers.openai.com/api/docs/api-reference/responses/input-tokens | ||||
| 306 | + | ||||
| 307 | +## 16. Model listing | ||||
| 308 | + | ||||
| 309 | +`GET /v1/models` (SDK `client.models.list()` auto-paginates) returns `{id, object:"model", created, owned_by, shutdown_date}`. **`shutdown_date` is the only lifecycle signal** (ISO date or null; documented as "The date when the model will shut down, or null if not announced"). No capability, context or pricing metadata is exposed, so a static registry is mandatory (`openai.models.json`). 131 ids were returned for this key on 2026-09-08; 55 carried a `shutdown_date`, several already in the past (see 18). Filtering heuristics for chat models: exclude ids matching `babbage|davinci|instruct|transcribe|tts|whisper|audio|realtime|image|sora|embedding|moderation|search-preview|search-api|deep-research|codex` and anything whose `shutdown_date < today`; keep the rest and look it up in the registry (unknown ids -> show as "unverified"). | ||||
| 310 | + | ||||
| 311 | +Source: https://developers.openai.com/api/docs/api-reference/models | ||||
| 312 | + | ||||
| 313 | +## 17. Aliases | ||||
| 314 | + | ||||
| 315 | +- `gpt-5.6` -> `gpt-5.6-sol` (documented; verified `response.model === "gpt-5.6-sol"`; **not** itself listed by `/v1/models`). | ||||
| 316 | +- `gpt-5.5` -> `gpt-5.5-2026-04-23`; `gpt-5.4` -> `-2026-03-05`; `gpt-5.4-mini`/`-nano` -> `-2026-03-17`; `gpt-5.2` -> `-2025-12-11`; `gpt-5.1` -> `-2025-11-13`; `gpt-5*` -> `-2025-08-07`; `gpt-4.1*` -> `-2025-04-14`; `gpt-4o` -> `-2024-08-06` (not the newest 11-20 snapshot); `gpt-4o-mini` -> `-2024-07-18`; `o3` -> `-2025-04-16`; `o4-mini` -> `-2025-04-16`; `o3-mini` -> `-2025-01-31`; `gpt-4-turbo` -> `-2024-04-09`; `gpt-3.5-turbo` -> `-0125`. `gpt-6-astra`, `gpt-5.6-*`, `gpt-5.3-codex`, `chat-latest` have no dated snapshot (echoed as-is). | ||||
| 317 | +- `*-chat-latest` aliases (gpt-5 / 5.1 / 5.2 / 5.3) are **dead** (404) although still listed; `chat-latest` is the live ChatGPT-Instant alias. | ||||
| 318 | + | ||||
| 319 | +## 18. Lifecycle / deprecations relevant to a chat app | ||||
| 320 | + | ||||
| 321 | +| Shutdown | Models | Replacement | | ||||
| 322 | +|---|---|---| | ||||
| 323 | +| 2026-07-23 (passed; 404 today) | gpt-5-chat-latest, gpt-5.1-chat-latest, gpt-5-codex, gpt-5.1-codex(-max/-mini), gpt-5.2-codex, o3-deep-research, o4-mini-deep-research, *-search-preview-2025-03-11 snapshots | gpt-6-astra / gpt-5.3-codex | | ||||
| 324 | +| 2026-08-10 (passed; 404) | gpt-5.2-chat-latest, gpt-5.3-chat-latest | chat-latest / gpt-6-astra | | ||||
| 325 | +| 2026-08-26 | Assistants API | Responses + Conversations | | ||||
| 326 | +| 2026-09-24 | sora-2, sora-2-pro, Videos API | - | | ||||
| 327 | +| 2026-09-28 | gpt-3.5-turbo-1106, gpt-3.5-turbo-instruct, babbage-002, davinci-002 | gpt-5.6-terra | | ||||
| 328 | +| 2026-10-23 | gpt-3.5-turbo(-0125), gpt-4(-0613), gpt-4-turbo, gpt-4.1-nano, gpt-4o-2024-05-13, gpt-image-1, o1, o1-pro, o3-mini, o4-mini | gpt-5.6-sol / terra / luna (o1-pro -> 5.6-sol `reasoning.mode: pro`) | | ||||
| 329 | +| 2026-11-30 | v1/prompts, Evals, Agent Builder | - | | ||||
| 330 | +| 2026-12-01 | gpt-image-1-mini, gpt-image-1.5, chatgpt-image-latest | gpt-image-2 | | ||||
| 331 | +| 2026-12-11 | gpt-5-2025-08-07, gpt-5-mini-2025-08-07, gpt-5-nano-2025-08-07, gpt-5-pro-2025-10-06, o3-2025-04-16, o3-pro-2025-06-10 (the bare aliases `gpt-5`, `o3` currently have `shutdown_date: null`) | gpt-5.6-sol / terra / luna | | ||||
| 332 | +| 2027-01-20 / 2027-02-26 | legacy audio/realtime/transcribe models | gpt-audio-1.5, gpt-realtime-2.1, gpt-transcribe | | ||||
| 333 | + | ||||
| 334 | +Fine-tuning: closed to new orgs since 2026-05-07; no new jobs at all after 2027-01-06. | ||||
| 335 | + | ||||
| 336 | +Source: https://developers.openai.com/api/docs/deprecations + `shutdown_date` from `/v1/models` | ||||
| 337 | + | ||||
| 338 | +## 19. Built-in tools, state, safety | ||||
| 339 | + | ||||
| 340 | +- **Conversation state**: `store` defaults to **true** (30-day retention, visible in dashboard logs) - set `store:false` for privacy-minded BYOK usage and manage history client-side; then use `include:["reasoning.encrypted_content"]` and replay all output items. `previous_response_id` (verified) and `conversation` (Conversations API, `conv_...`, not subject to 30-day TTL) require `store:true`. Server-side compaction: `context_management:[{type:"compaction", compact_threshold}]`. | ||||
| 341 | +- **Web search**: `{type:"web_search", search_context_size?:"low"|"medium"|"high", user_location?:{type:"approximate", country, city, region, timezone}, filters?:{allowed_domains[] | blocked_domains[]} (<=100), external_web_access?, search_content_types?:["text","image"], return_token_budget?}` (legacy `web_search_preview` still accepted). Output: `web_search_call` items (`action: {type:"search", queries[], query} | {type:"open_page", url} | {type:"find_in_page"}`) then a `message` whose `output_text` part has `annotations: [{type:"url_citation", start_index, end_index, url, title}]` and inline markdown links with `?utm_source=openai`. Events: `response.web_search_call.{in_progress,searching,completed}`, `response.output_text.annotation.added`. Verified on gpt-4.1-mini and gpt-5.5 (effort low); the gpt-5.5 default-effort attempt at 200 tokens ended with three searches and no message - budget again. Search context capped at 128k. Supported per docs on gpt-6, 5.6, 5.5, 5.4 family, 5.2, 5.1, 5(-mini/-nano), 4.1(-mini), 4o(-mini), o3, o4-mini, chat-latest; not on gpt-4.1-nano, o3-mini, o1, gpt-4, gpt-3.5. | ||||
| 342 | +- **Code execution**: `{type:"code_interpreter", container: {type:"auto", memory_limit?:"1g"|"4g"|"16g"|"64g", file_ids?} | "cntr_..."}` -> `code_interpreter_call {code, outputs, container_id}`; generated files as `container_file_citation` annotations; 100 RPM/org; events `response.code_interpreter_call*`. Also `shell` (hosted shell), `local_shell`, `apply_patch`, skills. | ||||
| 343 | +- **File search**: `{type:"file_search", vector_store_ids[], max_num_results?, filters?, ranking_options?}` -> `file_search_call` + `file_citation {file_id, filename, index}` annotations; `include:["file_search_call.results"]`. | ||||
| 344 | +- **Computer use**: `{type:"computer"}` (new) or `computer_use_preview {display_width, display_height, environment}`; `computer_call` -> reply with `computer_call_output {call_id, output:{type:"computer_screenshot", image_url}}`; docs recommend GPT-6 Astra with code execution instead. Not relevant for PolyLLM v1. | ||||
| 345 | +- **MCP**: `{type:"mcp", server_label, server_url | connector_id, server_description?, authorization?, headers?, allowed_tools?, require_approval:"never"|"always"|{...}, defer_loading?}` -> `mcp_list_tools`, `mcp_call`, `mcp_approval_request` (answer with `mcp_approval_response {approval_request_id, approve}`); events `response.mcp_call*`, `response.mcp_list_tools*`. Billing = tokens only. 8 first-party connectors (Gmail, Drive, Calendar, Dropbox, Teams, Outlook x2, SharePoint). | ||||
| 346 | +- **Citations/annotations**: `url_citation`, `file_citation`, `container_file_citation`, `file_path` types on `output_text.annotations[]`; streamed through `response.output_text.annotation.added`. | ||||
| 347 | +- **Safety**: send `safety_identifier` (hashed user id) on every request; `incomplete_details.reason:"content_filter"` and `refusal` parts must be surfaced; free Moderation API (`omni-moderation-latest`) available; revoke leaked keys immediately. `moderation` request param exists in the SDK types. | ||||
| 348 | + | ||||
| 349 | +Sources: https://developers.openai.com/api/docs/guides/conversation-state , https://developers.openai.com/api/docs/guides/tools-web-search , https://developers.openai.com/api/docs/guides/tools-code-interpreter , https://developers.openai.com/api/docs/guides/tools-file-search , https://developers.openai.com/api/docs/guides/tools-computer-use , https://developers.openai.com/api/docs/guides/tools-remote-mcp , https://developers.openai.com/api/docs/guides/safety-best-practices | ||||
| 350 | + | ||||
| 351 | +## 20. Probe results (2026-09-08, key = Tier 5 project key) | ||||
| 352 | + | ||||
| 353 | +Legend: OK = completed with expected output; INC = accepted but `status:"incomplete"` (reasoning consumed the <= 200-token cap); 4xx = rejected with the quoted message. | ||||
| 354 | + | ||||
| 355 | +| Probe | gpt-5.5 | gpt-5.4-mini | gpt-5.6-sol | gpt-6-astra | gpt-4.1-mini | o4-mini | | ||||
| 356 | +|---|---|---|---|---|---|---| | ||||
| 357 | +| (a) basic Responses, 200 tok | OK (`gpt-5.5-2026-04-23`, 12 reasoning tok) | OK (0 reasoning) | OK (0 reasoning) | OK (0 reasoning) | OK | OK (64 reasoning) | | ||||
| 358 | +| (b) streaming events | OK, reasoning item but no summary events | OK | OK, `reasoning_summary_*` events seen | OK, no summary events | OK | INC: summary events, then `response.incomplete` | | ||||
| 359 | +| (c) temperature / top_p | 400 `Unsupported parameter: 'temperature' is not supported with this model.` (OK with `effort:"none"`) | OK (echo 0.5) | 400 same (OK with `effort:"none"`) | 400 same; `effort:"none"` itself rejected | OK | 400 same | | ||||
| 360 | +| (c) frequency/presence_penalty | 400 `Unsupported parameter: 'frequency_penalty' is not supported with this model.` | OK | 400 | 400 | OK | 400 | | ||||
| 361 | +| (c) seed | 400 `unknown_parameter` `Unknown parameter: 'seed'.` (all models) | | | | | | | ||||
| 362 | +| (c) stop | 400 `unknown_parameter` `Unknown parameter: 'stop'. Did you mean 'store'?` (all models) | | | | | | | ||||
| 363 | +| (c) logprobs | 400 `logprobs are not supported with reasoning models.` | OK | 400 | 400 | OK | 400 | | ||||
| 364 | +| (c) effort none | OK | OK | OK | 400 `Unsupported value: 'none' is not supported with the 'gpt-6-astra' model. Supported values are: 'low', 'medium', 'high', 'xhigh', and 'max'.` | 400 `Unsupported parameter: 'reasoning.effort' is not supported with this model.` | 400 `Supported values are: 'low', 'medium', and 'high'.` | | ||||
| 365 | +| (c) effort minimal | 400 `... Supported values are: 'none', 'low', 'medium', 'high', and 'xhigh'.` | 400 same | 400 `... 'none', 'low', 'medium', 'high', 'xhigh', and 'max'.` | 400 | 400 | 400 | | ||||
| 366 | +| (c) effort low/medium/high | OK | OK | OK | OK | 400 | OK | | ||||
| 367 | +| (c) effort xhigh | OK | OK | OK | OK | 400 | 400 | | ||||
| 368 | +| (c) effort max | 400 | 400 | OK | OK | 400 | 400 | | ||||
| 369 | +| (c) reasoning.summary auto | OK (echo `detailed`) | OK | OK | OK | OK (ignored) | OK | | ||||
| 370 | +| (c) text.verbosity low | OK | OK | OK | OK | 400 `Unsupported value: 'low' is not supported with the 'gpt-4.1-mini' model. Supported values are: 'medium'.` | 400 same wording | | ||||
| 371 | +| (c) max_output_tokens 5 | 400 `integer_below_min_value` `Expected a value >= 16, but got 5 instead.` (all) | | | | | | | ||||
| 372 | +| (c) truncation auto, prompt_cache_key, store:false+encrypted | OK on all six | | | | | | | ||||
| 373 | +| (d) function call, streaming | OK: `function_call` item, args `{"city":"Paris","unit":"c"}`, round trip -> "18°C and cloudy" | OK (+ `previous_response_id` variant OK) | OK | OK | OK | INC (192 reasoning tokens, no call) | | ||||
| 374 | +| (e) json_schema strict | OK parsed | OK | OK | OK | OK | INC | | ||||
| 375 | +| (f) vision 2x2 PNG | INC at default effort; OK with `effort:"none"` | OK | INC default; OK with `effort:"none"` | OK (`effort:"low"`, detail high) | OK | INC (even at `effort:"low"`) | | ||||
| 376 | +| (h) web_search tool | OK with `effort:"low"`: `web_search_call` + `url_citation` annotation; default effort at 200 tok: 3 searches, no message | OK (3 searches + message, no citation) | not probed | not probed | OK with citation | not probed | | ||||
| 377 | +| (i) Chat Completions | `temperature:0.3` -> 400 `Only the default (1) value is supported`; `reasoning_effort`+`verbosity` OK | - | - | `max_tokens` -> 400 `Use 'max_completion_tokens' instead.` | OK streaming, usage in last chunk | OK streaming | | ||||
| 378 | +| (g) invalid key | 401 `invalid_api_key` (`AuthenticationError`) - see section 14 | | | | | | | ||||
| 379 | + | ||||
| 380 | +Other probes: `gpt-5.6` alias OK; 26-model sweep OK except `gpt-5.3-chat-latest` (404); `gpt-5-nano` INC at default effort; `o1` INC at 32 tokens; `gpt-5.5-pro` OK (7.3 s, "pong"); `gpt-4.1-nano`, `gpt-4-turbo`, `gpt-3.5-turbo`, `o3-mini` still answer despite announced shutdowns; `gpt-5-search-api` works only via `chat.completions`; PDF input OK on gpt-5.4-mini; `previous_response_id` OK (remembered "7"); `service_tier:"flex"` OK; `input_audio` -> 400 `Audio input is not available.`; `prompt_cache_options.ttl:"30m"` OK on gpt-5.6-sol. | ||||
| 381 | + | ||||
| 382 | +## 21. Adapter recommendations (summary) | ||||
| 383 | + | ||||
| 384 | +1. Responses API only; Chat Completions merely as an optional legacy toggle (needed for `gpt-audio-*`, `*-search-*`). | ||||
| 385 | +2. Registry-driven parameter gating: sampling params allowed iff model is non-reasoning or `effort === "none"`; never send `seed`/`stop`; `max_output_tokens >= 16` and default it high (>= 4k) for reasoning models; effort dropdown filtered per model (`reasoningEfforts` in the JSON). | ||||
| 386 | +3. Stream handler keyed on `event.type`; treat `response.incomplete` as a normal terminal event (show partial text + "reasoning exhausted budget" hint when `reason === "max_output_tokens"` and no text). | ||||
| 387 | +4. Default `store:false` + `include:["reasoning.encrypted_content"]`, replay full `output` items each turn; offer `previous_response_id` mode only when the user opts into server-side storage. | ||||
| 388 | +5. Persist `usage` (incl. `cached_tokens`, `cache_write_tokens`, `reasoning_tokens`) per message and price with the registry table; web-search turns cost 5-9k input tokens. | ||||
| 389 | +6. Model list = `GET /v1/models` filtered by regex + `shutdown_date`, then joined with `openai.models.json`; show shutdown badges for dates < 90 days away. | ||||
| 390 | +7. Errors: map on `status` + `code` + `param`; `code` can be null for "Unsupported parameter" - fall back to message prefix; never retry 401/400/quota-429. | ||||
added
docs/provider-research/openai.models.json
+4472 −0
Large diff (4,472 lines) is not loaded by default to keep the page fast.Load diff
added
docs/provider-research/xai.md
+280 −0
Large diff (280 lines) is not loaded by default to keep the page fast.Load diff
added
docs/provider-research/xai.models.json
+356 −0
Large diff (356 lines) is not loaded by default to keep the page fast.Load diff
added
docs/provider-test-matrix.md
+34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added
drizzle.config.ts
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
drizzle/0000_foamy_loners.sql
+370 −0
Large diff (371 lines) is not loaded by default to keep the page fast.Load diff
added
drizzle/meta/0000_snapshot.json
+2748 −0
Large diff (2,749 lines) is not loaded by default to keep the page fast.Load diff
added
drizzle/meta/_journal.json
+13 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added
e2e/full-flow.spec.ts
+156 −0
Large diff (156 lines) is not loaded by default to keep the page fast.Load diff
added
e2e/helpers.ts
+44 −0
Large diff (44 lines) is not loaded by default to keep the page fast.Load diff
added
e2e/mobile.spec.ts
+12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added
eslint.config.mjs
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
next.config.ts
+51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added
package.json
+80 −0
Large diff (80 lines) is not loaded by default to keep the page fast.Load diff
added
playwright.config.ts
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
pnpm-lock.yaml
+9267 −0
Large diff (9,267 lines) is not loaded by default to keep the page fast.Load diff
added
postcss.config.mjs
+2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
public/icon.svg
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
public/robots.txt
+6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/00-models-list.ts
+44 −0
Large diff (44 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/01-basic-stream.ts
+83 −0
Large diff (83 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/02-params.ts
+68 −0
Large diff (68 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/03-tools.ts
+107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/04-vision.ts
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/05-errors-misc.ts
+104 −0
Large diff (104 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/06-availability.ts
+26 −0
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/07-signature.ts
+47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/08-rest-sse.ts
+20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/lib.ts
+89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/package.json
+24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added
research/gemini/pnpm-lock.yaml
+91 −0
Large diff (91 lines) is not loaded by default to keep the page fast.Load diff
Diff truncated — file too large.