spb/polyllm

1# PolyLLM — final production audit (v0.2.0, 2026-09-08)23## Build4- `pnpm typecheck` — 0 errors. `pnpm lint` — 0 errors. `pnpm build` (on M3U96a, Next 16 production build) — ok.5- `pnpm test` — vitest: unit (encryption, parameter normalization, model normalization for 3 providers, pricing, error6  normalization, stream accumulator, titles) + integration (registry sync with a real key against Postgres, encrypted key vault).78## Security9- No secrets committed: `.env`, `.env.*` (except `.env.example`) and `deploy/*.mld.json` are gitignored; the repo was grepped for key prefixes before the first commit.10- User provider keys: AES-256-GCM, data key derived with HKDF from `API_KEY_ENCRYPTION_SECRET`, AAD = user id + provider;11  verified in Postgres during E2E (`encrypted_key` starts with `v1.` and never contains the key); browser DOM never contains the key; API returns `keyHint` only.12- Passwords: Argon2id (19 MiB, t=2). Sessions: HttpOnly, SameSite, Secure in production, 30-day expiry, cookie cache 5 min; revoke one / others / all.13- Email verification mandatory (unverified login refused — verified by E2E). Password reset tokens expire in 1 h; sessions revoked on reset.14- Rate limits: Better Auth (sign-in 8/min, sign-up 4/min, reset 4/min, resend 3/min…) + per-user limits on chat, arena, key save/validate, model sync, uploads, search, sharing.15- Headers: CSP, HSTS, X-Frame-Options DENY, nosniff, Referrer-Policy, Permissions-Policy (verified on https://www.polyllm.io).16- Logging: structured JSON, key-shaped strings redacted, sensitive field names redacted, prompts never logged.17- Audit log per user (account, login, key added/replaced/removed, password/email events).1819## Providers (documentation re-audited 2026-09-08; live test matrix `docs/provider-test-matrix.md`)20Nine providers, each with a dated research report (`docs/provider-research/<id>.md`), real probes (`research/<id>/`) and an21adapter whose parameter translation only sends what the model accepts. The live matrix is **all ✅ for 13 capabilities × 922providers** (auth, model list, text, streaming, system prompt, vision, tools, structured output, reasoning, token usage,23invalid key, error normalization, long response).2425| Provider | SDK / endpoint | Models listed | Highlights verified |26| --- | --- | --- | --- |27| OpenAI | `openai` 7.10, Responses API | 65 (+ alias gpt-5.6) | encrypted reasoning replay, sampling only at effort none, no seed/stop |28| Anthropic | `@anthropic-ai/sdk` 0.124, Messages API | 11 | live capabilities, adaptive thinking, Haiku budget mode |29| Google Gemini | `@google/genai` 2.21 | 18 chat | thinkingLevel/budget per family, thought signatures, free-tier quotas |30| xAI | `openai` @ api.x.ai | 7 | reasoning_content, live pricing, 400 on bad key |31| Mistral AI | `openai` @ api.mistral.ai (translated bodies) | 28 ids / 10 billing models | listing capabilities, `reasoning_effort` none/high, thinking content arrays, `document_url` PDFs, 422 strictness |32| DeepSeek | `openai` @ api.deepseek.com | 3 | thinking on by default, `reasoning_effort` low/high/max, json_object only, cache-hit tokens |33| Kimi (Moonshot AI) | `openai` @ api.moonshot.ai | 4 | `/models/{id}` capabilities, fixed temperature/top_p, reasoning_content replay |34| OpenRouter | `openai` @ openrouter.ai | 323 after filters | `supported_parameters` gating, `usage.cost`, reasoning effort/budget, web plugin, upstream provider surfaced |35| Cerebras | `openai` @ api.cerebras.ai | 3 (gemma retired) | strict validation whitelist, `reasoning_effort`, tools+response_format rule, strict schema |3637## Email (Resend)38- Sender: `PolyLLM <polyllm@mail.spboucher.ai>` (the key is not authorized for `spboucher.ai` or `polyllm.io`; `mail.spboucher.ai` verified — probed).39- Verification email sent in production (Resend ids logged), welcome email after verification, reset/changed/email-change/deletion/new-login templates (HTML + plain text).4041## UI42- Desktop and mobile (390 px) screenshots of every screen in light and dark: `qa/out/` (48 screens) — no horizontal overflow, no console errors (Playwright `e2e/visual.spec.ts`).43- Real browser workflow (`e2e/full-flow.spec.ts`, 10 steps): sign up → verification refused → verify link → add 4 real keys → select model → real stream (Claude Haiku 4.5) → persistence after reload + second provider (GPT-5.4 nano) → capability-driven config → usage → change password → logout → login. All green.4445## Production46- Node M3U96a, PM2 `polyllm-web` (Next, :8240) + `polyllm-ngrok` (`www.polyllm.io`), registry entry in mld.47- `https://www.polyllm.io/api/health/ready` → `{"status":"ok","database":"ok"}`; TLS via ngrok; HSTS.48- Startup model sync on the node: 65 + 11 + 18 + 7 + 28 + 3 + 4 + 323 + 3 models (nine providers).49- Production login tested and a real provider request streamed through ngrok (`e2e/prod-smoke.spec.ts`, account `spbou4+polyllm-test@icloud.com`).50- Backups: `deploy/backup/install.sh` (daily pg_dump 03:45, 14-day retention) — to run on the node.5152## Known limitations (honest)53- Built-in tools are limited to calculator / clock / random (side-effect free); provider-native web search and code execution are wired where supported.54- Audio input/output and image generation are catalogued as capabilities but not exposed in the chat UI (text, images, PDF, text files only).55- Gemini web-search grounding and code execution were not live-tested (free-tier quota); Anthropic/OpenAI/xAI/OpenRouter web search paths are implemented from the audited docs and covered by the research probes, not by the matrix. Kimi `$web_search` and Mistral Conversations-API tools are not exposed (round trips not stable / different API).56- OpenRouter `reasoning_details` signatures are not replayed yet (works without; docs recommend replay).57- Conversation search is `ILIKE`-based (adequate for personal volumes); no full-text index yet.58- English only (architecture ready for locales).59