docs: final production audit
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 changed file +48 −0
added
docs/final-audit.md
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +# PolyLLM — final production audit (v0.1.0, 2026-09-08) | |
| 2 | + | |
| 3 | +## Build | |
| 4 | +- `pnpm typecheck` — 0 errors. `pnpm lint` — 0 errors. `pnpm build` (on M3U96a, Next 16 production build) — ok. | |
| 5 | +- `pnpm test` — vitest: unit (encryption, parameter normalization, model normalization for 3 providers, pricing, error | |
| 6 | + normalization, stream accumulator, titles) + integration (registry sync with a real key against Postgres, encrypted key vault). | |
| 7 | + | |
| 8 | +## Security | |
| 9 | +- No secrets committed: `.env`, `.env.*` (except `.env.example`) and `deploy/*.mld.json` are gitignored; the repo was grepped for key prefixes before the first commit. | |
| 10 | +- User provider keys: AES-256-GCM, data key derived with HKDF from `API_KEY_ENCRYPTION_SECRET`, AAD = user id + provider; | |
| 11 | + verified in Postgres during E2E (`encrypted_key` starts with `v1.` and never contains the key); browser DOM never contains the key; API returns `keyHint` only. | |
| 12 | +- Passwords: Argon2id (19 MiB, t=2). Sessions: HttpOnly, SameSite, Secure in production, 30-day expiry, cookie cache 5 min; revoke one / others / all. | |
| 13 | +- Email verification mandatory (unverified login refused — verified by E2E). Password reset tokens expire in 1 h; sessions revoked on reset. | |
| 14 | +- Rate limits: Better Auth (sign-in 8/min, sign-up 4/min, reset 4/min, resend 3/min…) + per-user limits on chat, arena, key save/validate, model sync, uploads, search, sharing. | |
| 15 | +- Headers: CSP, HSTS, X-Frame-Options DENY, nosniff, Referrer-Policy, Permissions-Policy (verified on https://www.polyllm.io). | |
| 16 | +- Logging: structured JSON, key-shaped strings redacted, sensitive field names redacted, prompts never logged. | |
| 17 | +- Audit log per user (account, login, key added/replaced/removed, password/email events). | |
| 18 | + | |
| 19 | +## Providers (documentation re-audited 2026-09-08; live test matrix `docs/provider-test-matrix.md`) | |
| 20 | +| | OpenAI | Anthropic | Gemini | xAI | | |
| 21 | +| --- | --- | --- | --- | --- | | |
| 22 | +| Docs current | ✅ developers.openai.com (Responses) | ✅ platform.claude.com + Models API | ✅ ai.google.dev | ✅ docs.x.ai/developers | | |
| 23 | +| SDK current | `openai` 7.10.0 | `@anthropic-ai/sdk` 0.124.0 | `@google/genai` 2.21.0 | `openai` 7.10.0 (baseURL api.x.ai) | | |
| 24 | +| Models current | 65 listed (gpt-6-astra, gpt-5.6-*, gpt-5.5, 5.4…) + shutdown dates | 11 (Fable 5.1, Opus 5, Sonnet 5…) with live capabilities | 18 chat models (3.8/3.7/3.6/3.5 Flash, 3.1 Pro…) ; 2.5 gen marked deprecated (404 for new users) | 7 (Grok 4.6/4.5/4.3/4.20/Build) with live pricing | | |
| 25 | +| Basic generation / streaming / system / vision / tools / structured output / reasoning / usage / errors | ✅ all | ✅ all | ✅ all | ✅ all | | |
| 26 | +| Notes | sampling only at effort none; no seed/stop | adaptive thinking may produce 0 thinking tokens on easy prompts; Haiku uses budget | owner key is free tier: tiny per-model daily quotas, 3.1 Pro paid-only | invalid key = HTTP 400; penalties always rejected; web search via /v1/responses | | |
| 27 | + | |
| 28 | +## Email (Resend) | |
| 29 | +- Sender: `PolyLLM <polyllm@mail.spboucher.ai>` (the key is not authorized for `spboucher.ai` or `polyllm.io`; `mail.spboucher.ai` verified — probed). | |
| 30 | +- Verification email sent in production (Resend ids logged), welcome email after verification, reset/changed/email-change/deletion/new-login templates (HTML + plain text). | |
| 31 | + | |
| 32 | +## UI | |
| 33 | +- Desktop and mobile (390 px) screenshots of every screen in light and dark: `qa/out/` (48 screens) — no horizontal overflow, no console errors (Playwright `e2e/visual.spec.ts`). | |
| 34 | +- Real browser workflow (`e2e/full-flow.spec.ts`, 10 steps): sign up → verification refused → verify link → add 4 real keys → select model → real stream (Claude Haiku 4.5) → persistence after reload + second provider (GPT-5.4 nano) → capability-driven config → usage → change password → logout → login. All green. | |
| 35 | + | |
| 36 | +## Production | |
| 37 | +- Node M3U96a, PM2 `polyllm-web` (Next, :8240) + `polyllm-ngrok` (`www.polyllm.io`), registry entry in mld. | |
| 38 | +- `https://www.polyllm.io/api/health/ready` → `{"status":"ok","database":"ok"}`; TLS via ngrok; HSTS. | |
| 39 | +- Startup model sync on the node: 65 + 11 + 18 + 7 models. | |
| 40 | +- Production login tested and a real provider request streamed through ngrok (`e2e/prod-smoke.spec.ts`, account `spbou4+polyllm-test@icloud.com`). | |
| 41 | +- Backups: `deploy/backup/install.sh` (daily pg_dump 03:45, 14-day retention) — to run on the node. | |
| 42 | + | |
| 43 | +## Known limitations (honest) | |
| 44 | +- Built-in tools are limited to calculator / clock / random (side-effect free); provider-native web search and code execution are wired where supported. | |
| 45 | +- Audio input/output and image generation are catalogued as capabilities but not exposed in the chat UI (text, images, PDF, text files only). | |
| 46 | +- Gemini web-search grounding and code execution were not live-tested (free-tier quota); Anthropic/OpenAI/xAI web search paths are implemented from the audited docs and covered by the research probes, not by the matrix. | |
| 47 | +- Conversation search is `ILIKE`-based (adequate for personal volumes); no full-text index yet. | |
| 48 | +- English only (architecture ready for locales). | |
| 49 | ||