spb/polyllm

1# Workstream E — Usage analytics, provider management, custom endpoints23Status: complete for this phase. `pnpm typecheck`, `pnpm lint` (0 errors) and `pnpm test` (159 tests, 28 new) are green.4No dev server was started; no commit was made. One additive migration was generated **and applied locally**5(`drizzle/0002_endpoints.sql`) — it must run on prod via the mld hook (`pnpm db:migrate`).67## 1. Files89### Created10| Area | File |11| --- | --- |12| Usage API | `src/lib/usage/time.ts` (tz-aware ranges/buckets, pure), `src/lib/usage/savings.ts` (savings math, pure), `src/app/api/usage/export.csv/route.ts` |13| Usage UI | `src/components/usage/usage-dashboard.tsx`, `lazy-charts.tsx`, `kpi-tiles.tsx`, `usage-filters.tsx`, `insight-cards.tsx`, `recent-activity.tsx`, `types.ts` |14| Providers | `src/components/providers/provider-row.tsx`, `src/app/api/providers/test/route.ts` |15| Custom endpoints | `src/lib/ai/providers/custom/index.ts`, `src/lib/endpoints/service.ts`, `src/lib/endpoints/ssrf.ts`, `src/app/api/endpoints/route.ts`, `src/app/api/endpoints/[id]/route.ts`, `src/app/api/endpoints/[id]/test/route.ts`, `src/app/api/endpoints/[id]/sync/route.ts`, `src/components/endpoints/endpoint-sheet.tsx`, `endpoint-row.tsx`, `presets.ts`, `src/app/app/settings/endpoints/page.tsx` |16| Settings | `src/components/settings/section.tsx` (`SettingsSection`, `SettingsGroup`, `SettingsRow`), `src/components/settings/sections.ts` (nav) |17| Schema | `drizzle/0002_endpoints.sql` (+ `drizzle/meta/0002_snapshot.json`, journal entry) |18| Tests | `tests/unit/usage-savings.test.ts`, `usage-time.test.ts`, `endpoints-ssrf.test.ts`, `custom-endpoints.test.ts` |1920### Changed21| File | Change |22| --- | --- |23| `src/lib/ai/core/types.ts` | `"custom"` appended to `PROVIDER_IDS`; new `KEYED_PROVIDER_IDS` (everything except custom). `parseModelKey` unchanged (splits on first `/` → provider `custom`). |24| `src/lib/ai/providers/index.ts` | `ADAPTERS.custom = customPlaceholderAdapter` (fails loudly, never calls a server); `PROVIDER_META.custom`. |25| `src/lib/ai/providers/env-keys.ts` | `OWNER_KEY_ENV.custom = ""`; `ownerKey()` returns `undefined` for it. |26| `src/lib/ai/registry/catalog.ts` | `CATALOGS.custom = new Map()`. |27| `src/lib/client/providers.ts` | `PROVIDERS.custom` (color `--p-custom`, links to Settings → Endpoints). `PROVIDER_ORDER` unchanged (keyed providers only). |28| `src/components/brand/provider-icon.tsx` | `case "custom"` glyph (plug). |29| `src/components/marketing/mock-data.ts` | one-line `custom` entry to keep `Record<ProviderId,string>` exhaustive (F's file — exhaustiveness only). |30| `scripts/provider-matrix.ts` | `TEST_MODELS.custom` placeholder (skipped: no owner key). |31| `src/db/schema-workspace.ts` | `custom_endpoints.discovered_models jsonb` + `discovered_at` (discovery snapshot so `/api/models` never calls the endpoint). |32| `src/app/api/models/route.ts` | additive: appends `listCustomModels(userId)`; pushes `"custom"` into `connectedProviders` when ≥ 1 endpoint is `valid`. |33| `src/lib/usage/service.ts` | rewritten (see API). `UsageRange` kept as a deprecated alias. |34| `src/app/api/usage/route.ts` | uses `usageQueryFromUrl`. |35| `src/components/usage/charts.tsx` | dataviz pass (≤ 24 px bars, 4 px data-end radius, 2 px surface gaps, 10 % area wash, hairline grid, tooltips everywhere), new `LatencyByModel` (two small multiples — never a dual axis), `Sparkline`, `ChartEmpty`; labels parsed from wall-clock bucket keys. |36| `src/app/app/usage/page.tsx` | thin server page → `<UsageDashboard />` in `Suspense` (uses `useSearchParams`). |37| `src/components/providers/add-key-dialog.tsx` | `ResponsiveDialog` (sheet on phones), paste button (clipboard API when available), show/hide, whitespace check, latency in the toast. Public API (`AddKeyDialog`, `useAddKeyDialog`) unchanged. |38| `src/app/app/settings/providers/page.tsx` | redesign (rows, statuses, Test connection with latency, More menu / ActionSheet, link card to Endpoints). |39| `src/app/app/settings/layout.tsx`, `page.tsx` | phone = section list + 48 px back bar per section; desktop = side nav; "Endpoints" added. |40| `src/app/app/settings/{account,security,appearance,data}/page.tsx` | `SettingsSection` header, 44 px targets / 16 px inputs on phones, audit log → stacked rows below `md`, session actions full-width on phones, CSV usage export link on Data. |41| `src/lib/client/types.ts` | append-only re-exports (`PublicEndpoint`, `EndpointInput`, `ManualModel`, `UsageSummary`, `SavingsOpportunity`, …). |42| `.env.example` | `ALLOW_PRIVATE_ENDPOINTS=0` documented. |4344## 2. API routes4546### `GET /api/usage`47Query: `range=today|7d|30d|90d|custom|all` (default 30d) · `from`,`to` = `YYYY-MM-DD` (custom, inclusive, clamped to today and 366 days; ≤ 2 days → hourly) · `tz` = IANA zone (validated, default UTC; the client sends `Intl.DateTimeFormat().resolvedOptions().timeZone`) · `provider` · `modelKey` · `projectId` (join `usage_records.conversation_id → conversations.project_id`, scoped to the user).4849Response `UsageSummary` (`src/lib/usage/service.ts`):50```ts51{52  range: { key, from: string|null, to: string, bucket: "hour"|"day", tz, days },53  filters: { provider, modelKey, projectId },54  kpis: { requests, failures, stopped, errorRate, inputTokens, outputTokens, cachedTokens, reasoningTokens, totalTokens,55          costUsd, unpricedRequests, avgLatencyMs, avgTtftMs|null, avgTokensPerSec|null, avgContextTokens|null },56  series: { bucket: "YYYY-MM-DD" | "YYYY-MM-DDTHH:00", requests, failures, inputTokens, outputTokens, costUsd }[], // gap-filled in tz57  byProvider: { provider, requests, failures, inputTokens, outputTokens, costUsd }[],58  byModel: { modelKey, provider, requests, failures, inputTokens, outputTokens, cachedTokens, reasoningTokens, costUsd,59             avgLatencyMs, avgTtftMs|null, tokensPerSec|null, avgContextTokens|null }[],60  projection: { dailyAverageCost, estimatedMonthlyCost /* daily avg × 30 */, observedDays,61                previous: { costUsd, requests } | null, costTrendPct|null, requestsTrendPct|null },62  savings: SavingsOpportunity[], // top 3, see below63  recent: RecentRecord[60] (+ conversationTitle, projectId via left join),64  facets: { providers[], models: {modelKey, provider, requests}[24], projects: {id,name,icon,color}[] },65  generatedAt66}67```68tok/s = `outputTokens / max(latency − ttft, 1 ms)` on successful streaming rows. Avg context = avg input tokens of successful requests.6970**Savings** (`computeSavings`, pure, tested): for every model that cost money, candidates = same provider, not deprecated, priced, covering every capability the source has among {vision, tools, structuredOutput, reasoning, files}, context ≥ 1.1 × avg prompt; same `family` preferred. The real token volume (input, cached at cached rate, output) is re-priced with `estimateCost`; the **most capable sibling that still saves ≥ 40 %** wins (Opus → Sonnet, not Haiku). Min $0.01. Each row carries `headline` ("Switching eligible tasks from Claude Opus 4 to Claude Sonnet 4 could have saved ~$17.30") and a `rationale`.7172### `GET /api/usage/export.csv` — same query params → `text/csv` attachment (`polyllm-usage-<from>_<to>.csv`, ≤ 50 000 rows, formula-injection-safe quoting, `X-Row-Count` header).7374### `POST /api/providers/test` `{ provider }` → `{ ok, latencyMs, modelsAvailable|null, error?, errorCode?, connections }`75Validates the **stored encrypted key** with `validateApiKey` (never returns the key), refreshes the model list on success. Rate limit `keyValidate`. Rejects `custom`.7677### Endpoints (all user-scoped, key material never returned; `PublicEndpoint` carries `keyHint`, `hasKey`, `headerNames` only)78| Method | Path | Body → Response |79| --- | --- | --- |80| GET | `/api/endpoints` | → `{ endpoints: PublicEndpoint[], allowPrivate }` |81| POST | `/api/endpoints` | `{ name, baseUrl, apiKey?, headers?: Record<string,string>, modelsPath? ("" = manual only), manualModels?: ManualModel[], validate?: boolean=true }` → 201 `{ endpoint, test?: { ok, latencyMs, modelsAvailable, error? } }` |82| GET | `/api/endpoints/[id]` | → `{ endpoint }` |83| PATCH | `/api/endpoints/[id]` | partial input; `apiKey: null` clears the key, `headers: null` clears headers; changing URL/key/headers/path resets status to `unverified` → `{ endpoint }` |84| DELETE | `/api/endpoints/[id]` | → `{ ok: true }` |85| POST | `/api/endpoints/[id]/test` | → `{ ok, latencyMs, modelsAvailable, error?, errorCode?, endpoint }` |86| POST | `/api/endpoints/[id]/sync` | → `{ ok, models: PolyModel[], latencyMs, error?, endpoint }` (discovery snapshot persisted) |8788`ManualModel = { id, displayName?, contextTokens?, vision?, tools?, reasoning? }` — also acts as capability override for a discovered id.89Probe = `GET {baseUrl}{modelsPath}` (accepts `{data:[{id}]}`, bare arrays, Ollama `{models:[{name}]}`); with discovery off and manual models present → a 1-token chat completion against the first manual model.9091Encryption: `encryptSecret(value, { userId, provider: "custom:<endpointId>" })` → AAD `"<userId>|custom:<endpointId>"` for both the key and the headers JSON. Audit actions: `endpoint.created|updated|deleted`.9293SSRF guard (`src/lib/endpoints/ssrf.ts`): http/https only, no credentials in URL, blocks localhost/*.local/*.internal/single-label hosts, IPv4 private/loopback/link-local/CGNAT/test/multicast, IPv6 ::1/ULA/link-local/mapped-v4/NAT64, plus DNS resolution of the hostname at create/update/probe. Bypass only with `ALLOW_PRIVATE_ENDPOINTS=1`. DNS rebinding after the check is out of scope.9495## 3. Exact integration hook for chat and Arena (integrator wires — A's and C's files were not edited)9697`src/lib/endpoints/service.ts` exports98```ts99resolveCustomEndpoint(userId, modelKey): Promise<{ adapter: AIProviderAdapter; apiKey: string; model: PolyModel; endpoint: PublicEndpoint } | null>100isCustomModelKey(key): boolean101```102Custom keys look like `custom/cep_x:llama3.1:8b` → `getModel()` returns `null` for them (they are not in the registry table) and `getDecryptedKey(userId, "custom")` returns `null`. Wire both services like this:103104**`src/lib/chat/service.ts` → `prepareTurn`** — replace105```ts106const model = await getModel(input.modelKey);107if (!model) throw new ApiError(404, "Unknown model. Refresh your model list.", "MODEL_NOT_FOUND");108const apiKey = await getDecryptedKey(ctx.userId, model.provider);109if (!apiKey) throw new ApiError(400, `No API key configured for ${model.provider}. …`, "NO_PROVIDER_KEY", { provider: model.provider });110```111with112```ts113const custom = isCustomModelKey(input.modelKey) ? await resolveCustomEndpoint(ctx.userId, input.modelKey) : null;114const model = custom?.model ?? (await getModel(input.modelKey));115if (!model) throw new ApiError(404, isCustomModelKey(input.modelKey) ? "This custom endpoint no longer exists. Check Settings → Endpoints." : "Unknown model. Refresh your model list.", "MODEL_NOT_FOUND");116const apiKey = custom?.apiKey ?? (await getDecryptedKey(ctx.userId, model.provider));117if (!apiKey) throw new ApiError(400, `No API key configured for ${model.provider}. Add one in Settings → Providers.`, "NO_PROVIDER_KEY", { provider: model.provider });118```119add `adapter?: AIProviderAdapter` to `PreparedTurn`, return `adapter: custom?.adapter` from every `return { conversation, model, apiKey, … }`, and in **`runTurn`** replace `const adapter = getAdapter(turn.model.provider);` with `const adapter = turn.adapter ?? getAdapter(turn.model.provider);`. Also skip `recordProviderOutcome` when `turn.model.provider === "custom"` (there is no `provider_connections` row; it is a no-op update anyway) and keep `touchRecent(userId, model.key)` as is (works with any key).120121**`src/lib/arena/service.ts`** — `createArenaSession`: for each key, `const custom = isCustomModelKey(key) ? await resolveCustomEndpoint(userId, key) : null; const m = custom?.model ?? await getModel(key); … if (!custom && !(await getDecryptedKey(userId, m.provider))) missing.push(m.provider);`. `runArenaModel`: same three-line substitution as `prepareTurn`, then `const adapter = custom?.adapter ?? getAdapter(model.provider);`.122123`apiKey` for key-less endpoints is the sentinel `NO_KEY_SENTINEL` ("polyllm-no-key") — the OpenAI SDK needs a non-empty bearer; Ollama & co. ignore it. `model.pricing` is `null` → `estimateCost().known === false` → `costUsd` stays `null` in `usage_records` (the dashboard counts them as "unpriced"). `usage_records.provider` will be `"custom"` (valid `ProviderId`).124125Model picker (B): `/api/models` already returns custom models with `provider: "custom"`, `family` = endpoint name, `metadata.endpointId/endpointName/endpointStatus`; `connectedProviders` includes `"custom"` when at least one endpoint is valid. If B's category chips iterate `PROVIDER_ORDER`, custom models only appear in "All"/search — add a "Custom" chip keyed on `provider === "custom"` if desired.126127## 4. Coming soon / known limits (nothing fake shipped)128- Custom endpoints have **no pricing** (never invented) — their requests show `—` cost and are counted in `kpis.unpricedRequests`. A per-endpoint price sheet is a candidate follow-up.129- Discovered custom models start **text-only**; users unlock vision/tools/reasoning per id through the manual-models editor. `structuredOutput`/`files` are always off for custom endpoints (server support varies too much to assume).130- Stored **header values are never shown**; editing any header value re-saves all headers (UI explains this).131- SSRF: DNS rebinding after the check and redirects to private hosts are not followed (`redirect: "manual"` on discovery; the OpenAI SDK follows redirects for chat — acceptable risk noted).132- Usage "all time" projection uses the first recorded request as the start of the observation window.133- No `Coming soon` badges were needed.134135## 5. QA checklist (integration phase, single dev server)136Usage (`/app/usage`) at 375 / 390 / 430 / 1440:137- [ ] 48 px bar with hamburger, refresh, CSV icon on phones; `PageHeader` + buttons on desktop.138- [ ] Segmented Today/7d/30d/90d/Custom scrolls horizontally without page overflow; Custom opens a bottom sheet (dialog on desktop) with two `type=date` fields (16 px), presets, validation (end before start), Apply.139- [ ] Provider/model/project chips filter everything; "Filtered by …" pills clear individually; URL query updates (deep link reload keeps state).140- [ ] KPI tiles 2-col on phones, 4-col desktop; deltas coloured by direction (cost up = red).141- [ ] Charts lazy-load with skeletons; no default recharts colours; hour labels for Today, day labels otherwise, in the browser's zone; tooltips on every plot; stacked segments show 2 px surface gaps; failure segment only when failures exist.142- [ ] Cost by provider donut folds > 6 providers into "Other"; list shows provider icons + values.143- [ ] Projection card and Savings card (or the honest "nothing to save" state); "Why this sibling?" expands; catalog link works.144- [ ] Latency by model = two side-by-side small multiples (TTFT ascending, tok/s descending).145- [ ] Recent requests: rows on phones (tap → conversation), table from `md`; export CSV downloads with the current filters.146- [ ] Empty states: no data (with "Show all time"), filters with no match ("Clear filters"), API error with Retry. Refetch keeps the previous render at 70 % opacity (no skeleton flash).147148Providers (`/app/settings/providers`):149- [ ] Rows sorted connected → failed → not connected; statuses Connected / Validation failed / Not validated yet / Not connected.150- [ ] Add key opens a bottom sheet on phones (dialog on desktop) with Paste (only where `clipboard.readText` exists), show/hide, docs link; success toast shows models + latency.151- [ ] Test connection shows latency inline (green/red) and toasts; failure shows the error under "Last error".152- [ ] More → Replace / Open console / Remove (ActionSheet on phones, dropdown on desktop); Remove uses ConfirmDialog.153- [ ] "Custom endpoints" card links to `/app/settings/endpoints` with counts.154155Endpoints (`/app/settings/endpoints`):156- [ ] Empty state with preset chips; Add opens a full-height sheet on phones.157- [ ] Presets fill URL/path; the private-address warning appears for localhost/LAN URLs and explains tunnel vs `ALLOW_PRIVATE_ENDPOINTS=1` (info tone when the server allows private).158- [ ] Headers rows add/remove; manual models rows with vision/tools/reasoning switches; discovery toggle hides/shows models path.159- [ ] "Save & test connection" → latency + discovered model chips; failure keeps the sheet open with the error; "Save without testing" → status "Not tested".160- [ ] Row actions: Test, More → Edit / Refresh model list / Remove. Models appear in the picker as provider "Custom" (family = endpoint name) after `refreshModels`.161- [ ] Server rejects `http://localhost:11434/v1` with 400 `PRIVATE_HOST` when `ALLOW_PRIVATE_ENDPOINTS` is unset; accepts it when set to 1.162163Settings shell:164- [ ] Phone `/app/settings` shows the six-section list (≥ 60 px rows); each section has a back chevron bar; desktop `/app/settings` redirects to Account and shows the side nav with "Endpoints".165- [ ] Account: audit log as stacked rows on phones, table on desktop; inputs 44 px / 16 px on phones. Security: session buttons full-width on phones. Appearance: switches have 44 px hit areas. Data: JSON export + usage CSV link.166