# Workstream E — Usage analytics, provider management, custom endpoints

Status: complete for this phase. pnpm typecheck, pnpm lint (0 errors) and pnpm test (159 tests, 28 new) are green. No dev server was started; no commit was made. One additive migration was generated and applied locally (drizzle/0002_endpoints.sql) — it must run on prod via the mld hook (pnpm db:migrate).

# 1. Files

# Created

# Changed

# 2. API routes

# GET /api/usage

Query: range=today|7d|30d|90d|custom|all (default 30d) · from,to = YYYY-MM-DD (custom, inclusive, clamped to today and 366 days; ≤ 2 days → hourly) · tz = IANA zone (validated, default UTC; the client sends Intl.DateTimeFormat().resolvedOptions().timeZone) · provider · modelKey · projectId (join usage_records.conversation_id → conversations.project_id, scoped to the user).

Response UsageSummary (src/lib/usage/service.ts):

{
  range: { key, from: string|null, to: string, bucket: "hour"|"day", tz, days },
  filters: { provider, modelKey, projectId },
  kpis: { requests, failures, stopped, errorRate, inputTokens, outputTokens, cachedTokens, reasoningTokens, totalTokens,
          costUsd, unpricedRequests, avgLatencyMs, avgTtftMs|null, avgTokensPerSec|null, avgContextTokens|null },
  series: { bucket: "YYYY-MM-DD" | "YYYY-MM-DDTHH:00", requests, failures, inputTokens, outputTokens, costUsd }[], // gap-filled in tz
  byProvider: { provider, requests, failures, inputTokens, outputTokens, costUsd }[],
  byModel: { modelKey, provider, requests, failures, inputTokens, outputTokens, cachedTokens, reasoningTokens, costUsd,
             avgLatencyMs, avgTtftMs|null, tokensPerSec|null, avgContextTokens|null }[],
  projection: { dailyAverageCost, estimatedMonthlyCost /* daily avg × 30 */, observedDays,
                previous: { costUsd, requests } | null, costTrendPct|null, requestsTrendPct|null },
  savings: SavingsOpportunity[], // top 3, see below
  recent: RecentRecord[60] (+ conversationTitle, projectId via left join),
  facets: { providers[], models: {modelKey, provider, requests}[24], projects: {id,name,icon,color}[] },
  generatedAt
}

tok/s = outputTokens / max(latency − ttft, 1 ms) on successful streaming rows. Avg context = avg input tokens of successful requests.

Savings (computeSavings, pure, tested): for every model that cost money, candidates = same provider, not deprecated, priced, covering every capability the source has among {vision, tools, structuredOutput, reasoning, files}, context ≥ 1.1 × avg prompt; same family preferred. The real token volume (input, cached at cached rate, output) is re-priced with estimateCost; the most capable sibling that still saves ≥ 40 % wins (Opus → Sonnet, not Haiku). Min $0.01. Each row carries headline ("Switching eligible tasks from Claude Opus 4 to Claude Sonnet 4 could have saved ~$17.30") and a rationale.

# GET /api/usage/export.csv — same query params → text/csv attachment (polyllm-usage-<from>_<to>.csv, ≤ 50 000 rows, formula-injection-safe quoting, X-Row-Count header).

# POST /api/providers/test { provider } → { ok, latencyMs, modelsAvailable|null, error?, errorCode?, connections }

Validates the stored encrypted key with validateApiKey (never returns the key), refreshes the model list on success. Rate limit keyValidate. Rejects custom.

# Endpoints (all user-scoped, key material never returned; PublicEndpoint carries keyHint, hasKey, headerNames only)

ManualModel = { id, displayName?, contextTokens?, vision?, tools?, reasoning? } — also acts as capability override for a discovered id. Probe = GET {baseUrl}{modelsPath} (accepts {data:[{id}]}, bare arrays, Ollama {models:[{name}]}); with discovery off and manual models present → a 1-token chat completion against the first manual model.

Encryption: encryptSecret(value, { userId, provider: "custom:<endpointId>" }) → AAD "<userId>|custom:<endpointId>" for both the key and the headers JSON. Audit actions: endpoint.created|updated|deleted.

SSRF guard (src/lib/endpoints/ssrf.ts): http/https only, no credentials in URL, blocks localhost/.local/.internal/single-label hosts, IPv4 private/loopback/link-local/CGNAT/test/multicast, IPv6 ::1/ULA/link-local/mapped-v4/NAT64, plus DNS resolution of the hostname at create/update/probe. Bypass only with ALLOW_PRIVATE_ENDPOINTS=1. DNS rebinding after the check is out of scope.

# 3. Exact integration hook for chat and Arena (integrator wires — A's and C's files were not edited)

src/lib/endpoints/service.ts exports

resolveCustomEndpoint(userId, modelKey): Promise<{ adapter: AIProviderAdapter; apiKey: string; model: PolyModel; endpoint: PublicEndpoint } | null>
isCustomModelKey(key): boolean

Custom keys look like custom/cep_x:llama3.1:8b → getModel() returns null for them (they are not in the registry table) and getDecryptedKey(userId, "custom") returns null. Wire both services like this:

src/lib/chat/service.ts → prepareTurn — replace

const model = await getModel(input.modelKey);
if (!model) throw new ApiError(404, "Unknown model. Refresh your model list.", "MODEL_NOT_FOUND");
const apiKey = await getDecryptedKey(ctx.userId, model.provider);
if (!apiKey) throw new ApiError(400, `No API key configured for ${model.provider}. …`, "NO_PROVIDER_KEY", { provider: model.provider });

with

const custom = isCustomModelKey(input.modelKey) ? await resolveCustomEndpoint(ctx.userId, input.modelKey) : null;
const model = custom?.model ?? (await getModel(input.modelKey));
if (!model) throw new ApiError(404, isCustomModelKey(input.modelKey) ? "This custom endpoint no longer exists. Check Settings → Endpoints." : "Unknown model. Refresh your model list.", "MODEL_NOT_FOUND");
const apiKey = custom?.apiKey ?? (await getDecryptedKey(ctx.userId, model.provider));
if (!apiKey) throw new ApiError(400, `No API key configured for ${model.provider}. Add one in Settings → Providers.`, "NO_PROVIDER_KEY", { provider: model.provider });

add adapter?: AIProviderAdapter to PreparedTurn, return adapter: custom?.adapter from every return { conversation, model, apiKey, … }, and in runTurn replace const adapter = getAdapter(turn.model.provider); with const adapter = turn.adapter ?? getAdapter(turn.model.provider);. Also skip recordProviderOutcome when turn.model.provider === "custom" (there is no provider_connections row; it is a no-op update anyway) and keep touchRecent(userId, model.key) as is (works with any key).

src/lib/arena/service.ts — createArenaSession: for each key, const custom = isCustomModelKey(key) ? await resolveCustomEndpoint(userId, key) : null; const m = custom?.model ?? await getModel(key); … if (!custom && !(await getDecryptedKey(userId, m.provider))) missing.push(m.provider);. runArenaModel: same three-line substitution as prepareTurn, then const adapter = custom?.adapter ?? getAdapter(model.provider);.

apiKey for key-less endpoints is the sentinel NO_KEY_SENTINEL ("polyllm-no-key") — the OpenAI SDK needs a non-empty bearer; Ollama & co. ignore it. model.pricing is null → estimateCost().known === false → costUsd stays null in usage_records (the dashboard counts them as "unpriced"). usage_records.provider will be "custom" (valid ProviderId).

Model picker (B): /api/models already returns custom models with provider: "custom", family = endpoint name, metadata.endpointId/endpointName/endpointStatus; connectedProviders includes "custom" when at least one endpoint is valid. If B's category chips iterate PROVIDER_ORDER, custom models only appear in "All"/search — add a "Custom" chip keyed on provider === "custom" if desired.

# 4. Coming soon / known limits (nothing fake shipped)

• Custom endpoints have no pricing (never invented) — their requests show — cost and are counted in kpis.unpricedRequests. A per-endpoint price sheet is a candidate follow-up.
• Discovered custom models start text-only; users unlock vision/tools/reasoning per id through the manual-models editor. structuredOutput/files are always off for custom endpoints (server support varies too much to assume).
• Stored header values are never shown; editing any header value re-saves all headers (UI explains this).
• SSRF: DNS rebinding after the check and redirects to private hosts are not followed (redirect: "manual" on discovery; the OpenAI SDK follows redirects for chat — acceptable risk noted).
• Usage "all time" projection uses the first recorded request as the start of the observation window.
• No Coming soon badges were needed.

# 5. QA checklist (integration phase, single dev server)

Usage (/app/usage) at 375 / 390 / 430 / 1440:

• 48 px bar with hamburger, refresh, CSV icon on phones; PageHeader + buttons on desktop.
• Segmented Today/7d/30d/90d/Custom scrolls horizontally without page overflow; Custom opens a bottom sheet (dialog on desktop) with two type=date fields (16 px), presets, validation (end before start), Apply.
• Provider/model/project chips filter everything; "Filtered by …" pills clear individually; URL query updates (deep link reload keeps state).
• KPI tiles 2-col on phones, 4-col desktop; deltas coloured by direction (cost up = red).
• Charts lazy-load with skeletons; no default recharts colours; hour labels for Today, day labels otherwise, in the browser's zone; tooltips on every plot; stacked segments show 2 px surface gaps; failure segment only when failures exist.
• Cost by provider donut folds > 6 providers into "Other"; list shows provider icons + values.
• Projection card and Savings card (or the honest "nothing to save" state); "Why this sibling?" expands; catalog link works.
• Latency by model = two side-by-side small multiples (TTFT ascending, tok/s descending).
• Recent requests: rows on phones (tap → conversation), table from md; export CSV downloads with the current filters.
• Empty states: no data (with "Show all time"), filters with no match ("Clear filters"), API error with Retry. Refetch keeps the previous render at 70 % opacity (no skeleton flash).

Providers (/app/settings/providers):

• Rows sorted connected → failed → not connected; statuses Connected / Validation failed / Not validated yet / Not connected.
• Add key opens a bottom sheet on phones (dialog on desktop) with Paste (only where clipboard.readText exists), show/hide, docs link; success toast shows models + latency.
• Test connection shows latency inline (green/red) and toasts; failure shows the error under "Last error".
• More → Replace / Open console / Remove (ActionSheet on phones, dropdown on desktop); Remove uses ConfirmDialog.
• "Custom endpoints" card links to /app/settings/endpoints with counts.

Endpoints (/app/settings/endpoints):

• Empty state with preset chips; Add opens a full-height sheet on phones.
• Presets fill URL/path; the private-address warning appears for localhost/LAN URLs and explains tunnel vs ALLOW_PRIVATE_ENDPOINTS=1 (info tone when the server allows private).
• Headers rows add/remove; manual models rows with vision/tools/reasoning switches; discovery toggle hides/shows models path.
• "Save & test connection" → latency + discovered model chips; failure keeps the sheet open with the error; "Save without testing" → status "Not tested".
• Row actions: Test, More → Edit / Refresh model list / Remove. Models appear in the picker as provider "Custom" (family = endpoint name) after refreshModels.
• Server rejects http://localhost:11434/v1 with 400 PRIVATE_HOST when ALLOW_PRIVATE_ENDPOINTS is unset; accepts it when set to 1.

Settings shell:

• Phone /app/settings shows the six-section list (≥ 60 px rows); each section has a back chevron bar; desktop /app/settings redirects to Account and shows the side nav with "Endpoints".
• Account: audit log as stacked rows on phones, table on desktop; inputs 44 px / 16 px on phones. Security: session buttons full-width on phones. Appearance: switches have 44 px hit areas. Data: JSON export + usage CSV link.