SPB Git

spb/zyquo-router Public MIT

One local endpoint, every AI provider — a private OpenAI-compatible LLM gateway for your Mac (170 models, 12 providers).

macosswiftaillmgateway
Swift 95.7% Python 2.3% Shell 1.2% Makefile 0.9%

phase0: research + Zyquo Cloud study — ROUTER-RESEARCH.md, PROVIDER-REUSE.md, PLAN.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 11 days ago (Jul 31, 2026)

Showing 5 changed files with +2,793 and −0

added .gitignore +9 −0
@@ -0,0 +1,9 @@
1 +.build/
2 +*.app
3 +*.icns
4 +AppIcon.iconset/
5 +.DS_Store
6 +.env.test-keys
7 +dist/
8 +*.zip
9 +*.dmg
added CLAUDE.md +309 −0
@@ -0,0 +1,309 @@
1 +# CLAUDE.md — Zyquo Router
2 +
3 +## Project Identity
4 +
5 +**Zyquo Router** is the **gateway** member of the **Zyquo** family: a legendary, native macOS app written in **Swift + SwiftUI**, built **without the Xcode IDE** (Swift Package Manager + command-line toolchain). It turns the user's Mac into a **local LLM gateway**: the user stores their provider API keys once (same encrypted vault as the rest of the family — NO Keychain), picks a **port**, hits Start, and Zyquo Router serves a **standardized, OpenAI-compatible HTTP API** on `http://localhost:<port>` that fronts **every model of every provider from Zyquo Cloud** — one unified endpoint, one request format, **streaming and non-streaming**, for all of them.
6 +
7 +Any tool that speaks the OpenAI API (SDKs, CLIs, IDE plugins, scripts, other apps) can point at `http://localhost:<port>/v1` and instantly use Anthropic, OpenAI, xAI, Mistral, Gemini, Qwen/DashScope, DeepSeek, Kimi, Perplexity, Together, DeepInfra, and Cerebras through a single normalized interface — with per-request model routing, live traffic logs, usage/cost tracking, and access control. Think "OpenRouter / LiteLLM, but native, local, private, and gorgeous."
8 +
9 +**Naming conventions (use consistently everywhere):**
10 +- Display name / product name: `Zyquo Router`
11 +- App bundle: `Zyquo Router.app`
12 +- Bundle identifier: `com.zyquo.router`
13 +- Executable / SPM target: `ZyquoRouter` (no space)
14 +- Data folder: `~/Library/Application Support/ZyquoRouter/`
15 +- Repo module prefix in file headers: `Zyquo Router`
16 +
17 +---
18 +
19 +## 📋 MANDATORY FILE HEADER — EVERY CODE FILE
20 +
21 +**Every single code file you write** (all `.swift` files, plus `Makefile`, shell scripts, `Package.swift`, verification scripts — anything containing code) **MUST begin with this header comment**, adapted to the file's comment syntax:
22 +
23 +```swift
24 +//
25 +// <FileName>.swift
26 +// Zyquo Router
27 +//
28 +// Author: Simon-Pierre Boucher
29 +// Mail: contact@spboucher.ai
30 +//
31 +```
32 +
33 +For shell scripts / Makefiles:
34 +
35 +```bash
36 +#
37 +# <filename>
38 +# Zyquo Router
39 +#
40 +# Author: Simon-Pierre Boucher
41 +# Mail: contact@spboucher.ai
42 +#
43 +```
44 +
45 +No exceptions. If you ever create or refactor a file and the header is missing, add it. Before declaring the project done, run a sweep over the repository to verify every code file carries the header.
46 +
47 +---
48 +
49 +## 🧭 METHODOLOGY — WORK METHODICALLY, KEEP EVERYTHING COHERENT
50 +
51 +You must execute this project **strictly in phase order (0 → 8)**. Do not jump ahead, do not interleave phases, do not build the dashboard UI before the HTTP server routes a real request end-to-end, and do not write any code before Phase 0 research + Zyquo Cloud study are complete.
52 +
53 +**Working rules:**
54 +
55 +1. **One phase at a time.** At the start of each phase, write a checklist into `docs/PLAN.md`; check items off as you go. At the end of each phase, run a **phase checkpoint**: build (`swift build`), run what's runnable, fix all warnings/errors, write a 3–5 line phase summary in `docs/PLAN.md` before moving on.
56 +2. **Phase gates:** Phase 0 is complete only when `docs/ROUTER-RESEARCH.md` and `docs/PROVIDER-REUSE.md` are complete. Phase 2 is complete only when the embedded HTTP server answers `GET /v1/models` on the chosen port. Phase 3 is complete only when `POST /v1/chat/completions` works end-to-end — streaming AND non-streaming — against at least one model of at least three structurally different providers (one OpenAI-compatible, Anthropic, Gemini), verified with `curl` and the official OpenAI SDK. Phase 4 spec is the contract for all UI in Phase 6. Phase 7 is complete only when the full compatibility matrix is green with real keys. Phase 8 is complete only when `spctl` says "Notarized Developer ID".
57 +3. **Single source of truth, everywhere:**
58 + - Provider/model behavior → ported from Zyquo Cloud's client layer (Phase 0.B); never re-invent upstream request formats. **All Zyquo Cloud models are exposed by the router.**
59 + - The public API surface → defined once in `docs/API.md` (Phase 3) and implemented exactly; the served behavior, the docs, and the in-app API reference must never drift apart.
60 + - Colors, fonts, spacing, radii → only from `ZyquoTheme` design tokens. Zero raw hex or magic numbers in views.
61 + - HTTP serving, translation, and routing → only in the `Server/`, `Translate/`, `Router/` layers; never leak into Views/ViewModels.
62 + - Product naming → per the conventions above. Never `Zyquo` alone, never `ZyquoRouter` in user-facing text.
63 +4. **Coherence sweeps:** after Phases 3, 6, and 8 (uniform naming — always `RouteRequest`, `UpstreamCall`, `ProviderClient`, `APIKeyRecord`; no dead code; headers present; folders match Phase 2).
64 +5. **Compile early, compile often.** Never accumulate more than one file of unbuilt changes.
65 +6. **Commit discipline:** one logical unit per commit, phase-prefixed. Never commit secrets or logged request bodies.
66 +7. **Security is a design constraint from the first line of server code:** the server binds to `127.0.0.1` (localhost) by default; exposing on the LAN (`0.0.0.0`) is an explicit opt-in with a mandatory local access token; provider keys are NEVER returned by any endpoint, never logged, and requests/responses in logs are redacted by default.
67 +
68 +---
69 +
70 +## ⚠️ PHASE 0 — MANDATORY RESEARCH + ZYQUO CLOUD STUDY (DO THIS FIRST, BEFORE ANY CODE)
71 +
72 +Two mandatory research tracks, each producing a document. No Swift until both are done.
73 +
74 +### 0.A — `docs/ROUTER-RESEARCH.md` — how to build an LLM gateway/router, correctly (INTENSIVE WEB RESEARCH)
75 +
76 +Do NOT rely on training data. Perform **several intensive web research sessions** to nail every detail of building a production-quality LLM router. Research and document at minimum:
77 +
78 +1. **The OpenAI API specification — the standard you are implementing.** Document the CURRENT `POST /v1/chat/completions` contract exhaustively: request schema (`model`, `messages` with roles and multimodal content parts incl. `image_url`/base64, `temperature`, `top_p`, `max_tokens`/`max_completion_tokens`, `stop`, `n`, `frequency_penalty`, `presence_penalty`, `seed`, `response_format` / JSON mode, `tools` + `tool_choice` function calling, `stream`, `stream_options` incl. `include_usage`, `user`), the non-streaming response schema (`id`, `object: "chat.completion"`, `created`, `model`, `choices[].message`, `finish_reason` values, `usage` with prompt/completion/total tokens), and the **SSE streaming format** exactly (`data: {chat.completion.chunk}` events, delta structure for content AND for tool_calls arguments, role delta on first chunk, `finish_reason` on last content chunk, optional final usage chunk, terminating `data: [DONE]`). Also document `GET /v1/models` (list) and the **OpenAI error response format** (`{"error": {"message", "type", "param", "code"}}`) with correct HTTP status codes (400/401/403/404/429/500/502/503). Check whether the newer OpenAI `/v1/responses` API is worth also exposing; decide and document (chat/completions is mandatory; responses optional).
79 +2. **How existing gateways do it — study the references:** **LiteLLM** (proxy config, model naming `provider/model`, param translation tables, error mapping), **OpenRouter** (unified model IDs `vendor/model`, extra headers, streaming normalization, usage/cost accounting), and local servers that expose OpenAI-compatible endpoints (**Ollama**'s OpenAI compatibility layer, **LM Studio** server, **vLLM**). Extract concrete patterns: model namespacing, how they handle provider-specific params (pass-through via extra body), how they normalize finish reasons and usage, how they surface upstream errors, retry/fallback policies, and what compatibility pitfalls trip up real clients (SDKs are strict about chunk shapes).
80 +3. **Translation matrices — the hard core.** For each NON-OpenAI-shaped upstream, document the exact bidirectional mapping:
81 + - **Anthropic Messages API:** system prompt extraction (top-level `system` vs. messages), content blocks, `max_tokens` required, tool_use/tool_result ↔ OpenAI `tool_calls`/`tool` role messages, image blocks, stop_reason ↔ finish_reason mapping, and the **SSE event model** (`message_start`, `content_block_start/delta/stop`, `message_delta`, `message_stop`, `input_json_delta` for streamed tool arguments) → how each maps onto OpenAI chunk deltas.
82 + - **Gemini `generateContent` / `streamGenerateContent`:** `contents`/`parts`, `systemInstruction`, `generationConfig` param names, function calling ↔ tools, inline image data, candidate/finishReason mapping, and its streaming format → OpenAI chunks.
83 + - Confirm which of the remaining providers (xAI, Mistral, DashScope/Qwen, DeepSeek, Kimi, Perplexity, Together, DeepInfra, Cerebras) are OpenAI-compatible enough for near-pass-through, and document every known deviation per provider (unsupported params to strip, extra params to allow, usage/finish quirks, Perplexity citations, DeepSeek/Qwen reasoning content fields — decide how reasoning is surfaced, e.g., a `reasoning_content` field preserved in the normalized response).
84 +4. **HTTP server in Swift without heavyweight deps:** evaluate and choose the serving approach — **SwiftNIO** directly (preferred: Apple-maintained, SPM-clean, full control over SSE/chunked responses, graceful shutdown, backpressure) vs. `Network.framework` listener vs. embedding a small framework (Hummingbird/Vapor — heavier). Document the choice with rationale, plus: binding localhost vs. 0.0.0.0, port-in-use detection, concurrent connection handling with structured concurrency, request body size limits, timeouts (long ones for streaming), keep-alive, CORS headers (so browser-based tools can call the router), and clean cancel when a client disconnects mid-stream (must cancel the upstream call too).
85 +5. **Gateway concerns:** request logging with redaction, token usage extraction per provider, **cost calculation** from per-model pricing, rate limiting per API key, retries with exponential backoff on upstream 429/5xx, optional **fallback chains** (if model A fails → try model B), and health checks.
86 +
87 +### 0.B — `docs/PROVIDER-REUSE.md` — study the Zyquo Cloud repo and reuse its providers
88 +
89 +**Before writing provider code, read and study the Zyquo Cloud repository** (sibling project). Locate it on disk (check the user's projects folder; if not found, ask the user for its path). Document and reuse:
90 +
91 +1. **Exactly how each provider's API is called** in Zyquo Cloud: base URLs, auth headers, request/response `Codable` models, the shared `OpenAICompatibleClient`, native `AnthropicClient`/`GeminiClient`, SSE parsing. The router calls upstreams the **exact same way** — port or factor this code to be identical to Cloud's.
92 +2. **The complete model catalog** (`ModelCatalog` / `docs/PROVIDERS.md`) with capabilities (vision, tools, reasoning, context, max output) and pricing. **The router exposes ALL of these models** via `GET /v1/models`, using stable namespaced IDs: **`provider/model-id`** (e.g., `anthropic/claude-...`, `openai/gpt-...`, `deepseek/deepseek-chat`), while also accepting the bare upstream model ID when it is unambiguous. Aliases (user-defined friendly names, e.g., `fast` → a chosen model) supported.
93 +3. **The secure key vault** from Cloud (custom AES-256-GCM, **NO Keychain**): reuse the same `SecureKeyStore` design and vault format so users manage keys identically across the family.
94 +4. Provider streaming quirks so the translation layer normalizes them all into byte-perfect OpenAI chunks.
95 +
96 +---
97 +
98 +## PHASE 1 — Project Setup (No Xcode IDE)
99 +
100 +- **Toolchain:** SPM. `Package.swift`, executable target `ZyquoRouter`. Dependencies: Foundation + SwiftUI + CryptoKit + **SwiftNIO** (or the serving choice justified in Phase 0.A.4) + optionally Apple `swift-markdown` for the in-app docs. Nothing else.
101 +- **App bundle:** `Makefile` builds release, assembles `Zyquo Router.app`, signs (Phase 8; ad-hoc for `make dev`).
102 +- **Info.plist:** `CFBundleDisplayName` = `Zyquo Router`, bundle ID `com.zyquo.router`, `LSMinimumSystemVersion` (macOS 13.0+), `NSHighResolutionCapable`, `LSApplicationCategoryType` (`public.app-category.developer-tools`). Universal (arm64 + x86_64) for release. Optional `LSUIElement`-style behavior later via a "run in menu bar only" setting (still a regular app by default).
103 +- **Entry point:** `@main` SwiftUI `App`; proper activation from terminal; the server lifecycle is owned by the app (start/stop survives window close if enabled).
104 +
105 +---
106 +
107 +## PHASE 2 — Architecture + Server Skeleton
108 +
109 +```
110 +Sources/ZyquoRouter/
111 +├── App/ # @main, windows, menu bar extra, server lifecycle
112 +├── DesignSystem/ # ZyquoTheme — family tokens, Router palette
113 +├── Models/ # AIModel, ProviderConfig, RouteRule, Alias, RequestLogEntry, UsageRecord, LocalAPIKey…
114 +├── Server/
115 +│ ├── HTTPServer.swift # NIO bootstrap, bind host/port, lifecycle, graceful shutdown
116 +│ ├── Routes.swift # /v1/chat/completions, /v1/models, /health, (optional /v1/embeddings, /v1/responses)
117 +│ ├── SSEWriter.swift # spec-exact chunked SSE emission, [DONE], client-disconnect handling
118 +│ ├── AuthMiddleware.swift # optional local API keys (Bearer), per-key limits
119 +│ └── CORS.swift
120 +├── Router/
121 +│ ├── RequestRouter.swift # model-id → provider resolution, aliases, fallback chains
122 +│ ├── RetryPolicy.swift # backoff on 429/5xx, timeout handling
123 +│ └── UsageMeter.swift # tokens + cost per request/key/model/provider
124 +├── Translate/
125 +│ ├── OpenAINormalizer.swift # canonical internal request/response ⇄ OpenAI wire format
126 +│ ├── AnthropicTranslator.swift
127 +│ ├── GeminiTranslator.swift
128 +│ └── CompatAdjuster.swift # per-provider param strip/translate table for OpenAI-compatible upstreams
129 +├── Providers/ # PORTED FROM ZYQUO CLOUD (all models)
130 +│ ├── ProviderProtocol.swift
131 +│ ├── OpenAICompatibleClient.swift
132 +│ ├── AnthropicClient.swift
133 +│ └── GeminiClient.swift
134 +├── Services/
135 +│ ├── SecureKeyStore.swift # reused from Zyquo Cloud (no Keychain)
136 +│ ├── RequestLogStore.swift # ring-buffer + persisted log, redaction
137 +│ └── PersistenceService.swift
138 +├── ViewModels/
139 +└── Views/
140 +```
141 +
142 +- **Structured concurrency end-to-end:** each inbound request is a task; upstream streaming is an `AsyncSequence` piped into `SSEWriter`; client disconnect cancels the upstream task immediately.
143 +- **PHASE GATE:** server starts on a user-chosen port, `GET /health` returns ok, `GET /v1/models` returns the full namespaced catalog as spec-shaped JSON, port-in-use is detected with a clean error, and Stop shuts down gracefully (in-flight requests drain or cancel cleanly).
144 +
145 +---
146 +
147 +## PHASE 3 — THE STANDARDIZED API (THE CORE)
148 +
149 +Implement the unified endpoint per `docs/ROUTER-RESEARCH.md`, and write **`docs/API.md`** — the exact public contract — as you build. **PHASE GATE:** streaming + non-streaming chat completions verified with `curl` AND the official OpenAI Python/JS SDK pointed at `http://localhost:<port>/v1`, against one OpenAI-compatible provider, Anthropic, and Gemini.
150 +
151 +### 3.A — `POST /v1/chat/completions` (mandatory, spec-exact)
152 +- Accept the full OpenAI request schema (0.A.1) including multimodal image content and `tools`/`tool_choice`. Resolve `model` (namespaced `provider/model`, unambiguous bare ID, or alias) via `RequestRouter`; return the proper OpenAI 404-style error for unknown models.
153 +- **Non-streaming:** call the upstream via the ported clients, translate the response into a spec-exact `chat.completion` object — correct `choices`, mapped `finish_reason`, real `usage` (from upstream when provided; estimated and flagged otherwise), and the namespaced model echoed back.
154 +- **Streaming:** translate every upstream's stream (OpenAI-style SSE, Anthropic event blocks, Gemini chunks) into **byte-perfect OpenAI `chat.completion.chunk` SSE** — role delta first, content deltas, streamed `tool_calls` argument deltas, `finish_reason` chunk, optional usage chunk when `stream_options.include_usage` is set, then `data: [DONE]`. Strict SDKs must parse it without complaint.
155 +- **Tool calling** and **JSON mode / response_format** translated per provider where supported; cleanly rejected with a helpful OpenAI-format error where the upstream can't do it.
156 +- **Reasoning models:** preserve reasoning output in a consistent field (per the Phase 0 decision, e.g., `reasoning_content` on the message/delta) so DeepSeek-R1-style models work through the router.
157 +- Provider-specific extras (e.g., Perplexity search options) accepted via pass-through extra body keys, documented in `docs/API.md`.
158 +- **Error mapping:** upstream failures become OpenAI-format errors with correct status codes (401 upstream auth → 401 with clear "provider key invalid" message; 429 → 429 with retry-after if given; timeouts → 504-style; unknown → 502) — never leak raw provider payload shapes or any key material.
159 +- **Retries & fallbacks:** RetryPolicy on transient upstream errors; optional user-configured fallback chains (model list tried in order), surfaced honestly in the response `model` field.
160 +
161 +### 3.B — Supporting endpoints
162 +- `GET /v1/models` — full catalog with namespaced IDs (+ aliases), OpenAI list shape; optionally enriched metadata (context, pricing) under an `x-zyquo` extension key.
163 +- `GET /health` — status, uptime, version.
164 +- **Optional (nice, after the mandatory works):** `POST /v1/embeddings` routed to providers offering embeddings; `POST /v1/responses` if Phase 0 deemed it worthwhile.
165 +
166 +### 3.C — Access control & network posture
167 +- Default bind `127.0.0.1`; **LAN exposure (`0.0.0.0`) is explicit opt-in** and forces at least one **local API key**.
168 +- Local API keys: user-generated bearer tokens (`zyquo-sk-…`) with per-key enable/disable, optional per-key rate limits and model allow-lists; keys hashed at rest; shown once at creation.
169 +- CORS configurable (default permissive for localhost tooling).
170 +- Provider keys live only in the reused encrypted vault; no endpoint ever returns them.
171 +
172 +---
173 +
174 +## PHASE 4 — DESIGN SYSTEM & UI (LIGHT THEME, PIXEL-PERFECT, "CONTROL ROOM" IDENTITY)
175 +
176 +Same design DNA and `ZyquoTheme` token system as the family, with a **"Router / signal" identity**: a **graphite-cyan** story — switching, signal, uptime. The app is a **dashboard/control room**, not a chat app.
177 +
178 +### 4.1 — Light theme
179 +
180 +| Token | Value (light) | Usage |
181 +|---|---|---|
182 +| `background` | `#FAFBFC` (crisp cool off-white) | Canvas |
183 +| `surface` | `#FFFFFF` | Cards, panels |
184 +| `surfaceSecondary` | `#F1F4F6` | Hover, log rows, code blocks |
185 +| `accent` | `#0891B2` (signal cyan) paired with graphite `#374151` | Start button, active states, links, charts |
186 +| `accentSubtle` | `#E5F5F9` | Selected rows, active-server tint |
187 +| `textPrimary` `#191C1F` · `textSecondary` `#697077` · `textTertiary` `#9BA3AA` · `border` `#E4E8EB` | | |
188 +| `success`/`warning`/`danger` | `#2FA36B`/`#D9822B`/`#D64545` | Server running / degraded / stopped & errors |
189 +| chart tokens | cyan (requests), graphite (latency), per-provider hues | Dashboard charts |
190 +
191 +Family rules apply (no pure black on white, 0.5pt hairlines, ultra-soft shadows on floating panels only, dark theme derived — a deep graphite "ops room" feel; **light theme is flagship**). Typography/spacing/radii identical to the family; **SF Mono everywhere data lives**: endpoints, model IDs, logs, JSON, keys.
192 +
193 +### 4.2 — Layout & screens (exact spec)
194 +
195 +Left navigator (240pt, translucent) + detail. Default 1280×820, min 1020×660. Sections: **Dashboard**, **Models**, **Requests**, **Keys** (Provider Keys + Local API Keys), **Playground**, **Docs**, plus footer settings gear and a permanent **server status pill** (● Running on :8787 / ○ Stopped) with a Start/Stop control — visible from every screen.
196 +
197 +- **Dashboard:** the hero. A large **server card**: status, big Start/Stop button (accent), editable **port field**, bind selector (Localhost only / LAN with warning), the endpoint URL `http://localhost:8787/v1` with a one-click **Copy** and copy-as `curl` / OpenAI-Python / OpenAI-JS snippets. Below: live tiles — requests/min sparkline, tokens in/out today, **estimated cost today**, error rate, active streams count, uptime; a per-provider breakdown bar. Everything updates live and smoothly.
198 +- **Models:** the full namespaced catalog (search, filter by provider/capability): each row = `provider/model-id` (mono, one-click copy), capability badges (vision/tools/reasoning), context, pricing, enable/disable toggle (disabled models 404 through the API), favorite. **Aliases** editor (e.g., `fast` → `cerebras/...`; `best` → `anthropic/...`). **Fallback chains** editor (ordered model lists with drag-reorder).
199 +- **Requests (live traffic):** a streaming log table — time, method/path, model, provider, status chip, latency, tokens in/out, cost, stream badge; click a row → detail pane with redacted request/response JSON (pretty-printed, mono), timing waterfall (queue → upstream TTFB → stream duration), and error detail when failed. Filters (provider, model, status, key), pause/clear, and a **redaction toggle** (bodies hidden by default; revealing requires an explicit per-session switch).
200 +- **Keys:** two tabs. *Provider Keys* — identical UX to Zyquo Cloud (masked fields, per-provider Test button with status dot + latency, reused encrypted vault). *Local API Keys* — create/name/revoke `zyquo-sk-…` tokens (shown once, copy button), per-key rate limit and model allow-list, per-key usage mini-chart.
201 +- **Playground:** a built-in tester that calls **the router's own local endpoint** (not the upstreams directly — it must eat its own dogfood): model picker (namespaced IDs), message composer, params, streaming toggle; response pane shows streamed output AND the raw request/response JSON side-by-side with copy-as-code.
202 +- **Docs:** a beautiful in-app rendering of `docs/API.md` — endpoint reference, schemas, streaming format, error codes, and ready-to-paste snippets (curl, Python `OpenAI(base_url=...)`, JS, LangChain) each with copy buttons. Generated from the same source of truth as the implementation.
203 +- **Empty/first-run state:** a stunning onboarding card — "Add a provider key → pick a port → Start" three-step, with the endpoint revealed on start. Must look App-Store-feature quality.
204 +
205 +**Settings** (native tabs, 720×540): 1. **Server** (default port, bind, auto-start server at app launch, launch app at login, keep serving when window closed, request body size limit, timeouts) 2. **Logging** (retention, redaction default, export logs) 3. **Usage & Pricing** (pricing table view/override, cost currency, reset counters) 4. **Appearance** (Light/Dark/System; accent: cyan default + graphite, sky, emerald, violet, copper; font size) 5. **Shortcuts** 6. **Advanced** (reveal data folder, export/import config incl. aliases/chains — never keys in plaintext).
206 +
207 +### 4.3 — Motion & 4.4 quality gate
208 +Family motion standard; router-specific: the live request table streams new rows without jank, sparkline/tiles update smoothly (≤4Hz), the status pill transitions cleanly (stopped→starting→running), Start/Stop feels instant. **Quality gate:** review every state — stopped, starting, running, port conflict, LAN warning, no provider keys yet, request streaming live, upstream failing (429/500), key revoked, logs empty/full, redaction on/off. Consistent tokens, aligned mono columns, no clipped IDs. If it looks "developer-made", iterate.
209 +
210 +### Menu bar extra (first-class here)
211 +A toggleable menu bar item: status dot + port, Start/Stop, requests/min, today's cost, and "Copy endpoint URL". The router is exactly the kind of app that lives in the menu bar; make this excellent.
212 +
213 +---
214 +
215 +## PHASE 5 — APP ICON: ULTRA-LEGENDARY "ROUTER" ICON, DESIGNED IN SVG
216 +
217 +Designed in SVG first (`assets/icon/zyquo-router.svg`) → `.icns`. Sibling of the family, in the established style: **Big Sur squircle with a clean near-white soft-3D background (#FCFCFD, faint inner-edge shadow), gentle top lighting, soft realistic drop shadows; the dominant hero is the very bold, thick, chunky geometric charcoal-black "Z" (#0B0B14 → #1C1C2A gradient, subtle top-face sheen, fully opaque, ~50% of the icon) whose bottom stroke is a vivid electric-blue-to-indigo parallelogram accent (#2B6BFF → #4C2BE0)** — identical Z treatment to the other Zyquo icons.
218 +
219 +**Signature motif — the Z that routes.** Two directions (render both, keep the best):
220 +1. *Z-hub:* from the Z's right side, **three thin gradient connection lines fan out** to three small soft-3D node endpoints (tiny rounded squares or spheres, cyan-to-indigo gradients) at the icon's right edge — one Z in, many providers out; a small glowing cyan "port" dot sits at the line origin. Lines kept thin and secondary.
221 +2. *Z-switch:* the Z centered over a subtle backdrop of **crossing signal paths** (two thin curved lines swapping positions, like a rail switch/interchange, light cyan-gray, low contrast) with small direction chevrons — routing made visual.
222 +- Accent color leaning **cyan** (#22B8D4 blended into the family blue-indigo) in the connection lines/nodes so Router is distinguishable beside Cloud (cloud), Local (server stack), Agent (robot), Atlas (globe), MLX (forge) while remaining unmistakably the same set.
223 +- **Precision & iteration:** clean paths, `viewBox="0 0 1024 1024"`, optical centering; render 16→1024, inspect, refine; simplified small-size variant (drop lines/nodes, keep the black Z with blue base) for 16/32px.
224 +
225 +**Pipeline (Makefile):** SVG → PNGs (16→1024 incl. `@2x`) via `rsvg-convert` or CoreGraphics rasterizer → `AppIcon.iconset` → `iconutil -c icns`. SVG is the source of truth. Derive the monochrome **menu bar template icon** (18×18pt — critical for this app; a mini Z-with-signal-dot glyph) and in-app wordmark from the same SVG.
226 +
227 +---
228 +
229 +## PHASE 6 — Features (This is where Zyquo Router becomes LEGENDARY)
230 +
231 +### The gateway
232 +- One OpenAI-compatible endpoint on a user-chosen port fronting **every provider/model from Zyquo Cloud**, streaming and non-streaming, spec-exact
233 +- Namespaced model IDs + user aliases + fallback chains; enable/disable models; multimodal (images) and tool calling translated per provider; reasoning content preserved
234 +- Retries with backoff; honest error mapping in OpenAI format; per-request cancellation propagated upstream on client disconnect
235 +
236 +### Control & security
237 +- Localhost by default; LAN opt-in gated behind mandatory local API keys (`zyquo-sk-…`, hashed at rest, per-key limits and model allow-lists)
238 +- Provider keys in the reused encrypted vault (NO Keychain), never logged, never served
239 +- Redacted-by-default request logging with explicit reveal; export logs
240 +
241 +### Observability
242 +- Live dashboard: req/min, tokens, **cost estimates** from the catalog's pricing, error rate, latency, per-provider/per-model/per-key breakdowns, uptime
243 +- Full request inspector with timing waterfall and pretty JSON
244 +- Usage history persisted; daily/weekly summaries
245 +
246 +### Developer experience
247 +- One-click copy: endpoint URL, curl, OpenAI Python/JS snippets pre-filled with the port and a local key
248 +- In-app **Docs** rendered from `docs/API.md`; built-in **Playground** that calls the router's own endpoint
249 +- Health endpoint; auto-start server at launch; launch-at-login; keep-serving with window closed; first-class **menu bar extra**
250 +- Shortcuts: ⌘R start/stop server, ⌘K command palette, ⌘1–6 sections, ⌘F filter logs, ⌘⇧C copy endpoint URL
251 +
252 +---
253 +
254 +## PHASE 7 — VERIFICATION (MANDATORY)
255 +
256 +The user will provide **real API keys** (same providers as Zyquo Cloud). You MUST:
257 +
258 +1. **Full compatibility matrix:** for **every provider and every chat model** in the catalog, drive requests through the local endpoint (never directly at upstreams) and verify: non-streaming completion correct; **streaming chunk-shape spec-exact** (validated by parsing with the official OpenAI SDK, not just eyeballing); usage present/estimated correctly; finish_reason mapped; errors in OpenAI format. Additionally verify **tool calling** on tool-capable models, **vision** on vision-capable models, and **reasoning content** on reasoning models. Produce the table: provider → model → non-stream ✅/❌ → stream ✅/❌ → tools/vision/reasoning ✅/❌/n-a → notes. **Fix every failure until green.**
259 +2. **Client-compatibility tests:** point the official **OpenAI Python SDK** and **OpenAI JS SDK** (and `curl`) at `http://localhost:<port>/v1` with a local key and run scripted checks for both modes incl. streamed tool calls. The SDKs must work unmodified.
260 +3. **Gateway behavior:** port conflict handling; graceful shutdown with an active stream; client-disconnect cancels upstream (verify no orphaned upstream usage); retry on injected 429; fallback chain triggers correctly and reports the actually-used model; local-key auth (valid/invalid/revoked/rate-limited paths); LAN mode refuses to start without a key.
261 +4. **Security checks:** confirm provider keys never appear in any response, log file, or error; logs are redacted by default; vault round-trips.
262 +5. Never commit, log, or embed the user's keys anywhere; keys live only in the encrypted vault or env vars during testing.
263 +
264 +---
265 +
266 +## PHASE 8 — SIGNING & NOTARIZATION (REAL, NOT AD-HOC)
267 +
268 +The user has an existing, working signing/notarization setup for another project. **Before doing anything, read and inspect the folder:**
269 +
270 +```
271 +/Users/simon-pierreboucher/Desktop/other/OTHER/zyquo-term
272 +```
273 +
274 +Locate the **Developer ID Application identity name**, **Team ID**, **notarytool keychain profile (or Apple ID + app-specific password)**, entitlements, and any config there. **Reuse the exact same identity, Team ID, and notarytool credentials/profile for Zyquo Router.** Never invent placeholders, never print secrets, never commit them.
275 +
276 +Then implement `make release`:
277 +1. Build universal release (arm64 + x86_64, `lipo`), assemble `Zyquo Router.app`.
278 +2. `entitlements.plist` with **Hardened Runtime**; minimal set — network **client** (upstream calls) and network **server** behavior (verify: for a non-sandboxed Developer ID app, listening on localhost needs no special entitlement; if you adopt App Sandbox instead, you'd need `com.apple.security.network.server` + `.client` — prefer NOT sandboxing this developer tool, document the choice). Note that macOS may show the local-network permission prompt when binding non-localhost; handle and explain it in-app.
279 +3. `codesign --force --options runtime --timestamp --entitlements entitlements.plist --sign "Developer ID Application: <identity from zyquo-term>" "Zyquo Router.app"` — sign nested code first.
280 +4. `ditto -c -k --keepParent` → `xcrun notarytool submit "Zyquo Router.zip" --keychain-profile "<profile from zyquo-term>" --wait`.
281 +5. `xcrun stapler staple "Zyquo Router.app"`; verify `spctl -a -vv` = "accepted, source=Notarized Developer ID" and `stapler validate`.
282 +6. Optional signed+stapled DMG (`hdiutil`).
283 +7. On failure: `notarytool log`, fix, resubmit until it passes. Keep `make dev` (ad-hoc) for iteration.
284 +
285 +---
286 +
287 +## Engineering Standards
288 +
289 +- Swift 5.9+ (Swift 6 mode if the toolchain allows); zero warnings
290 +- Server built on structured concurrency; every request a cancellable task; upstream cancellation on client disconnect guaranteed; backpressure-aware SSE writing
291 +- All wire types `Codable` and spec-exact; the translation layer fully unit-testable with recorded fixtures per provider (build fixture tests for chunk translation — Anthropic events → OpenAI chunks, Gemini → OpenAI chunks)
292 +- Robust, human-readable errors everywhere (port in use → suggest next free port; provider key invalid → name the provider; upstream down → 502 with detail)
293 +- Provider layer identical to Zyquo Cloud with ALL models; design tokens only; UI strings centralized; `docs/API.md` always in sync with behavior (add a CI-style check script that exercises the running server against the documented schemas)
294 +- `README.md` (build + quickstart incl. pointing the OpenAI SDK at the router) + `docs/` (ROUTER-RESEARCH, PROVIDER-REUSE, API, PLAN)
295 +- Commit in logical, phase-prefixed increments; never commit secrets or unredacted logs
296 +
297 +## Definition of Done
298 +
299 +- `make release` produces a **Developer ID–signed, notarized, stapled** `Zyquo Router.app` (verified by `spctl`), built without the Xcode IDE
300 +- The server starts on any user-chosen port and serves a **spec-exact OpenAI-compatible API** — `chat/completions` (streaming + non-streaming, tools, vision, reasoning) and `models` — fronting **every provider/model from Zyquo Cloud** with namespaced IDs, aliases, and fallback chains
301 +- The official OpenAI Python and JS SDKs work against the router unmodified; the Phase 7 compatibility matrix is fully green with real keys
302 +- Security posture holds: localhost by default, LAN only with mandatory local keys, provider keys only in the reused encrypted vault (NO Keychain), redacted logs, nothing leaks
303 +- Live dashboard, request inspector, usage/cost tracking, Playground, in-app Docs, and a first-class menu bar extra all work beautifully
304 +- The cyan-signal SVG icon exists in the family style (chunky black Z with electric-blue base on the white soft-3D squircle, routing-lines motif), striking at all sizes, embedded as `.icns` + menu bar template glyph
305 +- The cyan-graphite light theme matches the Phase 4 spec and passes the design quality gate; dark theme derived and correct
306 +- Naming coherent everywhere: `Zyquo Router` user-facing, `com.zyquo.router`, `ZyquoRouter` target/data folder
307 +- **Every code file starts with the mandatory Author/Mail header** (verified by a repo-wide sweep)
308 +- `docs/PLAN.md` shows every phase completed; `docs/ROUTER-RESEARCH.md`, `docs/PROVIDER-REUSE.md`, and `docs/API.md` are complete and traceable to the implementation
309 +- Zyquo Router feels like a polished, legendary native Mac gateway — the definitive local LLM router
added docs/PLAN.md +41 −0
@@ -0,0 +1,41 @@
1 +# Zyquo Router — PLAN
2 +
3 +Phase-by-phase execution log. One phase at a time, checkpoint at each gate.
4 +
5 +---
6 +
7 +## Phase 0 — Mandatory research + Zyquo Cloud study
8 +
9 +- [x] 0.A.1 OpenAI API spec documented exhaustively (chat/completions request + response + SSE streaming + models + errors; /v1/responses decision)
10 +- [x] 0.A.2 Reference gateways studied (LiteLLM, OpenRouter, Ollama/LM Studio/vLLM compat layers) with concrete patterns extracted
11 +- [x] 0.A.3 Translation matrices: Anthropic Messages ⇄ OpenAI, Gemini generateContent ⇄ OpenAI, per-provider deviation table for OpenAI-compatible upstreams, reasoning-content decision
12 +- [x] 0.A.4 Swift HTTP serving approach evaluated and chosen with rationale (SwiftNIO vs Network.framework vs Hummingbird/Vapor), incl. SSE, cancellation, CORS, timeouts
13 +- [x] 0.A.5 Gateway concerns: logging/redaction, usage extraction, cost calc, rate limiting, retries/backoff, fallback chains, health
14 +- [x] `docs/ROUTER-RESEARCH.md` complete
15 +- [x] 0.B.1 Zyquo Cloud provider layer studied (base URLs, auth, Codable models, OpenAICompatibleClient, AnthropicClient, GeminiClient, SSE parsing)
16 +- [x] 0.B.2 Complete model catalog + capabilities + pricing extracted
17 +- [x] 0.B.3 SecureKeyStore vault design (AES-256-GCM, NO Keychain) documented for reuse
18 +- [x] 0.B.4 Provider streaming quirks documented
19 +- [x] `docs/PROVIDER-REUSE.md` complete
20 +
21 +**Phase gate: PASSED (2026-07-30).**
22 +
23 +**Phase 0 summary:** `docs/ROUTER-RESEARCH.md` (1,646 lines) compiled from four intensive
24 +web-research tracks: full current OpenAI chat/completions + SSE contract, gateway patterns
25 +from LiteLLM/OpenRouter/Ollama/LM Studio/vLLM, exact bidirectional Anthropic and Gemini
26 +translation matrices plus a 9-provider OpenAI-compat deviation table, SwiftNIO chosen for
27 +serving, and ops (redaction, usage/cost, rate limits, retries, fallbacks). 12 binding
28 +decisions (D1–D12) recorded up front. `docs/PROVIDER-REUSE.md` (788 lines) documents Zyquo
29 +Cloud's 12 providers / 170-model catalog / `vault.zq` AES-256-GCM SecureKeyStore (no
30 +Keychain) and a file-by-file port plan; key caveats: Cloud has no native GeminiClient
31 +(Router translates Gemini natively per D10) and no tool-calling in Cloud's wire types
32 +(Router extends them). All 12 real provider keys smoke-tested OK (HTTP 200).
33 +
34 +## Phase 1 — Project setup (SPM, no Xcode IDE) — pending
35 +## Phase 2 — Architecture + server skeleton — pending
36 +## Phase 3 — Standardized API (the core) — pending
37 +## Phase 4 — Design system & UI spec — pending
38 +## Phase 5 — App icon — pending
39 +## Phase 6 — Features — pending
40 +## Phase 7 — Verification with real keys — pending
41 +## Phase 8 — Signing & notarization — pending
added docs/PROVIDER-REUSE.md +788 −0
@@ -0,0 +1,788 @@
1 +# PROVIDER-REUSE.md — Zyquo Router
2 +Study of the Zyquo Cloud repository (../zyquo-cloud) for provider-layer reuse.
3 +
4 +<!--
5 + PROVIDER-REUSE.md
6 + Zyquo Router
7 +
8 + Author: Simon-Pierre Boucher
9 + Mail: contact@spboucher.ai
10 +-->
11 +
12 +Source repo studied: `/Users/simon-pierreboucher/Desktop/zyquo-cloud` (Swift 5.9 SPM executable, macOS 13+,
13 +single dependency `swift-markdown` — see `Package.swift:12-39`). Catalog and research were compiled/verified
14 +by Cloud with **live keys on 2026-07-30** (`docs/PROVIDERS.md:8-13`, Phase 7 amendments at
15 +`docs/PROVIDERS.md:1638-1719`). Every claim below is traceable to a file/line in that repo.
16 +
17 +---
18 +
19 +## 1. How each provider's API is called in Zyquo Cloud
20 +
21 +### 1.1 Architecture — two clients, one registry
22 +
23 +Cloud's entire provider layer is **four files** in `Sources/ZyquoCloud/Providers/` plus one shared
24 +networking service:
25 +
26 +| File | Role |
27 +|---|---|
28 +| `Sources/ZyquoCloud/Providers/ProviderProtocol.swift` | `ChatRequest`, `ChatEvent`, `protocol ProviderClient`, `ProviderError` |
29 +| `Sources/ZyquoCloud/Providers/OpenAICompatibleClient.swift` | One client for **11 of 12** providers (everything except Anthropic) + custom endpoints |
30 +| `Sources/ZyquoCloud/Providers/AnthropicClient.swift` | Native Anthropic Messages API (`/v1/messages`) |
31 +| `Sources/ZyquoCloud/Providers/ProviderRegistry.swift` | `wireFormat` → client resolution |
32 +| `Sources/ZyquoCloud/Services/StreamingService.swift` | `SSEParser`, `SSEEvent`, shared `URLSession`, retry/backoff, error mapping |
33 +
34 +**There is NO `GeminiClient.swift` in Zyquo Cloud.** Gemini is driven through its **OpenAI-compatible
35 +endpoint** `https://generativelanguage.googleapis.com/v1beta/openai` with Bearer auth
36 +(`Sources/ZyquoCloud/Models/ProviderID.swift:64`), through `OpenAICompatibleClient` like the other
37 +OpenAI-shaped providers. Cloud's research notes the compat endpoint's limitations and that a native
38 +client would be needed for rich thinking display (`docs/PROVIDERS.md:560`), but Cloud shipped
39 +compat-only. Zyquo Router's CLAUDE.md assumes a native `GeminiClient` exists — **it does not**; the
40 +Router must either keep the compat path (near-pass-through, easiest) or write the native translator
41 +itself per Phase 0.A research (see §5 caveats).
42 +
43 +The protocol (`ProviderProtocol.swift:31-42`):
44 +
45 +```swift
46 +protocol ProviderClient {
47 + var providerID: ProviderID { get }
48 + func streamChat(_ request: ChatRequest, apiKey: String) -> AsyncThrowingStream<ChatEvent, Error>
49 + func complete(_ request: ChatRequest, apiKey: String) async throws -> Message
50 + func listModelIDs(apiKey: String) async throws -> [String]
51 +}
52 +```
53 +
54 +The provider-agnostic request/event types (`ProviderProtocol.swift:13-28`):
55 +
56 +```swift
57 +struct ChatRequest {
58 + var model: AIModel
59 + var systemPrompt: String?
60 + var messages: [Message]
61 + var parameters: ChatParameters
62 + var stream: Bool = true
63 +}
64 +enum ChatEvent {
65 + case reasoningDelta(String)
66 + case textDelta(String)
67 + case citations([Citation])
68 + case usage(TokenUsage)
69 + case finished(reason: String?)
70 +}
71 +```
72 +
73 +A default-protocol extension provides `testKey(_:fallbackModel:) async throws -> TimeInterval`
74 +(`ProviderProtocol.swift:49-68`): calls `listModelIDs` where available, else a 16-token
75 +`"Reply with exactly: OK"` completion (Perplexity has no `/models` — returns 404,
76 +`ProviderID.swift:77-79`, `docs/PROVIDERS.md:27`).
77 +
78 +`ProviderRegistry` (`ProviderRegistry.swift:13-34`) resolves clients purely from
79 +`ProviderID.wireFormat`:
80 +
81 +```swift
82 +switch model.provider.wireFormat {
83 +case .anthropicMessages: return AnthropicClient()
84 +case .openAIChatCompletions: return OpenAICompatibleClient(provider: model.provider,
85 + baseURLOverride: model.customBaseURL)
86 +}
87 +```
88 +
89 +`WireFormat` has exactly two cases (`ProviderID.swift:88-93`): `.openAIChatCompletions` (11/12
90 +providers) and `.anthropicMessages`.
91 +
92 +### 1.2 Provider identity, base URLs, auth
93 +
94 +`ProviderID` is a 13-case enum (`Sources/ZyquoCloud/Models/ProviderID.swift:12-25`):
95 +`openai, anthropic, xai, mistral, gemini, qwen, deepseek, kimi, perplexity, together, deepinfra,
96 +cerebras, custom`. Exact `defaultBaseURL` constants (`ProviderID.swift:58-74`):
97 +
98 +| Provider | `defaultBaseURL` | Auth header | Wire format | `/models`? |
99 +|---|---|---|---|---|
100 +| OpenAI | `https://api.openai.com/v1` | `Authorization: Bearer <key>` | OpenAI (origin) | ✅ |
101 +| Anthropic | `https://api.anthropic.com/v1` | `x-api-key: <key>` + `anthropic-version: 2023-06-01` | Anthropic Messages | ✅ (`?limit=100`) |
102 +| xAI | `https://api.x.ai/v1` | Bearer | OpenAI-compat | ✅ |
103 +| Mistral | `https://api.mistral.ai/v1` | Bearer | OpenAI-compat | ✅ |
104 +| Google Gemini | `https://generativelanguage.googleapis.com/v1beta/openai` | Bearer (compat endpoint) | OpenAI-compat | ✅ (IDs prefixed `models/` — stripped) |
105 +| Qwen (DashScope intl) | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | Bearer | OpenAI-compat | ✅ |
106 +| DeepSeek | `https://api.deepseek.com` | Bearer | OpenAI-compat | ✅ |
107 +| Kimi (Moonshot) | `https://api.moonshot.ai/v1` | Bearer | OpenAI-compat | ✅ |
108 +| Perplexity | `https://api.perplexity.ai` | Bearer | OpenAI-compat + search extras | ❌ (404) |
109 +| Together AI | `https://api.together.xyz/v1` | Bearer | OpenAI-compat | ✅ (bare array, not `{"data":[…]}`) |
110 +| DeepInfra | `https://api.deepinfra.com/v1/openai` | Bearer | OpenAI-compat | ✅ |
111 +| Cerebras | `https://api.cerebras.ai/v1` | Bearer | OpenAI-compat | ✅ |
112 +| custom | `nil` — from user endpoint config (`baseURLOverride`) | Bearer | OpenAI-compat | varies |
113 +
114 +Paths are appended with `base.appendingPathComponent(path)` so multi-segment bases
115 +(`…/compatible-mode/v1`, `…/v1beta/openai`) are preserved
116 +(`OpenAICompatibleClient.swift:229-241`). Chat endpoint path is `"chat/completions"`; Anthropic's is
117 +`"messages"`. Anthropic constants: `apiVersion = "2023-06-01"`, `defaultMaxTokens = 8192`
118 +(`AnthropicClient.swift:19-20`).
119 +
120 +### 1.3 `OpenAICompatibleClient` — request/response wire types
121 +
122 +All wire types are **private nested Codable structs** (an adaptation point for the Router — see §5).
123 +
124 +**Request** (`OpenAICompatibleClient.swift:28-103`) — `WireRequest`:
125 +`model, messages: [WireMessage], stream, streamOptions (stream_options.include_usage), temperature,
126 +topP (top_p), maxTokens (max_tokens), maxCompletionTokens (max_completion_tokens), frequencyPenalty,
127 +presencePenalty, reasoningEffort (reasoning_effort), enableThinking (enable_thinking)`.
128 +`WireMessage { role: String, content: WireContent }`; `WireContent` encodes either a plain string or
129 +`[WirePart]` where `WirePart` is `.text(String)` or `.imageURL(String)` (encoded as
130 +`{"type":"image_url","image_url":{"url":"data:<mime>;base64,…"}}`). Images are always sent as
131 +base64 data URIs (`OpenAICompatibleClient.swift:302-320`).
132 +
133 +**Body construction** (`buildBody(_:)`, `OpenAICompatibleClient.swift:257-300`) is gated by the
134 +model's `ParameterSupport` (see §2): only supported params are encoded. Notable per-provider logic:
135 +
136 +- `wantsStreamOptions` (`OpenAICompatibleClient.swift:243-255`): `stream_options:{include_usage:true}`
137 + sent to **openai, xai, gemini, deepseek, kimi, together, cerebras, custom**; **omitted** for
138 + mistral (rejects unknown params), qwen, deepinfra, perplexity (usage arrives automatically).
139 +- Mistral `reasoning_effort` only accepts `"high"`/`"none"` → client maps `low → none`, else `high`
140 + (`OpenAICompatibleClient.swift:286-293`; `docs/PROVIDERS.md:1703`).
141 +- Qwen/DashScope `enable_thinking` is only legal on **streaming** requests
142 + (`OpenAICompatibleClient.swift:294-297`).
143 +- `usesMaxCompletionTokens` switches `max_tokens` → `max_completion_tokens` (OpenAI reasoning
144 + models, Cerebras, Kimi K-series) (`OpenAICompatibleClient.swift:277-283`).
145 +- System prompt injected as a leading `{"role":"system"}` message
146 + (`OpenAICompatibleClient.swift:259-261`).
147 +- Text-file attachments are inlined into the message text fenced with the file name
148 + (`OpenAICompatibleClient.swift:306-309`).
149 +
150 +**Response** (`OpenAICompatibleClient.swift:107-223`) — decoded with plain `JSONDecoder`, unknown
151 +fields ignored:
152 +
153 +- `WireChunk { choices: [WireChoice]?, usage: WireUsage?, citations: [String]?, searchResults: [WireSearchResult]? }`
154 + (`search_results` = Perplexity).
155 +- `WireChoice { delta: WireDelta?, message: WireDelta?, text: String?, finishReason: String? }` — the
156 + same struct decodes both streaming chunks (`delta`) and non-streaming responses (`message`);
157 + `text` is the Together completions-style fallback (`choices[].text` instead of `delta.content`,
158 + `OpenAICompatibleClient.swift:119-131`; `docs/PROVIDERS.md:1712-1713`).
159 +- `WireDelta { content, reasoningContent ("reasoning_content"), reasoning }` with a **custom
160 + `init(from:)`** (`OpenAICompatibleClient.swift:143-167`): `content` is normally a string, but
161 + Mistral reasoning models return an **array of `ContentChunk`s**
162 + (`{type:"thinking"|"text",…}` with nested `thinking:[{type:"text",text:…}]`) which are flattened
163 + into text + reasoning.
164 +- `WireUsage { promptTokens, completionTokens, completionTokensDetails.reasoningTokens }` →
165 + `TokenUsage` (`OpenAICompatibleClient.swift:187-210`).
166 +- `/models` decoding tries `{"data":[{id}]}` then a **bare array** (Together), and strips the
167 + `models/` prefix Gemini's compat endpoint adds (`listModelIDs`,
168 + `OpenAICompatibleClient.swift:407-421`).
169 +
170 +**Streaming loop** (`streamChat`, `OpenAICompatibleClient.swift:324-372`): wraps everything in an
171 +`AsyncThrowingStream` whose backing `Task` is cancelled `onTermination`; iterates
172 +`StreamingService.sseEvents(for:provider:)`; breaks on `data: [DONE]`; **silently skips
173 +undecodable chunks** (keep-alives, unknown shapes); yields `.reasoningDelta` (from
174 +`reasoning_content` or `reasoning`), `.textDelta` (from `delta.content` or `choices[].text`),
175 +`.citations` (once, Perplexity), `.usage`, then a final `.finished(reason:)` carrying the last
176 +`finish_reason` seen.
177 +
178 +**Non-streaming** (`complete`, `OpenAICompatibleClient.swift:374-405`): plain POST via
179 +`StreamingService.postJSON`, decodes the same `WireChunk` (reading `choice.message ?? choice.delta`),
180 +builds a `Message` with `reasoning`, `citations`, `usage`, and `estimatedCost` computed from
181 +`model.pricing.cost(inputTokens:outputTokens:)`. If `parameterSupport.requiresStreaming` is set the
182 +client transparently aggregates the stream instead (`completeViaStream`,
183 +`OpenAICompatibleClient.swift:425-454`).
184 +
185 +**Perplexity citations** (`OpenAICompatibleClient.swift:468-476`): top-level `citations` (array of
186 +URL strings) merged with `search_results` titles into numbered `Citation` values
187 +(`Sources/ZyquoCloud/Models/Message.swift:93-105`).
188 +
189 +### 1.4 `AnthropicClient` — native Messages API
190 +
191 +`Sources/ZyquoCloud/Providers/AnthropicClient.swift`. Headers: `x-api-key` + `anthropic-version:
192 +2023-06-01` (`AnthropicClient.swift:142-154`). **Never** routed through the OpenAI client
193 +(`wantsStreamOptions` explicitly notes "never routed here", `OpenAICompatibleClient.swift:252-253`).
194 +
195 +**Request** (`WireRequest`, `AnthropicClient.swift:24-48`): `model`, **mandatory `max_tokens`**
196 +(default 8192 when the user set none), `messages: [WireMessage]` (system messages filtered out),
197 +top-level `system: String?`, `stream`, `temperature`, `top_p`, and
198 +`thinking: {type: "enabled", budget_tokens: 8000} | {type: "disabled"}` when the model has
199 +`thinkingToggle` (`AnthropicClient.swift:156-181`). Content is block-structured: `WireBlock.text`
200 +and `WireBlock.image(mediaType:base64:)` encoding
201 +`{"type":"image","source":{"type":"base64","media_type":…,"data":…}}`
202 +(`AnthropicClient.swift:55-78`). Empty text is sent as `" "` (Anthropic rejects empty blocks,
203 +`AnthropicClient.swift:196`). Note per-model gating: Claude 4.7+/5 removed temperature/top_p —
204 +encoded in each model's `ParameterSupport` (`AnthropicClient.swift:171-174`,
205 +`ModelCatalogData.swift:249,257,266`).
206 +
207 +**Streaming** (`streamChat`, `AnthropicClient.swift:202-259`) decodes the **named SSE events**
208 +(the event name comes from `sse.event ?? event.type`):
209 +
210 +| Event | Handling |
211 +|---|---|
212 +| `message_start` | `usage.inputTokens` captured from `message.usage.input_tokens` |
213 +| `content_block_delta` | `delta.text` → `.textDelta`; `delta.thinking` → `.reasoningDelta` |
214 +| `message_delta` | `usage.output_tokens` (final count) + `delta.stop_reason` |
215 +| `error` | thrown as `ProviderError.serverError(status: 200, message:)` (mid-stream error events) |
216 +| `message_stop`, `ping`, `content_block_start/stop`, unknown | ignored |
217 +
218 +There is **no `[DONE]` sentinel** on Anthropic streams. `.usage` and `.finished(reason: stopReason)`
219 +are yielded at stream end. Non-streaming (`complete`, `AnthropicClient.swift:261-287`) joins
220 +`content[].type == "text"` and `"thinking"` blocks. `listModelIDs` GETs `messages`-sibling
221 +`models?limit=100` (`AnthropicClient.swift:289-299`).
222 +
223 +### 1.5 SSE parsing — `StreamingService` / `SSEParser`
224 +
225 +`Sources/ZyquoCloud/Services/StreamingService.swift`. `SSEEvent { event: String?, data: String }`
226 +(`StreamingService.swift:12-17`). `SSEParser` (`StreamingService.swift:22-47`) is a line-fed
227 +incremental parser: accumulates `event:`/`data:` fields (multi-line `data:` joined with `\n`),
228 +emits a complete event at the blank-line separator, **ignores `:` comment lines** (DeepSeek sends
229 +`: keep-alive`) and `id:`/`retry:`/unknown fields. Unit-tested in
230 +`Tests/ZyquoCloudTests/SSEParserTests.swift`.
231 +
232 +`StreamingService.sseEvents(for:provider:)` (`StreamingService.swift:63-118`):
233 +- Shared `URLSession` with `timeoutIntervalForRequest = 120`, `timeoutIntervalForResource = 900`
234 + (long streams), `User-Agent: ZyquoCloud/1.0 (macOS)` (`StreamingService.swift:53-59`).
235 +- Uses `session.bytes(for:)`; non-2xx → drains the full error body and throws
236 + `ProviderError.from(status:body:provider:)`.
237 +- **Splits bytes on `\n` manually** (handling `\r\n`) because `AsyncBytes.lines` *skips empty
238 + lines*, which are the SSE event separators (`StreamingService.swift:80-95`) — a real pitfall the
239 + Router must keep.
240 +- Flushes a trailing partial line/event at EOF; checks `Task.isCancelled` per byte-boundary;
241 + cancellation surfaces as `ProviderError.cancelled`.
242 +
243 +**Retry policy**: `postJSON` (`StreamingService.swift:122-156`) retries **3 attempts** on 429/5xx
244 +with `Retry-After` honored when present, else exponential backoff `pow(2, attempt) * 2` seconds
245 +(4s, 8s). `getJSON` delegates to `postJSON` (same mapping). **Streaming requests are NOT retried**
246 +(single attempt).
247 +
248 +### 1.6 Error types
249 +
250 +`ProviderError` (`ProviderProtocol.swift:73-146`), a `LocalizedError`:
251 +`invalidAPIKey(ProviderID)`, `rateLimited(ProviderID, retryAfter:)`, `serverError(ProviderID,
252 +status:, message:)`, `badRequest(ProviderID, message:)`, `networkError(underlying:)`,
253 +`invalidResponse(ProviderID, detail:)`, `missingAPIKey(ProviderID)`, `noModelAvailable(ProviderID)`,
254 +`cancelled`. Status mapping (`from(status:body:provider:)`): 401/403 → `invalidAPIKey`; 429 →
255 +`rateLimited` (retryAfter not parsed here); 400/404/422 → `badRequest`; else `serverError`.
256 +`extractMessage(from:)` (`ProviderProtocol.swift:127-145`) probes the provider error shapes
257 +`{"error":{"message":…}}`, `{"error":"…"}`, `{"message":…}`, `{"detail":…}`, and Gemini's
258 +top-level array form, falling back to the first 300 bytes of the body. The Router's error
259 +translation to OpenAI wire format (`{"error":{message,type,param,code}}` + proper status) maps
260 +directly onto these cases.
261 +
262 +---
263 +
264 +## 2. The complete model catalog
265 +
266 +**Swift types** (`Sources/ZyquoCloud/Models/AIModel.swift`):
267 +
268 +- `AIModel` (`AIModel.swift:15-43`): `id` (exact upstream model ID), `provider: ProviderID`,
269 + `displayName`, `contextWindow: Int`, `maxOutputTokens: Int?`, `capabilities: ModelCapabilities`,
270 + `pricing: ModelPricing?`, `parameterSupport: ParameterSupport`, `isLegacy`, `isRecommended`,
271 + `customBaseURL: URL?`.
272 +- `ModelCapabilities` (`AIModel.swift:47-60`): `vision, tools, reasoning, streaming (default true),
273 + jsonMode, citations`.
274 +- `ModelPricing` (`AIModel.swift:64-72`): **USD per 1M tokens**, `inputPerMTok` / `outputPerMTok`,
275 + with `func cost(inputTokens:outputTokens:) -> Double`. Cached/tiered pricing intentionally
276 + simplified to base rate; UI labels costs "estimates". **This is where the Router's cost meter
277 + gets its numbers.**
278 +- `ParameterSupport` (`AIModel.swift:77-93`): `temperature, topP, frequencyPenalty,
279 + presencePenalty, usesMaxCompletionTokens, reasoningEffort, thinkingToggle, requiresStreaming`;
280 + preset `.openAIDefault = ParameterSupport(frequencyPenalty: true, presencePenalty: true)`.
281 +- `TokenUsage` (`AIModel.swift:96-111`): `inputTokens, outputTokens, reasoningTokens?` + `+`.
282 +
283 +**Data lives in** `Sources/ZyquoCloud/Services/ModelCatalogData.swift` (1365 lines; `static let all
284 += openai + anthropic + xai + mistral + gemini + qwen + deepseek + kimi + perplexity + together +
285 +deepinfra + cerebras`, `ModelCatalogData.swift:1361-1363`), generated from `docs/PROVIDERS.md` and
286 +`docs/research/*.md` on **2026-07-30**, live-verified (Phase 7 amendments,
287 +`docs/PROVIDERS.md:1638-1719`). Runtime access via `@MainActor final class ModelCatalog:
288 +ObservableObject` (`Sources/ZyquoCloud/Services/ModelCatalog.swift:17-72`) which layers dynamic
289 +`/models` refreshes (`applyLiveListing`, `unknownLiveIDs`) and user custom models on top, and
290 +provides `cheapestModel(for:)` (non-reasoning preferred, min `outputPerMTok`).
291 +
292 +Scope note (`ModelCatalogData.swift:13-16`): **chat-completions-capable chat models only** —
293 +embeddings, audio, realtime, image/video gen, moderation, and Responses-API-only models are
294 +excluded by design.
295 +
296 +**Total: 170 models across 12 providers.** The Router exposes ALL of them as `provider/model-id`
297 +(e.g. `anthropic/claude-opus-5`, `qwen/qwen3.7-max`, `together/moonshotai/Kimi-K3` — note Together/
298 +DeepInfra IDs already contain `/`, so the Router's namespacing must split on the **first** `/`
299 +only). Legend: V=vision, T=tools, R=reasoning, J=jsonMode, C=citations; prices are USD per MTok
300 +in/out; ✩=isRecommended, †=isLegacy.
301 +
302 +### OpenAI (27) — `ModelCatalogData.swift:33-239`
303 +
304 +Preset `openAIReasoning` (`ModelCatalogData.swift:26-29`): no temperature/top_p,
305 +`max_completion_tokens`, `reasoning_effort`.
306 +
307 +| Model ID | Ctx | Max out | Caps | $ in | $ out | Notes |
308 +|---|---|---|---|---|---|---|
309 +| `gpt-5.6-sol` ✩ | 1,050,000 | 128,000 | V T R J | 5.00 | 30.00 | openAIReasoning |
310 +| `gpt-5.6-terra` ✩ | 1,050,000 | 128,000 | V T R J | 2.50 | 15.00 | openAIReasoning |
311 +| `gpt-5.6-luna` | 1,050,000 | 128,000 | V T R J | 1.00 | 6.00 | openAIReasoning |
312 +| `chat-latest` | 128,000 | — | V T J | 5.00 | 30.00 | freq/pres pen; rejects `max_tokens` (uses `max_completion_tokens`) |
313 +| `gpt-5.5` | 400,000 | — | V T R J | 5.00 | 30.00 | openAIReasoning |
314 +| `gpt-5.4` | 400,000 | 128,000 | V T R J | 2.50 | 15.00 | openAIReasoning |
315 +| `gpt-5.4-mini` | 400,000 | — | V T R J | 0.75 | 4.50 | openAIReasoning |
316 +| `gpt-5.4-nano` | 400,000 | — | V T R J | 0.20 | 1.25 | openAIReasoning |
317 +| `gpt-5.3-chat-latest` | 128,000 | — | V T J | — | — | `max_completion_tokens` |
318 +| `gpt-5.2` | 400,000 | 128,000 | V T R J | 1.75 | 14.00 | openAIReasoning |
319 +| `gpt-5.2-chat-latest` | 128,000 | 16,000 | V T J | 1.75 | 14.00 | `max_completion_tokens` |
320 +| `gpt-5.1` | 400,000 | 128,000 | V T R J | 1.25 | 10.00 | openAIReasoning |
321 +| `gpt-5` | 400,000 | 128,000 | V T R J | 1.25 | 10.00 | openAIReasoning |
322 +| `gpt-5-mini` | 400,000 | 128,000 | V T R J | 0.25 | 2.00 | openAIReasoning |
323 +| `gpt-5-nano` | 400,000 | 128,000 | V T R J | 0.05 | 0.40 | openAIReasoning |
324 +| `o3` | 200,000 | 100,000 | V T R J | 2.00 | 8.00 | openAIReasoning |
325 +| `o4-mini` | 200,000 | 100,000 | V T R J | 1.10 | 4.40 | openAIReasoning |
326 +| `o3-mini` † | 200,000 | 100,000 | T R J | 1.10 | 4.40 | openAIReasoning |
327 +| `o1` † | 200,000 | 100,000 | V T R J | 15.00 | 60.00 | openAIReasoning |
328 +| `gpt-4.1` † | 1,047,576 | 32,768 | V T J | 2.00 | 8.00 | .openAIDefault |
329 +| `gpt-4.1-mini` † | 1,047,576 | 32,768 | V T J | 0.40 | 1.60 | .openAIDefault |
330 +| `gpt-4.1-nano` † | 1,047,576 | 32,768 | V T J | 0.10 | 0.40 | .openAIDefault |
331 +| `gpt-4o` † | 128,000 | 16,384 | V T J | 2.50 | 10.00 | .openAIDefault |
332 +| `gpt-4o-mini` † | 128,000 | 16,384 | V T J | 0.15 | 0.60 | .openAIDefault |
333 +| `gpt-4-turbo` † | 128,000 | 4,096 | V T J | 10.00 | 30.00 | .openAIDefault |
334 +| `gpt-4` † | 8,192 | 8,192 | T | 30.00 | 60.00 | .openAIDefault |
335 +| `gpt-3.5-turbo` † | 16,385 | 4,096 | T J | 0.50 | 1.50 | .openAIDefault |
336 +
337 +### Anthropic (11) — `ModelCatalogData.swift:243-327`
338 +
339 +| Model ID | Ctx | Max out | Caps | $ in | $ out | Notes |
340 +|---|---|---|---|---|---|---|
341 +| `claude-opus-5` ✩ | 1,000,000 | 128,000 | V T R J | 5.00 | 25.00 | no temp/top_p; thinkingToggle |
342 +| `claude-sonnet-5` ✩ | 1,000,000 | 128,000 | V T R J | 3.00 | 15.00 | no temp/top_p; thinkingToggle |
343 +| `claude-fable-5` | 1,000,000 | 128,000 | V T R J | 10.00 | 50.00 | thinking always on — **no toggle**; no temp/top_p |
344 +| `claude-opus-4-8` | 1,000,000 | 128,000 | V T R J | 5.00 | 25.00 | no temp/top_p; thinkingToggle |
345 +| `claude-opus-4-7` | 1,000,000 | 128,000 | V T R J | 5.00 | 25.00 | no temp/top_p; thinkingToggle |
346 +| `claude-opus-4-6` | 1,000,000 | 128,000 | V T R J | 5.00 | 25.00 | temp/top_p OK; thinkingToggle |
347 +| `claude-sonnet-4-6` | 1,000,000 | 128,000 | V T R J | 3.00 | 15.00 | temp/top_p OK; thinkingToggle |
348 +| `claude-haiku-4-5-20251001` | 200,000 | 64,000 | V T R J | 1.00 | 5.00 | temp/top_p OK; thinkingToggle |
349 +| `claude-opus-4-5-20251101` † | 200,000 | 64,000 | V T R J | 5.00 | 25.00 | thinkingToggle |
350 +| `claude-sonnet-4-5-20250929` † | 1,000,000 | 64,000 | V T R J | 3.00 | 15.00 | thinkingToggle |
351 +| `claude-opus-4-1-20250805` † | 200,000 | 32,000 | V T R J | 15.00 | 75.00 | thinkingToggle |
352 +
353 +### xAI (5) — `ModelCatalogData.swift:331-369`
354 +
355 +| Model ID | Ctx | Max out | Caps | $ in | $ out | Notes |
356 +|---|---|---|---|---|---|---|
357 +| `grok-4.5` ✩ | 500,000 | — | V T R J | 2.00 | 6.00 | reasoning_effort |
358 +| `grok-4.3` | 1,000,000 | — | V T R J | 1.25 | 2.50 | reasoning_effort |
359 +| `grok-4.20` | 1,000,000 | — | V T R J | 1.25 | 2.50 | rejects reasoning_effort |
360 +| `grok-4.20-non-reasoning` | 1,000,000 | — | V T J | 1.25 | 2.50 | |
361 +| `grok-code-fast-1` ✩ | 256,000 | — | V T R J | 1.00 | 2.00 | rejects reasoning_effort |
362 +
363 +Note: `grok-4.20`, `grok-4.20-non-reasoning`, `grok-code-fast-1` resolve on chat completions but do
364 +**not** appear in xAI `/models` (`docs/PROVIDERS.md:1706`).
365 +
366 +### Mistral (10) — `ModelCatalogData.swift:373-451`
367 +
368 +| Model ID | Ctx | Caps | $ in | $ out | Notes |
369 +|---|---|---|---|---|---|
370 +| `mistral-medium-latest` ✩ (Medium 3.5) | 262,144 | V T R J | 1.50 | 7.50 | freq/pres + reasoning_effort |
371 +| `mistral-large-latest` ✩ (Large 3) | 262,144 | V T J | 0.50 | 1.50 | .openAIDefault |
372 +| `mistral-small-latest` ✩ (Small 4) | 262,144 | V T R J | 0.15 | 0.60 | freq/pres + reasoning_effort |
373 +| `codestral-latest` | 256,000 | T J | 0.30 | 0.90 | .openAIDefault |
374 +| `ministral-14b-latest` | 262,144 | V T J | 0.20 | 0.20 | .openAIDefault |
375 +| `ministral-8b-latest` | 262,144 | V T J | 0.15 | 0.15 | .openAIDefault |
376 +| `ministral-3b-latest` | 131,072 | V T J | 0.10 | 0.10 | .openAIDefault |
377 +| `magistral-medium-latest` † | 131,072 | T R J | 2.00 | 5.00 | ThinkChunk content arrays |
378 +| `devstral-latest` † (Devstral 2) | 262,144 | T J | 0.40 | 2.00 | |
379 +| `open-mistral-nemo` † | 131,072 | T J | 0.15 | 0.15 | |
380 +
381 +### Google Gemini (14) — `ModelCatalogData.swift:455-559` (all max out 65,536 except Gemma 32,768)
382 +
383 +| Model ID | Ctx | Caps | $ in | $ out | Notes |
384 +|---|---|---|---|---|---|
385 +| `gemini-3.6-flash` ✩ | 1,048,576 | V T R J | 1.50 | 7.50 | reasoning_effort |
386 +| `gemini-3.5-flash` | 1,048,576 | V T R J | 1.50 | 9.00 | reasoning_effort |
387 +| `gemini-3.5-flash-lite` ✩ | 1,048,576 | V T R J | 0.30 | 2.50 | reasoning_effort |
388 +| `gemini-3.1-pro-preview` ✩ | 1,048,576 | V T R J | 2.00 | 12.00 | reasoning_effort |
389 +| `gemini-3.1-flash-lite` | 1,048,576 | V T R J | 0.25 | 1.50 | reasoning_effort |
390 +| `gemini-2.5-pro` | 1,048,576 | V T R J | 1.25 | 10.00 | reasoning_effort |
391 +| `gemini-2.5-flash` | 1,048,576 | V T R J | 0.30 | 2.50 | reasoning_effort |
392 +| `gemini-2.5-flash-lite` | 1,048,576 | V T R J | 0.10 | 0.40 | reasoning_effort |
393 +| `gemini-pro-latest` | 1,048,576 | V T R J | — | — | rolling alias, pricing varies |
394 +| `gemini-flash-latest` | 1,048,576 | V T R J | — | — | rolling alias |
395 +| `gemini-flash-lite-latest` | 1,048,576 | V T R J | — | — | rolling alias |
396 +| `gemini-3-flash-preview` | 1,048,576 | V T R J | 0.50 | 3.00 | reasoning_effort |
397 +| `gemma-4-26b-a4b-it` | 262,144 | J | — | — | max out 32,768 |
398 +| `gemma-4-31b-it` | 262,144 | J | — | — | max out 32,768 |
399 +
400 +### Alibaba Qwen / DashScope (32) — `ModelCatalogData.swift:563-799` (pricing mostly unrecorded)
401 +
402 +| Model ID | Ctx | Max out | Caps | $ in | $ out | Notes |
403 +|---|---|---|---|---|---|---|
404 +| `qwen3.7-max` ✩ | 1,000,000 | — | T R J | 2.50 | 7.50 | enable_thinking |
405 +| `qwen3.7-plus` ✩ | 1,000,000 | 65,536 | V T R J | 0.32 | 1.28 | enable_thinking |
406 +| `qwen3.7-flash` ✩ | 1,000,000 | 65,536 | V T R J | 0.03 | 0.13 | enable_thinking |
407 +| `qwen3.6-plus` | 1,000,000 | — | V T R J | — | — | enable_thinking |
408 +| `qwen3.6-flash` | 1,000,000 | — | V T R J | — | — | enable_thinking |
409 +| `qwen3.5-plus` | 1,000,000 | — | V T R J | — | — | enable_thinking |
410 +| `qwen3.5-flash` | 1,000,000 | — | V T R J | — | — | enable_thinking |
411 +| `qwen-max` | 128,000 | — | T R J | — | — | enable_thinking |
412 +| `qwen-plus` | 1,000,000 | — | T R J | — | — | enable_thinking |
413 +| `qwen-turbo` † | 1,000,000 | — | T R J | — | — | enable_thinking |
414 +| `qwen-flash` | 1,000,000 | — | T R J | — | — | enable_thinking |
415 +| `qwen3-coder-plus` | 1,000,000 | — | T J | — | — | |
416 +| `qwen3-coder-flash` | 1,000,000 | — | T J | — | — | |
417 +| `qwen3-coder-next` | 262,144 | — | T J | — | — | |
418 +| `qwen3-coder-480b-a35b-instruct` | 262,144 | — | T J | — | — | |
419 +| `qwen3-vl-plus` | 1,000,000 | 65,536 | V T R J | — | — | enable_thinking |
420 +| `qwen3-vl-flash` | 1,000,000 | 65,536 | V T R J | — | — | enable_thinking |
421 +| `qwen3-vl-235b-a22b-instruct` | 131,072 | — | V T J | — | — | |
422 +| `qwen3-vl-235b-a22b-thinking` | 131,072 | — | V T R J | — | — | |
423 +| `qvq-max` | 131,072 | — | V R J | — | — | **requiresStreaming** |
424 +| `qwq-plus` | 131,072 | — | T R J | — | — | **requiresStreaming** |
425 +| `qwen3.5-397b-a17b` | 262,144 | — | T R J | — | — | enable_thinking |
426 +| `qwen3.5-122b-a10b` | 262,144 | — | T R J | — | — | enable_thinking |
427 +| `qwen3.5-35b-a3b` | 262,144 | — | T R J | — | — | enable_thinking |
428 +| `qwen3-235b-a22b-instruct-2507` | 262,144 | — | T J | — | — | |
429 +| `qwen3-235b-a22b-thinking-2507` | 262,144 | — | T R J | — | — | |
430 +| `qwen3-next-80b-a3b-instruct` | 262,144 | — | T J | — | — | |
431 +| `qwen3-next-80b-a3b-thinking` | 262,144 | — | T R J | — | — | |
432 +| `deepseek-v4-pro` | 1,000,000 | — | T R J | — | — | 3rd-party hosted; enable_thinking |
433 +| `deepseek-v4-flash` | 1,000,000 | — | T R J | — | — | 3rd-party hosted; enable_thinking |
434 +| `glm-5.2` | 198,000 | — | T R J | — | — | 3rd-party hosted; enable_thinking |
435 +| `kimi-k2.7-code` | 262,144 | — | T R J | — | — | 3rd-party hosted; enable_thinking |
436 +
437 +### DeepSeek (2) — `ModelCatalogData.swift:803-820`
438 +
439 +| Model ID | Ctx | Max out | Caps | $ in | $ out | Notes |
440 +|---|---|---|---|---|---|---|
441 +| `deepseek-v4-flash` ✩ | 1,000,000 | 384,000 | T R J | 0.14 | 0.28 | reasoning_effort + thinkingToggle |
442 +| `deepseek-v4-pro` ✩ | 1,000,000 | 384,000 | T R J | 0.435 | 0.87 | reasoning_effort + thinkingToggle |
443 +
444 +(`deepseek-chat`/`deepseek-reasoner` were retired 2026-07-24, `docs/PROVIDERS.md:42-44`.)
445 +
446 +### Kimi / Moonshot (12) — `ModelCatalogData.swift:824-920`
447 +
448 +| Model ID | Ctx | Max out | Caps | $ in | $ out | Notes |
449 +|---|---|---|---|---|---|---|
450 +| `kimi-k3` ✩ | 1,048,576 | 131,072 | V T R J | 3.00 | 15.00 | no temp/top_p; max_completion_tokens; reasoning_effort (thinking always on) |
451 +| `kimi-k2.7-code` ✩ | 262,144 | — | V T R J | 0.95 | 4.00 | no temp/top_p; max_completion_tokens |
452 +| `kimi-k2.7-code-highspeed` | 262,144 | — | V T R J | 1.90 | 8.00 | no temp/top_p; max_completion_tokens |
453 +| `kimi-k2.6` | 262,144 | — | V T R J | 0.95 | 4.00 | + thinkingToggle |
454 +| `kimi-k2.5` | 262,144 | — | V T R J | 0.60 | 3.00 | + thinkingToggle |
455 +| `moonshot-v1-8k` † | 8,192 | — | T J | 0.20 | 2.00 | temp capped 1.0 upstream |
456 +| `moonshot-v1-32k` † | 32,768 | — | T J | 1.00 | 3.00 | |
457 +| `moonshot-v1-128k` † | 131,072 | — | T J | 2.00 | 5.00 | |
458 +| `moonshot-v1-auto` † | 131,072 | — | T J | — | — | |
459 +| `moonshot-v1-8k-vision-preview` † | 8,192 | — | V T J | 0.20 | 2.00 | |
460 +| `moonshot-v1-32k-vision-preview` † | 32,768 | — | V T J | 1.00 | 3.00 | |
461 +| `moonshot-v1-128k-vision-preview` † | 131,072 | — | V T J | 2.00 | 5.00 | |
462 +
463 +### Perplexity (4) — `ModelCatalogData.swift:924-955` (all `citations: true`; no `/models` endpoint)
464 +
465 +| Model ID | Ctx | Max out | Caps | $ in | $ out | Notes |
466 +|---|---|---|---|---|---|---|
467 +| `sonar` ✩ | 128,000 | 128,000 | J C | 1.00 | 1.00 | |
468 +| `sonar-pro` ✩ | 200,000 | 8,000 | J C | 3.00 | 15.00 | |
469 +| `sonar-reasoning-pro` | 128,000 | — | R J C | 2.00 | 8.00 | reasoning as inline `<think>` blocks |
470 +| `sonar-deep-research` | 128,000 | — | R J C | 2.00 | 8.00 | multi-minute agentic runs — skipped in Cloud's automated sweep |
471 +
472 +### Together AI (16) — `ModelCatalogData.swift:959-1077`
473 +
474 +| Model ID | Ctx | Caps | $ in | $ out | Notes |
475 +|---|---|---|---|---|---|
476 +| `moonshotai/Kimi-K3` ✩ | 1,000,000 | T R J | 3.00 | 15.00 | |
477 +| `moonshotai/Kimi-K2.7-Code` | 262,144 | T R J | 0.95 | 4.00 | |
478 +| `moonshotai/Kimi-K2.6` | 262,144 | T R J | 1.20 | 4.50 | |
479 +| `deepseek-ai/DeepSeek-V4-Pro` ✩ | 512,000 | T R J | 1.74 | 3.48 | |
480 +| `zai-org/GLM-5.2` | 512,000 | T R J | 1.40 | 4.40 | |
481 +| `Qwen/Qwen3.7-Max` | 1,000,000 | T R J | 1.25 | 3.75 | **requiresStreaming** |
482 +| `Qwen/Qwen3.7-Plus` | 1,000,000 | T J | 0.32 | 1.28 | **requiresStreaming** |
483 +| `Qwen/Qwen3.6-Plus` | 1,000,000 | T J | 0.50 | 3.00 | **requiresStreaming** |
484 +| `Qwen/Qwen3.5-9B` | 262,144 | T J | 0.17 | 0.25 | **requiresStreaming**; streams completions-style (`choices[].text`) |
485 +| `meta-llama/Llama-3.3-70B-Instruct-Turbo` | 131,072 | T J | 1.04 | 1.04 | |
486 +| `openai/gpt-oss-120b` ✩ | 131,072 | T R J | 0.15 | 0.60 | reasoning_effort |
487 +| `openai/gpt-oss-20b` | 131,072 | T R J | 0.05 | 0.20 | reasoning_effort |
488 +| `nvidia/nemotron-3-ultra-550b-a55b` | 512,288 | T R J | 0.60 | 3.60 | |
489 +| `MiniMaxAI/MiniMax-M3` | 524,288 | T R J | 0.30 | 1.20 | |
490 +| `google/gemma-4-31B-it` | 262,144 | T J | 0.39 | 0.97 | **requiresStreaming**; vision disabled 2026-07-30 (empty answers on image input) |
491 +| `thinkingmachines/Inkling` | 524,288 | T R J | 1.00 | 4.05 | |
492 +
493 +### DeepInfra (34) — `ModelCatalogData.swift:1081-1328`
494 +
495 +| Model ID | Ctx | Caps | $ in | $ out | Notes |
496 +|---|---|---|---|---|---|
497 +| `anthropic/claude-fable-5` | 1,000,000 | V T R J | 10.00 | 50.00 | proxied frontier |
498 +| `anthropic/claude-opus-5` | 1,000,000 | V T R J | 5.00 | 25.00 | proxied |
499 +| `anthropic/claude-sonnet-5` | 1,000,000 | V T R J | 2.00 | 10.00 | proxied |
500 +| `anthropic/claude-opus-4-8` | 1,000,000 | V T R J | 5.00 | 25.00 | proxied |
501 +| `anthropic/claude-haiku-4-5` | 200,000 | V T R J | 1.00 | 5.00 | proxied |
502 +| `google/gemini-3.1-pro` | 1,000,000 | V T R J | 2.00 | 12.00 | proxied |
503 +| `google/gemini-3.5-flash` | 1,000,000 | V T R J | 1.50 | 9.00 | proxied |
504 +| `google/gemini-3.1-flash-lite` | 1,000,000 | V T J | 0.25 | 1.50 | proxied |
505 +| `google/gemini-2.5-pro` | 1,000,000 | V T R J | 1.25 | 10.00 | proxied |
506 +| `google/gemini-2.5-flash` | 1,000,000 | V T R J | 0.30 | 2.50 | proxied |
507 +| `deepseek-ai/DeepSeek-V4-Pro` ✩ | 1,048,576 | T R J | 1.30 | 2.60 | |
508 +| `deepseek-ai/DeepSeek-V4-Flash` ✩ | 1,048,576 | T J | 0.09 | 0.18 | |
509 +| `deepseek-ai/DeepSeek-V3.1` | 163,840 | T R J | 0.25 | 0.95 | |
510 +| `deepseek-ai/DeepSeek-R1-0528` | 163,840 | R | 0.50 | 2.15 | reasoning only |
511 +| `moonshotai/Kimi-K2.7-Code` | 262,144 | T R J | 0.74 | 3.50 | |
512 +| `moonshotai/Kimi-K2.6` | 262,144 | T R J | 0.75 | 3.50 | |
513 +| `moonshotai/Kimi-K2.5` | 262,144 | T J | 0.45 | 2.25 | **requiresStreaming** |
514 +| `zai-org/GLM-5.2` ✩ | 1,048,576 | T R J | 0.75 | 2.40 | |
515 +| `zai-org/GLM-4.7` | 202,752 | T R J | 0.40 | 1.75 | |
516 +| `Qwen/Qwen3.7-Max` | 256,000 | T R J | 2.50 | 7.50 | |
517 +| `Qwen/Qwen3.5-397B-A17B` | 262,144 | T R J | 0.45 | 3.00 | |
518 +| `Qwen/Qwen3-235B-A22B-Instruct-2507` | 262,144 | T J | 0.09 | 0.55 | |
519 +| `Qwen/Qwen3-235B-A22B-Thinking-2507` | 262,144 | T R J | 0.23 | 2.30 | |
520 +| `Qwen/Qwen3-Coder-480B-A35B-Instruct-Turbo` | 262,144 | T J | 0.30 | 1.00 | |
521 +| `Qwen/Qwen3-VL-235B-A22B-Instruct` | 262,144 | V T J | 0.20 | 0.88 | |
522 +| `meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8` | 1,048,576 | V T J | 0.20 | 0.80 | |
523 +| `meta-llama/Llama-4-Scout-17B-16E-Instruct` | 327,680 | V T J | 0.10 | 0.30 | |
524 +| `meta-llama/Llama-3.3-70B-Instruct-Turbo` | 131,072 | T J | 0.10 | 0.32 | |
525 +| `meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo` | 131,072 | T J | 0.02 | 0.04 | |
526 +| `openai/gpt-oss-120b` ✩ | 131,072 | T R J | 0.037 | 0.17 | reasoning_effort |
527 +| `openai/gpt-oss-20b` | 131,072 | T R J | 0.03 | 0.14 | reasoning_effort |
528 +| `MiniMaxAI/MiniMax-M3` | 524,288 | T R J | 0.30 | 1.20 | |
529 +| `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B` | 262,144 | T R J | 0.50 | 2.20 | |
530 +| `mistralai/Mistral-Small-3.2-24B-Instruct-2506` | 128,000 | V T J | 0.075 | 0.20 | |
531 +
532 +(`google/gemma-4-31B-it` was **removed** from DeepInfra 2026-07-30 — endpoint hangs 60s+ with zero
533 +bytes, `ModelCatalogData.swift:1326-1327`, `docs/PROVIDERS.md:1714-1715`.)
534 +
535 +### Cerebras (3) — `ModelCatalogData.swift:1332-1357` (all max out 40,000; `max_completion_tokens` required)
536 +
537 +| Model ID | Ctx | Caps | $ in | $ out | Notes |
538 +|---|---|---|---|---|---|
539 +| `gpt-oss-120b` ✩ | 131,072 | T R J | 0.35 | 0.75 | reasoning_effort |
540 +| `gemma-4-31b` | 131,072 | V T R J | 0.99 | 1.49 | reasoning_effort |
541 +| `zai-glm-4.7` † | 131,072 | T R J | 2.25 | 2.75 | discontinued 2026-08-17 |
542 +
543 +---
544 +
545 +## 3. The secure key vault — `SecureKeyStore`
546 +
547 +`Sources/ZyquoCloud/Services/SecureKeyStore.swift` (174 lines). **Confirmed: deliberately NOT the
548 +macOS Keychain** — the header says so verbatim (`SecureKeyStore.swift:8`). The only `Security.framework`
549 +usage in the file is `SecRandomCopyBytes(kSecRandomDefault, …)` for salt generation
550 +(`SecureKeyStore.swift:161-166`) — no `SecItem*` / keychain item APIs anywhere. A repo grep confirms
551 +no other Keychain usage in Zyquo Cloud.
552 +
553 +### File location & format
554 +
555 +- Vault file: `~/Library/Application Support/ZyquoCloud/vault.zq` — built as
556 + `PersistenceService.shared.rootDirectory.appendingPathComponent("vault.zq")`
557 + (`SecureKeyStore.swift:56-57`; root dir from
558 + `Sources/ZyquoCloud/Services/PersistenceService.swift:24-27`). **Router equivalent:**
559 + `~/Library/Application Support/ZyquoRouter/vault.zq` (same format).
560 +- Binary layout (`SecureKeyStore.swift:11-13`): **`[salt 32B][AES-GCM nonce 12B][ciphertext+tag 16B]`**
561 + — everything after the salt is the CryptoKit `AES.GCM.SealedBox.combined` representation
562 + (nonce ‖ ciphertext ‖ tag), opened with `AES.GCM.SealedBox(combined:)` + `AES.GCM.open(_:using:)`
563 + (`SecureKeyStore.swift:71-72`) and produced by `AES.GCM.seal(_:using:).combined`
564 + (`SecureKeyStore.swift:84-85`). Nonce is generated by CryptoKit per seal (fresh every save).
565 +- Plaintext: a JSON dictionary `{"openai": "sk-…", "anthropic": "sk-ant-…", …}` — keys are
566 + **`ProviderID.rawValue` strings** (`SecureKeyStore.swift:13`, `key(for:)`/`setKey(_:for:)` at
567 + `SecureKeyStore.swift:94-108`).
568 +- Written atomically with `.completeFileProtection` (`SecureKeyStore.swift:91`); the directory is
569 + created on demand.
570 +
571 +### Master key derivation (`SecureKeyStore.swift:118-159`)
572 +
573 +```swift
574 +HKDF<SHA256>.deriveKey(
575 + inputKeyMaterial: SymmetricKey(data: machineEntropy() ‖ pepper),
576 + salt: <32-byte vault salt>,
577 + info: Data("ZyquoCloud.vault.v1".utf8),
578 + outputByteCount: 32 // AES-256 key
579 +)
580 +```
581 +
582 +- `machineEntropy` = **IOPlatformUUID** (read from IOKit's `IOPlatformExpertDevice` registry entry
583 + via `IORegistryEntryCreateCFProperty(…, kIOPlatformUUIDKey, …)`,
584 + `SecureKeyStore.swift:138-148`) ‖ `NSHomeDirectory()` — binds the vault to this machine **and**
585 + account. Injectable via init for tests (`SecureKeyStore.swift:52-59`,
586 + `Tests/ZyquoCloudTests/SecureKeyStoreTests.swift`).
587 +- **Pepper**: 30 compiled-in bytes, stored XOR `0x5A` so the value never appears verbatim in the
588 + binary, reassembled at runtime (`pepper()`, `SecureKeyStore.swift:152-159`).
589 +- Salt: 32 random bytes (`SecRandomCopyBytes`) generated on first save; **re-saves reuse the
590 + existing salt** (`existingSalt()`, `SecureKeyStore.swift:169-173`) so the derived key stays stable.
591 +- Constants: `saltLength = 32`, `keyLength = 32` (`SecureKeyStore.swift:45-46`).
592 +
593 +### Public API surface (`SecureKeyStore.swift:61-114`)
594 +
595 +```swift
596 +init(vaultURL: URL? = nil, machineEntropy: (() throws -> Data)? = nil)
597 +func loadKeys() throws -> [String: String] // empty dict if no vault file
598 +func saveKeys(_ keys: [String: String]) throws // atomic full-dictionary rewrite
599 +func key(for provider: ProviderID) throws -> String?
600 +func setKey(_ apiKey: String, for provider: ProviderID) throws
601 +func deleteKey(for provider: ProviderID) throws
602 +static func redacted(_ apiKey: String) -> String // "••••abcd" (last 4 only)
603 +```
604 +
605 +Errors: `VaultError.corrupted` ("The key vault is damaged or belongs to another machine." — also
606 +thrown when GCM auth fails, i.e. wrong machine) and `VaultError.machineIdentityUnavailable`
607 +(`SecureKeyStore.swift:31-43`).
608 +
609 +The UI-facing wrapper is `@MainActor final class KeyVaultStore: ObservableObject`
610 +(`Sources/ZyquoCloud/ViewModels/KeyVaultStore.swift:16-93`): per-provider `KeyStatus`
611 +(`unset/saved/testing/verified(latency:)/failed(message:)`), redacted display strings, and
612 +`testKey(for:catalog:)` using `ProviderRegistry` + `ModelCatalog.cheapestModel(for:)`. Keys are
613 +decrypted on demand and never retained beyond the call (`KeyVaultStore.swift:55-61`).
614 +
615 +**Router decision for identical UX**: reuse `SecureKeyStore` byte-for-byte in design. Two constants
616 +differ per app: the vault path root (`ZyquoRouter` folder) and the HKDF `info` string
617 +(`"ZyquoCloud.vault.v1"`). Keeping `info` identical AND pointing at Cloud's vault file would let the
618 +apps literally share one vault; the Zyquo Router CLAUDE.md asks for the **same design and format**
619 +(users manage keys identically), not necessarily the same file — decide in Phase 2/3. If the Router
620 +keeps its own vault, use `info: "ZyquoRouter.vault.v1"` and its own `vault.zq` under
621 +`~/Library/Application Support/ZyquoRouter/`. Either way: no Keychain, ever.
622 +
623 +---
624 +
625 +## 4. Provider streaming quirks the Router's translation layer must normalize
626 +
627 +All recorded in code and in `docs/PROVIDERS.md` (cross-provider notes at lines 32-46; Phase 7
628 +amendments at lines 1638-1719):
629 +
630 +1. **Reasoning content field zoo** — normalize to one field (Router decision: `reasoning_content`
631 + on message/delta, per OpenAI-compat majority):
632 + - `delta.reasoning_content`: DeepSeek (on by default for v4-flash), Qwen, Kimi K-series, some
633 + DeepInfra models (`OpenAICompatibleClient.swift:133-146`).
634 + - `delta.reasoning` (no `_content`): some hosts (decoded as fallback,
635 + `OpenAICompatibleClient.swift:344`); Together exposes `message.reasoning` for hosted
636 + reasoning models (`docs/PROVIDERS.md:28`).
637 + - Anthropic: `thinking_delta` inside `content_block_delta` events; non-streaming `thinking`
638 + content blocks (`AnthropicClient.swift:226-232,270-271`).
639 + - Mistral: `content` arrives as an **array of ThinkChunk/TextChunk objects**
640 + (`{type:"thinking",thinking:[{type:"text",text:…}]}`) — must be flattened
641 + (`OpenAICompatibleClient.swift:143-184`; `docs/PROVIDERS.md:1704`).
642 + - Perplexity `sonar-reasoning-pro`: reasoning arrives as inline **`<think>…</think>` text** in
643 + the content itself (`docs/PROVIDERS.md:27,37`) — Cloud does not split it; the Router must
644 + decide (pass through or extract).
645 + - OpenAI: **no reasoning text at all** on chat completions — only
646 + `usage.completion_tokens_details.reasoning_tokens` (`docs/PROVIDERS.md:178`).
647 +2. **Usage-in-stream behavior** (`OpenAICompatibleClient.swift:243-255`; `docs/PROVIDERS.md:38-39`):
648 + `stream_options:{include_usage:true}` needed for OpenAI, xAI, Gemini-compat, DeepSeek, Kimi,
649 + Together, Cerebras (final chunk has empty `choices` + `usage`); Qwen, DeepInfra, Perplexity
650 + include usage automatically (and Mistral rejects the param). Anthropic splits usage:
651 + `input_tokens` in `message_start`, final `output_tokens` in `message_delta`.
652 +3. **Keep-alive comments**: DeepSeek sends `: keep-alive` SSE comment lines — `SSEParser` drops
653 + any `:`-prefixed line (`StreamingService.swift:36`). Undecodable data chunks are skipped, not
654 + fatal (`OpenAICompatibleClient.swift:339-342`).
655 +4. **Finish-reason quirks**: Together emits nonstandard `finish_reason: "eos"`
656 + (`docs/PROVIDERS.md:35`) — the Router must map it to `stop`. Anthropic uses `stop_reason`
657 + (`end_turn`, `max_tokens`, `stop_sequence`, `tool_use`) delivered in `message_delta`, which the
658 + Router maps to OpenAI `finish_reason` (`stop`/`length`/`stop`/`tool_calls`).
659 +5. **Together completions-style streams**: some models (`Qwen/Qwen3.5-9B`,
660 + `google/gemma-4-31B-it`) stream tokens in `choices[].text` instead of `delta.content`
661 + (`OpenAICompatibleClient.swift:122-124`; `docs/PROVIDERS.md:1712-1713`).
662 +6. **Models that reject non-streaming calls** (`ParameterSupport.requiresStreaming`,
663 + `AIModel.swift:88-90`): Qwen `qvq-max`/`qwq-plus`; Together `Qwen3.7-Max/-Plus`, `Qwen3.6-Plus`,
664 + `Qwen3.5-9B`, `google/gemma-4-31B-it`; DeepInfra `moonshotai/Kimi-K2.5`. For a non-streaming
665 + router request, aggregate the stream (Cloud's `completeViaStream` pattern).
666 +7. **Perplexity citations**: top-level `citations: [String]` (URLs) + `search_results` (titles) on
667 + chunks and responses — surfaced once per stream (`OpenAICompatibleClient.swift:356-359,468-476`).
668 + No `/models` endpoint (404): the Router must serve Perplexity models from the built-in catalog
669 + only.
670 +8. **Gemini compat endpoint**: `/models` IDs prefixed `models/` (stripped,
671 + `OpenAICompatibleClient.swift:411,420`); unknown fields like `extra_content`/`thought_signature`
672 + in deltas must be ignored; native-only features (thought summaries, `thoughtsTokenCount`) are
673 + not exposed on the compat endpoint (`docs/PROVIDERS.md:560`).
674 +9. **Unknown-field tolerance generally**: OpenAI adds `obfuscation` fields; decoders must ignore
675 + unknown JSON keys everywhere (`docs/PROVIDERS.md:34-35`).
676 +10. **Anthropic mid-stream errors**: an `error` SSE event can arrive with HTTP 200
677 + (`AnthropicClient.swift:240-243`) — the Router must convert it into an OpenAI-format error
678 + (or a terminal chunk if the stream already started).
679 +11. **Parameter strip tables** (per-model `ParameterSupport`): OpenAI reasoning models and Claude
680 + 4.7+/5 reject `temperature`/`top_p`; Cerebras + OpenAI reasoning + Kimi K-series require
681 + `max_completion_tokens`; Mistral `reasoning_effort` accepts only `high`/`none`; Qwen
682 + `enable_thinking` only when streaming; xAI `grok-4.20`/`grok-code-fast-1` reject
683 + `reasoning_effort`. This is exactly the Router's `CompatAdjuster` data.
684 +12. **Vision minimums**: xAI and Qwen reject images smaller than 8px (`docs/PROVIDERS.md:1707`).
685 +13. **Anthropic streams have no `data: [DONE]`** — termination is the `message_stop` event; the
686 + Router's SSEWriter must synthesize `[DONE]` itself for downstream OpenAI clients.
687 +
688 +---
689 +
690 +## 5. Reuse plan — files to port from Cloud → Router
691 +
692 +### 5.1 Port nearly verbatim (change header `Zyquo Cloud` → `Zyquo Router`; keep type names)
693 +
694 +| Source (zyquo-cloud) | Destination (zyquo-router) | Changes |
695 +|---|---|---|
696 +| `Sources/ZyquoCloud/Models/ProviderID.swift` | `Sources/ZyquoRouter/Models/ProviderID.swift` | header only (keep all base URLs, `WireFormat`, `supportsModelListing`) |
697 +| `Sources/ZyquoCloud/Models/AIModel.swift` | `Sources/ZyquoRouter/Models/AIModel.swift` | header only (`AIModel`, `ModelCapabilities`, `ModelPricing`, `ParameterSupport`, `TokenUsage`) |
698 +| `Sources/ZyquoCloud/Services/ModelCatalogData.swift` | `Sources/ZyquoRouter/Services/ModelCatalogData.swift` | header only — **all 170 models, verbatim**; keep in sync with Cloud going forward |
699 +| `Sources/ZyquoCloud/Services/ModelCatalog.swift` | `Sources/ZyquoRouter/Services/ModelCatalog.swift` | header; keep `@MainActor ObservableObject` for the UI, but the server-side `RequestRouter` needs a `Sendable` snapshot of `all` (don't hop to the main actor per request) |
700 +| `Sources/ZyquoCloud/Services/StreamingService.swift` | `Sources/ZyquoRouter/Services/StreamingService.swift` | header; change User-Agent to `ZyquoRouter/1.0 (macOS)`; consider moving retry policy into `Router/RetryPolicy.swift` (see 5.3) |
701 +| `Sources/ZyquoCloud/Services/SecureKeyStore.swift` | `Sources/ZyquoRouter/Services/SecureKeyStore.swift` | header; vault root becomes `~/Library/Application Support/ZyquoRouter/`; decide HKDF `info` (`"ZyquoRouter.vault.v1"` for a separate vault, or keep Cloud's string + path to share one vault); keep pepper mechanism (may reuse the same obfuscated bytes) |
702 +| `Sources/ZyquoCloud/Services/PersistenceService.swift` | `Sources/ZyquoRouter/Services/PersistenceService.swift` | header; root folder `ZyquoRouter`; drop `Conversation` specifics, keep generic `load/save` |
703 +| `Sources/ZyquoCloud/Providers/ProviderProtocol.swift` | `Sources/ZyquoRouter/Providers/ProviderProtocol.swift` | header; **extend** (see 5.2) |
704 +| `Sources/ZyquoCloud/Providers/ProviderRegistry.swift` | `Sources/ZyquoRouter/Providers/ProviderRegistry.swift` | header only |
705 +| `Sources/ZyquoCloud/Providers/OpenAICompatibleClient.swift` | `Sources/ZyquoRouter/Providers/OpenAICompatibleClient.swift` | header + extensions (see 5.2) |
706 +| `Sources/ZyquoCloud/Providers/AnthropicClient.swift` | `Sources/ZyquoRouter/Providers/AnthropicClient.swift` | header + extensions (see 5.2) |
707 +| `Tests/ZyquoCloudTests/SSEParserTests.swift` | `Tests/ZyquoRouterTests/SSEParserTests.swift` | header/module rename |
708 +| `Tests/ZyquoCloudTests/SecureKeyStoreTests.swift` | `Tests/ZyquoRouterTests/SecureKeyStoreTests.swift` | header/module rename |
709 +| (subset of) `Sources/ZyquoCloud/Models/Message.swift` | `Sources/ZyquoRouter/Models/Message.swift` | keep `Message`, `Attachment`, `Citation`; drop chat-UI fields if unused (`isStreaming`, `errorText`) or keep for log display |
710 +
711 +`KeyVaultStore.swift` (ViewModel) ports with light edits for the Router's Keys screen (Provider
712 +Keys tab is spec'd "identical UX to Zyquo Cloud").
713 +
714 +### 5.2 What Cloud's clients DON'T give the Router (must be added, not just ported)
715 +
716 +Cloud is a chat app; its clients expose a **UI-oriented, lossy** event stream. The Router needs a
717 +**spec-complete OpenAI surface**. Gaps found in the code:
718 +
719 +1. **No tool calling anywhere.** `WireRequest` has no `tools`/`tool_choice`; `WireDelta` decodes no
720 + `tool_calls`; `ChatEvent` has no tool case; Anthropic's client sends no `tools` and ignores
721 + `tool_use`/`input_json_delta` (grep of `Sources/ZyquoCloud/Providers/` for "tool" → zero hits).
722 + The Router must extend the wire types and `ChatEvent` (e.g. `.toolCallDelta(index:id:name:argumentsDelta:)`)
723 + and implement Anthropic `tool_use`/`tool_result` ↔ OpenAI `tool_calls`/`role:"tool"` translation
724 + per `docs/ROUTER-RESEARCH.md`.
725 +2. **No `response_format`/JSON mode, `stop`, `n`, `seed`, `logprobs`, `user`** in `WireRequest` —
726 + the catalog tracks `jsonMode` capability but Cloud never sends it. Add these fields (gated by
727 + `CompatAdjuster`).
728 +3. **Lossy events**: `ChatEvent` collapses per-choice structure (only `choices.first` is read,
729 + `OpenAICompatibleClient.swift:343`), drops chunk `id`/`created`/`model`/`system_fingerprint`,
730 + and merges role/content deltas. Fine for a chat UI; the Router must emit **byte-perfect
731 + `chat.completion.chunk`s**. Recommended approach: keep Cloud's request-construction +
732 + `StreamingService`/`SSEParser` + auth/quirk logic **as-is**, but widen the response path —
733 + either (a) make the wire response types non-private and add a raw-chunk streaming API
734 + (`streamRawChunks(_:apiKey:) -> AsyncThrowingStream<Data, Error>` yielding upstream SSE data
735 + payloads for OpenAI-compatible providers, enabling near-pass-through), plus a translated path
736 + for Anthropic; or (b) enrich `ChatEvent` to carry everything (finish per choice, tool deltas,
737 + ids, raw usage). Option (a) is closest to how LiteLLM/OpenRouter behave for compat upstreams and
738 + preserves fidelity; the Anthropic translator then builds spec-exact chunks from the named
739 + events already parsed in `AnthropicClient.streamChat`.
740 +4. **`ChatRequest` is UI-shaped** (`model: AIModel`, `messages: [Message]` with attachments). The
741 + Router receives OpenAI wire JSON; it should define a canonical internal request
742 + (`Translate/OpenAINormalizer.swift`) and map it into the ported clients' body builders — or
743 + refactor `buildBody` to accept the canonical type. Keep `ParameterSupport` gating exactly as
744 + Cloud does; it *is* the per-provider param strip/translate table (`CompatAdjuster` seed data).
745 +5. **`ChatParameters`** lives in `Sources/ZyquoCloud/Models/Conversation.swift` (temperature, topP,
746 + maxTokens, frequency/presencePenalty, reasoningEffort, thinkingEnabled) — port the struct (or
747 + inline it) since both clients consume it.
748 +6. **Retry semantics**: Cloud retries only non-streaming calls (3 attempts in
749 + `StreamingService.postJSON`). The Router's `RetryPolicy` (Phase 2 layout) should own this and
750 + also handle pre-first-byte retry for streaming + `Retry-After` propagation to clients.
751 +7. **Gemini**: no native client exists (see §1.1). Near-term: reuse the compat path (verified live
752 + by Cloud, including vision via data URIs and `reasoning_effort`). If Phase 3's gate demands the
753 + native `generateContent` translation, `GeminiTranslator.swift` is **net-new work** guided by
754 + `docs/ROUTER-RESEARCH.md` — nothing to port from Cloud beyond `docs/PROVIDERS.md:524-696`
755 + research.
756 +8. **Usage estimation**: when upstream omits usage, Cloud just shows nothing. The Router spec
757 + requires estimated-and-flagged usage — net-new (token estimation), though `TokenUsage` and
758 + `ModelPricing.cost` port directly for the metering.
759 +
760 +### 5.3 Destination layout (matches Router CLAUDE.md Phase 2)
761 +
762 +```
763 +Sources/ZyquoRouter/
764 +├── Models/ ProviderID.swift, AIModel.swift, Message.swift (+ChatParameters) ← ported
765 +├── Providers/ ProviderProtocol.swift, ProviderRegistry.swift,
766 +│ OpenAICompatibleClient.swift, AnthropicClient.swift ← ported + extended
767 +│ (GeminiClient.swift only if native path chosen — net-new)
768 +├── Services/ StreamingService.swift, SecureKeyStore.swift,
769 +│ ModelCatalog.swift, ModelCatalogData.swift, PersistenceService.swift ← ported
770 +├── Translate/ OpenAINormalizer.swift, AnthropicTranslator.swift,
771 +│ GeminiTranslator.swift, CompatAdjuster.swift ← new (seeded by ParameterSupport + §4)
772 +├── Router/ RequestRouter.swift, RetryPolicy.swift, UsageMeter.swift ← new (RetryPolicy absorbs postJSON backoff)
773 +└── Server/ … ← new
774 +```
775 +
776 +### 5.4 Porting checklist
777 +
778 +- [ ] Rewrite every file header comment `Zyquo Cloud` → `Zyquo Router` (mandatory header sweep).
779 +- [ ] `ZyquoCloud/1.0` User-Agent → `ZyquoRouter/1.0`; HKDF info + vault path decision recorded.
780 +- [ ] Keep type names Cloud uses: `ProviderClient`, `ProviderID`, `WireFormat`, `AIModel`,
781 + `ModelCatalog`, `SecureKeyStore`, `SSEParser`, `TokenUsage`, `ProviderError` (Router
782 + CLAUDE.md's uniform-naming rule already includes `ProviderClient`).
783 +- [ ] `ModelCatalogData.swift` stays byte-identical to Cloud's (modulo header) — single source of
784 + truth for the `GET /v1/models` catalog and pricing/cost metering.
785 +- [ ] Extend wire types for tools / response_format / stop / n / seed; add raw-chunk streaming path.
786 +- [ ] Port fixture-worthy behaviors into unit tests: SSE blank-line handling, `: keep-alive`,
787 + Mistral ThinkChunk flattening, Together `choices[].text`, Anthropic event mapping, Gemini
788 + `models/` prefix strip, `finish_reason:"eos"` → `stop`.
added docs/ROUTER-RESEARCH.md +1646 −0
@@ -0,0 +1,1646 @@
1 +# ROUTER-RESEARCH.md — Zyquo Router
2 +
3 +How to build a production-quality local LLM gateway, correctly. Compiled from intensive
4 +web research (2026-07-30) against current official documentation, SDK sources, and
5 +reference gateway implementations. Sections: (1) the OpenAI API specification the router
6 +implements, (2) how existing gateways do it, (3) translation matrices for non-OpenAI
7 +upstreams, (4) HTTP serving in Swift, (5) gateway concerns. Sources cited inline.
8 +
9 +## Binding decisions (executive summary)
10 +
11 +| # | Decision | Rationale (detail in section) |
12 +|---|----------|-------------------------------|
13 +| D1 | Implement `POST /v1/chat/completions`, `GET /v1/models`, `GET /v1/models/{id}`, `GET /health`. **Do NOT expose `POST /v1/responses` in v1.** | No upstream except OpenAI speaks it; its statefulness conflicts with the local/private posture; cheap to add later (§1.6). |
14 +| D2 | Accept both `max_tokens` and `max_completion_tokens`; treat `max_completion_tokens` as canonical. | OpenAI deprecated `max_tokens`; real clients send either (§1.1, §2.4). |
15 +| D3 | **SwiftNIO directly** (NIOCore/NIOPosix/NIOHTTP1 + NIOExtras), `NIOAsyncChannel` structured-concurrency APIs. Hummingbird 2 is the documented runner-up. | SPM-clean, Apple-maintained, full control over SSE flush/backpressure/disconnect, Swift 6-ready (§4.1). |
16 +| D4 | Model namespace `provider/model-id`; bare IDs accepted when unambiguous; user aliases; disabled models 404. | LiteLLM/OpenRouter convention; avoids collisions like `deepseek-chat` on multiple hosts (§2.1, §2.5). |
17 +| D5 | Param policy: known-param translation table per provider (strip/rename/clamp), unknown keys **passed through** to the upstream body. | Matches vLLM/OpenRouter behavior; enables provider extras (Perplexity search, Qwen `enable_thinking`) without schema churn (§2.5, §3.3). |
18 +| D6 | Reasoning output normalized to DeepSeek-style **`reasoning_content`** on message and delta, with optional `reasoning_details` for signature round-trips. | Most tooling already understands DeepSeek's convention (§3.4). |
19 +| D7 | Errors always in OpenAI `{"error":{...}}` shape: upstream 401→401 "provider key invalid (<provider>)", 429→429 with Retry-After, timeout→504, other upstream→502. Never leak raw provider payloads or key material. | §1.5, §2.5. |
20 +| D8 | Retries: exponential backoff + jitter on 429/5xx/timeouts, respect `Retry-After`, **never retry once the first streamed byte has been forwarded**. Fallback chains report the actually-used model in `model`. | §5.5–5.6. |
21 +| D9 | Usage: upstream-first; estimated (and flagged via `x_zyquo.usage_estimated`) only when the upstream provides none. Cost computed from the catalog's per-model pricing incl. cached-token rates. | §5.2–5.3. |
22 +| D10 | **Gemini is translated natively** (`generateContent`/`streamGenerateContent?alt=sse`), even though Zyquo Cloud reaches Gemini through its OpenAI-compat endpoint. The Phase 3 gate requires a structurally different third upstream, and native translation avoids the compat layer's gaps (strict tool schemas, thinking metadata). | §3.2; PROVIDER-REUSE §1. |
23 +| D11 | Streaming contract is byte-exact per §1.3: role-delta first chunk, content/tool-argument deltas, finish_reason chunk, optional usage chunk (empty `choices`) only when `stream_options.include_usage`, then `data: [DONE]`. Upstream streams are read to true EOF. | §1.3, §2.4. |
24 +| D12 | Security: bind 127.0.0.1 by default; 0.0.0.0 opt-in forces ≥1 local API key (`zyquo-sk-…`, hashed at rest); logs redacted by default; provider keys never serialized into any response, log, or error. | §4.2–4.3, §5.1. |
25 +
26 +## 1. The OpenAI API specification
27 +
28 +> Research date: 2026-07-30. Primary sources: the OpenAI API reference (https://platform.openai.com/docs/api-reference/chat, mirrored at https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create), the streaming-events reference (https://developers.openai.com/api/reference/resources/chat/subresources/completions/streaming-events), and the official SDK type definitions, which are generated from OpenAI's OpenAPI spec and are therefore authoritative for the wire format (https://github.com/openai/openai-python/tree/main/src/openai/types/chat, https://github.com/openai/openai-node). This section is the implementation contract for Zyquo Router's public surface: **whatever the router serves must match this, byte-shape for byte-shape.**
29 +
30 +---
31 +
32 +### 1.1 `POST /v1/chat/completions` — request schema
33 +
34 +Headers: `Authorization: Bearer <key>`, `Content-Type: application/json`. Only `model` and `messages` are required; every other parameter is optional and, when omitted, must be treated as "provider default" (the router must NOT inject its own defaults into upstream calls).
35 +
36 +#### 1.1.1 `model` (string, required)
37 +
38 +Model ID, e.g. `"gpt-4o"`. For Zyquo Router this is the namespaced `provider/model-id`, an unambiguous bare ID, or an alias. The response must echo a model string back (the router echoes the namespaced ID actually used).
39 +
40 +#### 1.1.2 `messages` (array, required)
41 +
42 +Ordered conversation. Each element is an object with a `role` and role-specific fields. Current roles (per `ChatCompletionMessageParam` in openai-python, https://github.com/openai/openai-python/tree/main/src/openai/types/chat):
43 +
44 +| role | fields | notes |
45 +|---|---|---|
46 +| `system` | `content` (string or array of `text` parts), optional `name` | Classic system prompt. |
47 +| `developer` | `content` (string or array of `text` parts), optional `name` | Introduced with o1; for OpenAI reasoning models `developer` replaces `system` ("with o1 models and newer, developer messages replace the previous system messages"). **Gateway rule: accept both; treat `developer` exactly like `system` when translating to upstreams that only know system prompts.** |
48 +| `user` | `content` (string or array of content parts), optional `name` | Content parts may be multimodal (below). |
49 +| `assistant` | `content` (string, array of `text`/`refusal` parts, or `null`), optional `name`, optional `refusal`, optional `tool_calls`, optional deprecated `function_call`, optional `audio` | "The contents of the assistant message. Required unless `tool_calls` or `function_call` is specified." So `content: null` + `tool_calls` is a legal and common history message. |
50 +| `tool` | `content` (string or array of `text` parts, required), `tool_call_id` (string, required) | "Tool call that this message is responding to." One tool message per tool call ID. |
51 +| `function` | `content`, `name` | Deprecated legacy of the pre-tools function API. Accept and map to `tool` semantics if seen. |
52 +
53 +**User content parts** (array form of `content`):
54 +
55 +- Text part: `{"type": "text", "text": "..."}`
56 +- Image part (`ChatCompletionContentPartImageParam`, https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_content_part_image_param.py):
57 +
58 +```json
59 +{
60 + "type": "image_url",
61 + "image_url": {
62 + "url": "https://example.com/cat.png",
63 + "detail": "auto"
64 + }
65 +}
66 +```
67 +
68 + - `image_url.url` (required): "Either a URL of the image or the base64 encoded image data." The base64 form is a **data URI**: `"data:image/jpeg;base64,/9j/4AAQ..."` (`data:<mime>;base64,<payload>`; supported mimes: png, jpeg, webp, non-animated gif).
69 + - `image_url.detail` (optional): `"auto"` (default) | `"low"` | `"high"` — "Specifies the detail level of the image."
70 +- Audio part: `{"type": "input_audio", "input_audio": {"data": "<base64>", "format": "wav"|"mp3"}}` (audio-capable models only; the router may reject with a clean error for providers without audio-in).
71 +- File part: `{"type": "file", "file": {"file_id": "..."} }` or `{"file_data": "<base64 data URI>", "filename": "..."}` (PDF input on OpenAI; per-provider support varies).
72 +
73 +Full request example with roles + multimodal:
74 +
75 +```json
76 +{
77 + "model": "gpt-4o",
78 + "messages": [
79 + {"role": "system", "content": "You are a terse assistant."},
80 + {"role": "user", "content": [
81 + {"type": "text", "text": "What is in this image?"},
82 + {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KG...", "detail": "high"}}
83 + ]},
84 + {"role": "assistant", "content": null, "tool_calls": [
85 + {"id": "call_abc123", "type": "function",
86 + "function": {"name": "lookup", "arguments": "{\"q\":\"cats\"}"}}
87 + ]},
88 + {"role": "tool", "tool_call_id": "call_abc123", "content": "{\"result\":\"a cat\"}"},
89 + {"role": "user", "content": "Thanks — summarize."}
90 + ]
91 +}
92 +```
93 +
94 +#### 1.1.3 Sampling & length parameters
95 +
96 +Types/defaults/deprecations verified against `CompletionCreateParams` (https://github.com/openai/openai-python/blob/main/src/openai/types/chat/completion_create_params.py) and the API reference:
97 +
98 +| param | type | default | notes |
99 +|---|---|---|---|
100 +| `temperature` | number \| null | 1 | 0–2. "Higher values like 0.8 will make the output more random." Reasoning models (o-series, gpt-5) reject non-default values — the router passes through and lets upstreams reject, or strips per-provider via CompatAdjuster. |
101 +| `top_p` | number \| null | 1 | Nucleus sampling. "We generally recommend altering this or `temperature` but not both." |
102 +| `max_completion_tokens` | integer \| null | none | **The current parameter.** "An upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens." |
103 +| `max_tokens` | integer \| null | none | **Deprecated:** "This value is now deprecated in favor of `max_completion_tokens`, and is not compatible with o-series models." Still accepted for older models. (Sources: https://community.openai.com/t/why-was-max-tokens-changed-to-max-completion-tokens/938077, https://github.com/simonw/llm/issues/724, https://github.com/vercel/ai/issues/7863 — gpt-5 rejects `max_tokens` outright.) **Gateway rule: accept BOTH; if only `max_tokens` is given, treat it as `max_completion_tokens`; if both are given, prefer `max_completion_tokens`. Translate to each upstream's native cap (e.g., Anthropic's required `max_tokens`).** |
104 +| `stop` | string \| string[] \| null | null | Up to 4 stop sequences. "Not supported with latest reasoning models `o3` and `o4-mini`." |
105 +| `n` | integer \| null | 1 | Number of choices. Most non-OpenAI upstreams only support n=1 — the router should reject `n > 1` for those with a clear 400. |
106 +| `frequency_penalty` | number \| null | 0 | −2.0 to 2.0. |
107 +| `presence_penalty` | number \| null | 0 | −2.0 to 2.0. |
108 +| `seed` | integer \| null | none | Beta. "Best effort to sample deterministically … same `seed` and parameters should return the same result." Pairs with `system_fingerprint` in the response. |
109 +| `logit_bias` | map<string,int> \| null | null | Token-ID → bias −100..100. OpenAI-specific token IDs — pass through only to OpenAI-tokenizer upstreams; strip elsewhere. |
110 +| `logprobs` | boolean \| null | false | "Whether to return log probabilities of the output tokens." Fills `choices[].logprobs`. |
111 +| `top_logprobs` | integer \| null | none | 0–20; requires `logprobs: true`. |
112 +
113 +#### 1.1.4 `response_format`
114 +
115 +Three variants (see https://developers.openai.com/api/docs/guides/structured-outputs and `shared_params/response_format_*.py` in openai-python):
116 +
117 +```json
118 +{"type": "text"}
119 +{"type": "json_object"}
120 +{
121 + "type": "json_schema",
122 + "json_schema": {
123 + "name": "weather_report",
124 + "description": "optional",
125 + "schema": {
126 + "type": "object",
127 + "properties": {"city": {"type": "string"}, "temp_c": {"type": "number"}},
128 + "required": ["city", "temp_c"],
129 + "additionalProperties": false
130 + },
131 + "strict": true
132 + }
133 +}
134 +```
135 +
136 +- `json_object` = legacy JSON mode ("an older method of generating JSON responses"); the prompt must mention JSON or OpenAI errors.
137 +- `json_schema` = Structured Outputs. `json_schema.name` is required ("Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64"); `strict: true` means "the model will always follow the exact schema defined" (subset of JSON Schema: all fields `required`, `additionalProperties: false`).
138 +- Router: translate to each provider's equivalent (Gemini `responseMimeType`/`responseSchema`, provider-specific json modes) or reject with a helpful 400 where unsupported.
139 +
140 +#### 1.1.5 Tools / function calling
141 +
142 +```json
143 +{
144 + "tools": [
145 + {
146 + "type": "function",
147 + "function": {
148 + "name": "get_weather",
149 + "description": "Get current weather for a city",
150 + "parameters": {
151 + "type": "object",
152 + "properties": {"city": {"type": "string"}},
153 + "required": ["city"],
154 + "additionalProperties": false
155 + },
156 + "strict": true
157 + }
158 + }
159 + ],
160 + "tool_choice": "auto",
161 + "parallel_tool_calls": true
162 +}
163 +```
164 +
165 +- `tools[]`: currently `type: "function"` for the public wire format (newer OpenAI additions include `custom` tools and hosted tools on the Responses API; a gateway needs only `function`). `function.parameters` is a JSON Schema object; `function.strict` optional.
166 +- `tool_choice` — union per `ChatCompletionToolChoiceOptionParam` (https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_tool_choice_option_param.py):
167 + - `"none"` — never call tools ("default when no tools are present"),
168 + - `"auto"` — model decides (default when tools present),
169 + - `"required"` — model must call at least one tool,
170 + - named function: `{"type": "function", "function": {"name": "get_weather"}}`,
171 + - (newer) allowed-tools form `{"type": "allowed_tools", ...}` — pass-through/optional for a gateway.
172 +- `parallel_tool_calls` (boolean, default true): "Whether to enable parallel function calling during tool use."
173 +- Deprecated legacy: `functions` ("Deprecated in favor of `tools`") and `function_call` ("Deprecated in favor of `tool_choice`") — accept, map to tools/tool_choice internally.
174 +
175 +#### 1.1.6 Streaming controls
176 +
177 +- `stream` (boolean \| null, default false): "If set to true, the model response data will be streamed to the client as it is generated using server-sent events."
178 +- `stream_options` (object \| null — "Only set this when you set `stream: true`"):
179 + - `include_usage` (boolean): "If set, an additional chunk will be streamed before the `data: [DONE]` message. The `usage` field on this chunk shows the token usage statistics for the entire request, and the `choices` field will always be an empty array. All other chunks will also include a `usage` field, but with a null value." (Source: https://community.openai.com/t/usage-stats-now-available-when-using-streaming-with-the-chat-completions-api-or-completions-api/738156, and the SDK type docstring.)
180 + - `include_obfuscation` (boolean, newer): adds random `obfuscation` padding fields to chunks; a local gateway should not emit it.
181 +
182 +#### 1.1.7 Identity, caching & misc parameters
183 +
184 +| param | notes |
185 +|---|---|
186 +| `user` (string) | Legacy end-user ID. "This field is being replaced by `safety_identifier` and `prompt_cache_key`." Accept it; useful as a per-key attribution hint. |
187 +| `safety_identifier` (string) | "A stable identifier used to help detect users … Maximum length of 64 characters." Pass through to OpenAI only. |
188 +| `prompt_cache_key` (string) | Cache-affinity hint, "Replaces the `user` field" for caching. Pass through to OpenAI only. |
189 +| `store` (bool), `metadata` (map, ≤16 keys) | OpenAI-side storage for distillation/evals. Pass through to OpenAI; strip elsewhere. |
190 +| `service_tier` | `"auto" \| "default" \| "flex" \| "scale" \| "priority" \| "fast"`. OpenAI-only; strip elsewhere. |
191 +| `reasoning_effort` | For reasoning models. Current values per SDK: "`none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`" (model-dependent subsets). The router should pass this through to reasoning-capable upstreams that accept it. |
192 +| `verbosity` | `"low" \| "medium" \| "high"` (gpt-5 family). Pass through to OpenAI. |
193 +| `modalities`, `audio`, `prediction`, `web_search_options` | Audio-out, predicted outputs, built-in web search — OpenAI-specific; a gateway may pass through to OpenAI and strip/400 elsewhere. |
194 +
195 +**Unknown-key policy:** OpenAI itself returns 400 `unrecognized argument` for unknown top-level keys, but a *gateway* should follow LiteLLM/OpenRouter practice: accept unknown keys and pass them through to the upstream body (this is how provider-specific extras like Perplexity `search_domain_filter` or Qwen `enable_thinking` travel). Document accepted extras in `docs/API.md`.
196 +
197 +---
198 +
199 +### 1.2 Non-streaming response — `chat.completion` object
200 +
201 +Fields verified against `ChatCompletion` (https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion.py) and `CompletionUsage` (https://github.com/openai/openai-python/blob/main/src/openai/types/completion_usage.py):
202 +
203 +```json
204 +{
205 + "id": "chatcmpl-B9MBs8CjcvOU2jLn4n570S5qMJKcT",
206 + "object": "chat.completion",
207 + "created": 1741569952,
208 + "model": "gpt-4o-2024-08-06",
209 + "system_fingerprint": "fp_50cad350e4",
210 + "service_tier": "default",
211 + "choices": [
212 + {
213 + "index": 0,
214 + "message": {
215 + "role": "assistant",
216 + "content": "Hello! How can I assist you today?",
217 + "refusal": null,
218 + "annotations": []
219 + },
220 + "logprobs": null,
221 + "finish_reason": "stop"
222 + }
223 + ],
224 + "usage": {
225 + "prompt_tokens": 19,
226 + "completion_tokens": 10,
227 + "total_tokens": 29,
228 + "prompt_tokens_details": {"cached_tokens": 0, "audio_tokens": 0},
229 + "completion_tokens_details": {
230 + "reasoning_tokens": 0,
231 + "audio_tokens": 0,
232 + "accepted_prediction_tokens": 0,
233 + "rejected_prediction_tokens": 0
234 + }
235 + }
236 +}
237 +```
238 +
239 +Field-by-field:
240 +
241 +- `id` (string): "A unique identifier for the chat completion." Convention `chatcmpl-<base62>`; the router generates its own (`chatcmpl-` prefix keeps naive clients happy).
242 +- `object`: literal `"chat.completion"`.
243 +- `created` (integer): Unix seconds.
244 +- `model` (string): "The model used" — router echoes the namespaced ID actually served (incl. after fallback).
245 +- `system_fingerprint` (string, optional, now marked deprecated in OpenAI docs): backend-config fingerprint for use with `seed`. Optional — the router may omit or emit a static value.
246 +- `service_tier` (optional): echo only for OpenAI upstreams.
247 +- `choices[]`:
248 + - `index` (integer),
249 + - `message`:
250 + - `role`: always `"assistant"`,
251 + - `content` (string \| null): null when the model only called tools,
252 + - `refusal` (string \| null): structured-outputs refusal message,
253 + - `tool_calls` (array, optional): each `{"id": "call_…", "type": "function", "function": {"name": "...", "arguments": "<JSON string>"}}` — **`arguments` is a string containing JSON, not an object**,
254 + - `annotations` (array, optional): e.g. `url_citation` items from web search,
255 + - `audio` (optional, audio-out models),
256 + - reasoning models on other providers add `reasoning_content` (DeepSeek et al.) — not an OpenAI field, but the de-facto extension the router preserves (Phase 0 decision),
257 + - `logprobs` (object \| null): `{"content": [{token, logprob, bytes, top_logprobs: […]}], "refusal": […]}` when requested,
258 + - `finish_reason` — exact literal set per the SDK: **`"stop" | "length" | "tool_calls" | "content_filter" | "function_call"`**:
259 + - `stop` — natural stop or stop sequence hit,
260 + - `length` — token cap reached (`max_completion_tokens` or context limit),
261 + - `tool_calls` — the model called tools,
262 + - `content_filter` — content omitted by a filter,
263 + - `function_call` — deprecated legacy (only when using deprecated `functions`).
264 + Every upstream stop reason must be mapped into this set (e.g., Anthropic `end_turn`→`stop`, `max_tokens`→`length`, `tool_use`→`tool_calls`, `stop_sequence`→`stop`).
265 +- `usage`:
266 + - `prompt_tokens`, `completion_tokens`, `total_tokens` (integers, total = prompt + completion),
267 + - `prompt_tokens_details` (optional): `cached_tokens` ("Cached tokens present in the prompt"), `audio_tokens`, and newer `cache_write_tokens`,
268 + - `completion_tokens_details` (optional): `reasoning_tokens` ("Tokens generated by the model for reasoning"), `audio_tokens`, `accepted_prediction_tokens`, `rejected_prediction_tokens`.
269 + - Gateway rule: use upstream-reported usage when present; when absent, estimate and flag (e.g. `"x_zyquo": {"usage_estimated": true}` — extension keys are tolerated by both SDKs).
270 +
271 +Tool-call response example (non-streaming):
272 +
273 +```json
274 +{
275 + "id": "chatcmpl-abc123",
276 + "object": "chat.completion",
277 + "created": 1699896916,
278 + "model": "gpt-4o-mini",
279 + "choices": [
280 + {
281 + "index": 0,
282 + "message": {
283 + "role": "assistant",
284 + "content": null,
285 + "tool_calls": [
286 + {
287 + "id": "call_abc123",
288 + "type": "function",
289 + "function": {"name": "get_weather", "arguments": "{\n\"city\": \"Boston\"\n}"}
290 + }
291 + ]
292 + },
293 + "logprobs": null,
294 + "finish_reason": "tool_calls"
295 + }
296 + ],
297 + "usage": {"prompt_tokens": 82, "completion_tokens": 17, "total_tokens": 99}
298 +}
299 +```
300 +
301 +---
302 +
303 +### 1.3 The SSE streaming format — byte-level contract
304 +
305 +Sources: streaming reference (https://developers.openai.com/api/reference/resources/chat/subresources/completions/streaming-events, https://platform.openai.com/docs/api-reference/chat-streaming/streaming), the cookbook (https://cookbook.openai.com/examples/how_to_stream_completions), and `ChatCompletionChunk` SDK types.
306 +
307 +**Transport.** Response headers: `Content-Type: text/event-stream; charset=utf-8`, `Cache-Control: no-cache`, chunked transfer (no `Content-Length`), keep connection open. Each event is exactly:
308 +
309 +```
310 +data: <one-line JSON>\n
311 +\n
312 +```
313 +
314 +i.e. the 6 bytes `data: `, the JSON serialized **without newlines**, then `\n\n`. OpenAI emits only `data:` lines — no `event:`, `id:`, or `retry:` fields, no SSE comments. The stream terminates with the sentinel:
315 +
316 +```
317 +data: [DONE]\n
318 +\n
319 +```
320 +
321 +(`[DONE]` is not JSON; both official SDKs special-case this exact string.) An HTTP-level error that occurs *before* streaming starts is a plain JSON error body with a proper status code; the status is sent before any chunk, so a request that fails validation must NOT return 200 + SSE.
322 +
323 +**Chunk object** (`object: "chat.completion.chunk"`): same `id` ("Each chunk has the same ID"), `created`, and `model` across all chunks of one completion; `choices[]` with `{index, delta, logprobs, finish_reason}`; `usage` null/absent except the final usage chunk. Delta fields: `role`, `content`, `refusal`, `tool_calls[]` (each with `index`, optional `id`, optional `type: "function"`, optional `function.name`, optional `function.arguments` fragment), deprecated `function_call`.
324 +
325 +Chunk sequence rules:
326 +
327 +1. **First chunk** carries the role delta: `"delta": {"role": "assistant", "content": ""}` (OpenAI includes the empty `content` string; emit it — some clients concatenate blindly). May also carry `refusal: null` — harmless.
328 +2. **Content chunks**: `"delta": {"content": "<fragment>"}`, `finish_reason: null`.
329 +3. **Final content chunk**: `"delta": {}` (empty object) with `"finish_reason": "stop"` (or `length`/`tool_calls`/`content_filter`). The finish_reason travels on a chunk whose delta is empty — not alongside content.
330 +4. **Optional usage chunk** (only when `stream_options.include_usage` is true): `"choices": []` (empty array — quoted from the SDK: choices "can also be empty for the last chunk if you set `stream_options: {\"include_usage\": true}`") and a populated `usage` object. "All other chunks will also include a `usage` field, but with a null value" when include_usage is set.
331 +5. `data: [DONE]`.
332 +
333 +#### (a) Plain-text transcript (`stream: true`, `stream_options: {"include_usage": true}`)
334 +
335 +```
336 +data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"usage":null}
337 +
338 +data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"content":"Hello"},"logprobs":null,"finish_reason":null}],"usage":null}
339 +
340 +data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"content":" there"},"logprobs":null,"finish_reason":null}],"usage":null}
341 +
342 +data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"content":"!"},"logprobs":null,"finish_reason":null}],"usage":null}
343 +
344 +data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null}
345 +
346 +data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[],"usage":{"prompt_tokens":19,"completion_tokens":3,"total_tokens":22,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}}}
347 +
348 +data: [DONE]
349 +
350 +```
351 +
352 +(Without `include_usage`, the usage chunk is absent and no `usage` key appears on chunks.)
353 +
354 +#### (b) Streamed tool call transcript
355 +
356 +Tool-call arguments stream as string fragments. **The first tool_call delta for a given `index` carries `id`, `type`, and `function.name` (with `"arguments":""`); every subsequent delta for that index carries ONLY `index` and `function.arguments` fragments — no id, no name.** Parallel tool calls interleave via `index` 0,1,…; clients accumulate by index and concatenate `arguments`.
357 +
358 +```
359 +data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"role":"assistant","content":null},"logprobs":null,"finish_reason":null}]}
360 +
361 +data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_KSEnFnucOtZNQqEZF9wvfIWm","type":"function","function":{"name":"get_weather","arguments":""}}]},"logprobs":null,"finish_reason":null}]}
362 +
363 +data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"ci"}}]},"logprobs":null,"finish_reason":null}]}
364 +
365 +data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ty\": \"Bos"}}]},"logprobs":null,"finish_reason":null}]}
366 +
367 +data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ton\"}"}}]},"logprobs":null,"finish_reason":null}]}
368 +
369 +data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"tool_calls"}]}
370 +
371 +data: [DONE]
372 +
373 +```
374 +
375 +#### What strict SDK clients require
376 +
377 +- **openai-python** parses SSE by splitting on blank lines, reads only `data:` payloads, stops at the exact string `[DONE]`, and constructs a pydantic `ChatCompletionChunk`. Pydantic will **fail the whole stream** if `id`, `object`, `created`, `model`, or `choices` is missing/wrong-typed, if `object` isn't exactly `"chat.completion.chunk"`, or if `finish_reason` is a value outside the literal set. Extra unknown fields are tolerated (kept as extras) — so `reasoning_content` in deltas and `x_zyquo` extensions are safe. (See https://github.com/openai/openai-python; a real-world failure mode of non-conforming servers: https://github.com/janhq/jan/issues/8280 — "the chat.completion.chunk schema … requires choices[]; type validation failed" halts the stream.)
378 +- **openai-node** is looser at runtime (TypeScript types are compile-time), but its stream accumulator (`stream.finalChatCompletion()`, and the Vercel AI SDK on top of it) indexes `choices[0].delta`, accumulates `tool_calls` strictly by `index`, and expects `id`/`function.name` on the first delta of each tool call; missing `index` or re-sending `name` fragments corrupts accumulation (see https://ai-sdk.dev/providers/openai-compatible-providers on buffering unreliable tool-call deltas).
379 +- Both SDKs ignore SSE `event:` fields **only** if the line isn't a `data:` line; do not emit named events or comments — emit *only* `data:` lines exactly as above.
380 +- Keep every JSON chunk on a single line; UTF-8; never split a multibyte character across chunks inside one JSON string (JSON-escape or buffer to codepoint boundaries).
381 +
382 +---
383 +
384 +### 1.4 `GET /v1/models` and `GET /v1/models/{id}`
385 +
386 +Source: https://developers.openai.com/api/reference/resources/models (mirrors https://platform.openai.com/docs/api-reference/models).
387 +
388 +`GET /v1/models` →
389 +
390 +```json
391 +{
392 + "object": "list",
393 + "data": [
394 + {"id": "gpt-4o", "object": "model", "created": 1686935002, "owned_by": "openai"},
395 + {"id": "gpt-4o-mini", "object": "model", "created": 1686935002, "owned_by": "openai"}
396 + ]
397 +}
398 +```
399 +
400 +Model object: `id` (string — "The model identifier, which can be referenced in the API endpoints"), `object` (always `"model"`), `created` (Unix seconds), `owned_by` (string — "The organization that owns the model").
401 +
402 +`GET /v1/models/{model}` → a single model object:
403 +
404 +```json
405 +{"id": "gpt-4o", "object": "model", "created": 1686935002, "owned_by": "openai"}
406 +```
407 +
408 +Unknown ID → 404 with the model-not-found error (§1.5). Router mapping: `id` = namespaced `provider/model-id` (aliases listed too), `owned_by` = provider name; enriched metadata (context window, pricing, capabilities) goes under an `x_zyquo` extension key on each entry — both SDKs tolerate extra fields.
409 +
410 +---
411 +
412 +### 1.5 Error response format
413 +
414 +Every error is JSON with a single `error` object (see https://developers.openai.com/api/docs/guides/error-codes and https://community.openai.com/t/openai-chat-list-of-error-codes-and-types/357791):
415 +
416 +```json
417 +{
418 + "error": {
419 + "message": "<human-readable description>",
420 + "type": "<error family>",
421 + "param": "<offending request parameter or null>",
422 + "code": "<machine-readable code or null>"
423 + }
424 +}
425 +```
426 +
427 +All four keys are always present (`param`/`code` may be `null`). The official SDKs map HTTP status → typed exceptions (`BadRequestError` 400, `AuthenticationError` 401, `PermissionDeniedError` 403, `NotFoundError` 404, `UnprocessableEntityError` 422, `RateLimitError` 429, `InternalServerError` ≥500, per https://github.com/openai/openai-python#handling-errors).
428 +
429 +| HTTP | `type` | typical `code` values | when |
430 +|---|---|---|---|
431 +| 400 | `invalid_request_error` | `null`, `invalid_value`, `unsupported_parameter`, `context_length_exceeded`, `string_above_max_length`, `invalid_image_format` | Malformed body, bad param, context overflow |
432 +| 401 | `invalid_request_error` / `authentication_error` | `invalid_api_key`, `no_organization` | Missing/invalid API key ("Incorrect API key provided: …") |
433 +| 403 | `permission_error` / `invalid_request_error` | `unsupported_country_region_territory`, `insufficient_permissions` | Key valid but not allowed (region, scoped key) |
434 +| 404 | `invalid_request_error` | `model_not_found` | Unknown model / resource |
435 +| 422 | `invalid_request_error` | — | Semantically invalid (rare) |
436 +| 429 | `rate_limit_error` | `rate_limit_exceeded` | "Rate limit reached for …" — retriable; honor/emit `Retry-After` |
437 +| 429 | `insufficient_quota` | `insufficient_quota` | "You exceeded your current quota, plans & billing…" — NOT retriable (note: type AND code are both `insufficient_quota`) |
438 +| 500 | `server_error` | `null` | "The server had an error while processing your request" |
439 +| 502 | (gateway) `server_error` / `api_error` | `bad_gateway` | Used by gateways (OpenRouter/LiteLLM) for upstream failure — the router's choice for "upstream returned garbage / is down" |
440 +| 503 | `server_error` / `service_unavailable` | `service_unavailable`, `slow_down` | "The engine is currently overloaded, please try again later" |
441 +| 504 | (gateway) `timeout_error` | `timeout` | Gateway convention for upstream timeout |
442 +
443 +**The exact model-not-found error** (HTTP 404) as OpenAI returns it (sources: https://community.openai.com/t/openai-error-invalidrequesterror-the-model-gpt-4-does-not-exist-or-you-do-not-have-access-to-it/376230, https://community.openai.com/t/api-returning-404-model-not-found-all-of-a-sudden-why-and-how-to-fix/679777):
444 +
445 +```json
446 +{
447 + "error": {
448 + "message": "The model `gpt-5-nonexistent` does not exist or you do not have access to it.",
449 + "type": "invalid_request_error",
450 + "param": null,
451 + "code": "model_not_found"
452 + }
453 +}
454 +```
455 +
456 +The router must reproduce this shape verbatim (with its namespaced ID in backticks) for unknown/disabled models — SDK error-handling paths and agent frameworks string-match parts of it.
457 +
458 +**Streaming errors:** if the failure happens before any chunk, return the JSON error with the real status code (no SSE). If the upstream dies mid-stream, OpenAI's own behavior is to emit an error payload as a `data:` line (`{"error": {...}}`) and close without `[DONE]`; a gateway should do the same — strict clients surface it as a stream error rather than hanging.
459 +
460 +**Gateway error-mapping rules (Phase 3 contract):** upstream 401 (provider key bad) → 401 with "provider key for `<provider>` was rejected" (never echo the key); upstream 429 → 429 + `Retry-After` when given; upstream timeout → 504 `timeout`; upstream 5xx/unparseable → 502 `bad_gateway` with sanitized detail; never leak raw provider error shapes or key material.
461 +
462 +---
463 +
464 +### 1.6 `POST /v1/responses` — evaluate and decide
465 +
466 +Summary (sources: https://platform.openai.com/docs/guides/migrate-to-responses, https://developers.openai.com/api/docs/guides/migrate-to-responses):
467 +
468 +- **Shape:** flatter request — `input` (string or item array) + top-level `instructions` instead of a `messages` array; response is a typed `output` array of *items* (`message`, `reasoning` with `encrypted_content`, `function_call`, `function_call_output`, hosted-tool items) plus an `output_text` convenience; server-side state via `store: true` / `previous_response_id`; built-in hosted tools (web search, file search, code interpreter, computer use).
469 +- **Streaming:** *semantic events*, not chunk deltas — named SSE events like `response.created`, `response.output_item.added`, `response.output_text.delta`, `response.completed` — a completely different event model from `chat.completion.chunk`.
470 +- **Adoption (as of mid-2026):** OpenAI recommends Responses "for all new projects" and newest OpenAI-native features (encrypted reasoning, hosted tools) land there first, but they state Chat Completions "remains supported" indefinitely as the industry standard. Critically for a *multi-provider gateway*: the entire compatible-provider ecosystem (DeepSeek, Qwen, Mistral, xAI, Together, Cerebras, Ollama, LM Studio, vLLM…) standardized on **chat/completions**, and gateways (LiteLLM, OpenRouter) still treat it as the lingua franca (OpenRouter exposes chat/completions; LiteLLM added a Responses *bridge* that internally converts to chat/completions).
471 +
472 +**DECISION for Zyquo Router: do NOT expose `/v1/responses` in v1.** Rationale:
473 +
474 +1. Zero translation leverage — none of our 12 upstream providers speak Responses natively except OpenAI itself; we'd be building a second full bidirectional translation layer (items + semantic streaming events) purely as a front-end conversion to the same canonical internal request.
475 +2. Statefulness (`store`, `previous_response_id`, `encrypted_content`) implies server-side conversation storage — out of scope and against the router's privacy posture.
476 +3. Client compatibility target is met without it: every OpenAI-SDK-based tool can (and with third-party base URLs, typically does) use `chat.completions`.
477 +4. Cheap to add later as a stateless subset (`input`/`instructions` → messages; emit `response.output_text.delta` events) once the chat/completions core is green — track as a post-v1 enhancement in `docs/PLAN.md`. `/health` should not advertise it; requests to `/v1/responses` return a 404 OpenAI-format error with a message pointing to `/v1/chat/completions`.
478 +
479 +---
480 +
481 +### 1.7 `POST /v1/embeddings` (optional gateway endpoint)
482 +
483 +Source: https://platform.openai.com/docs/api-reference/embeddings and `embedding_create_params.py` / `create_embedding_response.py` in openai-python.
484 +
485 +Request:
486 +
487 +```json
488 +{
489 + "model": "text-embedding-3-small",
490 + "input": "The food was delicious and the waiter...",
491 + "encoding_format": "float",
492 + "dimensions": 512,
493 + "user": "optional-end-user-id"
494 +}
495 +```
496 +
497 +- `input` (required): "string or array of tokens. To embed multiple inputs in a single request, pass an array of strings or array of token arrays." (Max ~2048 inputs per request; each within the model's token limit.)
498 +- `model` (required); `encoding_format`: `"float"` (default) or `"base64"`; `dimensions`: "Only supported in `text-embedding-3` and later models"; `user`: abuse-monitoring ID.
499 +
500 +Response:
501 +
502 +```json
503 +{
504 + "object": "list",
505 + "data": [
506 + {
507 + "object": "embedding",
508 + "index": 0,
509 + "embedding": [0.0023064255, -0.009327292, -0.0028842222]
510 + }
511 + ],
512 + "model": "text-embedding-3-small",
513 + "usage": {"prompt_tokens": 8, "total_tokens": 8}
514 +}
515 +```
516 +
517 +Notes for the router: `data[]` is ordered by `index` matching the input array; `usage` has only `prompt_tokens` + `total_tokens` (no completion tokens); `encoding_format: "base64"` returns each embedding as a base64 string of little-endian float32 — support it, since openai-python requests base64 by default when numpy is available. Route only to providers that offer embeddings; others 404 with `model_not_found`.
518 +
519 +---
520 +
521 +### 1.8 Contract checklist for Zyquo Router (derived from this section)
522 +
523 +- [ ] Accept full request schema §1.1 incl. both `max_tokens` and `max_completion_tokens`, `developer` role, multimodal parts, all three `response_format` variants, all `tool_choice` forms, deprecated `functions`/`function_call`.
524 +- [ ] Emit spec-exact `chat.completion` (§1.2) with mapped `finish_reason` from the closed literal set and real-or-flagged `usage` incl. details sub-objects when upstreams provide them.
525 +- [ ] Emit byte-exact SSE (§1.3): role-first chunk, single-line JSON `data:` events, empty-delta finish chunk, empty-choices usage chunk only under `include_usage`, terminating `data: [DONE]`, correct tool-call delta id/name/index rules.
526 +- [ ] `GET /v1/models` + `/v1/models/{id}` per §1.4 with namespaced IDs.
527 +- [ ] All errors per §1.5 incl. verbatim model-not-found shape and gateway 502/504 conventions.
528 +- [ ] `/v1/responses`: not exposed in v1 (documented 404 with pointer); `/v1/embeddings`: optional, per §1.7.
529 +## 2. How existing gateways do it
530 +
531 +Research date: 2026-07-30. Sources fetched live from docs.litellm.ai, openrouter.ai/docs, docs.ollama.com, lmstudio.ai/docs, docs.vllm.ai, and GitHub issue trackers. This section extracts the concrete, battle-tested patterns from the reference gateways that Zyquo Router should copy — and the compatibility landmines it must avoid.
532 +
533 +---
534 +
535 +### 2.1 LiteLLM proxy — the reference for config, translation, and error mapping
536 +
537 +LiteLLM is the most complete open-source implementation of exactly what Zyquo Router is: an OpenAI-compatible front for ~100 providers. Its patterns are the closest prior art.
538 +
539 +#### 2.1.1 Model naming: `provider/model`
540 +
541 +LiteLLM routes on a **provider prefix in the model string**: `openai/gpt-4o`, `azure/gpt-4o`, `anthropic/claude-...`, `bedrock/anthropic.claude-instant-v1`, `ollama/mistral`, `gemini/gemini-2.5-pro`. The prefix selects the client implementation; the remainder is the upstream model ID sent to the provider. (Source: https://docs.litellm.ai/docs/proxy/configs)
542 +
543 +The proxy adds a second layer of indirection: a **user-facing `model_name` alias** mapped to one or more concrete deployments in `config.yaml`:
544 +
545 +```yaml
546 +model_list:
547 + - model_name: gpt-4o # what clients send in "model"
548 + litellm_params:
549 + model: azure/gpt-4o-eu # provider/upstream-id actually called
550 + api_base: https://endpoint-europe.openai.azure.com/
551 + api_key: "os.environ/AZURE_API_KEY_EU" # env-var indirection, secrets never in config
552 + rpm: 6 # per-deployment rate limit
553 + model_info: # optional metadata (pricing/context overrides)
554 + max_input_tokens: 128000
555 +
556 +litellm_settings:
557 + drop_params: true
558 + num_retries: 3
559 + request_timeout: 10
560 + fallbacks: [{"gpt-4o": ["claude-sonnet"]}]
561 +
562 +router_settings:
563 + routing_strategy: simple-shuffle
564 + model_group_alias: {"gpt-4": "gpt-4o"} # request-time alias remapping
565 +
566 +general_settings:
567 + master_key: sk-1234 # local bearer key gating the proxy
568 +```
569 +
570 +Key ideas to steal (https://docs.litellm.ai/docs/proxy/configs):
571 +
572 +- **Two-level naming**: public alias (`model_name`) → concrete `provider/model` deployment. Zyquo Router's aliases (`fast` → `cerebras/...`) are exactly this.
573 +- Multiple entries with the same `model_name` = load balancing group (Zyquo Router doesn't need multi-deployment balancing, but the alias→target indirection is the same shape).
574 +- `model_group_alias` maps well-known client names (`gpt-4`) onto configured groups — useful for tools that hardcode OpenAI model names.
575 +- Wildcards (`model_name: "*"`, `model: openai/*`) allow pass-through of any model given credentials — Zyquo Router's "accept unambiguous bare IDs" is a constrained version of this.
576 +- `os.environ/VAR` indirection keeps keys out of the config file (Zyquo Router: keys live only in the vault; config export never includes them).
577 +
578 +#### 2.1.2 Param translation and drop tables
579 +
580 +LiteLLM maintains, per provider, a **mapping table of which OpenAI params the provider supports** (queryable via `litellm.get_supported_openai_params(model)`), and translates names where they differ. Handling of *unsupported* params is explicit policy, not accident (https://docs.litellm.ai/docs/completion/drop_params):
581 +
582 +- **Default: raise an exception** if a param is sent to a model that doesn't support it — loud failure over silent behavior change.
583 +- **`drop_params: true`** (global, per-deployment, or per-request): silently strip unsupported params instead of erroring. Most proxies run with this on.
584 +- **`additional_drop_params: ["response_format"]`** — per-deployment list of specific params to strip even if nominally supported; supports JSONPath-ish nested syntax (`tools[*].input_examples`, `parent.child`, `array[0]`).
585 +- **`allowed_openai_params: ["tools"]`** — the inverse escape hatch: force-forward a param LiteLLM believes is unsupported (settable in config or per-request via `extra_body`).
586 +- **Provider-specific extras** ride through `extra_body` on the OpenAI SDK and are passed to the upstream unchanged.
587 +
588 +Lesson for Zyquo Router's `CompatAdjuster`: implement a **per-provider param table** (supported / rename / strip / pass-through-extras) as data, not scattered `if`s, and make the strip-vs-error policy explicit and configurable.
589 +
590 +#### 2.1.3 Error mapping to OpenAI exceptions
591 +
592 +LiteLLM maps every upstream failure onto **exception types that inherit from the OpenAI SDK's own exceptions**, so client code catching `openai.RateLimitError` works against any provider (https://docs.litellm.ai/docs/exception_mapping). Status-code taxonomy:
593 +
594 +| Status | Exceptions |
595 +|---|---|
596 +| 400 | `BadRequestError`, `UnsupportedParamsError`, `ContextWindowExceededError`, `ContentPolicyViolationError` |
597 +| 401 | `AuthenticationError` |
598 +| 403 | `PermissionDeniedError` |
599 +| 404 | `NotFoundError` |
600 +| 408 | `Timeout` |
601 +| 422 | `UnprocessableEntityError` |
602 +| 429 | `RateLimitError` |
603 +| 500 | `APIConnectionError`, `APIError` |
604 +| 503 | `ServiceUnavailableError` |
605 +| ≥500 | `InternalServerError` |
606 +
607 +Every mapped exception carries `status_code`, `message`, and **`llm_provider`** (which upstream failed) — Zyquo Router should likewise name the provider in error messages ("Anthropic key invalid") without leaking payloads. Note that `ContextWindowExceededError` and `ContentPolicyViolationError` are *distinguished subtypes of 400*: this is what makes context-window fallbacks and content-policy fallbacks possible. A `_should_retry(status_code)` helper centralizes the retryability decision (429, 5xx, timeouts → retry; 4xx auth/validation → don't).
608 +
609 +#### 2.1.4 Retries, fallbacks, cooldowns
610 +
611 +(Sources: https://docs.litellm.ai/docs/routing, https://docs.litellm.ai/docs/proxy/reliability)
612 +
613 +- **`num_retries: 3`** with exponential backoff for `RateLimitError`, immediate retry for transient errors; `retry_after` sets a minimum wait. A `RetryPolicy` can set retry counts **per exception class** (e.g., `AuthenticationErrorRetries=0`, `RateLimitErrorRetries=3`, `TimeoutErrorRetries=2`) — never retry auth failures.
614 +- **Three fallback kinds**, configured as ordered maps `{"primary": ["fallback1", "fallback2"]}`:
615 + - `fallbacks` — general retryable errors (429/5xx) after retries exhaust;
616 + - `context_window_fallbacks` — prompt too big → reroute to a bigger-context model;
617 + - `content_policy_fallbacks` — content filter tripped → reroute to a laxer model;
618 + - `default_fallbacks: ["claude-opus"]` — catch-all.
619 +- **Per-request fallbacks** via a `fallbacks: [...]` array in the request body, and `disable_fallbacks: true` to opt out per request.
620 +- **Cooldowns**: `allowed_fails: 3` failures per minute puts a deployment on a `cooldown_time: 30`s bench so the router stops hammering a failing upstream. For a single-deployment-per-model local router this maps to "mark provider degraded, surface in dashboard, fail fast or fall back."
621 +- Execution order: retries on the primary first, then fallbacks in order until success or exhaustion. The actually-used deployment is reported via an `x-litellm-model-id` response header — Zyquo Router should report the actually-used model in the response `model` field (OpenRouter's approach, §2.2.4) and/or a header.
622 +
623 +#### 2.1.5 Usage & cost tracking
624 +
625 +(Source: https://docs.litellm.ai/docs/completion/token_usage)
626 +
627 +- Pricing lives in one community-maintained JSON file, `model_prices_and_context_window.json` (https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) — the de-facto industry pricing database. Entry shape:
628 +
629 +```json
630 +{
631 + "gpt-4o": {
632 + "max_tokens": 4000,
633 + "input_cost_per_token": 1.5e-06,
634 + "output_cost_per_token": 2e-06,
635 + "litellm_provider": "openai",
636 + "mode": "chat"
637 + }
638 +}
639 +```
640 +
641 +- `completion_cost(response)` computes USD from the usage in a response + this table; `cost_per_token(model, prompt_tokens, completion_tokens)` is the primitive. Every response carries `response_cost` in hidden params.
642 +- `register_model({...})` lets users override/add pricing — Zyquo Router's Settings ▸ Usage & Pricing "pricing override" is the same feature.
643 +- When upstream doesn't return usage (some streams), tokens are **estimated with a tokenizer** (tiktoken default, provider-specific where known) — and Zyquo Router should flag estimated usage as such.
644 +
645 +Pattern: **pricing is data keyed by model ID, cost computed at response time from `usage`, estimation as fallback**. Zyquo Router already has per-model pricing in the Zyquo Cloud catalog; reuse it as the single pricing source.
646 +
647 +#### 2.1.6 Streaming normalization
648 +
649 +LiteLLM wraps every provider stream in a `CustomStreamWrapper` that re-emits **uniform OpenAI-shaped chunk objects** (`choices[0].delta.content`, etc.) regardless of upstream wire format, plus a `stream_chunk_builder(chunks)` helper that reassembles a full `chat.completion` from chunks (useful for logging/cost of streamed requests — Zyquo Router's request log needs exactly this) (https://docs.litellm.ai/docs/completion/stream). It also guards against pathological streams: `REPEATED_STREAMING_CHUNK_LIMIT = 100` aborts with an `InternalServerError` if the same chunk repeats endlessly.
650 +
651 +Cautionary tales from LiteLLM's own tracker — even the reference implementation gets chunk shape wrong:
652 +
653 +- Its synthetic usage chunk violated the OpenAI spec by carrying a **non-empty `choices`** array (spec: the `include_usage` final chunk has `"choices": []`) — https://github.com/BerriAI/litellm/issues/28735
654 +- It **lost vLLM's usage** because vLLM sends usage in a separate empty-choices chunk *after* the `finish_reason` chunk and LiteLLM stopped reading at `finish_reason` — https://github.com/BerriAI/litellm/issues/25389 . Lesson: when consuming upstreams, read until the stream actually ends, not until `finish_reason`.
655 +- Grok returned usage in the wrong chunk (extra empty final chunk) — https://github.com/BerriAI/litellm/issues/17136 ; and some providers reject `stream_options` as an unknown param, so the gateway must know per provider whether it can request usage-in-stream — https://github.com/BerriAI/litellm/issues/23847
656 +
657 +---
658 +
659 +### 2.2 OpenRouter — the reference for unified IDs, streaming discipline, and honest accounting
660 +
661 +#### 2.2.1 Unified model IDs and variants
662 +
663 +- IDs are **`vendor/model-name`** (`anthropic/claude-3.5-sonnet`, `openai/gpt-4o`), plus a permanent `canonical_slug` that survives renames. (https://openrouter.ai/docs/models)
664 +- **Variant suffixes** append behavior to a slug: `:free` (free tier), `:thinking` (reasoning mode), `:nitro` (= `provider.sort: "throughput"`), `:floor` (sort by price). A suffix-on-the-ID is a very ergonomic way to encode routing preferences without extra params (https://openrouter.ai/docs/models, https://openrouter.ai/docs/features/provider-routing). Zyquo Router could reserve this pattern for future use (e.g., `model:nostore` to skip logging).
665 +- `GET /api/v1/models` returns rich metadata per model: `id`, `canonical_slug`, `name`, `context_length`, `architecture` (input/output modalities, tokenizer), `pricing` (**USD per token as strings** — `"0"` means free; string avoids float precision issues), `supported_parameters` (array of OpenAI params this model accepts — clients can pre-check!), `top_provider`. Zyquo Router's plan to enrich `/v1/models` under an `x-zyquo` extension key mirrors this; adopting `context_length`, `pricing`, and `supported_parameters` fields is directly useful for the Playground and Docs UI.
666 +
667 +#### 2.2.2 Provider-specific params & headers
668 +
669 +- Default posture: **unsupported params are silently ignored by the receiving provider**; setting `"provider": {"require_parameters": true}` restricts routing to providers that support *every* param in the request (https://openrouter.ai/docs/features/provider-routing). For a single-upstream-per-model router, the equivalent decision is strip-vs-reject per param (see §2.5).
670 +- Extra attribution headers `HTTP-Referer` and `X-Title` are optional and additive — the API remains pure OpenAI otherwise (https://openrouter.ai/docs/api-reference/overview).
671 +- Provider-specific features arrive as **extra top-level body keys** (e.g., `models`, `provider`, `plugins`, `transforms`) that OpenAI SDKs send via `extra_body` — the standard pass-through idiom Zyquo Router should adopt for things like Perplexity search options.
672 +
673 +#### 2.2.3 Streaming normalization — the details that matter
674 +
675 +(Source: https://openrouter.ai/docs/api-reference/streaming)
676 +
677 +- **Keep-alive SSE comments**: OpenRouter periodically emits `: OPENROUTER PROCESSING` comment lines to hold connections open during long prefill/queue waits. Per the SSE spec, lines starting with `:` are comments and must be ignored — but naive client loops that `JSON.parse` every line crash on them. Zyquo Router should (a) emit its own `: keep-alive` comments during long upstream silences (Anthropic thinking, queued requests), and (b) tolerate/strip comments when *consuming* upstream SSE.
678 +- Termination is always `data: [DONE]`.
679 +- **Mid-stream errors**: once tokens have flowed you can't change the HTTP status, so errors arrive as a final SSE event that is still a valid `chat.completion.chunk` with an added `error` object and `finish_reason: "error"`:
680 +
681 +```
682 +data: {"id":"gen-abc123","object":"chat.completion.chunk","created":1730000000,
683 + "model":"...","error":{"code":429,"message":"Rate limit exceeded",
684 + "metadata":{"error_type":"rate_limit_exceeded"}},
685 + "choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}]}
686 +```
687 +
688 + (https://openrouter.ai/docs/api-reference/errors) This is the pattern Zyquo Router needs for "upstream died mid-stream": emit a well-formed error chunk, then `[DONE]`, then close — never a bare connection reset.
689 +- **Cancellation**: aborting the connection cancels the upstream *only for streaming requests on providers that support cancellation*; otherwise the upstream finishes and bills anyway. Zyquo Router must cancel the upstream `URLSession`/NIO task on client disconnect (its clients are direct HTTP calls, so cancellation is always possible) — this is Phase 7 test 3.
690 +- Usage stats ride in the **final chunk** (`chunk.usage`).
691 +
692 +#### 2.2.4 Usage accounting
693 +
694 +(Source: https://openrouter.ai/docs/use-cases/usage-accounting)
695 +
696 +- Usage (with **cost**) is now included in every response automatically (the old `usage: {include: true}` opt-in is deprecated). Shape extends OpenAI's `usage`:
697 +
698 +```json
699 +"usage": {
700 + "prompt_tokens": 194,
701 + "completion_tokens": 2,
702 + "total_tokens": 196,
703 + "cost": 0.95,
704 + "cost_details": {"upstream_inference_cost": 19},
705 + "prompt_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 100},
706 + "completion_tokens_details": {"reasoning_tokens": 0}
707 +}
708 +```
709 +
710 + Putting `cost` inside `usage` is a compatible extension (SDKs ignore unknown fields on responses) — a good idea for Zyquo Router's estimated-cost surfacing.
711 +- For streams, usage appears **in the last SSE message**; there's also a post-hoc `GET /api/v1/generation?id=...` stats endpoint keyed by the response `id` for auditing. Zyquo Router's request-log detail view is the local equivalent.
712 +
713 +#### 2.2.5 Error format
714 +
715 +(Source: https://openrouter.ai/docs/api-reference/errors)
716 +
717 +- Error shape: `{"error": {"code": <number>, "message": "...", "metadata": {...}}}` with HTTP status = `error.code` for pre-stream failures. Status vocabulary: **400** bad params/CORS, **401** bad key, **402** out of credits, **403** moderation/guardrail, **408** timeout, **429** rate limited, **502** "your chosen model is down or we received an invalid response", **503** "no available provider meets your routing requirements".
718 +- `metadata` carries structured context without leaking raw payloads: moderation errors include `reasons`, a truncated `flagged_input` (max 100 chars), `provider_name`, `model_slug`; provider errors include a canonical `error_type` plus the original `provider_code`. This "canonical code + original provider code" pair is exactly the honest-but-normalized surfacing Zyquo Router wants.
719 +- **429/503 responses include a standard `Retry-After` header** — Zyquo Router should propagate upstream `Retry-After` to its own clients.
720 +- Note: OpenRouter uses a numeric `error.code`; the strict OpenAI format is `{"error": {"message", "type", "param", "code"}}` with string-ish `code`. Zyquo Router should keep the **OpenAI field set** (per §1) and put router-specific context in the message and/or an extension key, since OpenAI SDKs construct exceptions from `type`/`code`.
721 +
722 +#### 2.2.6 Fallbacks & routing preferences
723 +
724 +(Sources: https://openrouter.ai/docs/guides/routing/model-fallbacks, https://openrouter.ai/docs/features/provider-routing)
725 +
726 +- **Model fallbacks**: an extra `models: ["primary", "fallback1", ...]` array in the body (the `model` field is the first attempt; via OpenAI SDK it goes in `extra_body`). Fallback triggers on *any* error: context-length validation, moderation flags, rate limits, downtime. **The response's `model` field always reports the model actually used, and pricing follows the actually-used model.** This "honest `model` echo" is the contract Zyquo Router's CLAUDE.md already mandates for its fallback chains.
727 +- **Provider preferences** (`provider` object): `order` (try providers in this order), `allow_fallbacks` (default true), `require_parameters`, `ignore`/`only` allow/deny lists, `sort` by price/throughput/latency, `max_price`. Default load balancing prefers providers without recent outages, weighted by **inverse square of price**. Mostly N/A for a local single-key-per-provider router, but the *cooldown-on-recent-outage* idea maps to LiteLLM cooldowns.
728 +- `finish_reason` is **normalized to exactly five values** — `stop`, `length`, `tool_calls`, `content_filter`, `error` — with the raw upstream value preserved in a separate `native_finish_reason` field (https://openrouter.ai/docs/api-reference/overview). Recommended verbatim for Zyquo Router (OpenAI's own set is the first four plus `function_call` legacy; `error` only ever appears mid-stream).
729 +
730 +---
731 +
732 +### 2.3 Local OpenAI-compatible servers — what "compatible enough" looks like
733 +
734 +These show which subset of the spec real clients actually depend on, and which deviations are tolerated.
735 +
736 +#### 2.3.1 Ollama (`http://localhost:11434/v1`)
737 +
738 +(Source: https://docs.ollama.com/api/openai-compatibility)
739 +
740 +- Endpoints: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/v1/models/{model}`, `/v1/embeddings`, `/v1/responses`.
741 +- Supported on chat: `model`, `messages`, `temperature`, `top_p`, `max_tokens`, `frequency_penalty`, `presence_penalty`, `seed`, `stop`, `stream`, `stream_options.include_usage`, `response_format` (JSON mode), `tools`, `reasoning_effort`/`reasoning`, vision via **base64 images only**.
742 +- **Not supported**: `logprobs`, `user`, `n`, `tool_choice`, `logit_bias`, image **URLs**. Unsupported params are ignored rather than erroring.
743 +- Auth: any/no API key accepted (pure localhost trust). Zyquo Router improves on this with optional local keys.
744 +- Notable: Ollama had to add `max_completion_tokens` support because OpenAI deprecated `max_tokens` (https://github.com/ollama/ollama/issues/7125) — see §2.4.5.
745 +
746 +Takeaways: even a hugely popular compat layer omits `n`, `logprobs`, `logit_bias` and (long) omitted `tool_choice` — clients broadly tolerate missing niche params, but `tools`, `response_format`, `stream_options.include_usage`, and vision are table stakes in 2026.
747 +
748 +#### 2.3.2 LM Studio (`http://localhost:1234/v1`)
749 +
750 +(Source: https://lmstudio.ai/docs/app/api/endpoints/openai)
751 +
752 +- Endpoints: `/v1/models`, `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, and **`/v1/responses`** (added specifically so OpenAI Codex CLI works against it).
753 +- Compatibility story is purely "change `base_url` on the official SDK" — the same acceptance test Zyquo Router's Phase 3 gate uses.
754 +- Signal: local servers are converging on also exposing `/v1/responses` because new OpenAI tooling (Codex) speaks only the Responses API. Relevant to the Phase 0 "is `/v1/responses` worth it" decision: *optional now, trending toward expected*.
755 +
756 +#### 2.3.3 vLLM OpenAI-compatible server
757 +
758 +(Source: https://docs.vllm.ai/en/latest/serving/online_serving/)
759 +
760 +- Implements `/v1/chat/completions` (+ batch), `/v1/completions` (no `suffix`), `/v1/responses`, `/v1/embeddings`, transcription/translation.
761 +- Known deviations: `user` is ignored; `parallel_tool_calls` defaults to `true`; extra sampling params (`top_k`, `best_of`, guided decoding) accepted via `extra_body` — the standard pass-through idiom again.
762 +- vLLM's habit of sending **usage in a separate empty-`choices` chunk after the `finish_reason` chunk** is spec-correct but broke LiteLLM's consumer (https://github.com/BerriAI/litellm/issues/25389) — Zyquo Router's normalizer must handle both orderings when consuming OpenAI-compatible upstreams.
763 +
764 +---
765 +
766 +### 2.4 Compatibility pitfalls that trip up real clients
767 +
768 +Strict SDKs (openai-python uses Pydantic models; openai-node/Vercel AI SDK use zod-like validation) parse every chunk. These are the documented, real-world failure modes a gateway must design around:
769 +
770 +1. **Usage chunk shape.** With `stream_options: {"include_usage": true}`, OpenAI's contract is: every content chunk has `"usage": null`, and one **final extra chunk before `[DONE]`** has `"choices": []` and the full `usage` object. Emitting usage with non-empty choices violates the spec (LiteLLM bug https://github.com/BerriAI/litellm/issues/28735); conversely, clients that stop at `finish_reason` miss the usage chunk (https://github.com/BerriAI/litellm/issues/25389); AutoGen crashed on the empty-choices chunk itself (https://github.com/microsoft/autogen/issues/5078); llama.cpp put usage in a slightly different chunk than OpenAI and broke clients (https://github.com/ggml-org/llama.cpp/issues/15443). **Rule: emit usage exactly like OpenAI (separate final empty-choices chunk, only when requested), and when consuming, read to true end-of-stream and tolerate usage in any late chunk.**
771 +2. **Streamed tool-call deltas.** The first `tool_calls` delta must carry `index`, `id`, `type: "function"`, and `function.name`; subsequent deltas carry only `index` + `function.arguments` fragments. Real breakage: Gemini-behind-a-compat-shim omitting `index` (https://github.com/anomalyco/opencode/issues/17902); providers sending `function.name` only in a *later* chunk (https://github.com/anomalyco/opencode/issues/24137, https://github.com/anomalyco/opencode/issues/26412); vLLM omitting `"type":"function"` under forced `tool_choice` (https://github.com/vllm-project/vllm/issues/16340). **Rule: the translator owns tool-call chunk assembly — always emit index/id/type/name complete in the first delta for each call, arguments-only after.**
772 +3. **Role delta discipline.** The first chunk of each choice must have `delta: {"role": "assistant"}` (optionally with `content: ""`); an **empty-string role** instead of `"assistant"` breaks strict parsers (https://github.com/anomalyco/opencode/issues/28427). Later deltas must omit `role` entirely rather than repeat it as `""`.
773 +4. **All-or-nothing chunk validation.** Some client stacks silently drop an entire chunk if any field fails validation — provider quirks then surface as *silently missing content*, which is undebuggable (https://github.com/Effect-TS/effect-smol/issues/2337, https://github.com/pydantic/pydantic-ai/issues/3658 — OpenRouter `reasoning_details` variant missing a field broke pydantic-ai). **Rule: every field Zyquo Router emits must be exactly typed (`created` as integer epoch seconds, `object` exactly `"chat.completion.chunk"`, `index` present on every choice/tool_call); when *adding* fields (e.g. `reasoning_content`), add only well-formed, consistently shaped ones.** Also: null-valued token-detail fields inside `usage` broke the OpenAI Agents SDK (https://github.com/openai/openai-agents-python/issues/1179) — omit detail objects rather than sending them with `null` members.
774 +5. **`max_tokens` vs `max_completion_tokens`.** OpenAI deprecated `max_tokens` in favor of `max_completion_tokens`; o-series/reasoning models hard-reject `max_tokens` ("Unsupported parameter"). Every ecosystem project had to patch (Ollama https://github.com/ollama/ollama/issues/7125, simonw/llm https://github.com/simonw/llm/issues/724, Home Assistant https://github.com/home-assistant/core/issues/137039, Spring AI https://github.com/spring-projects/spring-ai/issues/3300). **Rule: accept both on ingress, normalize internally to one limit value, emit whichever the upstream requires (per-provider table), never forward both.**
775 +6. **`system_fingerprint`.** Optional in practice — OpenAI itself returns `null`/absent for many models (https://github.com/openai/openai-python/issues/1038, https://github.com/openai/openai-openapi/issues/167), and SDK type defs treat it as optional (https://github.com/openai/openai-node/issues/443). Gateways may safely omit it or set `null`; do not fabricate values (clients use it for determinism tracking with `seed`).
776 +7. **`n > 1`.** Most non-OpenAI upstreams don't support multiple choices (Ollama: unsupported; Anthropic/Gemini: no direct equivalent). Options: reject with a clear 400, or fan out N upstream calls. LiteLLM/Ollama precedent: reject or ignore. `choices` must still always be an **array** with correct `index` fields even for n=1 (OpenRouter: "choices is always an array" — https://openrouter.ai/docs/api-reference/overview).
777 +8. **Keep-alives, buffering, and timeouts.** Long prefills (big prompts, reasoning models) can be silent for 30s+; intermediaries and client idle timeouts kill the connection (e.g. https://github.com/microsoft/agent-framework/issues/6941). SSE **comment lines** (`: keep-alive`) every 15–30s are the only spec-compatible heartbeat; also send `Cache-Control: no-cache`, `Connection: keep-alive`, `X-Accel-Buffering: no`, and never gzip SSE (compression layers buffer the stream). Conversely, when *consuming*, tolerate comment lines from upstreams (OpenRouter emits `: OPENROUTER PROCESSING` — https://openrouter.ai/docs/api-reference/streaming). Streaming responses need effectively unlimited write timeouts; non-streaming needs a generous but bounded upstream timeout mapped to 504/408.
778 +9. **CORS for browser clients.** A cross-origin `fetch` to the router fails before the first byte without correct `Access-Control-Allow-Origin` + preflight handling for `POST` with `Authorization`/`Content-Type: application/json` headers. OpenRouter even classifies CORS problems under 400 (https://openrouter.ai/docs/api-reference/errors). Zyquo Router's default-permissive-on-localhost CORS (with `OPTIONS` preflight support) is the right call for browser-based dev tools.
779 +10. **Mid-stream failure surfacing.** After the first chunk, the status line is committed (200). The only clean options are OpenRouter's error-chunk-with-`finish_reason:"error"` (§2.2.3) followed by `[DONE]`, or an abrupt close (which strict SDKs report as a network error). Emit the error chunk.
780 +
781 +---
782 +
783 +### 2.5 Recommended pattern set for Zyquo Router
784 +
785 +Synthesis of the above into the concrete policy for our gateway:
786 +
787 +1. **Model namespacing** — `provider/model-id` (LiteLLM/OpenRouter convention), full catalog from Zyquo Cloud; also accept unambiguous bare upstream IDs (resolve via catalog; ambiguous → 400 listing candidates) and user aliases (LiteLLM `model_group_alias` pattern). `GET /v1/models` returns OpenAI list shape with `id` = namespaced ID, enriched per-model metadata (`context_length`, pricing as decimal strings, capability flags, `supported_parameters`) under an `x-zyquo` key, following OpenRouter's metadata precedent.
788 +2. **Unknown/unsupported-param policy** — data-driven per-provider tables (LiteLLM style): *rename* where names differ (`max_tokens`/`max_completion_tokens`, `stop`→`stop_sequences`), *strip silently* what the upstream would reject (default `drop_params: true` behavior, log at debug in the request inspector), *reject with a helpful OpenAI-format 400* only when silently dropping would change semantics materially (e.g. `tools` on a no-tools model, `n>1`), and *pass through* unknown extra body keys to OpenAI-compatible upstreams (the `extra_body` idiom) — documented per provider in `docs/API.md`.
789 +3. **finish_reason & usage normalization** — normalize `finish_reason` to `stop | length | tool_calls | content_filter` (+ `error` mid-stream only), preserving the raw upstream value as `native_finish_reason` (OpenRouter pattern). Usage: real upstream numbers when available (request usage-in-stream from upstreams that support it, per-provider flag); tokenizer-estimated otherwise, flagged (e.g. `"x-zyquo": {"usage_estimated": true}`); cost computed from the catalog's pricing at response time and exposed OpenRouter-style as extra `usage` fields. Streaming usage emitted **only** when the client sends `stream_options.include_usage`, as a final `"choices": []` chunk before `[DONE]` — byte-exact per §2.4.1–4.
790 +4. **Upstream error surfacing** — map to OpenAI error JSON `{"error":{"message","type","param","code"}}` with LiteLLM's status taxonomy (401 provider-key invalid naming the provider, 429 with propagated `Retry-After`, 400 subtypes for context-window/content-policy, 502 "upstream returned an invalid response", 503 provider unavailable, 504/408 timeouts); include a canonical machine `code` plus the upstream's original code in the message (OpenRouter's `error_type`+`provider_code` idea) — never raw upstream payloads or key material. Mid-stream: OpenRouter-style error chunk with `finish_reason: "error"`, then `[DONE]`.
791 +5. **Retry/fallback policy** — per-error-class retry policy (LiteLLM `RetryPolicy`): retries with exponential backoff + jitter on 429/5xx/timeouts (honoring `Retry-After`), zero retries on 400/401/403; then user-configured fallback chains (ordered model lists) triggered on retry exhaustion, context-window and content-policy errors; the response `model` field reports the model actually used and cost follows it (OpenRouter contract). Optional per-provider cooldown state feeding the dashboard's "degraded" indicator.
792 +6. **Health & liveness** — `GET /health` (status, version, uptime, active streams) never touches upstreams; per-provider "Test key" in the UI does a minimal authenticated upstream call and reports latency; SSE keep-alive comments every ~20s of upstream silence; client disconnect cancels the upstream task immediately (guaranteed, since we own the upstream HTTP call).
793 +
794 +---
795 +
796 +*Primary sources:* https://docs.litellm.ai/docs/proxy/configs · https://docs.litellm.ai/docs/completion/drop_params · https://docs.litellm.ai/docs/exception_mapping · https://docs.litellm.ai/docs/routing · https://docs.litellm.ai/docs/proxy/reliability · https://docs.litellm.ai/docs/completion/token_usage · https://docs.litellm.ai/docs/completion/stream · https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json · https://openrouter.ai/docs/api-reference/streaming · https://openrouter.ai/docs/api-reference/errors · https://openrouter.ai/docs/api-reference/overview · https://openrouter.ai/docs/use-cases/usage-accounting · https://openrouter.ai/docs/guides/routing/model-fallbacks · https://openrouter.ai/docs/features/provider-routing · https://openrouter.ai/docs/models · https://docs.ollama.com/api/openai-compatibility · https://lmstudio.ai/docs/app/api/endpoints/openai · https://docs.vllm.ai/en/latest/serving/online_serving/ · GitHub issues linked inline (§2.4).
797 +## 3. Translation matrices
798 +
799 +> Research date: 2026-07-30. Verified against live official documentation (URLs cited inline).
800 +> This section is the contract for `Translate/AnthropicTranslator.swift`, `Translate/GeminiTranslator.swift`,
801 +> and `Translate/CompatAdjuster.swift`. The router's canonical internal format IS the OpenAI
802 +> `chat/completions` wire format (Section 1); every non-OpenAI upstream is mapped bidirectionally onto it.
803 +
804 +---
805 +
806 +### 3.1 Anthropic Messages API ⇄ OpenAI chat/completions
807 +
808 +Primary sources:
809 +- Messages API reference: https://platform.claude.com/docs/en/api/messages (docs.anthropic.com 301-redirects here)
810 +- Streaming: https://platform.claude.com/docs/en/docs/build-with-claude/streaming
811 +- Tool use: https://platform.claude.com/docs/en/docs/agents-and-tools/tool-use/overview
812 +- Errors: https://platform.claude.com/docs/en/api/errors
813 +
814 +#### 3.1.1 Endpoint & auth
815 +
816 +| | OpenAI (what our client sends us) | Anthropic (what we send upstream) |
817 +|---|---|---|
818 +| Endpoint | `POST /v1/chat/completions` | `POST https://api.anthropic.com/v1/messages` |
819 +| Auth header | `Authorization: Bearer zyquo-sk-…` (local key) | `x-api-key: <ANTHROPIC_KEY>` |
820 +| Version header | — | `anthropic-version: 2023-06-01` (required) |
821 +| Content type | `application/json` | `application/json` |
822 +
823 +The `anthropic-version` header is **mandatory**; requests without it are rejected. Pin `2023-06-01` (the stable version used by all official SDKs).
824 +
825 +#### 3.1.2 Request translation (OpenAI → Anthropic)
826 +
827 +##### Parameter map
828 +
829 +| OpenAI request field | Anthropic field | Rule |
830 +|---|---|---|
831 +| `model` | `model` | Strip the `anthropic/` namespace prefix. |
832 +| `messages[role=system]`, `messages[role=developer]` | top-level `system` | **Extract** all system/developer messages (in order), join text with `"\n\n"`. Anthropic has no `system` role inside `messages`. `developer` is treated identically to `system`. |
833 +| `messages[role=user/assistant/tool]` | `messages` | See message-shape rules below. |
834 +| `max_tokens` / `max_completion_tokens` | `max_tokens` | **REQUIRED by Anthropic.** If the client omits both, the router MUST synthesize a value. Strategy: use the model's catalog `max_output` (from Zyquo Cloud's `ModelCatalog`); fall back to `4096` if unknown. `max_completion_tokens` wins if both present. |
835 +| `temperature` | `temperature` | OpenAI range **0–2**, Anthropic range **0–1** (default 1.0). Strategy: `min(temperature, 1.0)` (clamp). Do NOT divide by 2 — halving changes semantics for the common 0–1 sub-range that clients actually use. Log a warning when clamping. |
836 +| `top_p` | `top_p` | Same 0–1 range, pass through. Anthropic advises using temperature OR top_p, not both — pass both if given (API accepts it). |
837 +| — (extra body `top_k`) | `top_k` | Not an OpenAI param; accept as pass-through extra key. |
838 +| `stop` (string or array) | `stop_sequences` (array) | Wrap a bare string in a 1-element array. |
839 +| `n` | — | **Unsupported.** If `n > 1` → reject with OpenAI 400 error (`invalid_request_error`, param `n`). |
840 +| `frequency_penalty`, `presence_penalty`, `logit_bias`, `seed`, `logprobs`, `top_logprobs` | — | **Unsupported → strip silently** (LiteLLM behavior); optionally record in the request log that params were dropped. |
841 +| `stream` | `stream` | Pass through. |
842 +| `stream_options` | — | Router-side only (controls our usage chunk emission). Never forwarded. |
843 +| `user` | `metadata.user_id` | Direct map (Anthropic wants an opaque non-PII id — pass as-is). |
844 +| `tools` | `tools` | See tools mapping. |
845 +| `tool_choice` | `tool_choice` | See tool_choice mapping. |
846 +| `parallel_tool_calls: false` | `tool_choice.disable_parallel_tool_use: true` | Set on whatever `tool_choice` object we send (`auto` if none was specified). `parallel_tool_calls: true` → omit (default). |
847 +| `response_format` | `output_config.format` / workaround | See JSON-mode strategy. |
848 +| `reasoning_effort` (OpenAI standard) | `output_config.effort` / `thinking` | See §3.4 (reasoning). |
849 +| extra body `thinking` | `thinking` | Pass-through extra key for power users: `{"type":"enabled"\|"adaptive"\|"disabled","budget_tokens":≥1024,"display":"summarized"\|"omitted"}`. |
850 +
851 +##### Message-shape rules (the tricky part)
852 +
853 +Anthropic enforces constraints that OpenAI does not:
854 +
855 +1. **Roles are only `user` and `assistant`** inside `messages`.
856 +2. **Turns must alternate.** Consecutive same-role messages must be **merged** into a single message whose `content` is an array of blocks, preserving order.
857 +3. **The first message must be `user`.** If the client's first non-system message is `assistant`, prepend a placeholder user message (e.g. a single text block `"(continue)"`) or reject — the router merges/prepends (LiteLLM's approach) so real clients keep working.
858 +4. **OpenAI `tool` role messages → `user` messages containing `tool_result` blocks.** Consecutive `tool` messages (parallel tool results) merge into ONE user message with multiple `tool_result` blocks. `tool_result` blocks must come FIRST in that user message's content array if user text follows.
859 +5. **Assistant `tool_calls` → `tool_use` content blocks**, after any text content, with `input` as a **parsed JSON object** (OpenAI `arguments` is a JSON *string* — parse it; if unparseable, send `{}` and log).
860 +
861 +##### Content-part map
862 +
863 +| OpenAI content part | Anthropic content block |
864 +|---|---|
865 +| `{"type":"text","text":T}` | `{"type":"text","text":T}` |
866 +| `{"type":"image_url","image_url":{"url":"data:image/png;base64,XXX"}}` | `{"type":"image","source":{"type":"base64","media_type":"image/png","data":"XXX"}}` — media_type parsed from the data URL; allowed: `image/jpeg`, `image/png`, `image/gif`, `image/webp`. |
867 +| `{"type":"image_url","image_url":{"url":"https://…"}}` | `{"type":"image","source":{"type":"url","url":"https://…"}}` — Anthropic supports URL sources natively. |
868 +| plain string `content` | plain string `content` (both APIs accept a bare string). |
869 +
870 +##### Side-by-side request example (tools + image + system)
871 +
872 +OpenAI request received by the router:
873 +
874 +```json
875 +{
876 + "model": "anthropic/claude-sonnet-4-5",
877 + "max_tokens": 1024,
878 + "temperature": 1.4,
879 + "messages": [
880 + {"role": "system", "content": "You are terse."},
881 + {"role": "user", "content": [
882 + {"type": "text", "text": "What's in this image, and what's the weather there?"},
883 + {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ..."}}
884 + ]},
885 + {"role": "assistant", "content": null, "tool_calls": [
886 + {"id": "call_abc123", "type": "function",
887 + "function": {"name": "get_weather", "arguments": "{\"location\": \"Paris\"}"}}
888 + ]},
889 + {"role": "tool", "tool_call_id": "call_abc123", "content": "18°C, sunny"}
890 + ],
891 + "tools": [
892 + {"type": "function", "function": {
893 + "name": "get_weather",
894 + "description": "Get current weather",
895 + "parameters": {"type": "object",
896 + "properties": {"location": {"type": "string"}}, "required": ["location"]}}}
897 + ],
898 + "tool_choice": "auto",
899 + "parallel_tool_calls": false
900 +}
901 +```
902 +
903 +Anthropic request the router sends upstream:
904 +
905 +```json
906 +{
907 + "model": "claude-sonnet-4-5",
908 + "max_tokens": 1024,
909 + "temperature": 1.0,
910 + "system": "You are terse.",
911 + "messages": [
912 + {"role": "user", "content": [
913 + {"type": "text", "text": "What's in this image, and what's the weather there?"},
914 + {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": "/9j/4AAQ..."}}
915 + ]},
916 + {"role": "assistant", "content": [
917 + {"type": "tool_use", "id": "call_abc123", "name": "get_weather",
918 + "input": {"location": "Paris"}}
919 + ]},
920 + {"role": "user", "content": [
921 + {"type": "tool_result", "tool_use_id": "call_abc123", "content": "18°C, sunny"}
922 + ]}
923 + ],
924 + "tools": [
925 + {"name": "get_weather", "description": "Get current weather",
926 + "input_schema": {"type": "object",
927 + "properties": {"location": {"type": "string"}}, "required": ["location"]}}
928 + ],
929 + "tool_choice": {"type": "auto", "disable_parallel_tool_use": true}
930 +}
931 +```
932 +
933 +Note the tool_use `id` is preserved verbatim in both directions so multi-turn tool loops round-trip.
934 +
935 +##### Tools & tool_choice map
936 +
937 +| OpenAI | Anthropic |
938 +|---|---|
939 +| `tools[].function.name` | `tools[].name` |
940 +| `tools[].function.description` | `tools[].description` |
941 +| `tools[].function.parameters` (JSON Schema) | `tools[].input_schema` (JSON Schema draft 2020-12) |
942 +| `tools[].function.strict: true` | `tools[].strict: true` (now supported natively) |
943 +| `tool_choice: "auto"` (or omitted with tools) | `{"type": "auto"}` |
944 +| `tool_choice: "required"` | `{"type": "any"}` |
945 +| `tool_choice: "none"` | `{"type": "none"}` |
946 +| `tool_choice: {"type":"function","function":{"name":N}}` | `{"type": "tool", "name": N}` |
947 +| `parallel_tool_calls: false` | `disable_parallel_tool_use: true` on the tool_choice object |
948 +
949 +##### JSON mode / `response_format` strategy
950 +
951 +Anthropic historically had **no native JSON mode**; the current API (2026) has **structured output** via
952 +`output_config.format` (source: https://platform.claude.com/docs/en/api/messages — `output_config: { format: { type: "json_schema", schema: {...} } }`).
953 +
954 +Router strategy, in priority order:
955 +
956 +1. `response_format: {"type":"json_schema","json_schema":{"schema":S,...}}` → `output_config: {"format": {"type":"json_schema","schema": S}}` on models that support it (catalog capability flag).
957 +2. On models WITHOUT structured-output support — **tool trick**: define a single synthetic tool (`name: "json_output"`, `input_schema: S`), force it with `tool_choice: {"type":"tool","name":"json_output"}`, and return the streamed/collected `input` object as the assistant `content` string (finish_reason `stop`, not `tool_calls`).
958 +3. `response_format: {"type":"json_object"}` → append to `system`: `"You must respond with valid JSON only, no prose, no markdown fences."` and optionally **prefill** the assistant turn with `{` (append `{"role":"assistant","content":"{"}` and re-prepend `{` to the returned text). Prefill is Anthropic-sanctioned steering. Document in `docs/API.md` that json_object on Anthropic is best-effort.
959 +4. `{"type":"text"}` → no-op.
960 +
961 +##### Extended thinking (request side)
962 +
963 +- `thinking: {"type":"enabled","budget_tokens":N}` (N ≥ 1024, must be < `max_tokens`) or `{"type":"adaptive"}`; `display: "summarized"|"omitted"` controls whether thinking text is streamed.
964 +- Newer models also take `output_config.effort: "low"|"medium"|"high"|"xhigh"|"max"`.
965 +- Router mapping for the standard OpenAI `reasoning_effort` param: `low → output_config.effort "low"` (or `thinking budget 1024`), `medium → "medium"` (8192), `high → "high"` (24576) — per-model capability gate from the catalog. Raw `thinking` extra-body always wins if provided.
966 +- **Multi-turn constraint:** in tool-use loops with thinking enabled, Anthropic expects prior `thinking` blocks (with `signature`) to be passed back. See §3.4 for how the router preserves signatures via `reasoning_details`.
967 +
968 +#### 3.1.3 Response translation (Anthropic → OpenAI), non-streaming
969 +
970 +Anthropic response:
971 +
972 +```json
973 +{
974 + "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
975 + "type": "message",
976 + "role": "assistant",
977 + "model": "claude-sonnet-4-5",
978 + "content": [
979 + {"type": "text", "text": "It's 18°C and sunny in Paris."},
980 + {"type": "tool_use", "id": "toolu_01A09q90qw90lq917835lq9",
981 + "name": "get_weather", "input": {"location": "Paris"}}
982 + ],
983 + "stop_reason": "tool_use",
984 + "stop_sequence": null,
985 + "usage": {
986 + "input_tokens": 412, "output_tokens": 61,
987 + "cache_creation_input_tokens": 0, "cache_read_input_tokens": 128
988 + }
989 +}
990 +```
991 +
992 +Router emits:
993 +
994 +```json
995 +{
996 + "id": "chatcmpl-9f3c1a2b7d4e",
997 + "object": "chat.completion",
998 + "created": 1753872000,
999 + "model": "anthropic/claude-sonnet-4-5",
1000 + "choices": [{
1001 + "index": 0,
1002 + "message": {
1003 + "role": "assistant",
1004 + "content": "It's 18°C and sunny in Paris.",
1005 + "tool_calls": [{
1006 + "id": "toolu_01A09q90qw90lq917835lq9",
1007 + "type": "function",
1008 + "function": {"name": "get_weather", "arguments": "{\"location\":\"Paris\"}"}
1009 + }]
1010 + },
1011 + "finish_reason": "tool_calls"
1012 + }],
1013 + "usage": {
1014 + "prompt_tokens": 540,
1015 + "completion_tokens": 61,
1016 + "total_tokens": 601,
1017 + "prompt_tokens_details": {"cached_tokens": 128}
1018 + }
1019 +}
1020 +```
1021 +
1022 +Rules:
1023 +
1024 +- **`id`**: generate a fresh `chatcmpl-<hex>` (keep the upstream `msg_…` id in the request log for tracing). `object: "chat.completion"`, `created`: gateway clock (Unix seconds). `model`: echo the **namespaced** router id.
1025 +- **Content blocks →** concatenate all `text` block texts into `message.content` (`null` if none and tool_calls exist); each `tool_use` block → one `tool_calls[]` entry with `arguments` = `JSON.stringify(input)`; `thinking` blocks → `message.reasoning_content` (§3.4), `signature` → `reasoning_details`; `redacted_thinking` → `reasoning_details` only (opaque `data`).
1026 +- **`stop_reason` → `finish_reason`:**
1027 +
1028 +| Anthropic `stop_reason` | OpenAI `finish_reason` | Note |
1029 +|---|---|---|
1030 +| `end_turn` | `stop` | |
1031 +| `max_tokens` | `length` | |
1032 +| `stop_sequence` | `stop` | OpenAI has no separate value; log the matched `stop_sequence`. |
1033 +| `tool_use` | `tool_calls` | |
1034 +| `refusal` | `content_filter` | Closest OpenAI semantic; `stop_details` (category/explanation) goes to the request log only. |
1035 +| `pause_turn` | `stop` | Long-running server-tool turns; router doesn't use server tools, treat as stop. |
1036 +| `model_context_window_exceeded` | `length` | |
1037 +
1038 +- **`usage`:** Anthropic's `input_tokens` EXCLUDES cache reads/writes. Normalize:
1039 + `prompt_tokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokens`;
1040 + `completion_tokens = output_tokens` (already includes thinking tokens);
1041 + `total_tokens = prompt + completion`;
1042 + `prompt_tokens_details.cached_tokens = cache_read_input_tokens`;
1043 + `completion_tokens_details.reasoning_tokens = usage.output_tokens_details.thinking_tokens` when present.
1044 +
1045 +#### 3.1.4 SSE event model → OpenAI `chat.completion.chunk` (streaming)
1046 +
1047 +Anthropic stream grammar (source: https://platform.claude.com/docs/en/docs/build-with-claude/streaming):
1048 +
1049 +```
1050 +message_start
1051 +( content_block_start → content_block_delta* → content_block_stop )*
1052 +message_delta+
1053 +message_stop
1054 +```
1055 +with `ping` events anywhere and possible `error` events. Each SSE frame is
1056 +`event: <name>\ndata: <json>\n\n`. `message_delta.usage.output_tokens` is **cumulative**.
1057 +
1058 +Delta types inside `content_block_delta`: `text_delta` (`.text`), `input_json_delta` (`.partial_json`, a partial JSON **string** for `tool_use.input`), `thinking_delta` (`.thinking`), `signature_delta` (`.signature`, arrives just before the thinking block's `content_block_stop`).
1059 +
1060 +##### Event → chunk mapping table
1061 +
1062 +The translator keeps two counters: `toolIdx` = number of `tool_use` blocks seen so far (this is the OpenAI `tool_calls[].index`, 0-based, **independent of the Anthropic block `index`**), and cumulative usage.
1063 +
1064 +| Anthropic event | Emitted OpenAI chunk delta | Notes |
1065 +|---|---|---|
1066 +| `message_start` | `{"delta":{"role":"assistant","content":""},"finish_reason":null}` | First chunk; role delta exactly once. Capture `message.usage.input_tokens` (+cache fields) for the final usage chunk. |
1067 +| `ping` | *(nothing)* — or forward as SSE comment `: keep-alive` | Comments keep clients' sockets warm without confusing SDK parsers. |
1068 +| `content_block_start` (`type:"text"`) | *(nothing)* | |
1069 +| `content_block_delta` / `text_delta` | `{"delta":{"content": text}}` | |
1070 +| `content_block_start` (`type:"tool_use"`) | `{"delta":{"tool_calls":[{"index": toolIdx, "id": block.id, "type": "function", "function": {"name": block.name, "arguments": ""}}]}}` | id + name announced once, `arguments:""` starts the accumulator — exactly the shape the OpenAI SDKs expect. |
1071 +| `content_block_delta` / `input_json_delta` | `{"delta":{"tool_calls":[{"index": toolIdx, "function": {"arguments": partial_json}}]}}` | No id/name repetition. Empty `partial_json` frames may be skipped. |
1072 +| `content_block_stop` (tool_use) | *(nothing)*; `toolIdx += 1` | |
1073 +| `content_block_delta` / `thinking_delta` | `{"delta":{"reasoning_content": thinking}}` | §3.4 convention. |
1074 +| `content_block_delta` / `signature_delta` | `{"delta":{"reasoning_details":[{"type":"anthropic.signature","signature":…,"index":blockIdx}]}}` — or drop if client didn't opt in | Needed only to round-trip thinking in tool loops. |
1075 +| `content_block_start/stop` (`type:"thinking"`) | *(nothing)* | |
1076 +| `message_delta` | `{"delta":{},"finish_reason": map(stop_reason)}` | finish_reason chunk (empty delta object, per OpenAI spec). Capture cumulative `usage.output_tokens`. |
1077 +| `message_stop` | If `stream_options.include_usage`: `{"choices":[],"usage":{…}}` chunk; then `data: [DONE]` | Usage chunk has an EMPTY `choices` array per OpenAI spec. Then terminate. |
1078 +| `error` | Emit `data: {"error":{"message":…,"type":"api_error","code":upstream_type}}` then close | OpenAI has no in-band stream-error spec; this LiteLLM-style error frame is the least-bad option — document it in `docs/API.md`. Map `overloaded_error` → our 529→503 semantics in logs. |
1079 +
1080 +Every emitted chunk carries the constant envelope:
1081 +`{"id":"chatcmpl-…","object":"chat.completion.chunk","created":C,"model":"anthropic/…","choices":[{"index":0,"delta":…,"finish_reason":…}]}` — same `id`/`created` for the whole stream.
1082 +
1083 +##### Full example transcript (tool-use stream)
1084 +
1085 +Anthropic events (left) → OpenAI chunks emitted by the router (right). Envelope fields elided for readability; every right-hand line is a full `chat.completion.chunk`.
1086 +
1087 +```
1088 +ANTHROPIC UPSTREAM → ZYQUO ROUTER EMITS (OpenAI SSE)
1089 +
1090 +event: message_start
1091 +data: {"type":"message_start","message":{"id":"msg_014p", → data: {"id":"chatcmpl-a1","object":"chat.completion.chunk",
1092 + "role":"assistant","content":[],"model":"claude-…", "created":1753872000,"model":"anthropic/claude-sonnet-4-5",
1093 + "usage":{"input_tokens":472,"output_tokens":2}, …}} "choices":[{"index":0,"delta":{"role":"assistant",
1094 + "content":""},"finish_reason":null}]}
1095 +
1096 +event: content_block_start
1097 +data: {"type":"content_block_start","index":0, → (nothing)
1098 + "content_block":{"type":"text","text":""}}
1099 +
1100 +event: ping
1101 +data: {"type":"ping"} → (nothing, or ": keep-alive" comment)
1102 +
1103 +event: content_block_delta
1104 +data: {…,"delta":{"type":"text_delta","text":"Okay,"}} → data: {…,"choices":[{"index":0,"delta":{"content":"Okay,"},
1105 + "finish_reason":null}]}
1106 +
1107 +event: content_block_delta
1108 +data: {…,"delta":{"type":"text_delta", → data: {…,"choices":[{"index":0,"delta":{"content":
1109 + "text":" checking the weather:"}} " checking the weather:"},"finish_reason":null}]}
1110 +
1111 +event: content_block_stop
1112 +data: {"type":"content_block_stop","index":0} → (nothing)
1113 +
1114 +event: content_block_start
1115 +data: {"type":"content_block_start","index":1, → data: {…,"choices":[{"index":0,"delta":{"tool_calls":[
1116 + "content_block":{"type":"tool_use", {"index":0,"id":"toolu_01T1x","type":"function",
1117 + "id":"toolu_01T1x","name":"get_weather","input":{}}} "function":{"name":"get_weather","arguments":""}}]},
1118 + "finish_reason":null}]}
1119 +
1120 +event: content_block_delta
1121 +data: {…,"delta":{"type":"input_json_delta", → (skipped — empty partial_json)
1122 + "partial_json":""}}
1123 +
1124 +event: content_block_delta
1125 +data: {…,"delta":{"type":"input_json_delta", → data: {…,"choices":[{"index":0,"delta":{"tool_calls":[
1126 + "partial_json":"{\"location\":"}} {"index":0,"function":{"arguments":"{\"location\":"}}]},
1127 + "finish_reason":null}]}
1128 +
1129 +event: content_block_delta
1130 +data: {…,"delta":{"type":"input_json_delta", → data: {…,"choices":[{"index":0,"delta":{"tool_calls":[
1131 + "partial_json":" \"San Francisco, CA\"}"}} {"index":0,"function":{"arguments":
1132 + " \"San Francisco, CA\"}"}}]},"finish_reason":null}]}
1133 +
1134 +event: content_block_stop
1135 +data: {"type":"content_block_stop","index":1} → (nothing; toolIdx→1)
1136 +
1137 +event: message_delta
1138 +data: {"type":"message_delta","delta":{"stop_reason": → data: {…,"choices":[{"index":0,"delta":{},
1139 + "tool_use","stop_sequence":null}, "finish_reason":"tool_calls"}]}
1140 + "usage":{"output_tokens":89}}
1141 +
1142 +event: message_stop → data: {…,"choices":[],"usage":{"prompt_tokens":472,
1143 +data: {"type":"message_stop"} "completion_tokens":89,"total_tokens":561}}
1144 + (only if stream_options.include_usage)
1145 + → data: [DONE]
1146 +```
1147 +
1148 +Thinking streams follow the same pattern: `content_block_start {type:"thinking"}` opens nothing, each `thinking_delta` → `{"delta":{"reasoning_content":"…"}}`, `signature_delta` → `reasoning_details` (or dropped), then the text block streams as normal `content` deltas.
1149 +
1150 +---
1151 +
1152 +### 3.2 Gemini `generateContent` / `streamGenerateContent` ⇄ OpenAI
1153 +
1154 +Primary sources:
1155 +- API reference: https://ai.google.dev/api/generate-content
1156 +- Part/Content schema: https://ai.google.dev/api/caching#Part
1157 +- Function calling: https://ai.google.dev/gemini-api/docs/function-calling
1158 +- Structured output: https://ai.google.dev/gemini-api/docs/structured-output
1159 +- Thinking: https://ai.google.dev/gemini-api/docs/thinking
1160 +- Google's own OpenAI-compat layer (used as a mapping oracle): https://ai.google.dev/gemini-api/docs/openai
1161 +
1162 +#### 3.2.1 Endpoint & auth
1163 +
1164 +| | Form |
1165 +|---|---|
1166 +| Non-streaming | `POST https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent` |
1167 +| Streaming (SSE) | `POST https://generativelanguage.googleapis.com/v1beta/models/{model}:streamGenerateContent?alt=sse` |
1168 +| Auth | Header `x-goog-api-key: <GEMINI_KEY>` (preferred) or query `?key=<GEMINI_KEY>` |
1169 +
1170 +**Always use `?alt=sse`** — without it, streamGenerateContent returns a chunked **JSON array**, not SSE. The model name is in the **path**, not the body. Use the header for auth so the key never appears in URLs/logs.
1171 +
1172 +#### 3.2.2 Request translation (OpenAI → Gemini)
1173 +
1174 +| OpenAI field | Gemini field | Rule |
1175 +|---|---|---|
1176 +| `model` | URL path | Strip `gemini/` prefix. |
1177 +| system/developer messages | `systemInstruction: {"parts":[{"text": joined}]}` | Join multiple with `"\n\n"`. |
1178 +| `messages[role=user]` | `contents[]` entry `role: "user"` | |
1179 +| `messages[role=assistant]` | `contents[]` entry `role: "model"` | **Role rename user/assistant → user/model.** |
1180 +| `messages[role=tool]` | `contents[]` entry `role: "user"` with `functionResponse` part(s) | See tool round-trip below. Consecutive `tool` messages merge into one user-role content with multiple `functionResponse` parts. |
1181 +| `max_tokens`/`max_completion_tokens` | `generationConfig.maxOutputTokens` | Optional on Gemini (nice: no synthesis needed). |
1182 +| `temperature` | `generationConfig.temperature` | Both 0–2. Pass through unchanged. |
1183 +| `top_p` | `generationConfig.topP` | camelCase rename. |
1184 +| extra `top_k` | `generationConfig.topK` | |
1185 +| `stop` | `generationConfig.stopSequences` | Wrap string → array. |
1186 +| `n` | `generationConfig.candidateCount` (1–8) | Router policy: support `n` here (one of the few upstreams that can) or clamp to 1 for uniformity — **decide once; recommend rejecting n>1 router-wide** for consistent behavior across providers. |
1187 +| `seed` | `generationConfig.seed` | |
1188 +| `presence_penalty` | `generationConfig.presencePenalty` | −2..2, same semantics. |
1189 +| `frequency_penalty` | `generationConfig.frequencyPenalty` | |
1190 +| `logit_bias`, `logprobs`, `user` | — | Strip. |
1191 +| `response_format {"type":"json_object"}` | `generationConfig.responseMimeType: "application/json"` | |
1192 +| `response_format {"type":"json_schema",…}` | `responseMimeType: "application/json"` + `generationConfig.responseJsonSchema` (standard JSON Schema; older models: `responseSchema` OpenAPI-subset) | Prefer `responseJsonSchema`; scrub unsupported keywords (`$schema`, `additionalProperties` on old models) defensively. |
1193 +| `tools` | `tools: [{"functionDeclarations":[{name, description, parameters}]}]` | ALL functions go into ONE `functionDeclarations` array. `parameters` is JSON-Schema-like; scrub `strict`. |
1194 +| `tool_choice` | `toolConfig.functionCallingConfig` | `"auto"`→`{"mode":"AUTO"}`; `"required"`→`{"mode":"ANY"}`; `"none"`→`{"mode":"NONE"}`; `{"function":{"name":N}}`→`{"mode":"ANY","allowedFunctionNames":[N]}`. (A `VALIDATED` mode also exists; unused by the router.) |
1195 +| `parallel_tool_calls` | — | No Gemini equivalent; strip. Gemini decides parallelism itself (multiple `functionCall` parts in one candidate). |
1196 +| `reasoning_effort` | `generationConfig.thinkingConfig` | Gemini 2.5: `low→thinkingBudget 1024`, `medium→8192`, `high→24576`; Gemini 3+: `thinkingLevel: "low"/"medium"/"high"`. (This is Google's own mapping in their OpenAI-compat layer.) Add `includeThoughts: true` when the client opted into reasoning output. |
1197 +| `stream` | endpoint choice | `stream:true` → `:streamGenerateContent?alt=sse`. |
1198 +
1199 +##### Content-part map
1200 +
1201 +| OpenAI part | Gemini part |
1202 +|---|---|
1203 +| `{"type":"text","text":T}` | `{"text": T}` |
1204 +| `image_url` with `data:` URL | `{"inlineData": {"mimeType": "image/png", "data": "<base64>"}}` |
1205 +| `image_url` with `https://` URL | **Gemini cannot fetch arbitrary URLs** (`fileData.fileUri` requires the Google File API). Router policy: download the image itself (size-capped, e.g. 20 MB) and convert to `inlineData`; on failure return an OpenAI 400 error naming the URL. |
1206 +
1207 +##### Tool round-trip (functionCall / functionResponse)
1208 +
1209 +OpenAI assistant `tool_calls` → Gemini model turn:
1210 +
1211 +```json
1212 +{"role": "model", "parts": [
1213 + {"functionCall": {"id": "call_abc123", "name": "get_weather",
1214 + "args": {"location": "Paris"}}}
1215 +]}
1216 +```
1217 +
1218 +(`args` is a parsed object — parse the OpenAI `arguments` string. `functionCall.id` / `functionResponse.id` exist in current v1beta for parallel-call matching; include them when the OpenAI ids are available.)
1219 +
1220 +OpenAI `tool` message → Gemini user turn:
1221 +
1222 +```json
1223 +{"role": "user", "parts": [
1224 + {"functionResponse": {"id": "call_abc123", "name": "get_weather",
1225 + "response": {"result": "18°C, sunny"}}}
1226 +]}
1227 +```
1228 +
1229 +Two traps:
1230 +1. **`functionResponse.response` must be a JSON OBJECT.** OpenAI tool content is a string → if it parses as a JSON object, pass it; otherwise wrap as `{"result": <string>}`.
1231 +2. **`name` is required**, but OpenAI `tool` messages carry only `tool_call_id`. The router must resolve `tool_call_id → name` from the preceding assistant message's `tool_calls` in the same request payload (always available in a well-formed OpenAI conversation).
1232 +3. **Thought signatures (Gemini 3+):** function-call parts may carry a `thoughtSignature` that should be echoed back on the following turn. Preserve it via `reasoning_details` (§3.4) and re-attach when translating the conversation back.
1233 +
1234 +##### Side-by-side minimal request
1235 +
1236 +```json
1237 +// OpenAI in // Gemini out
1238 +{ {
1239 + "model": "gemini/gemini-2.5-flash", // POST …/models/gemini-2.5-flash:generateContent
1240 + "messages": [ "systemInstruction": {"parts":[{"text":"Be brief."}]},
1241 + {"role":"system","content":"Be brief."}, "contents": [
1242 + {"role":"user","content":"Hi"}, {"role":"user","parts":[{"text":"Hi"}]},
1243 + {"role":"assistant","content":"Hello!"}, {"role":"model","parts":[{"text":"Hello!"}]},
1244 + {"role":"user","content":"Name a color"} {"role":"user","parts":[{"text":"Name a color"}]}
1245 + ], ],
1246 + "temperature": 0.7, "generationConfig": {
1247 + "max_tokens": 100, "temperature": 0.7,
1248 + "stop": ["\n\n"] "maxOutputTokens": 100,
1249 +} "stopSequences": ["\n\n"]
1250 + }
1251 + }
1252 +```
1253 +
1254 +#### 3.2.3 Response translation (Gemini → OpenAI)
1255 +
1256 +Gemini response:
1257 +
1258 +```json
1259 +{
1260 + "candidates": [{
1261 + "content": {"role": "model", "parts": [
1262 + {"functionCall": {"name": "get_weather", "args": {"location": "Paris"}}}
1263 + ]},
1264 + "finishReason": "STOP",
1265 + "index": 0,
1266 + "safetyRatings": [ … ]
1267 + }],
1268 + "usageMetadata": {
1269 + "promptTokenCount": 57, "candidatesTokenCount": 12,
1270 + "thoughtsTokenCount": 88, "totalTokenCount": 157
1271 + },
1272 + "modelVersion": "gemini-2.5-flash"
1273 +}
1274 +```
1275 +
1276 +Router emits:
1277 +
1278 +```json
1279 +{
1280 + "id": "chatcmpl-7be2f0c4",
1281 + "object": "chat.completion",
1282 + "created": 1753872000,
1283 + "model": "gemini/gemini-2.5-flash",
1284 + "choices": [{
1285 + "index": 0,
1286 + "message": {"role": "assistant", "content": null,
1287 + "tool_calls": [{"id": "call_9d1e2f3a", "type": "function",
1288 + "function": {"name": "get_weather", "arguments": "{\"location\":\"Paris\"}"}}]},
1289 + "finish_reason": "tool_calls"
1290 + }],
1291 + "usage": {
1292 + "prompt_tokens": 57,
1293 + "completion_tokens": 100,
1294 + "total_tokens": 157,
1295 + "completion_tokens_details": {"reasoning_tokens": 88}
1296 + }
1297 +}
1298 +```
1299 +
1300 +Rules:
1301 +
1302 +- Concatenate `parts[].text` (where `part.thought != true`) → `content`; parts with `"thought": true` → `reasoning_content`; each `functionCall` part → a `tool_calls[]` entry. **Gemini often omits ids** → synthesize `call_<12-hex>` (and remember name↔id for the return trip).
1303 +- **CRITICAL finish_reason rule:** Gemini reports `finishReason: "STOP"` even when the candidate contains `functionCall` parts. The router must emit `finish_reason: "tool_calls"` whenever any functionCall part is present, regardless of `finishReason`.
1304 +
1305 +| Gemini `finishReason` | OpenAI `finish_reason` |
1306 +|---|---|
1307 +| `STOP` (with functionCall parts) | `tool_calls` |
1308 +| `STOP` | `stop` |
1309 +| `MAX_TOKENS` | `length` |
1310 +| `SAFETY`, `PROHIBITED_CONTENT`, `BLOCKLIST`, `SPII`, `IMAGE_SAFETY` | `content_filter` |
1311 +| `RECITATION` | `content_filter` (recitation = copyright block) |
1312 +| `MALFORMED_FUNCTION_CALL` | map to a **502-style OpenAI error** on non-streaming (the candidate is unusable); on streaming, emit finish_reason `stop` + log |
1313 +| `LANGUAGE`, `OTHER`, unknown | `stop` (+ log the raw value) |
1314 +
1315 +- **Blocked prompts:** if `candidates` is empty and `promptFeedback.blockReason` is set (`SAFETY`, `BLOCKLIST`, `PROHIBITED_CONTENT`, `OTHER`, `IMAGE_SAFETY`), return an OpenAI 400 `invalid_request_error` with a clear message naming the block reason — never an empty 200.
1316 +- **usage:** `prompt_tokens = promptTokenCount`; `completion_tokens = candidatesTokenCount + thoughtsTokenCount` (OpenAI counts reasoning inside completion tokens); `total_tokens = totalTokenCount`; `completion_tokens_details.reasoning_tokens = thoughtsTokenCount`; `prompt_tokens_details.cached_tokens = cachedContentTokenCount`.
1317 +
1318 +#### 3.2.4 Streaming (`streamGenerateContent?alt=sse`) → OpenAI chunks
1319 +
1320 +Each SSE `data:` line is a **complete `GenerateContentResponse`** whose `candidates[0].content.parts` holds the *increment* — there is no delta envelope and **no `[DONE]` terminator** (the stream simply ends after the chunk carrying the final `finishReason`). `usageMetadata` appears on chunks with cumulative counts; the last chunk has the authoritative totals.
1321 +
1322 +```
1323 +GEMINI SSE → ROUTER EMITS
1324 +
1325 +data: {"candidates":[{"content":{"parts":[{"text":"The"}], → chunk 1 (synthesized role first):
1326 + "role":"model"},"index":0}], data: {…,"delta":{"role":"assistant","content":""},…}
1327 + "usageMetadata":{…},"modelVersion":"gemini-2.5-flash"} data: {…,"delta":{"content":"The"},"finish_reason":null}
1328 +
1329 +data: {"candidates":[{"content":{"parts":[{"text": → data: {…,"delta":{"content":" sky is blue."},
1330 + " sky is blue."}],"role":"model"},"index":0}],…} "finish_reason":null}
1331 +
1332 +data: {"candidates":[{"content":{"parts":[],"role": → data: {…,"delta":{},"finish_reason":"stop"}
1333 + "model"},"finishReason":"STOP","index":0}], → data: {…,"choices":[],"usage":{"prompt_tokens":8,
1334 + "usageMetadata":{"promptTokenCount":8, "completion_tokens":5,"total_tokens":13}}
1335 + "candidatesTokenCount":5,"totalTokenCount":13}} (if include_usage)
1336 +(stream ends — no [DONE] from Gemini) → data: [DONE] (router ALWAYS adds it)
1337 +```
1338 +
1339 +Rules:
1340 +- **Synthesize the role chunk**: Gemini has no role-only first frame; the router emits `{"role":"assistant","content":""}` before the first content delta.
1341 +- **Tool calls are NOT argument-streamed**: a `functionCall` part arrives complete in one chunk → emit ONE `tool_calls` delta containing `index`, generated `id`, `name`, and the FULL `arguments` string; strict SDKs accept whole-argument single deltas fine.
1342 +- Thought parts (`"thought": true`) → `reasoning_content` deltas.
1343 +- `finishReason` on the final chunk → the finish_reason chunk (apply the functionCall→`tool_calls` override).
1344 +- Router appends the OpenAI usage chunk (empty `choices`) and `data: [DONE]` itself.
1345 +- If Gemini aborts mid-stream with `finishReason: SAFETY`, emit `finish_reason: "content_filter"` and terminate normally.
1346 +
1347 +---
1348 +
1349 +### 3.3 OpenAI-compatible providers — deviation table
1350 +
1351 +All nine below speak the OpenAI chat/completions wire format closely enough for **near-pass-through**: the router's `CompatAdjuster` only needs a per-provider strip/rename/allow table plus finish/usage normalization. Auth is `Authorization: Bearer <key>` for all of them. (OpenAI itself, `api.openai.com/v1`, is the reference and needs no adjustment.)
1352 +
1353 +#### Summary matrix
1354 +
1355 +| Provider | Base URL | Strip / rename | Extra params to allow (pass-through) | Quirks |
1356 +|---|---|---|---|---|
1357 +| **xAI** | `https://api.x.ai/v1` | For Grok-4-family reasoning models: **strip `presence_penalty`, `frequency_penalty`, `stop`** (they 400, not ignore). Strip `reasoning_effort` on models that reject it. | `reasoning_effort` (model-gated: grok-3-mini; grok-4.3 `none/low/medium/high`; grok-4.5 `low/medium/high` only), `search_parameters` (Live Search), `deferred` | grok-3-mini returns `message.reasoning_content`; grok-4 does NOT expose reasoning content, only `usage.completion_tokens_details.reasoning_tokens`. Vision via standard `image_url` (jpeg/png, ≤20 MiB). Structured outputs supported. Chat Completions is now labeled a "legacy" endpoint (Responses API is primary) but remains fully supported. |
1358 +| **Mistral** | `https://api.mistral.ai/v1` | **Rename `seed` → `random_seed`.** Strip `logit_bias`, `user`, `logprobs`. | `safe_prompt` (bool), `prompt_mode: "reasoning"`, `prediction`, `prompt_cache_key`, tool_choice value `"any"` | `tool_choice` accepts `auto/none/any/required` (`any`≈`required`). Temperature recommended 0–0.7. `response_format` supports `json_object` AND `json_schema`. Magistral reasoning models return `message.content` as an ARRAY of chunks: `{"type":"thinking","thinking":[{"type":"text","text":…}]}` + `{"type":"text","text":…}` — router must flatten: thinking chunks → `reasoning_content`, text chunks → `content` (same in streaming deltas, which shape-shift between array and string). SSE ends with `[DONE]`. |
1359 +| **DashScope / Qwen (intl)** | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | Strip `logit_bias`. `n` forced to 1 when `tools` present. | `enable_thinking` (bool), `thinking_budget` (int) — both extra-body; `translation_options` | Hybrid-thinking models (qwen3/qwen-plus) emit `delta.reasoning_content` then `delta.content`. **Some open-source thinking models are streaming-only** (non-streaming call errors) → router should transparently stream-and-aggregate when a client asks non-streaming. `stream_options.include_usage` supported (usage in final chunk). Vision (qwen-vl) uses standard `image_url` parts. Newer qwen models enable thinking by default — set `enable_thinking:false` when the client didn't ask for reasoning. |
1360 +| **DeepSeek** | `https://api.deepseek.com` (alias `…/v1`) | For reasoner/thinking mode: `temperature`, `top_p`, `presence_penalty`, `frequency_penalty` are **silently ignored** (no strip needed, but don't pretend they work); `logprobs`/`top_logprobs` error → strip. **Strip `reasoning_content` from incoming assistant messages** except in tool-call loops (see quirks). | `thinking: {"type":"enabled"/"disabled"}` (extra-body), `reasoning_effort` | `message.reasoning_content` + `delta.reasoning_content` (the convention we adopt, §3.4). Multi-turn: reasoning_content must NOT be resent in ordinary turns, but in TOOL-CALL loops it **must be passed back** or the API 400s (current docs). JSON mode: `response_format {"type":"json_object"}` requires the word "json" in the prompt — router auto-appends an instruction if missing. Tools supported. Usage includes `prompt_cache_hit_tokens` / `prompt_cache_miss_tokens` → map hit tokens to `prompt_tokens_details.cached_tokens`. |
1361 +| **Kimi / Moonshot** | `https://api.moonshot.ai/v1` | **Clamp `temperature` to [0,1]** (out-of-range 400s; default 0.0). `stop` max 5 sequences × 32 bytes → truncate/reject. K3 reasoning model: strip `temperature`, `top_p`, penalties. | `partial: true` on last assistant message (prefill mode), `thinking: {"type":…}`, `reasoning_effort` (`low/high/max` on K3), `thinking.keep` | `n` 1–5. `response_format`: `json_object` and `json_schema` (strict). Vision + video via `image_url`/`video_url` (base64 or `ms://<file_id>`). Thinking models return `reasoning_content` (message + delta). `stream_options.include_usage` → usage in final pre-`[DONE]` chunk. |
1362 +| **Perplexity** | `https://api.perplexity.ai` (`POST /chat/completions`; docs now also expose a gateway at `/router/v1/chat/completions`) | Strip `frequency_penalty`/`presence_penalty`/`top_k` if absent from current schema; **reject `tools`** on Sonar search models (no function calling) with a clear OpenAI error. Messages after `system` must **strictly alternate user/assistant** — merge consecutive same-role messages like the Anthropic path. | `search_mode` (`web/academic/sec`), `search_domain_filter`, `search_recency_filter`, `search_after_date_filter`/`search_before_date_filter`, `web_search_options` (`search_context_size`, `user_location`), `return_images`, `return_related_questions`, `disable_search`, `reasoning_effort`, `language_preference` | Response carries top-level `citations: [urls]` and `search_results: [{title,url,snippet,date}]` — the router passes these through verbatim as extension keys on the normalized response (and on the final stream chunk). `usage` extras: `citation_tokens`, `num_search_queries`, `reasoning_tokens`, `cost{…}` → keep under usage extension, fold `reasoning_tokens` into `completion_tokens_details`. sonar-reasoning models emit `<think>…</think>` inside `content` → router extracts to `reasoning_content`. |
1363 +| **Together** | `https://api.together.xyz/v1` | Nothing mandatory to strip. | `top_k`, `min_p`, `repetition_penalty`, `safety_model`, `echo`, `context_length_exceeded_behavior`, `chat_template_kwargs` | **finish_reason may be `"eos"`** → normalize to `"stop"` (also normalize `function_call`→`tool_calls` if seen). Usage appears in the final chunk (and on some models in every chunk) — always take the LAST non-null usage. `json_schema` support is model-dependent → 400s surface as OpenAI errors. Open-weights models vary in tool-calling quality; nothing structural to translate. |
1364 +| **DeepInfra** | `https://api.deepinfra.com/v1/openai` | Strip `logit_bias` (unsupported on most models). | `min_p`, `repetition_penalty`, `service_tier` (`priority`/`flex`), `fail_fast` | Usage object includes non-standard **`estimated_cost`** (USD) → feed it straight into `UsageMeter` as authoritative cost when present. Final stream chunk carries usage. Self-described as "not 100% compatible with all OpenAI parameters" — treat unknown-param 400s as strippable and retry once without extras. |
1365 +| **Cerebras** | `https://api.cerebras.ai/v1` | **Vision: base64 data-URI images only — remote `image_url` http(s) URLs are rejected** → router inlines (download + base64) or rejects with a clear error. `response_format {"type":"json_object"}` is **incompatible with streaming** → reject that combination or fall back to json_schema. | `reasoning_effort` (`low/medium/high/none`), `clear_thinking` (Cerebras-specific: drop prior-turn reasoning), `service_tier`, `prediction`, `prompt_cache_key` | Extremely high tokens/s — the SSE writer must handle very fast chunk cadence (backpressure!). Usage included in stream. Extra response fields `time_info` (queue/prompt/completion latencies) and `service_tier_used` → log, don't forward. `max_completion_tokens` preferred name. Supports `logprobs`, penalties, `logit_bias` per current docs (verify in Phase 7). |
1366 +
1367 +Sources: xAI — https://docs.x.ai/developers/model-capabilities/legacy/chat-completions, https://www.promptfoo.dev/docs/providers/xai/ ; Mistral — https://docs.mistral.ai/api/ , https://docs.mistral.ai/capabilities/reasoning/ ; DashScope — https://www.alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope , https://www.alibabacloud.com/help/en/model-studio/deep-thinking ; DeepSeek — https://api-docs.deepseek.com/guides/thinking_mode ; Kimi — https://platform.kimi.ai/docs/api/chat (platform.moonshot.ai redirects here; API host remains api.moonshot.ai) ; Perplexity — https://docs.perplexity.ai/api-reference/chat-completions-post , https://docs.perplexity.ai/getting-started/quickstart ; Together — https://docs.together.ai/reference/chat-completions-1 ; DeepInfra — https://docs.deepinfra.com/chat/overview ; Cerebras — https://inference-docs.cerebras.ai/api-reference/chat-completions .
1368 +
1369 +#### Universal CompatAdjuster rules
1370 +
1371 +1. **Unknown-parameter resilience:** providers split between *ignore-unknown* (DeepSeek, Together) and *400-on-unknown* (xAI reasoning models, Moonshot on bad ranges). The per-provider strip table is authoritative; additionally, on a 400 whose message names a parameter, retry ONCE with that parameter removed, then surface the error.
1372 +2. **`max_tokens` vs `max_completion_tokens`:** accept both from clients; send whichever the provider documents (`max_completion_tokens` for Cerebras/Moonshot-K-series; `max_tokens` elsewhere). Never send both.
1373 +3. **Streaming usage:** request `stream_options: {"include_usage": true}` upstream wherever supported (DashScope, Moonshot, DeepSeek, Together, DeepInfra, Cerebras, xAI, Mistral); when the upstream can't provide usage, estimate tokens locally and flag `"x-zyquo": {"usage_estimated": true}`.
1374 +4. **finish_reason normalization set:** everything must land in `stop | length | tool_calls | content_filter`; map `eos→stop`, provider-specific refusal/safety values → `content_filter`, anything unknown → `stop` + log.
1375 +5. **Keep-alive noise:** some upstreams emit SSE comment lines (`: keep-alive`) or `ping`-ish frames — the SSE parser must skip comment lines and blank frames without erroring, and the router's own SSEWriter may emit comments to keep client sockets alive during long thinking phases.
1376 +
1377 +---
1378 +
1379 +### 3.4 Reasoning content — normalization decision
1380 +
1381 +**Decision: normalize on DeepSeek's `reasoning_content` convention** — a sibling of `content` on both the non-streaming `message` and the streaming `delta`:
1382 +
1383 +```json
1384 +// non-streaming // streaming
1385 +"message": { "delta": {
1386 + "role": "assistant", "reasoning_content": "Let me check…"
1387 + "reasoning_content": "Let me check…", }
1388 + "content": "The answer is 21."
1389 +}
1390 +```
1391 +
1392 +Rationale: it is the oldest and most widely recognized wire convention (DeepSeek-R1 era) — Qwen/DashScope, Moonshot/Kimi, and xAI grok-3-mini already emit exactly this field, so most client tooling (chat UIs, LangChain, Continue, aider, etc.) knows to render it and to NOT confuse it with `content`. OpenRouter's richer `reasoning` + `reasoning_details` model (https://openrouter.ai/docs/use-cases/reasoning-tokens) is adopted only as a *supplement*: the router additionally emits `reasoning_details` (array of provider-native structured blocks) when signatures/opacity must round-trip.
1393 +
1394 +#### Per-provider mapping into `reasoning_content`
1395 +
1396 +| Provider | Native form | → Router normalization |
1397 +|---|---|---|
1398 +| DeepSeek | `message.reasoning_content` / `delta.reasoning_content` | Pass through unchanged. |
1399 +| Qwen/DashScope | same field | Pass through. |
1400 +| Kimi/Moonshot | same field | Pass through. |
1401 +| xAI grok-3-mini | same field | Pass through. (grok-4: nothing exposed — only `reasoning_tokens` in usage.) |
1402 +| Anthropic | `thinking` content blocks; streaming `thinking_delta` (+ `signature_delta`, `redacted_thinking`) | Thinking text → `reasoning_content`; signature + redacted blocks → `reasoning_details: [{"type":"anthropic.thinking_signature",…}]`. |
1403 +| Gemini | parts with `"thought": true` (needs `thinkingConfig.includeThoughts`); `thoughtSignature` on parts | Thought text → `reasoning_content`; `thoughtSignature` → `reasoning_details: [{"type":"gemini.thought_signature",…}]`. |
1404 +| Mistral Magistral | content chunk `{"type":"thinking",…}` inside the content array | Flatten to `reasoning_content`; text chunks → `content`. |
1405 +| Perplexity sonar-reasoning | `<think>…</think>` prefix inside `content` | Extract tags → `reasoning_content`; strip from `content`. |
1406 +| Cerebras (reasoning models) | model-dependent (`reasoning` field or `<think>` tags per hosted model) | Same extraction pipeline; verify per model in Phase 7. |
1407 +
1408 +#### Usage normalization
1409 +
1410 +All reasoning token counts land in the OpenAI-standard `usage.completion_tokens_details.reasoning_tokens` (Anthropic `thinking_tokens`, Gemini `thoughtsTokenCount`, xAI/Perplexity `reasoning_tokens`), and reasoning tokens are INCLUDED in `completion_tokens` (OpenAI semantics).
1411 +
1412 +#### Request-side control
1413 +
1414 +The router accepts the OpenAI-standard **`reasoning_effort`** (`"minimal"|"low"|"medium"|"high"`, plus provider extras like `"none"`/`"max"`) and translates per provider: Anthropic → `output_config.effort` / `thinking.budget_tokens`; Gemini → `thinkingConfig.thinkingBudget`/`thinkingLevel` (Google's own compat mapping: low=1024, medium=8192, high=24576 on 2.5-series); DeepSeek/Kimi/Cerebras/xAI/Perplexity → pass `reasoning_effort` through (gated by catalog capability); Qwen → `enable_thinking:true` (+ `thinking_budget`); Mistral → `prompt_mode:"reasoning"`. On models with no reasoning capability, `reasoning_effort` is stripped (never 400 the client for asking).
1415 +
1416 +#### Echo-back rules (multi-turn)
1417 +
1418 +Incoming assistant messages may contain `reasoning_content`/`reasoning_details` from prior router responses. Before forwarding:
1419 +- **Strip `reasoning_content`** for all providers by default (DeepSeek 400s in plain turns if it leaks into context via unknown-field-strict paths; others ignore it but it wastes tokens),
1420 +- **except**: DeepSeek tool-call loops (must be passed back per current docs), Anthropic thinking+tools loops (reconstruct `thinking` blocks with signatures from `reasoning_details`), Gemini 3 (re-attach `thoughtSignature`), Mistral Magistral (replay ThinkChunk to preserve the trace), Kimi with `thinking.keep`.
1421 +This asymmetry is exactly why `reasoning_details` exists: it carries the provider-native, signed material that some upstreams demand back, while `reasoning_content` stays a clean display string.
1422 +
1423 +---
1424 +
1425 +### 3.5 Source index
1426 +
1427 +- Anthropic Messages API: https://platform.claude.com/docs/en/api/messages
1428 +- Anthropic streaming events: https://platform.claude.com/docs/en/docs/build-with-claude/streaming
1429 +- Anthropic errors: https://platform.claude.com/docs/en/api/errors
1430 +- Gemini generateContent reference: https://ai.google.dev/api/generate-content
1431 +- Gemini Part/Content schema: https://ai.google.dev/api/caching#Part
1432 +- Gemini function calling: https://ai.google.dev/gemini-api/docs/function-calling
1433 +- Gemini OpenAI-compat layer (mapping oracle): https://ai.google.dev/gemini-api/docs/openai
1434 +- xAI chat completions: https://docs.x.ai/developers/model-capabilities/legacy/chat-completions ; parameter-rejection field notes: https://www.promptfoo.dev/docs/providers/xai/ , https://github.com/vercel/ai/issues/12826
1435 +- Mistral API: https://docs.mistral.ai/api/ ; reasoning: https://docs.mistral.ai/capabilities/reasoning/
1436 +- DashScope OpenAI compat: https://www.alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope ; deep thinking: https://www.alibabacloud.com/help/en/model-studio/deep-thinking
1437 +- DeepSeek thinking mode: https://api-docs.deepseek.com/guides/thinking_mode
1438 +- Kimi/Moonshot chat API: https://platform.kimi.ai/docs/api/chat
1439 +- Perplexity chat completions: https://docs.perplexity.ai/api-reference/chat-completions-post ; quickstart: https://docs.perplexity.ai/getting-started/quickstart
1440 +- Together chat completions: https://docs.together.ai/reference/chat-completions-1
1441 +- DeepInfra OpenAI API: https://docs.deepinfra.com/chat/overview
1442 +- Cerebras chat completions: https://inference-docs.cerebras.ai/api-reference/chat-completions
1443 +- OpenRouter reasoning normalization (prior art): https://openrouter.ai/docs/use-cases/reasoning-tokens
1444 +## 4. HTTP server in Swift without heavyweight deps
1445 +
1446 +Research date: 2026-07-30. Target: an embedded HTTP/1.1 server inside a SwiftUI macOS 13+ app (SPM, no Xcode IDE) serving an OpenAI-compatible API on `http://localhost:<port>`, with spec-exact SSE streaming, long-lived streams, client-disconnect → upstream-cancellation, and graceful shutdown.
1447 +
1448 +### 4.1 Options evaluated
1449 +
1450 +#### Option A — SwiftNIO directly (NIOCore + NIOPosix + NIOHTTP1 + NIOExtras)
1451 +
1452 +- **State as of mid-2026:** swift-nio is at **2.101.3** (released ~2026-07-23), actively maintained by Apple, compatible with Swift **6.0–6.3** and strict concurrency. NIO 3 is expected "sometime around Swift 6" per the Swift.org server guidelines, with NIO 2 continuing to receive bug fixes afterwards — NIO 2.x is a safe multi-year foundation. Sources: [swift-nio releases](https://github.com/apple/swift-nio/releases), [Swift Package Index — swift-nio](https://swiftpackageindex.com/apple/swift-nio), [Swift.org concurrency adoption guidelines](https://www.swift.org/documentation/server/guides/libraries/concurrency-adoption-guidelines.html).
1453 +- **Structured concurrency:** modern NIO exposes **`NIOAsyncChannel`**, which "abstracts the notion of a NIO `Channel` into something that can safely be used in a structured concurrency context". The recommended split: protocol-specific logic (HTTP parsing/encoding via `configureHTTPServerPipeline`) stays as `ChannelHandler`s; business logic consumes/produces via the `NIOAsyncChannel` inbound `AsyncSequence` / outbound writer. `executeThenClose` scopes the channel's lifetime to a closure — the channel closes when the closure returns, which maps perfectly onto "one inbound request = one cancellable `Task`". Sources: [NIOAsyncChannel docs](https://swiftinit.org/docs/swift-nio/niocore/nioasyncchannel), [NIO public async APIs](https://github.com/apple/swift-nio/blob/main/docs/public-async-nio-apis.md), [executeThenClose discussion](https://forums.swift.org/t/nioasyncchannel-executethenclose-is-too-restrictive/73460), [Using SwiftNIO — Channels](https://swiftonserver.com/using-swiftnio-channels/), [Building a web app with only SwiftNIO (2026)](https://blog.alexseifert.com/2026/06/29/building-a-web-app-in-swift-using-only-swiftnio/).
1454 +- **Reference server:** Apple's `NIOHTTP1Server` example shows the canonical `ServerBootstrap` setup — `backlog: 256`, `so_reuseaddr` on server and child channels, `configureHTTPServerPipeline(withErrorHandling: true)`, explicit keep-alive state machine (idle → waiting-for-body → sending-response) and adding `Connection: close`/`keep-alive` headers for HTTP/1.0 or explicit-close requests. (The example itself is future-based; we use the NIOAsyncChannel equivalent.) Source: [NIOHTTP1Server main.swift](https://github.com/apple/swift-nio/blob/main/Sources/NIOHTTP1Server/main.swift).
1455 +- **Graceful shutdown:** `swift-nio-extras` ships **`ServerQuiescingHelper`** — "helps to quiesce a server by notifying user code when all previously open connections have closed"; call `initiateShutdown(promise:)` to stop accepting and drain. There is a full demo (`HTTPServerWithQuiescingDemo`). Sources: [QuiescingHelper.swift](https://github.com/apple/swift-nio-extras/blob/main/Sources/NIOExtras/QuiescingHelper.swift), [HTTPServerWithQuiescingDemo](https://github.com/apple/swift-nio-extras/blob/main/Sources/HTTPServerWithQuiescingDemo/main.swift).
1456 +- **Weight:** NIOCore/NIOPosix/NIOHTTP1 (+ optionally NIOExtras) — one Apple-maintained dependency tree, no routing framework, no ServiceLifecycle/Logging/Metrics transitive stack. Full control over every byte of the SSE wire format.
1457 +- **Cost:** we write our own tiny router (we have ~5 routes), our own request-body accumulation with a size limit, and our own SSE writer. For this app that's a feature: the OpenAI chunk stream must be byte-exact, and owning `SSEWriter` end-to-end removes a framework abstraction between us and the wire.
1458 +
1459 +#### Option B — Network.framework (`NWListener` + `NWProtocolFramer`)
1460 +
1461 +`NWListener` replaces the BSD bind/listen/accept sequence, but HTTP itself must be brought along: either a custom `NWProtocolFramer` (real boilerplate via `NWProtocolFramerImplementation`) or hand-wiring a C parser (`http_parser.c`) onto raw connections — Helge Heß's NWHTTPProtocol does exactly that and its author notes that "for production use it's suggested to not use a protocol framer for HTTP" and to hook the parser up directly instead. No HTTP/1.1 pipeline, no chunked-encoding helpers, no keep-alive management, no quiescing utilities — all DIY. It buys nothing over NIO for a localhost server (its strengths are Wi-Fi/cellular path handling and Bonjour, irrelevant here). Sources: [NWHTTPProtocol](https://github.com/helje5/NWHTTPProtocol), [Intro to Network.framework servers](http://www.alwaysrightinstitute.com/network-framework/), [Apple Network framework docs](https://developer.apple.com/documentation/network). **Rejected.**
1462 +
1463 +#### Option C — Hummingbird 2
1464 +
1465 +The strongest framework candidate. Built from scratch on SwiftNIO with Swift concurrency central; at **2.25.1** as of July 2026; Swift 6.1+; "designed to require the minimum number of dependencies". It has first-class pieces we'd otherwise hand-roll: a `ResponseBodyWriter` closure body with backpressure-aware `await writer.write(...)`, a `ServerSentEvent` type, `consumeWithInboundCloseHandler` for client-disconnect detection, and graceful shutdown via **swift-service-lifecycle** ("currently running requests continue being handled, while new connections and requests will not be accepted"), started with `Application.runService(gracefulShutdownSignals:)`. There is a dedicated `server-sent-events` example. Sources: [hummingbird repo](https://github.com/hummingbird-project/hummingbird), [SPI — hummingbird](https://swiftpackageindex.com/hummingbird-project/hummingbird), [What's new in Hummingbird 2](https://swiftonserver.com/whats-new-in-hummingbird-2/), [Hummingbird 2 announcement](https://hummingbird.codes/news/hummingbird-2/), [SSE example](https://github.com/hummingbird-project/hummingbird-examples), [swift-service-lifecycle](https://github.com/swift-server/swift-service-lifecycle).
1466 +
1467 +Why not choose it: (1) it drags in ServiceLifecycle/Logging/Metrics/Tracing abstractions designed for long-running server binaries with signal-driven lifecycles, whereas our lifecycle is owned by a SwiftUI app's Start/Stop button — bridging `runService` into an app-owned start/stop adds friction rather than removing it; (2) its router/middleware/extractor machinery is overhead for ~5 fixed routes; (3) an extra abstraction layer sits between us and the SSE bytes, and byte-exact OpenAI chunk emission is the core deliverable. Its SSE example is nonetheless the best public reference for the disconnect/shutdown patterns we will reimplement on raw NIO (see 4.2).
1468 +
1469 +#### Option D — Vapor
1470 +
1471 +Batteries-included (HTTP/2, TLS, auth, validation, WebSockets, …) and correspondingly heavy: bulks the executable, increases compile time, ~20–30 MB idle memory vs Hummingbird's ~5–10 MB, and a large transitive dependency graph. Everything it adds over NIO is something this app doesn't need. Sources: [Hummingbird vs Vapor discussion](https://github.com/hummingbird-project/hummingbird/discussions/150), [Beginner's guide to Hummingbird](https://theswiftdev.com/beginners-guide-to-server-side-swift-using-the-hummingbird-framework/). **Rejected.**
1472 +
1473 +#### Decision
1474 +
1475 +**SwiftNIO directly** (`NIOCore`, `NIOPosix`, `NIOHTTP1`, plus `NIOExtras` for `ServerQuiescingHelper`), using the `NIOAsyncChannel` structured-concurrency APIs. Rationale: Apple-maintained, SPM-clean, one dependency tree, macOS 13+ fine, Swift 6 strict-concurrency ready, full control over SSE emission/flushing/backpressure, first-class quiescing for graceful shutdown, and a natural one-request-one-`Task` model so client disconnect cancels the upstream call structurally. Hummingbird 2 is the documented fallback if raw-NIO plumbing proves costlier than expected — the migration path is easy since both are NIO underneath.
1476 +
1477 +### 4.2 Implementation specifics (SwiftNIO)
1478 +
1479 +**Bootstrap and binding.**
1480 +
1481 +```swift
1482 +let group = MultiThreadedEventLoopGroup.singleton
1483 +let quiesce = ServerQuiescingHelper(group: group)
1484 +
1485 +let serverChannel = try await ServerBootstrap(group: group)
1486 + .serverChannelOption(ChannelOptions.backlog, value: 256)
1487 + .serverChannelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)
1488 + .serverChannelInitializer { channel in
1489 + channel.pipeline.addHandler(quiesce.makeServerChannelHandler(channel: channel))
1490 + }
1491 + .childChannelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)
1492 + .bind(host: bindHost, port: port) { channel in
1493 + channel.eventLoop.makeCompletedFuture {
1494 + try channel.pipeline.syncOperations.configureHTTPServerPipeline(withErrorHandling: true)
1495 + return try NIOAsyncChannel<HTTPServerRequestPart, HTTPServerResponsePart>(
1496 + wrappingChannelSynchronously: channel)
1497 + }
1498 + }
1499 +```
1500 +
1501 +- **Bind host:** `"127.0.0.1"` by default (localhost only, unreachable from the LAN); `"0.0.0.0"` only when the user explicitly enables LAN exposure (which per our security policy forces a local API key). This is a plain string parameter to `bind(host:port:)` — no extra API.
1502 +- **Port-in-use:** `bind` throws; catch `IOError`/`NIOBSDSocket` errors and check `errno == EADDRINUSE` (48 on Darwin) → surface "Port 8787 is already in use" and probe upward (`try bind` on port+1, +2, …) to *suggest* the next free port (never silently switch). `EACCES` (ports < 1024 without privileges) gets its own message. `SO_REUSEADDR` on the server channel avoids spurious `EADDRINUSE` from sockets lingering in `TIME_WAIT` after a quick Stop→Start; note it does **not** let two live listeners share a port — a genuinely occupied port still fails, which is what we want. Sources: [ServerBootstrap docs](https://swiftinit.org/docs/swift-nio/nioposix/serverbootstrap), [Bind: address already in use](https://hea-www.harvard.edu/~fine/Tech/addrinuse.html), [NIOHTTP1Server example options](https://github.com/apple/swift-nio/blob/main/Sources/NIOHTTP1Server/main.swift).
1503 +
1504 +**Concurrent connections with structured concurrency.** The async `bind` returns a `NIOAsyncChannel` of accepted-connection `NIOAsyncChannel`s. The serving loop:
1505 +
1506 +```swift
1507 +try await serverChannel.executeThenClose { acceptedConnections in
1508 + try await withThrowingDiscardingTaskGroup { group in
1509 + for try await connection in acceptedConnections {
1510 + group.addTask { await handleConnection(connection) } // one Task per connection
1511 + }
1512 + }
1513 +}
1514 +```
1515 +
1516 +Inside `handleConnection`, `connection.executeThenClose { inbound, outbound in ... }` gives an `AsyncSequence` of `HTTPServerRequestPart` (`.head`, `.body` buffers, `.end`) and an outbound writer for `HTTPServerResponsePart`. Each request is parsed, dispatched to `Routes`, and answered; the loop iterates for keep-alive. Cancelling the connection's `Task` tears everything down cleanly — this is the backbone of both client-disconnect handling and graceful shutdown. Sources: [NIO public async APIs](https://github.com/apple/swift-nio/blob/main/docs/public-async-nio-apis.md), [NIOAsyncChannel docs](https://swiftinit.org/docs/swift-nio/niocore/nioasyncchannel).
1517 +
1518 +**Request body size limit.** Accumulate `.body` parts into a `ByteBuffer` with a hard cap (default e.g. 20 MiB — base64 images are large; user-configurable in Settings). On overflow: respond `413` in OpenAI error format, drain remaining parts (or close), never buffer further.
1519 +
1520 +**Timeouts.**
1521 +- *Header/idle timeout:* add `IdleStateHandler` (NIOCore) ahead of the HTTP handlers, or a per-request `withTimeout` wrapper, to kill connections that never complete a request (~30 s read idle).
1522 +- *Streaming:* once a response stream has started, the idle clock must apply to *write* progress, not total duration — chat completions can legitimately stream for many minutes. Practical policy: generous upstream time-to-first-byte timeout (e.g. 120 s, configurable), then no total cap while chunks keep flowing; abort if the upstream stalls (no chunk for N seconds, e.g. 300 s).
1523 +- Remove/suspend the idle handler for the duration of an SSE response, restore for keep-alive reuse.
1524 +
1525 +**Keep-alive.** `configureHTTPServerPipeline` parses `Connection` headers; our responder mirrors the NIOHTTP1Server example: honor `keepAlive` from the request head, set `Connection: keep-alive`/`close` explicitly for HTTP/1.0, close the channel after the response when keep-alive is false. After an SSE response we send the terminating `[DONE]` and `.end`; keeping the connection alive afterwards is legal (the response used chunked encoding with a proper terminator), but closing is also acceptable — OpenAI SDKs handle both. We keep it alive (SDKs reuse connections between calls). Source: [NIOHTTP1Server main.swift](https://github.com/apple/swift-nio/blob/main/Sources/NIOHTTP1Server/main.swift).
1526 +
1527 +**SSE emission (`SSEWriter`).** Response head:
1528 +
1529 +```
1530 +HTTP/1.1 200 OK
1531 +Content-Type: text/event-stream
1532 +Cache-Control: no-cache
1533 +Connection: keep-alive
1534 +X-Accel-Buffering: no ← harmless; defeats proxy buffering if any intermediary appears
1535 +Transfer-Encoding: chunked ← added automatically by HTTPResponseEncoder when no Content-Length
1536 +```
1537 +
1538 +Each OpenAI chunk is one SSE event: `data: {json}\n\n` (UTF-8; the OpenAI format is single-line JSON per `data:` line, terminated by `data: [DONE]\n\n`). **Flush per event:** with `NIOAsyncChannel`, every `try await outbound.write(.body(.byteBuffer(eventBuffer)))` is a `writeAndFlush` — each event leaves the process immediately; no coalescing layer may sit above it. `await`-ing each write is also the **backpressure** mechanism: a slow client suspends us, which suspends consumption of the upstream `AsyncSequence`, which propagates backpressure to the upstream HTTP read. Optionally set `TCP_NODELAY` on child channels so small event frames aren't Nagle-delayed. SSE format/caching rules: [MDN — Using server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events). (Note: OpenAI's stream is consumed by SDK parsers, not browser `EventSource`, so `event:`/`id:`/`retry:` fields are never used — `data:` lines only.)
1539 +
1540 +**Client-disconnect → cancel upstream.** Two complementary signals:
1541 +1. Structured: the write to a closed socket throws (`NIOAsyncWriterError`/`channelInactive`-driven); catching it in the request `Task` must cancel the upstream streaming `Task` — with structured concurrency this is automatic if the upstream call is a child task (`withThrowingTaskGroup` per request: one child consumes upstream and writes SSE; error/cancel of either cancels the other).
1542 +2. Proactive: watch the inbound side for EOF/half-closure while streaming — the pattern Hummingbird's SSE example uses (`consumeWithInboundCloseHandler` yielding a cancel event merged with the data stream). On raw NIO: run a second child task iterating `inbound`; when the sequence ends (client closed), cancel the group. This detects disconnects *between* writes, not only on the next failed write.
1543 +Cancellation must propagate into the provider client (`URLSession`/`AsyncHTTPClient` task cancelled) so upstream token spend stops — Phase 7 verifies "no orphaned upstream usage". Sources: [Hummingbird SSE example](https://github.com/hummingbird-project/hummingbird-examples), [NIOAsyncChannel docs](https://swiftinit.org/docs/swift-nio/niocore/nioasyncchannel).
1544 +
1545 +**CORS.** Needed so browser-based tools can call the router. Implement in a small `CORS.swift`:
1546 +- Preflight: `OPTIONS` with `Origin` + `Access-Control-Request-Method` → `204` with `Access-Control-Allow-Origin: *` (default; configurable to a specific origin), `Access-Control-Allow-Methods: GET, POST, OPTIONS`, `Access-Control-Allow-Headers: Authorization, Content-Type` (or echo the requested headers), `Access-Control-Max-Age: 600`.
1547 +- Actual responses: add `Access-Control-Allow-Origin: *` (echo origin + `Access-Control-Allow-Credentials: true` only if credentials mode is ever needed — default permissive `*` for localhost tooling, per our spec).
1548 +- SSE responses need the CORS headers too (the stream is fetched cross-origin by browser clients).
1549 +
1550 +**Graceful shutdown.** On Stop:
1551 +1. `quiesce.initiateShutdown(promise:)` — the `ServerQuiescingHelper` closes the *listening* channel (no new connections) and signals when all child channels have closed. Sources: [QuiescingHelper.swift](https://github.com/apple/swift-nio-extras/blob/main/Sources/NIOExtras/QuiescingHelper.swift), [HTTPServerWithQuiescingDemo](https://github.com/apple/swift-nio-extras/blob/main/Sources/HTTPServerWithQuiescingDemo/main.swift).
1552 +2. Wait up to a drain deadline (e.g. 10 s) for in-flight non-streaming requests to finish.
1553 +3. Long-lived SSE streams won't drain on their own: after the deadline (or immediately if the user chose "stop now"), cancel the connection task group — cancellation unwinds each request task, cancels upstream calls, and closes channels.
1554 +4. Do **not** `shutdownGracefully()` the singleton `EventLoopGroup` (it's shared and Start may be pressed again); just release the server channel. Semantics mirror Hummingbird/ServiceLifecycle: "currently running requests continue being handled, while new connections and requests will not be accepted" ([swift-service-lifecycle](https://github.com/swift-server/swift-service-lifecycle), [What's new in Hummingbird 2](https://swiftonserver.com/whats-new-in-hummingbird-2/)).
1555 +
1556 +### 4.3 macOS specifics
1557 +
1558 +- **Local network privacy prompt.** Since macOS 15 Sequoia there is an iOS-style *Local Network* permission (System Settings → Privacy & Security → Local Network). Binding and serving on `127.0.0.1` does not involve the local network and triggers nothing. Interacting with LAN peers can trigger the prompt; users can later toggle it, and there are known Sequoia bugs where access silently breaks after a restart until re-toggled. When the user enables LAN mode (`0.0.0.0`) the app must explain the prompt and the Settings toggle in the UI. Also note the responsible-process rules: tools run from Terminal inherit Terminal's exemption; root daemons are auto-granted. Sources: [Apple forums — local network privacy on Sequoia](https://developer.apple.com/forums/thread/763484), [mjtsai — Local Network Privacy on Sequoia](https://mjtsai.com/blog/2024/10/02/local-network-privacy-on-sequoia/), [Foldr — macOS 15 local network privacy](https://foldr.com/foldr-support/foldr-for-macos/macos-15-sequoia-local-network-privacy/), [Panic — granting local network access](https://help.panic.com/prompt/prompt-local-network/), [access lost after restart](https://developer.apple.com/forums/thread/769037?page=2).
1559 +- **App Sandbox vs Developer ID.** `com.apple.security.network.server` ("whether your app may listen for incoming network connections") is an **App Sandbox** entitlement; a **non-sandboxed, Hardened-Runtime Developer ID app needs no entitlement to listen on a socket** — "the sandbox was designed mainly for the App Store, while the hardened runtime was designed mainly for Developer ID". Decision (consistent with Phase 8 spec): ship non-sandboxed Developer ID + Hardened Runtime; document that if we ever adopt the sandbox we must add `com.apple.security.network.server` and `.client`. Sources: [Apple — com.apple.security.network.server](https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.security.network.server), [Hardened Runtime and Sandboxing (lapcatsoftware)](https://lapcatsoftware.com/articles/hardened-runtime-sandboxing.html).
1560 +- **Terminal vs Finder launch.** From Finder the app gets the standard GUI context. From Terminal (during `make dev`), the *Terminal* process is the responsible process for TCC purposes, so privacy prompts may attribute to Terminal; also a bare executable launched from Terminal needs explicit `NSApplication` activation to come frontmost (already planned in Phase 1's entry point). Environment differs too (Terminal's shell env vs launchd's user session) — never rely on env vars for configuration of the shipped app. Source: [Apple forums — CLI tools and local network privacy](https://developer.apple.com/forums/thread/767391).
1561 +
1562 +---
1563 +
1564 +## 5. Gateway concerns
1565 +
1566 +### 5.1 Request logging with redaction
1567 +
1568 +Pattern (validated against LiteLLM's proxy design):
1569 +- **Always log metadata:** timestamp, method/path, resolved model + provider, actually-used model (post-fallback), status, latency breakdown (queue → upstream TTFB → stream duration), tokens in/out, cost, local-key ID (its name/ID — never the token), stream flag, error class. This is safe and powers the whole dashboard.
1570 +- **Bodies are opt-in.** LiteLLM's `turn_off_message_logging` redacts `messages`/`prompt` and `choices[].message.content` (and `reasoning_content`) while still tracking spend; its bug tracker shows the classic failure mode — one storage path redacted, another (raw `proxy_server_request`) not. Lesson: **redact at the single ingestion point** of `RequestLogStore`, not per-sink. Store `"[redacted]"` placeholders (plus sizes/counts, e.g. message count, image count) unless the per-session "reveal bodies" switch is on. Sources: [LiteLLM logging docs](https://docs.litellm.ai/docs/proxy/logging), [redaction bug #16336](https://github.com/BerriAI/litellm/issues/16336), [message redaction overview](https://deepwiki.com/BerriAI/litellm/6.3-message-redaction-and-privacy-controls).
1571 +- **Keys never touch the log path:** strip/replace `Authorization`, `x-api-key`, `api-key`, and any `*key*`/`*token*` header before a request object is handed to the logger; scrub upstream error bodies (some providers echo the offending header) through the same filter. Verification (Phase 7): grep all logs/exports for known key material.
1572 +- Ring buffer in memory (e.g. last 1–5k entries) + persisted store with retention setting; export honors the current redaction state.
1573 +
1574 +### 5.2 Token usage extraction per provider
1575 +
1576 +- **Prefer upstream-reported usage.** Non-streaming: OpenAI-compatible providers return `usage`; Anthropic returns `usage.input_tokens/output_tokens` (in `message_start` + `message_delta` when streaming); Gemini returns `usageMetadata`. Streaming with OpenAI-compatible upstreams: request `stream_options: {"include_usage": true}` upstream where supported so a final usage chunk arrives; some compatible providers (per-provider quirk table from Section 3) send usage on the last chunk regardless, others never do.
1577 +- **Estimate only when the upstream gives nothing**, and **flag it** — e.g. `"usage": {..., "x_zyquo_estimated": true}` (or an `x-zyquo` extension block) so cost tiles can render "≈". Estimation options in Swift: [aespinilla/Tiktoken](https://github.com/aespinilla/Tiktoken) (pure-Swift tiktoken: cl100k_base etc.) and [narner/TiktokenSwift](https://github.com/narner/TiktokenSwift) (UniFFI bindings to the real tiktoken, incl. `o200k_base`). A dependency-free fallback — `chars/4` or `words × 4/3` — is acceptable for the *flagged-estimate* path given tokenizers differ per provider anyway; decide in Phase 3 whether pulling a tokenizer dep is worth it (recommendation: start with the heuristic, keep the field flagged, add Tiktoken later if users need tight estimates).
1578 +- Cached-token counts: OpenAI reports `usage.prompt_tokens_details.cached_tokens`; Anthropic reports `cache_read_input_tokens`/`cache_creation_input_tokens`. Preserve these in the normalized usage when present (they change cost — see 5.3).
1579 +
1580 +### 5.3 Cost calculation
1581 +
1582 +- Source of truth: the ported Zyquo Cloud model catalog's per-model pricing (USD per 1M input tokens / per 1M output tokens). `cost = prompt_tokens × in_price/1e6 + completion_tokens × out_price/1e6`.
1583 +- **Cached input tokens** are billed at a discount where reported (e.g. OpenAI cached input typically 50–90% off; Anthropic cache reads at 0.1× base input, cache writes at 1.25×): when the catalog has cached pricing and the upstream reports cached counts, split: `(prompt − cached) × in_price + cached × cached_price`. Where the catalog lacks a cached rate, fall back to full input price (over-estimate, never under).
1584 +- Reasoning tokens (OpenAI `completion_tokens_details.reasoning_tokens`) are already included in `completion_tokens` — do not double-count.
1585 +- Mark costs derived from estimated usage as estimated. Store per-request cost in `UsageRecord`; aggregate per key/model/provider/day for the dashboard. Settings allow a pricing override table (providers reprice frequently) and a display currency (store USD, convert for display only).
1586 +- Precedent: OpenRouter prices "using the model that was ultimately used, which will be returned in the `model` attribute of the response body" — cost must always be computed against the *actually-used* model after fallbacks. Source: [OpenRouter model fallbacks](https://openrouter.ai/docs/guides/routing/model-fallbacks).
1587 +
1588 +### 5.4 Rate limiting per local API key
1589 +
1590 +- **Algorithm: token bucket**, the production default (AWS, Stripe) because "real traffic is bursty" — it allows short bursts up to bucket capacity while enforcing an average rate; sliding-window counters give smoother limits but punish legitimate bursts. For a local single-process gateway, in-memory is all we need (no distributed store). Sources: [Arcjet — rate limiting algorithms](https://blog.arcjet.com/rate-limiting-algorithms-token-bucket-vs-sliding-window-vs-fixed-window/), [token bucket vs sliding window](https://medium.com/@tihomir.manushev/token-bucket-vs-sliding-window-the-rate-limiting-choice-that-shapes-your-apis-behavior-e04fb2646ee5), [APISIX — gateway rate limiting](https://apisix.apache.org/learning-center/api-gateway-rate-limiting/).
1591 +- Implementation: one `actor RateLimiter` holding `keyID → (tokens: Double, lastRefill: ContinuousClock.Instant)`. Lazy refill on each check: `tokens = min(capacity, tokens + elapsed × rate)`; admit iff `tokens ≥ 1` then decrement. Per-key config: requests/min (rate = rpm/60, capacity ≈ rpm burst allowance). No timers, O(1) per request, trivially Sendable.
1592 +- On limit: `429` in OpenAI error format (`type: "rate_limit_error"`-style) with `Retry-After: ceil((1 − tokens)/rate)` seconds. Count rejections in per-key stats (Keys screen mini-chart).
1593 +- Optional second dimension later: tokens-per-minute budget (LLM-style limits) using the same bucket with token-cost withdrawal after usage is known.
1594 +
1595 +### 5.5 Retries with exponential backoff + jitter
1596 +
1597 +Consensus best practice for LLM upstreams ([Zuplo 429 guide](https://zuplo.com/learning-center/http-429-too-many-requests-guide), [handling 429s in production LLM apps](https://www.getmaxim.ai/articles/handle-429-errors-in-production-llm-applications/), [retry strategies with backoff + jitter](https://callsphere.ai/blog/retry-strategies-llm-api-calls-exponential-backoff-jitter-tenacity)):
1598 +- **Retry on:** 429, 500, 502, 503, 504, connection reset/refused, and TTFB timeout. **Never** on 400/401/403/404/422 (client/config errors — fail fast with the mapped OpenAI error).
1599 +- **Schedule:** `delay = min(cap, base × 2^attempt) + random(0, jitter)` — e.g. base 1 s, cap 30 s, full jitter; **max 3 attempts** for these interactive, user-facing requests (total budget ≤ ~30 s before fallback/error).
1600 +- **Respect `Retry-After`** when the provider sends it: `wait = max(retryAfter, computedBackoff)`; if `Retry-After` exceeds our remaining budget, skip retrying this provider and go straight to fallback/error (propagating `Retry-After` to our own 429 response).
1601 +- **Idempotency / streaming rule:** a retry is only safe **before any response byte has been forwarded to the client**. Once the first SSE chunk has been written downstream, never retry or fall back — terminate the stream with an error event/close. (Chat completions are not idempotent upstream either: a "failed" request may still have consumed tokens; retrying after partial streaming double-bills and duplicates output.) So: retries apply to (a) whole non-streaming calls, (b) streaming calls that fail before the first upstream content delta.
1602 +- Jitter exists to desynchronize concurrent clients; even locally, parallel requests from one SDK justify it.
1603 +
1604 +### 5.6 Fallback chains
1605 +
1606 +Modeled on LiteLLM and OpenRouter:
1607 +- **Semantics:** an ordered model list tried in sequence. LiteLLM: "the router tries the primary model first; if it fails with a retry-able error (429, 5xx, context-limit, content-policy, timeout), it moves to the first fallback", in order. OpenRouter: a `models: [...]` array tried in order server-side. Sources: [LiteLLM reliability/fallbacks](https://docs.litellm.ai/docs/proxy/reliability), [LiteLLM router architecture](https://docs.litellm.ai/docs/router_architecture), [OpenRouter model fallbacks](https://openrouter.ai/docs/guides/routing/model-fallbacks).
1608 +- **Trigger errors:** exhausted retries on 429/5xx/timeout; upstream auth failure (missing/invalid provider key — jumping to a provider the user *has* a key for is exactly the point); model-not-available (404 upstream). Optionally context-window errors (LiteLLM has a distinct `context_window_fallbacks` class). **Not** on content-policy 400s by default (surprising model swaps on policy errors are a footgun; make it opt-in like LiteLLM's `content_policy_fallbacks`).
1609 +- **Same rule as retries:** no fallback after the first downstream byte.
1610 +- **Honest reporting of the actually-used model** (our Phase 3 spec requirement): OpenRouter returns "the model that was ultimately used … in the `model` attribute of the response body"; LiteLLM exposes the concrete deployment via `x-litellm-model-id` header / `_hidden_params`. Zyquo Router does both: the response/chunks' `model` field carries the namespaced ID that actually served the request, plus an `x-zyquo-served-model` response header and the fallback hop count in the request log. Beware LiteLLM's documented pitfall of fallbacks resetting the retry cycle and re-running fallback models ([issue #19985](https://github.com/BerriAI/litellm/issues/19985)) — our loop: `for model in chain { retryPolicy(model) }`, each model getting one bounded retry budget, no restarts.
1611 +- Per-chain config lives in the Models screen editor; a chain is addressable like a model/alias.
1612 +
1613 +### 5.7 Health checks
1614 +
1615 +- `GET /health` (no auth, no logging noise) returning:
1616 +
1617 +```json
1618 +{
1619 + "status": "ok",
1620 + "version": "1.0.0",
1621 + "uptime_seconds": 12345,
1622 + "server": { "host": "127.0.0.1", "port": 8787 },
1623 + "providers_configured": 7,
1624 + "active_streams": 2
1625 +}
1626 +```
1627 +
1628 +- `status` is `"ok"` if the server is accepting; no upstream probing on this path (it must be instant and side-effect-free — LiteLLM separates `/health` per-model probes, which cost real tokens, from cheap `/health/liveliness` liveness checks; ours is the cheap kind). Per-provider connectivity testing belongs to the Keys screen's explicit "Test" button, not the health endpoint. Source: [LiteLLM proxy docs](https://docs.litellm.ai/docs/proxy/reliability).
1629 +- Suitable for `curl`-based readiness in scripts and the Phase 7 harness; also the phase-gate check for Phase 2.
1630 +
1631 +### Sources (consolidated)
1632 +
1633 +- SwiftNIO: https://github.com/apple/swift-nio · https://github.com/apple/swift-nio/releases · https://swiftpackageindex.com/apple/swift-nio · https://github.com/apple/swift-nio/blob/main/docs/public-async-nio-apis.md · https://swiftinit.org/docs/swift-nio/niocore/nioasyncchannel · https://github.com/apple/swift-nio/blob/main/Sources/NIOHTTP1Server/main.swift · https://swiftonserver.com/using-swiftnio-channels/ · https://forums.swift.org/t/nioasyncchannel-executethenclose-is-too-restrictive/73460 · https://blog.alexseifert.com/2026/06/29/building-a-web-app-in-swift-using-only-swiftnio/
1634 +- NIOExtras quiescing: https://github.com/apple/swift-nio-extras/blob/main/Sources/NIOExtras/QuiescingHelper.swift · https://github.com/apple/swift-nio-extras/blob/main/Sources/HTTPServerWithQuiescingDemo/main.swift
1635 +- Swift.org server guidelines (NIO3 timing): https://www.swift.org/documentation/server/guides/libraries/concurrency-adoption-guidelines.html
1636 +- Hummingbird: https://github.com/hummingbird-project/hummingbird · https://swiftpackageindex.com/hummingbird-project/hummingbird · https://swiftonserver.com/whats-new-in-hummingbird-2/ · https://hummingbird.codes/news/hummingbird-2/ · https://github.com/hummingbird-project/hummingbird-examples (server-sent-events example) · https://github.com/swift-server/swift-service-lifecycle
1637 +- Vapor comparison: https://github.com/hummingbird-project/hummingbird/discussions/150 · https://theswiftdev.com/beginners-guide-to-server-side-swift-using-the-hummingbird-framework/
1638 +- Network.framework: https://github.com/helje5/NWHTTPProtocol · http://www.alwaysrightinstitute.com/network-framework/ · https://developer.apple.com/documentation/network
1639 +- SSE format: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
1640 +- macOS local network privacy: https://developer.apple.com/forums/thread/763484 · https://developer.apple.com/forums/thread/767391 · https://mjtsai.com/blog/2024/10/02/local-network-privacy-on-sequoia/ · https://foldr.com/foldr-support/foldr-for-macos/macos-15-sequoia-local-network-privacy/ · https://help.panic.com/prompt/prompt-local-network/ · https://developer.apple.com/forums/thread/769037?page=2
1641 +- Entitlements/sandbox: https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.security.network.server · https://lapcatsoftware.com/articles/hardened-runtime-sandboxing.html
1642 +- LiteLLM: https://docs.litellm.ai/docs/proxy/reliability · https://docs.litellm.ai/docs/router_architecture · https://docs.litellm.ai/docs/proxy/logging · https://github.com/BerriAI/litellm/issues/16336 · https://github.com/BerriAI/litellm/issues/19985 · https://deepwiki.com/BerriAI/litellm/6.3-message-redaction-and-privacy-controls
1643 +- OpenRouter: https://openrouter.ai/docs/guides/routing/model-fallbacks · https://openrouter.ai/blog/insights/reliability-failover/
1644 +- Tokenizers in Swift: https://github.com/aespinilla/Tiktoken · https://github.com/narner/TiktokenSwift
1645 +- Rate limiting: https://blog.arcjet.com/rate-limiting-algorithms-token-bucket-vs-sliding-window-vs-fixed-window/ · https://medium.com/@tihomir.manushev/token-bucket-vs-sliding-window-the-rate-limiting-choice-that-shapes-your-apis-behavior-e04fb2646ee5 · https://apisix.apache.org/learning-center/api-gateway-rate-limiting/
1646 +- Retries/backoff: https://zuplo.com/learning-center/http-429-too-many-requests-guide · https://www.getmaxim.ai/articles/handle-429-errors-in-production-llm-applications/ · https://callsphere.ai/blog/retry-strategies-llm-api-calls-exponential-backoff-jitter-tenacity
1647