spb/zyquo-router Public MIT
One local endpoint, every AI provider — a private OpenAI-compatible LLM gateway for your Mac (170 models, 12 providers).
Swift 95.7%
Python 2.3%
Shell 1.2%
Makefile 0.9%
1# ROUTER-RESEARCH.md — Zyquo Router23How to build a production-quality local LLM gateway, correctly. Compiled from intensive4web research (2026-07-30) against current official documentation, SDK sources, and5reference gateway implementations. Sections: (1) the OpenAI API specification the router6implements, (2) how existing gateways do it, (3) translation matrices for non-OpenAI7upstreams, (4) HTTP serving in Swift, (5) gateway concerns. Sources cited inline.89## Binding decisions (executive summary)1011| # | Decision | Rationale (detail in section) |12|---|----------|-------------------------------|13| D1 | Implement `POST /v1/chat/completions`, `GET /v1/models`, `GET /v1/models/{id}`, `GET /health`. **Do NOT expose `POST /v1/responses` in v1.** | No upstream except OpenAI speaks it; its statefulness conflicts with the local/private posture; cheap to add later (§1.6). |14| D2 | Accept both `max_tokens` and `max_completion_tokens`; treat `max_completion_tokens` as canonical. | OpenAI deprecated `max_tokens`; real clients send either (§1.1, §2.4). |15| D3 | **SwiftNIO directly** (NIOCore/NIOPosix/NIOHTTP1 + NIOExtras), `NIOAsyncChannel` structured-concurrency APIs. Hummingbird 2 is the documented runner-up. | SPM-clean, Apple-maintained, full control over SSE flush/backpressure/disconnect, Swift 6-ready (§4.1). |16| D4 | Model namespace `provider/model-id`; bare IDs accepted when unambiguous; user aliases; disabled models 404. | LiteLLM/OpenRouter convention; avoids collisions like `deepseek-chat` on multiple hosts (§2.1, §2.5). |17| D5 | Param policy: known-param translation table per provider (strip/rename/clamp), unknown keys **passed through** to the upstream body. | Matches vLLM/OpenRouter behavior; enables provider extras (Perplexity search, Qwen `enable_thinking`) without schema churn (§2.5, §3.3). |18| D6 | Reasoning output normalized to DeepSeek-style **`reasoning_content`** on message and delta, with optional `reasoning_details` for signature round-trips. | Most tooling already understands DeepSeek's convention (§3.4). |19| D7 | Errors always in OpenAI `{"error":{...}}` shape: upstream 401→401 "provider key invalid (<provider>)", 429→429 with Retry-After, timeout→504, other upstream→502. Never leak raw provider payloads or key material. | §1.5, §2.5. |20| D8 | Retries: exponential backoff + jitter on 429/5xx/timeouts, respect `Retry-After`, **never retry once the first streamed byte has been forwarded**. Fallback chains report the actually-used model in `model`. | §5.5–5.6. |21| D9 | Usage: upstream-first; estimated (and flagged via `x_zyquo.usage_estimated`) only when the upstream provides none. Cost computed from the catalog's per-model pricing incl. cached-token rates. | §5.2–5.3. |22| D10 | **Gemini is translated natively** (`generateContent`/`streamGenerateContent?alt=sse`), even though Zyquo Cloud reaches Gemini through its OpenAI-compat endpoint. The Phase 3 gate requires a structurally different third upstream, and native translation avoids the compat layer's gaps (strict tool schemas, thinking metadata). | §3.2; PROVIDER-REUSE §1. |23| D11 | Streaming contract is byte-exact per §1.3: role-delta first chunk, content/tool-argument deltas, finish_reason chunk, optional usage chunk (empty `choices`) only when `stream_options.include_usage`, then `data: [DONE]`. Upstream streams are read to true EOF. | §1.3, §2.4. |24| D12 | Security: bind 127.0.0.1 by default; 0.0.0.0 opt-in forces ≥1 local API key (`zyquo-sk-…`, hashed at rest); logs redacted by default; provider keys never serialized into any response, log, or error. | §4.2–4.3, §5.1. |2526## 1. The OpenAI API specification2728> Research date: 2026-07-30. Primary sources: the OpenAI API reference (https://platform.openai.com/docs/api-reference/chat, mirrored at https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create), the streaming-events reference (https://developers.openai.com/api/reference/resources/chat/subresources/completions/streaming-events), and the official SDK type definitions, which are generated from OpenAI's OpenAPI spec and are therefore authoritative for the wire format (https://github.com/openai/openai-python/tree/main/src/openai/types/chat, https://github.com/openai/openai-node). This section is the implementation contract for Zyquo Router's public surface: **whatever the router serves must match this, byte-shape for byte-shape.**2930---3132### 1.1 `POST /v1/chat/completions` — request schema3334Headers: `Authorization: Bearer <key>`, `Content-Type: application/json`. Only `model` and `messages` are required; every other parameter is optional and, when omitted, must be treated as "provider default" (the router must NOT inject its own defaults into upstream calls).3536#### 1.1.1 `model` (string, required)3738Model ID, e.g. `"gpt-4o"`. For Zyquo Router this is the namespaced `provider/model-id`, an unambiguous bare ID, or an alias. The response must echo a model string back (the router echoes the namespaced ID actually used).3940#### 1.1.2 `messages` (array, required)4142Ordered conversation. Each element is an object with a `role` and role-specific fields. Current roles (per `ChatCompletionMessageParam` in openai-python, https://github.com/openai/openai-python/tree/main/src/openai/types/chat):4344| role | fields | notes |45|---|---|---|46| `system` | `content` (string or array of `text` parts), optional `name` | Classic system prompt. |47| `developer` | `content` (string or array of `text` parts), optional `name` | Introduced with o1; for OpenAI reasoning models `developer` replaces `system` ("with o1 models and newer, developer messages replace the previous system messages"). **Gateway rule: accept both; treat `developer` exactly like `system` when translating to upstreams that only know system prompts.** |48| `user` | `content` (string or array of content parts), optional `name` | Content parts may be multimodal (below). |49| `assistant` | `content` (string, array of `text`/`refusal` parts, or `null`), optional `name`, optional `refusal`, optional `tool_calls`, optional deprecated `function_call`, optional `audio` | "The contents of the assistant message. Required unless `tool_calls` or `function_call` is specified." So `content: null` + `tool_calls` is a legal and common history message. |50| `tool` | `content` (string or array of `text` parts, required), `tool_call_id` (string, required) | "Tool call that this message is responding to." One tool message per tool call ID. |51| `function` | `content`, `name` | Deprecated legacy of the pre-tools function API. Accept and map to `tool` semantics if seen. |5253**User content parts** (array form of `content`):5455- Text part: `{"type": "text", "text": "..."}`56- Image part (`ChatCompletionContentPartImageParam`, https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_content_part_image_param.py):5758```json59{60 "type": "image_url",61 "image_url": {62 "url": "https://example.com/cat.png",63 "detail": "auto"64 }65}66```6768 - `image_url.url` (required): "Either a URL of the image or the base64 encoded image data." The base64 form is a **data URI**: `"data:image/jpeg;base64,/9j/4AAQ..."` (`data:<mime>;base64,<payload>`; supported mimes: png, jpeg, webp, non-animated gif).69 - `image_url.detail` (optional): `"auto"` (default) | `"low"` | `"high"` — "Specifies the detail level of the image."70- Audio part: `{"type": "input_audio", "input_audio": {"data": "<base64>", "format": "wav"|"mp3"}}` (audio-capable models only; the router may reject with a clean error for providers without audio-in).71- File part: `{"type": "file", "file": {"file_id": "..."} }` or `{"file_data": "<base64 data URI>", "filename": "..."}` (PDF input on OpenAI; per-provider support varies).7273Full request example with roles + multimodal:7475```json76{77 "model": "gpt-4o",78 "messages": [79 {"role": "system", "content": "You are a terse assistant."},80 {"role": "user", "content": [81 {"type": "text", "text": "What is in this image?"},82 {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KG...", "detail": "high"}}83 ]},84 {"role": "assistant", "content": null, "tool_calls": [85 {"id": "call_abc123", "type": "function",86 "function": {"name": "lookup", "arguments": "{\"q\":\"cats\"}"}}87 ]},88 {"role": "tool", "tool_call_id": "call_abc123", "content": "{\"result\":\"a cat\"}"},89 {"role": "user", "content": "Thanks — summarize."}90 ]91}92```9394#### 1.1.3 Sampling & length parameters9596Types/defaults/deprecations verified against `CompletionCreateParams` (https://github.com/openai/openai-python/blob/main/src/openai/types/chat/completion_create_params.py) and the API reference:9798| param | type | default | notes |99|---|---|---|---|100| `temperature` | number \| null | 1 | 0–2. "Higher values like 0.8 will make the output more random." Reasoning models (o-series, gpt-5) reject non-default values — the router passes through and lets upstreams reject, or strips per-provider via CompatAdjuster. |101| `top_p` | number \| null | 1 | Nucleus sampling. "We generally recommend altering this or `temperature` but not both." |102| `max_completion_tokens` | integer \| null | none | **The current parameter.** "An upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens." |103| `max_tokens` | integer \| null | none | **Deprecated:** "This value is now deprecated in favor of `max_completion_tokens`, and is not compatible with o-series models." Still accepted for older models. (Sources: https://community.openai.com/t/why-was-max-tokens-changed-to-max-completion-tokens/938077, https://github.com/simonw/llm/issues/724, https://github.com/vercel/ai/issues/7863 — gpt-5 rejects `max_tokens` outright.) **Gateway rule: accept BOTH; if only `max_tokens` is given, treat it as `max_completion_tokens`; if both are given, prefer `max_completion_tokens`. Translate to each upstream's native cap (e.g., Anthropic's required `max_tokens`).** |104| `stop` | string \| string[] \| null | null | Up to 4 stop sequences. "Not supported with latest reasoning models `o3` and `o4-mini`." |105| `n` | integer \| null | 1 | Number of choices. Most non-OpenAI upstreams only support n=1 — the router should reject `n > 1` for those with a clear 400. |106| `frequency_penalty` | number \| null | 0 | −2.0 to 2.0. |107| `presence_penalty` | number \| null | 0 | −2.0 to 2.0. |108| `seed` | integer \| null | none | Beta. "Best effort to sample deterministically … same `seed` and parameters should return the same result." Pairs with `system_fingerprint` in the response. |109| `logit_bias` | map<string,int> \| null | null | Token-ID → bias −100..100. OpenAI-specific token IDs — pass through only to OpenAI-tokenizer upstreams; strip elsewhere. |110| `logprobs` | boolean \| null | false | "Whether to return log probabilities of the output tokens." Fills `choices[].logprobs`. |111| `top_logprobs` | integer \| null | none | 0–20; requires `logprobs: true`. |112113#### 1.1.4 `response_format`114115Three variants (see https://developers.openai.com/api/docs/guides/structured-outputs and `shared_params/response_format_*.py` in openai-python):116117```json118{"type": "text"}119{"type": "json_object"}120{121 "type": "json_schema",122 "json_schema": {123 "name": "weather_report",124 "description": "optional",125 "schema": {126 "type": "object",127 "properties": {"city": {"type": "string"}, "temp_c": {"type": "number"}},128 "required": ["city", "temp_c"],129 "additionalProperties": false130 },131 "strict": true132 }133}134```135136- `json_object` = legacy JSON mode ("an older method of generating JSON responses"); the prompt must mention JSON or OpenAI errors.137- `json_schema` = Structured Outputs. `json_schema.name` is required ("Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64"); `strict: true` means "the model will always follow the exact schema defined" (subset of JSON Schema: all fields `required`, `additionalProperties: false`).138- Router: translate to each provider's equivalent (Gemini `responseMimeType`/`responseSchema`, provider-specific json modes) or reject with a helpful 400 where unsupported.139140#### 1.1.5 Tools / function calling141142```json143{144 "tools": [145 {146 "type": "function",147 "function": {148 "name": "get_weather",149 "description": "Get current weather for a city",150 "parameters": {151 "type": "object",152 "properties": {"city": {"type": "string"}},153 "required": ["city"],154 "additionalProperties": false155 },156 "strict": true157 }158 }159 ],160 "tool_choice": "auto",161 "parallel_tool_calls": true162}163```164165- `tools[]`: currently `type: "function"` for the public wire format (newer OpenAI additions include `custom` tools and hosted tools on the Responses API; a gateway needs only `function`). `function.parameters` is a JSON Schema object; `function.strict` optional.166- `tool_choice` — union per `ChatCompletionToolChoiceOptionParam` (https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_tool_choice_option_param.py):167 - `"none"` — never call tools ("default when no tools are present"),168 - `"auto"` — model decides (default when tools present),169 - `"required"` — model must call at least one tool,170 - named function: `{"type": "function", "function": {"name": "get_weather"}}`,171 - (newer) allowed-tools form `{"type": "allowed_tools", ...}` — pass-through/optional for a gateway.172- `parallel_tool_calls` (boolean, default true): "Whether to enable parallel function calling during tool use."173- Deprecated legacy: `functions` ("Deprecated in favor of `tools`") and `function_call` ("Deprecated in favor of `tool_choice`") — accept, map to tools/tool_choice internally.174175#### 1.1.6 Streaming controls176177- `stream` (boolean \| null, default false): "If set to true, the model response data will be streamed to the client as it is generated using server-sent events."178- `stream_options` (object \| null — "Only set this when you set `stream: true`"):179 - `include_usage` (boolean): "If set, an additional chunk will be streamed before the `data: [DONE]` message. The `usage` field on this chunk shows the token usage statistics for the entire request, and the `choices` field will always be an empty array. All other chunks will also include a `usage` field, but with a null value." (Source: https://community.openai.com/t/usage-stats-now-available-when-using-streaming-with-the-chat-completions-api-or-completions-api/738156, and the SDK type docstring.)180 - `include_obfuscation` (boolean, newer): adds random `obfuscation` padding fields to chunks; a local gateway should not emit it.181182#### 1.1.7 Identity, caching & misc parameters183184| param | notes |185|---|---|186| `user` (string) | Legacy end-user ID. "This field is being replaced by `safety_identifier` and `prompt_cache_key`." Accept it; useful as a per-key attribution hint. |187| `safety_identifier` (string) | "A stable identifier used to help detect users … Maximum length of 64 characters." Pass through to OpenAI only. |188| `prompt_cache_key` (string) | Cache-affinity hint, "Replaces the `user` field" for caching. Pass through to OpenAI only. |189| `store` (bool), `metadata` (map, ≤16 keys) | OpenAI-side storage for distillation/evals. Pass through to OpenAI; strip elsewhere. |190| `service_tier` | `"auto" \| "default" \| "flex" \| "scale" \| "priority" \| "fast"`. OpenAI-only; strip elsewhere. |191| `reasoning_effort` | For reasoning models. Current values per SDK: "`none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`" (model-dependent subsets). The router should pass this through to reasoning-capable upstreams that accept it. |192| `verbosity` | `"low" \| "medium" \| "high"` (gpt-5 family). Pass through to OpenAI. |193| `modalities`, `audio`, `prediction`, `web_search_options` | Audio-out, predicted outputs, built-in web search — OpenAI-specific; a gateway may pass through to OpenAI and strip/400 elsewhere. |194195**Unknown-key policy:** OpenAI itself returns 400 `unrecognized argument` for unknown top-level keys, but a *gateway* should follow LiteLLM/OpenRouter practice: accept unknown keys and pass them through to the upstream body (this is how provider-specific extras like Perplexity `search_domain_filter` or Qwen `enable_thinking` travel). Document accepted extras in `docs/API.md`.196197---198199### 1.2 Non-streaming response — `chat.completion` object200201Fields verified against `ChatCompletion` (https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion.py) and `CompletionUsage` (https://github.com/openai/openai-python/blob/main/src/openai/types/completion_usage.py):202203```json204{205 "id": "chatcmpl-B9MBs8CjcvOU2jLn4n570S5qMJKcT",206 "object": "chat.completion",207 "created": 1741569952,208 "model": "gpt-4o-2024-08-06",209 "system_fingerprint": "fp_50cad350e4",210 "service_tier": "default",211 "choices": [212 {213 "index": 0,214 "message": {215 "role": "assistant",216 "content": "Hello! How can I assist you today?",217 "refusal": null,218 "annotations": []219 },220 "logprobs": null,221 "finish_reason": "stop"222 }223 ],224 "usage": {225 "prompt_tokens": 19,226 "completion_tokens": 10,227 "total_tokens": 29,228 "prompt_tokens_details": {"cached_tokens": 0, "audio_tokens": 0},229 "completion_tokens_details": {230 "reasoning_tokens": 0,231 "audio_tokens": 0,232 "accepted_prediction_tokens": 0,233 "rejected_prediction_tokens": 0234 }235 }236}237```238239Field-by-field:240241- `id` (string): "A unique identifier for the chat completion." Convention `chatcmpl-<base62>`; the router generates its own (`chatcmpl-` prefix keeps naive clients happy).242- `object`: literal `"chat.completion"`.243- `created` (integer): Unix seconds.244- `model` (string): "The model used" — router echoes the namespaced ID actually served (incl. after fallback).245- `system_fingerprint` (string, optional, now marked deprecated in OpenAI docs): backend-config fingerprint for use with `seed`. Optional — the router may omit or emit a static value.246- `service_tier` (optional): echo only for OpenAI upstreams.247- `choices[]`:248 - `index` (integer),249 - `message`:250 - `role`: always `"assistant"`,251 - `content` (string \| null): null when the model only called tools,252 - `refusal` (string \| null): structured-outputs refusal message,253 - `tool_calls` (array, optional): each `{"id": "call_…", "type": "function", "function": {"name": "...", "arguments": "<JSON string>"}}` — **`arguments` is a string containing JSON, not an object**,254 - `annotations` (array, optional): e.g. `url_citation` items from web search,255 - `audio` (optional, audio-out models),256 - reasoning models on other providers add `reasoning_content` (DeepSeek et al.) — not an OpenAI field, but the de-facto extension the router preserves (Phase 0 decision),257 - `logprobs` (object \| null): `{"content": [{token, logprob, bytes, top_logprobs: […]}], "refusal": […]}` when requested,258 - `finish_reason` — exact literal set per the SDK: **`"stop" | "length" | "tool_calls" | "content_filter" | "function_call"`**:259 - `stop` — natural stop or stop sequence hit,260 - `length` — token cap reached (`max_completion_tokens` or context limit),261 - `tool_calls` — the model called tools,262 - `content_filter` — content omitted by a filter,263 - `function_call` — deprecated legacy (only when using deprecated `functions`).264 Every upstream stop reason must be mapped into this set (e.g., Anthropic `end_turn`→`stop`, `max_tokens`→`length`, `tool_use`→`tool_calls`, `stop_sequence`→`stop`).265- `usage`:266 - `prompt_tokens`, `completion_tokens`, `total_tokens` (integers, total = prompt + completion),267 - `prompt_tokens_details` (optional): `cached_tokens` ("Cached tokens present in the prompt"), `audio_tokens`, and newer `cache_write_tokens`,268 - `completion_tokens_details` (optional): `reasoning_tokens` ("Tokens generated by the model for reasoning"), `audio_tokens`, `accepted_prediction_tokens`, `rejected_prediction_tokens`.269 - Gateway rule: use upstream-reported usage when present; when absent, estimate and flag (e.g. `"x_zyquo": {"usage_estimated": true}` — extension keys are tolerated by both SDKs).270271Tool-call response example (non-streaming):272273```json274{275 "id": "chatcmpl-abc123",276 "object": "chat.completion",277 "created": 1699896916,278 "model": "gpt-4o-mini",279 "choices": [280 {281 "index": 0,282 "message": {283 "role": "assistant",284 "content": null,285 "tool_calls": [286 {287 "id": "call_abc123",288 "type": "function",289 "function": {"name": "get_weather", "arguments": "{\n\"city\": \"Boston\"\n}"}290 }291 ]292 },293 "logprobs": null,294 "finish_reason": "tool_calls"295 }296 ],297 "usage": {"prompt_tokens": 82, "completion_tokens": 17, "total_tokens": 99}298}299```300301---302303### 1.3 The SSE streaming format — byte-level contract304305Sources: streaming reference (https://developers.openai.com/api/reference/resources/chat/subresources/completions/streaming-events, https://platform.openai.com/docs/api-reference/chat-streaming/streaming), the cookbook (https://cookbook.openai.com/examples/how_to_stream_completions), and `ChatCompletionChunk` SDK types.306307**Transport.** Response headers: `Content-Type: text/event-stream; charset=utf-8`, `Cache-Control: no-cache`, chunked transfer (no `Content-Length`), keep connection open. Each event is exactly:308309```310data: <one-line JSON>\n311\n312```313314i.e. the 6 bytes `data: `, the JSON serialized **without newlines**, then `\n\n`. OpenAI emits only `data:` lines — no `event:`, `id:`, or `retry:` fields, no SSE comments. The stream terminates with the sentinel:315316```317data: [DONE]\n318\n319```320321(`[DONE]` is not JSON; both official SDKs special-case this exact string.) An HTTP-level error that occurs *before* streaming starts is a plain JSON error body with a proper status code; the status is sent before any chunk, so a request that fails validation must NOT return 200 + SSE.322323**Chunk object** (`object: "chat.completion.chunk"`): same `id` ("Each chunk has the same ID"), `created`, and `model` across all chunks of one completion; `choices[]` with `{index, delta, logprobs, finish_reason}`; `usage` null/absent except the final usage chunk. Delta fields: `role`, `content`, `refusal`, `tool_calls[]` (each with `index`, optional `id`, optional `type: "function"`, optional `function.name`, optional `function.arguments` fragment), deprecated `function_call`.324325Chunk sequence rules:3263271. **First chunk** carries the role delta: `"delta": {"role": "assistant", "content": ""}` (OpenAI includes the empty `content` string; emit it — some clients concatenate blindly). May also carry `refusal: null` — harmless.3282. **Content chunks**: `"delta": {"content": "<fragment>"}`, `finish_reason: null`.3293. **Final content chunk**: `"delta": {}` (empty object) with `"finish_reason": "stop"` (or `length`/`tool_calls`/`content_filter`). The finish_reason travels on a chunk whose delta is empty — not alongside content.3304. **Optional usage chunk** (only when `stream_options.include_usage` is true): `"choices": []` (empty array — quoted from the SDK: choices "can also be empty for the last chunk if you set `stream_options: {\"include_usage\": true}`") and a populated `usage` object. "All other chunks will also include a `usage` field, but with a null value" when include_usage is set.3315. `data: [DONE]`.332333#### (a) Plain-text transcript (`stream: true`, `stream_options: {"include_usage": true}`)334335```336data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"usage":null}337338data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"content":"Hello"},"logprobs":null,"finish_reason":null}],"usage":null}339340data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"content":" there"},"logprobs":null,"finish_reason":null}],"usage":null}341342data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"content":"!"},"logprobs":null,"finish_reason":null}],"usage":null}343344data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null}345346data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[],"usage":{"prompt_tokens":19,"completion_tokens":3,"total_tokens":22,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}}}347348data: [DONE]349350```351352(Without `include_usage`, the usage chunk is absent and no `usage` key appears on chunks.)353354#### (b) Streamed tool call transcript355356Tool-call arguments stream as string fragments. **The first tool_call delta for a given `index` carries `id`, `type`, and `function.name` (with `"arguments":""`); every subsequent delta for that index carries ONLY `index` and `function.arguments` fragments — no id, no name.** Parallel tool calls interleave via `index` 0,1,…; clients accumulate by index and concatenate `arguments`.357358```359data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"role":"assistant","content":null},"logprobs":null,"finish_reason":null}]}360361data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_KSEnFnucOtZNQqEZF9wvfIWm","type":"function","function":{"name":"get_weather","arguments":""}}]},"logprobs":null,"finish_reason":null}]}362363data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"ci"}}]},"logprobs":null,"finish_reason":null}]}364365data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ty\": \"Bos"}}]},"logprobs":null,"finish_reason":null}]}366367data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ton\"}"}}]},"logprobs":null,"finish_reason":null}]}368369data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"tool_calls"}]}370371data: [DONE]372373```374375#### What strict SDK clients require376377- **openai-python** parses SSE by splitting on blank lines, reads only `data:` payloads, stops at the exact string `[DONE]`, and constructs a pydantic `ChatCompletionChunk`. Pydantic will **fail the whole stream** if `id`, `object`, `created`, `model`, or `choices` is missing/wrong-typed, if `object` isn't exactly `"chat.completion.chunk"`, or if `finish_reason` is a value outside the literal set. Extra unknown fields are tolerated (kept as extras) — so `reasoning_content` in deltas and `x_zyquo` extensions are safe. (See https://github.com/openai/openai-python; a real-world failure mode of non-conforming servers: https://github.com/janhq/jan/issues/8280 — "the chat.completion.chunk schema … requires choices[]; type validation failed" halts the stream.)378- **openai-node** is looser at runtime (TypeScript types are compile-time), but its stream accumulator (`stream.finalChatCompletion()`, and the Vercel AI SDK on top of it) indexes `choices[0].delta`, accumulates `tool_calls` strictly by `index`, and expects `id`/`function.name` on the first delta of each tool call; missing `index` or re-sending `name` fragments corrupts accumulation (see https://ai-sdk.dev/providers/openai-compatible-providers on buffering unreliable tool-call deltas).379- Both SDKs ignore SSE `event:` fields **only** if the line isn't a `data:` line; do not emit named events or comments — emit *only* `data:` lines exactly as above.380- Keep every JSON chunk on a single line; UTF-8; never split a multibyte character across chunks inside one JSON string (JSON-escape or buffer to codepoint boundaries).381382---383384### 1.4 `GET /v1/models` and `GET /v1/models/{id}`385386Source: https://developers.openai.com/api/reference/resources/models (mirrors https://platform.openai.com/docs/api-reference/models).387388`GET /v1/models` →389390```json391{392 "object": "list",393 "data": [394 {"id": "gpt-4o", "object": "model", "created": 1686935002, "owned_by": "openai"},395 {"id": "gpt-4o-mini", "object": "model", "created": 1686935002, "owned_by": "openai"}396 ]397}398```399400Model object: `id` (string — "The model identifier, which can be referenced in the API endpoints"), `object` (always `"model"`), `created` (Unix seconds), `owned_by` (string — "The organization that owns the model").401402`GET /v1/models/{model}` → a single model object:403404```json405{"id": "gpt-4o", "object": "model", "created": 1686935002, "owned_by": "openai"}406```407408Unknown ID → 404 with the model-not-found error (§1.5). Router mapping: `id` = namespaced `provider/model-id` (aliases listed too), `owned_by` = provider name; enriched metadata (context window, pricing, capabilities) goes under an `x_zyquo` extension key on each entry — both SDKs tolerate extra fields.409410---411412### 1.5 Error response format413414Every error is JSON with a single `error` object (see https://developers.openai.com/api/docs/guides/error-codes and https://community.openai.com/t/openai-chat-list-of-error-codes-and-types/357791):415416```json417{418 "error": {419 "message": "<human-readable description>",420 "type": "<error family>",421 "param": "<offending request parameter or null>",422 "code": "<machine-readable code or null>"423 }424}425```426427All four keys are always present (`param`/`code` may be `null`). The official SDKs map HTTP status → typed exceptions (`BadRequestError` 400, `AuthenticationError` 401, `PermissionDeniedError` 403, `NotFoundError` 404, `UnprocessableEntityError` 422, `RateLimitError` 429, `InternalServerError` ≥500, per https://github.com/openai/openai-python#handling-errors).428429| HTTP | `type` | typical `code` values | when |430|---|---|---|---|431| 400 | `invalid_request_error` | `null`, `invalid_value`, `unsupported_parameter`, `context_length_exceeded`, `string_above_max_length`, `invalid_image_format` | Malformed body, bad param, context overflow |432| 401 | `invalid_request_error` / `authentication_error` | `invalid_api_key`, `no_organization` | Missing/invalid API key ("Incorrect API key provided: …") |433| 403 | `permission_error` / `invalid_request_error` | `unsupported_country_region_territory`, `insufficient_permissions` | Key valid but not allowed (region, scoped key) |434| 404 | `invalid_request_error` | `model_not_found` | Unknown model / resource |435| 422 | `invalid_request_error` | — | Semantically invalid (rare) |436| 429 | `rate_limit_error` | `rate_limit_exceeded` | "Rate limit reached for …" — retriable; honor/emit `Retry-After` |437| 429 | `insufficient_quota` | `insufficient_quota` | "You exceeded your current quota, plans & billing…" — NOT retriable (note: type AND code are both `insufficient_quota`) |438| 500 | `server_error` | `null` | "The server had an error while processing your request" |439| 502 | (gateway) `server_error` / `api_error` | `bad_gateway` | Used by gateways (OpenRouter/LiteLLM) for upstream failure — the router's choice for "upstream returned garbage / is down" |440| 503 | `server_error` / `service_unavailable` | `service_unavailable`, `slow_down` | "The engine is currently overloaded, please try again later" |441| 504 | (gateway) `timeout_error` | `timeout` | Gateway convention for upstream timeout |442443**The exact model-not-found error** (HTTP 404) as OpenAI returns it (sources: https://community.openai.com/t/openai-error-invalidrequesterror-the-model-gpt-4-does-not-exist-or-you-do-not-have-access-to-it/376230, https://community.openai.com/t/api-returning-404-model-not-found-all-of-a-sudden-why-and-how-to-fix/679777):444445```json446{447 "error": {448 "message": "The model `gpt-5-nonexistent` does not exist or you do not have access to it.",449 "type": "invalid_request_error",450 "param": null,451 "code": "model_not_found"452 }453}454```455456The router must reproduce this shape verbatim (with its namespaced ID in backticks) for unknown/disabled models — SDK error-handling paths and agent frameworks string-match parts of it.457458**Streaming errors:** if the failure happens before any chunk, return the JSON error with the real status code (no SSE). If the upstream dies mid-stream, OpenAI's own behavior is to emit an error payload as a `data:` line (`{"error": {...}}`) and close without `[DONE]`; a gateway should do the same — strict clients surface it as a stream error rather than hanging.459460**Gateway error-mapping rules (Phase 3 contract):** upstream 401 (provider key bad) → 401 with "provider key for `<provider>` was rejected" (never echo the key); upstream 429 → 429 + `Retry-After` when given; upstream timeout → 504 `timeout`; upstream 5xx/unparseable → 502 `bad_gateway` with sanitized detail; never leak raw provider error shapes or key material.461462---463464### 1.6 `POST /v1/responses` — evaluate and decide465466Summary (sources: https://platform.openai.com/docs/guides/migrate-to-responses, https://developers.openai.com/api/docs/guides/migrate-to-responses):467468- **Shape:** flatter request — `input` (string or item array) + top-level `instructions` instead of a `messages` array; response is a typed `output` array of *items* (`message`, `reasoning` with `encrypted_content`, `function_call`, `function_call_output`, hosted-tool items) plus an `output_text` convenience; server-side state via `store: true` / `previous_response_id`; built-in hosted tools (web search, file search, code interpreter, computer use).469- **Streaming:** *semantic events*, not chunk deltas — named SSE events like `response.created`, `response.output_item.added`, `response.output_text.delta`, `response.completed` — a completely different event model from `chat.completion.chunk`.470- **Adoption (as of mid-2026):** OpenAI recommends Responses "for all new projects" and newest OpenAI-native features (encrypted reasoning, hosted tools) land there first, but they state Chat Completions "remains supported" indefinitely as the industry standard. Critically for a *multi-provider gateway*: the entire compatible-provider ecosystem (DeepSeek, Qwen, Mistral, xAI, Together, Cerebras, Ollama, LM Studio, vLLM…) standardized on **chat/completions**, and gateways (LiteLLM, OpenRouter) still treat it as the lingua franca (OpenRouter exposes chat/completions; LiteLLM added a Responses *bridge* that internally converts to chat/completions).471472**DECISION for Zyquo Router: do NOT expose `/v1/responses` in v1.** Rationale:4734741. Zero translation leverage — none of our 12 upstream providers speak Responses natively except OpenAI itself; we'd be building a second full bidirectional translation layer (items + semantic streaming events) purely as a front-end conversion to the same canonical internal request.4752. Statefulness (`store`, `previous_response_id`, `encrypted_content`) implies server-side conversation storage — out of scope and against the router's privacy posture.4763. Client compatibility target is met without it: every OpenAI-SDK-based tool can (and with third-party base URLs, typically does) use `chat.completions`.4774. Cheap to add later as a stateless subset (`input`/`instructions` → messages; emit `response.output_text.delta` events) once the chat/completions core is green — track as a post-v1 enhancement in `docs/PLAN.md`. `/health` should not advertise it; requests to `/v1/responses` return a 404 OpenAI-format error with a message pointing to `/v1/chat/completions`.478479---480481### 1.7 `POST /v1/embeddings` (optional gateway endpoint)482483Source: https://platform.openai.com/docs/api-reference/embeddings and `embedding_create_params.py` / `create_embedding_response.py` in openai-python.484485Request:486487```json488{489 "model": "text-embedding-3-small",490 "input": "The food was delicious and the waiter...",491 "encoding_format": "float",492 "dimensions": 512,493 "user": "optional-end-user-id"494}495```496497- `input` (required): "string or array of tokens. To embed multiple inputs in a single request, pass an array of strings or array of token arrays." (Max ~2048 inputs per request; each within the model's token limit.)498- `model` (required); `encoding_format`: `"float"` (default) or `"base64"`; `dimensions`: "Only supported in `text-embedding-3` and later models"; `user`: abuse-monitoring ID.499500Response:501502```json503{504 "object": "list",505 "data": [506 {507 "object": "embedding",508 "index": 0,509 "embedding": [0.0023064255, -0.009327292, -0.0028842222]510 }511 ],512 "model": "text-embedding-3-small",513 "usage": {"prompt_tokens": 8, "total_tokens": 8}514}515```516517Notes for the router: `data[]` is ordered by `index` matching the input array; `usage` has only `prompt_tokens` + `total_tokens` (no completion tokens); `encoding_format: "base64"` returns each embedding as a base64 string of little-endian float32 — support it, since openai-python requests base64 by default when numpy is available. Route only to providers that offer embeddings; others 404 with `model_not_found`.518519---520521### 1.8 Contract checklist for Zyquo Router (derived from this section)522523- [ ] Accept full request schema §1.1 incl. both `max_tokens` and `max_completion_tokens`, `developer` role, multimodal parts, all three `response_format` variants, all `tool_choice` forms, deprecated `functions`/`function_call`.524- [ ] Emit spec-exact `chat.completion` (§1.2) with mapped `finish_reason` from the closed literal set and real-or-flagged `usage` incl. details sub-objects when upstreams provide them.525- [ ] Emit byte-exact SSE (§1.3): role-first chunk, single-line JSON `data:` events, empty-delta finish chunk, empty-choices usage chunk only under `include_usage`, terminating `data: [DONE]`, correct tool-call delta id/name/index rules.526- [ ] `GET /v1/models` + `/v1/models/{id}` per §1.4 with namespaced IDs.527- [ ] All errors per §1.5 incl. verbatim model-not-found shape and gateway 502/504 conventions.528- [ ] `/v1/responses`: not exposed in v1 (documented 404 with pointer); `/v1/embeddings`: optional, per §1.7.529## 2. How existing gateways do it530531Research date: 2026-07-30. Sources fetched live from docs.litellm.ai, openrouter.ai/docs, docs.ollama.com, lmstudio.ai/docs, docs.vllm.ai, and GitHub issue trackers. This section extracts the concrete, battle-tested patterns from the reference gateways that Zyquo Router should copy — and the compatibility landmines it must avoid.532533---534535### 2.1 LiteLLM proxy — the reference for config, translation, and error mapping536537LiteLLM is the most complete open-source implementation of exactly what Zyquo Router is: an OpenAI-compatible front for ~100 providers. Its patterns are the closest prior art.538539#### 2.1.1 Model naming: `provider/model`540541LiteLLM routes on a **provider prefix in the model string**: `openai/gpt-4o`, `azure/gpt-4o`, `anthropic/claude-...`, `bedrock/anthropic.claude-instant-v1`, `ollama/mistral`, `gemini/gemini-2.5-pro`. The prefix selects the client implementation; the remainder is the upstream model ID sent to the provider. (Source: https://docs.litellm.ai/docs/proxy/configs)542543The proxy adds a second layer of indirection: a **user-facing `model_name` alias** mapped to one or more concrete deployments in `config.yaml`:544545```yaml546model_list:547 - model_name: gpt-4o # what clients send in "model"548 litellm_params:549 model: azure/gpt-4o-eu # provider/upstream-id actually called550 api_base: https://endpoint-europe.openai.azure.com/551 api_key: "os.environ/AZURE_API_KEY_EU" # env-var indirection, secrets never in config552 rpm: 6 # per-deployment rate limit553 model_info: # optional metadata (pricing/context overrides)554 max_input_tokens: 128000555556litellm_settings:557 drop_params: true558 num_retries: 3559 request_timeout: 10560 fallbacks: [{"gpt-4o": ["claude-sonnet"]}]561562router_settings:563 routing_strategy: simple-shuffle564 model_group_alias: {"gpt-4": "gpt-4o"} # request-time alias remapping565566general_settings:567 master_key: sk-1234 # local bearer key gating the proxy568```569570Key ideas to steal (https://docs.litellm.ai/docs/proxy/configs):571572- **Two-level naming**: public alias (`model_name`) → concrete `provider/model` deployment. Zyquo Router's aliases (`fast` → `cerebras/...`) are exactly this.573- Multiple entries with the same `model_name` = load balancing group (Zyquo Router doesn't need multi-deployment balancing, but the alias→target indirection is the same shape).574- `model_group_alias` maps well-known client names (`gpt-4`) onto configured groups — useful for tools that hardcode OpenAI model names.575- Wildcards (`model_name: "*"`, `model: openai/*`) allow pass-through of any model given credentials — Zyquo Router's "accept unambiguous bare IDs" is a constrained version of this.576- `os.environ/VAR` indirection keeps keys out of the config file (Zyquo Router: keys live only in the vault; config export never includes them).577578#### 2.1.2 Param translation and drop tables579580LiteLLM maintains, per provider, a **mapping table of which OpenAI params the provider supports** (queryable via `litellm.get_supported_openai_params(model)`), and translates names where they differ. Handling of *unsupported* params is explicit policy, not accident (https://docs.litellm.ai/docs/completion/drop_params):581582- **Default: raise an exception** if a param is sent to a model that doesn't support it — loud failure over silent behavior change.583- **`drop_params: true`** (global, per-deployment, or per-request): silently strip unsupported params instead of erroring. Most proxies run with this on.584- **`additional_drop_params: ["response_format"]`** — per-deployment list of specific params to strip even if nominally supported; supports JSONPath-ish nested syntax (`tools[*].input_examples`, `parent.child`, `array[0]`).585- **`allowed_openai_params: ["tools"]`** — the inverse escape hatch: force-forward a param LiteLLM believes is unsupported (settable in config or per-request via `extra_body`).586- **Provider-specific extras** ride through `extra_body` on the OpenAI SDK and are passed to the upstream unchanged.587588Lesson for Zyquo Router's `CompatAdjuster`: implement a **per-provider param table** (supported / rename / strip / pass-through-extras) as data, not scattered `if`s, and make the strip-vs-error policy explicit and configurable.589590#### 2.1.3 Error mapping to OpenAI exceptions591592LiteLLM maps every upstream failure onto **exception types that inherit from the OpenAI SDK's own exceptions**, so client code catching `openai.RateLimitError` works against any provider (https://docs.litellm.ai/docs/exception_mapping). Status-code taxonomy:593594| Status | Exceptions |595|---|---|596| 400 | `BadRequestError`, `UnsupportedParamsError`, `ContextWindowExceededError`, `ContentPolicyViolationError` |597| 401 | `AuthenticationError` |598| 403 | `PermissionDeniedError` |599| 404 | `NotFoundError` |600| 408 | `Timeout` |601| 422 | `UnprocessableEntityError` |602| 429 | `RateLimitError` |603| 500 | `APIConnectionError`, `APIError` |604| 503 | `ServiceUnavailableError` |605| ≥500 | `InternalServerError` |606607Every mapped exception carries `status_code`, `message`, and **`llm_provider`** (which upstream failed) — Zyquo Router should likewise name the provider in error messages ("Anthropic key invalid") without leaking payloads. Note that `ContextWindowExceededError` and `ContentPolicyViolationError` are *distinguished subtypes of 400*: this is what makes context-window fallbacks and content-policy fallbacks possible. A `_should_retry(status_code)` helper centralizes the retryability decision (429, 5xx, timeouts → retry; 4xx auth/validation → don't).608609#### 2.1.4 Retries, fallbacks, cooldowns610611(Sources: https://docs.litellm.ai/docs/routing, https://docs.litellm.ai/docs/proxy/reliability)612613- **`num_retries: 3`** with exponential backoff for `RateLimitError`, immediate retry for transient errors; `retry_after` sets a minimum wait. A `RetryPolicy` can set retry counts **per exception class** (e.g., `AuthenticationErrorRetries=0`, `RateLimitErrorRetries=3`, `TimeoutErrorRetries=2`) — never retry auth failures.614- **Three fallback kinds**, configured as ordered maps `{"primary": ["fallback1", "fallback2"]}`:615 - `fallbacks` — general retryable errors (429/5xx) after retries exhaust;616 - `context_window_fallbacks` — prompt too big → reroute to a bigger-context model;617 - `content_policy_fallbacks` — content filter tripped → reroute to a laxer model;618 - `default_fallbacks: ["claude-opus"]` — catch-all.619- **Per-request fallbacks** via a `fallbacks: [...]` array in the request body, and `disable_fallbacks: true` to opt out per request.620- **Cooldowns**: `allowed_fails: 3` failures per minute puts a deployment on a `cooldown_time: 30`s bench so the router stops hammering a failing upstream. For a single-deployment-per-model local router this maps to "mark provider degraded, surface in dashboard, fail fast or fall back."621- Execution order: retries on the primary first, then fallbacks in order until success or exhaustion. The actually-used deployment is reported via an `x-litellm-model-id` response header — Zyquo Router should report the actually-used model in the response `model` field (OpenRouter's approach, §2.2.4) and/or a header.622623#### 2.1.5 Usage & cost tracking624625(Source: https://docs.litellm.ai/docs/completion/token_usage)626627- Pricing lives in one community-maintained JSON file, `model_prices_and_context_window.json` (https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) — the de-facto industry pricing database. Entry shape:628629```json630{631 "gpt-4o": {632 "max_tokens": 4000,633 "input_cost_per_token": 1.5e-06,634 "output_cost_per_token": 2e-06,635 "litellm_provider": "openai",636 "mode": "chat"637 }638}639```640641- `completion_cost(response)` computes USD from the usage in a response + this table; `cost_per_token(model, prompt_tokens, completion_tokens)` is the primitive. Every response carries `response_cost` in hidden params.642- `register_model({...})` lets users override/add pricing — Zyquo Router's Settings ▸ Usage & Pricing "pricing override" is the same feature.643- When upstream doesn't return usage (some streams), tokens are **estimated with a tokenizer** (tiktoken default, provider-specific where known) — and Zyquo Router should flag estimated usage as such.644645Pattern: **pricing is data keyed by model ID, cost computed at response time from `usage`, estimation as fallback**. Zyquo Router already has per-model pricing in the Zyquo Cloud catalog; reuse it as the single pricing source.646647#### 2.1.6 Streaming normalization648649LiteLLM wraps every provider stream in a `CustomStreamWrapper` that re-emits **uniform OpenAI-shaped chunk objects** (`choices[0].delta.content`, etc.) regardless of upstream wire format, plus a `stream_chunk_builder(chunks)` helper that reassembles a full `chat.completion` from chunks (useful for logging/cost of streamed requests — Zyquo Router's request log needs exactly this) (https://docs.litellm.ai/docs/completion/stream). It also guards against pathological streams: `REPEATED_STREAMING_CHUNK_LIMIT = 100` aborts with an `InternalServerError` if the same chunk repeats endlessly.650651Cautionary tales from LiteLLM's own tracker — even the reference implementation gets chunk shape wrong:652653- Its synthetic usage chunk violated the OpenAI spec by carrying a **non-empty `choices`** array (spec: the `include_usage` final chunk has `"choices": []`) — https://github.com/BerriAI/litellm/issues/28735654- It **lost vLLM's usage** because vLLM sends usage in a separate empty-choices chunk *after* the `finish_reason` chunk and LiteLLM stopped reading at `finish_reason` — https://github.com/BerriAI/litellm/issues/25389 . Lesson: when consuming upstreams, read until the stream actually ends, not until `finish_reason`.655- Grok returned usage in the wrong chunk (extra empty final chunk) — https://github.com/BerriAI/litellm/issues/17136 ; and some providers reject `stream_options` as an unknown param, so the gateway must know per provider whether it can request usage-in-stream — https://github.com/BerriAI/litellm/issues/23847656657---658659### 2.2 OpenRouter — the reference for unified IDs, streaming discipline, and honest accounting660661#### 2.2.1 Unified model IDs and variants662663- IDs are **`vendor/model-name`** (`anthropic/claude-3.5-sonnet`, `openai/gpt-4o`), plus a permanent `canonical_slug` that survives renames. (https://openrouter.ai/docs/models)664- **Variant suffixes** append behavior to a slug: `:free` (free tier), `:thinking` (reasoning mode), `:nitro` (= `provider.sort: "throughput"`), `:floor` (sort by price). A suffix-on-the-ID is a very ergonomic way to encode routing preferences without extra params (https://openrouter.ai/docs/models, https://openrouter.ai/docs/features/provider-routing). Zyquo Router could reserve this pattern for future use (e.g., `model:nostore` to skip logging).665- `GET /api/v1/models` returns rich metadata per model: `id`, `canonical_slug`, `name`, `context_length`, `architecture` (input/output modalities, tokenizer), `pricing` (**USD per token as strings** — `"0"` means free; string avoids float precision issues), `supported_parameters` (array of OpenAI params this model accepts — clients can pre-check!), `top_provider`. Zyquo Router's plan to enrich `/v1/models` under an `x-zyquo` extension key mirrors this; adopting `context_length`, `pricing`, and `supported_parameters` fields is directly useful for the Playground and Docs UI.666667#### 2.2.2 Provider-specific params & headers668669- Default posture: **unsupported params are silently ignored by the receiving provider**; setting `"provider": {"require_parameters": true}` restricts routing to providers that support *every* param in the request (https://openrouter.ai/docs/features/provider-routing). For a single-upstream-per-model router, the equivalent decision is strip-vs-reject per param (see §2.5).670- Extra attribution headers `HTTP-Referer` and `X-Title` are optional and additive — the API remains pure OpenAI otherwise (https://openrouter.ai/docs/api-reference/overview).671- Provider-specific features arrive as **extra top-level body keys** (e.g., `models`, `provider`, `plugins`, `transforms`) that OpenAI SDKs send via `extra_body` — the standard pass-through idiom Zyquo Router should adopt for things like Perplexity search options.672673#### 2.2.3 Streaming normalization — the details that matter674675(Source: https://openrouter.ai/docs/api-reference/streaming)676677- **Keep-alive SSE comments**: OpenRouter periodically emits `: OPENROUTER PROCESSING` comment lines to hold connections open during long prefill/queue waits. Per the SSE spec, lines starting with `:` are comments and must be ignored — but naive client loops that `JSON.parse` every line crash on them. Zyquo Router should (a) emit its own `: keep-alive` comments during long upstream silences (Anthropic thinking, queued requests), and (b) tolerate/strip comments when *consuming* upstream SSE.678- Termination is always `data: [DONE]`.679- **Mid-stream errors**: once tokens have flowed you can't change the HTTP status, so errors arrive as a final SSE event that is still a valid `chat.completion.chunk` with an added `error` object and `finish_reason: "error"`:680681```682data: {"id":"gen-abc123","object":"chat.completion.chunk","created":1730000000,683 "model":"...","error":{"code":429,"message":"Rate limit exceeded",684 "metadata":{"error_type":"rate_limit_exceeded"}},685 "choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}]}686```687688 (https://openrouter.ai/docs/api-reference/errors) This is the pattern Zyquo Router needs for "upstream died mid-stream": emit a well-formed error chunk, then `[DONE]`, then close — never a bare connection reset.689- **Cancellation**: aborting the connection cancels the upstream *only for streaming requests on providers that support cancellation*; otherwise the upstream finishes and bills anyway. Zyquo Router must cancel the upstream `URLSession`/NIO task on client disconnect (its clients are direct HTTP calls, so cancellation is always possible) — this is Phase 7 test 3.690- Usage stats ride in the **final chunk** (`chunk.usage`).691692#### 2.2.4 Usage accounting693694(Source: https://openrouter.ai/docs/use-cases/usage-accounting)695696- Usage (with **cost**) is now included in every response automatically (the old `usage: {include: true}` opt-in is deprecated). Shape extends OpenAI's `usage`:697698```json699"usage": {700 "prompt_tokens": 194,701 "completion_tokens": 2,702 "total_tokens": 196,703 "cost": 0.95,704 "cost_details": {"upstream_inference_cost": 19},705 "prompt_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 100},706 "completion_tokens_details": {"reasoning_tokens": 0}707}708```709710 Putting `cost` inside `usage` is a compatible extension (SDKs ignore unknown fields on responses) — a good idea for Zyquo Router's estimated-cost surfacing.711- For streams, usage appears **in the last SSE message**; there's also a post-hoc `GET /api/v1/generation?id=...` stats endpoint keyed by the response `id` for auditing. Zyquo Router's request-log detail view is the local equivalent.712713#### 2.2.5 Error format714715(Source: https://openrouter.ai/docs/api-reference/errors)716717- Error shape: `{"error": {"code": <number>, "message": "...", "metadata": {...}}}` with HTTP status = `error.code` for pre-stream failures. Status vocabulary: **400** bad params/CORS, **401** bad key, **402** out of credits, **403** moderation/guardrail, **408** timeout, **429** rate limited, **502** "your chosen model is down or we received an invalid response", **503** "no available provider meets your routing requirements".718- `metadata` carries structured context without leaking raw payloads: moderation errors include `reasons`, a truncated `flagged_input` (max 100 chars), `provider_name`, `model_slug`; provider errors include a canonical `error_type` plus the original `provider_code`. This "canonical code + original provider code" pair is exactly the honest-but-normalized surfacing Zyquo Router wants.719- **429/503 responses include a standard `Retry-After` header** — Zyquo Router should propagate upstream `Retry-After` to its own clients.720- Note: OpenRouter uses a numeric `error.code`; the strict OpenAI format is `{"error": {"message", "type", "param", "code"}}` with string-ish `code`. Zyquo Router should keep the **OpenAI field set** (per §1) and put router-specific context in the message and/or an extension key, since OpenAI SDKs construct exceptions from `type`/`code`.721722#### 2.2.6 Fallbacks & routing preferences723724(Sources: https://openrouter.ai/docs/guides/routing/model-fallbacks, https://openrouter.ai/docs/features/provider-routing)725726- **Model fallbacks**: an extra `models: ["primary", "fallback1", ...]` array in the body (the `model` field is the first attempt; via OpenAI SDK it goes in `extra_body`). Fallback triggers on *any* error: context-length validation, moderation flags, rate limits, downtime. **The response's `model` field always reports the model actually used, and pricing follows the actually-used model.** This "honest `model` echo" is the contract Zyquo Router's CLAUDE.md already mandates for its fallback chains.727- **Provider preferences** (`provider` object): `order` (try providers in this order), `allow_fallbacks` (default true), `require_parameters`, `ignore`/`only` allow/deny lists, `sort` by price/throughput/latency, `max_price`. Default load balancing prefers providers without recent outages, weighted by **inverse square of price**. Mostly N/A for a local single-key-per-provider router, but the *cooldown-on-recent-outage* idea maps to LiteLLM cooldowns.728- `finish_reason` is **normalized to exactly five values** — `stop`, `length`, `tool_calls`, `content_filter`, `error` — with the raw upstream value preserved in a separate `native_finish_reason` field (https://openrouter.ai/docs/api-reference/overview). Recommended verbatim for Zyquo Router (OpenAI's own set is the first four plus `function_call` legacy; `error` only ever appears mid-stream).729730---731732### 2.3 Local OpenAI-compatible servers — what "compatible enough" looks like733734These show which subset of the spec real clients actually depend on, and which deviations are tolerated.735736#### 2.3.1 Ollama (`http://localhost:11434/v1`)737738(Source: https://docs.ollama.com/api/openai-compatibility)739740- Endpoints: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/v1/models/{model}`, `/v1/embeddings`, `/v1/responses`.741- Supported on chat: `model`, `messages`, `temperature`, `top_p`, `max_tokens`, `frequency_penalty`, `presence_penalty`, `seed`, `stop`, `stream`, `stream_options.include_usage`, `response_format` (JSON mode), `tools`, `reasoning_effort`/`reasoning`, vision via **base64 images only**.742- **Not supported**: `logprobs`, `user`, `n`, `tool_choice`, `logit_bias`, image **URLs**. Unsupported params are ignored rather than erroring.743- Auth: any/no API key accepted (pure localhost trust). Zyquo Router improves on this with optional local keys.744- Notable: Ollama had to add `max_completion_tokens` support because OpenAI deprecated `max_tokens` (https://github.com/ollama/ollama/issues/7125) — see §2.4.5.745746Takeaways: even a hugely popular compat layer omits `n`, `logprobs`, `logit_bias` and (long) omitted `tool_choice` — clients broadly tolerate missing niche params, but `tools`, `response_format`, `stream_options.include_usage`, and vision are table stakes in 2026.747748#### 2.3.2 LM Studio (`http://localhost:1234/v1`)749750(Source: https://lmstudio.ai/docs/app/api/endpoints/openai)751752- Endpoints: `/v1/models`, `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, and **`/v1/responses`** (added specifically so OpenAI Codex CLI works against it).753- Compatibility story is purely "change `base_url` on the official SDK" — the same acceptance test Zyquo Router's Phase 3 gate uses.754- Signal: local servers are converging on also exposing `/v1/responses` because new OpenAI tooling (Codex) speaks only the Responses API. Relevant to the Phase 0 "is `/v1/responses` worth it" decision: *optional now, trending toward expected*.755756#### 2.3.3 vLLM OpenAI-compatible server757758(Source: https://docs.vllm.ai/en/latest/serving/online_serving/)759760- Implements `/v1/chat/completions` (+ batch), `/v1/completions` (no `suffix`), `/v1/responses`, `/v1/embeddings`, transcription/translation.761- Known deviations: `user` is ignored; `parallel_tool_calls` defaults to `true`; extra sampling params (`top_k`, `best_of`, guided decoding) accepted via `extra_body` — the standard pass-through idiom again.762- vLLM's habit of sending **usage in a separate empty-`choices` chunk after the `finish_reason` chunk** is spec-correct but broke LiteLLM's consumer (https://github.com/BerriAI/litellm/issues/25389) — Zyquo Router's normalizer must handle both orderings when consuming OpenAI-compatible upstreams.763764---765766### 2.4 Compatibility pitfalls that trip up real clients767768Strict SDKs (openai-python uses Pydantic models; openai-node/Vercel AI SDK use zod-like validation) parse every chunk. These are the documented, real-world failure modes a gateway must design around:7697701. **Usage chunk shape.** With `stream_options: {"include_usage": true}`, OpenAI's contract is: every content chunk has `"usage": null`, and one **final extra chunk before `[DONE]`** has `"choices": []` and the full `usage` object. Emitting usage with non-empty choices violates the spec (LiteLLM bug https://github.com/BerriAI/litellm/issues/28735); conversely, clients that stop at `finish_reason` miss the usage chunk (https://github.com/BerriAI/litellm/issues/25389); AutoGen crashed on the empty-choices chunk itself (https://github.com/microsoft/autogen/issues/5078); llama.cpp put usage in a slightly different chunk than OpenAI and broke clients (https://github.com/ggml-org/llama.cpp/issues/15443). **Rule: emit usage exactly like OpenAI (separate final empty-choices chunk, only when requested), and when consuming, read to true end-of-stream and tolerate usage in any late chunk.**7712. **Streamed tool-call deltas.** The first `tool_calls` delta must carry `index`, `id`, `type: "function"`, and `function.name`; subsequent deltas carry only `index` + `function.arguments` fragments. Real breakage: Gemini-behind-a-compat-shim omitting `index` (https://github.com/anomalyco/opencode/issues/17902); providers sending `function.name` only in a *later* chunk (https://github.com/anomalyco/opencode/issues/24137, https://github.com/anomalyco/opencode/issues/26412); vLLM omitting `"type":"function"` under forced `tool_choice` (https://github.com/vllm-project/vllm/issues/16340). **Rule: the translator owns tool-call chunk assembly — always emit index/id/type/name complete in the first delta for each call, arguments-only after.**7723. **Role delta discipline.** The first chunk of each choice must have `delta: {"role": "assistant"}` (optionally with `content: ""`); an **empty-string role** instead of `"assistant"` breaks strict parsers (https://github.com/anomalyco/opencode/issues/28427). Later deltas must omit `role` entirely rather than repeat it as `""`.7734. **All-or-nothing chunk validation.** Some client stacks silently drop an entire chunk if any field fails validation — provider quirks then surface as *silently missing content*, which is undebuggable (https://github.com/Effect-TS/effect-smol/issues/2337, https://github.com/pydantic/pydantic-ai/issues/3658 — OpenRouter `reasoning_details` variant missing a field broke pydantic-ai). **Rule: every field Zyquo Router emits must be exactly typed (`created` as integer epoch seconds, `object` exactly `"chat.completion.chunk"`, `index` present on every choice/tool_call); when *adding* fields (e.g. `reasoning_content`), add only well-formed, consistently shaped ones.** Also: null-valued token-detail fields inside `usage` broke the OpenAI Agents SDK (https://github.com/openai/openai-agents-python/issues/1179) — omit detail objects rather than sending them with `null` members.7745. **`max_tokens` vs `max_completion_tokens`.** OpenAI deprecated `max_tokens` in favor of `max_completion_tokens`; o-series/reasoning models hard-reject `max_tokens` ("Unsupported parameter"). Every ecosystem project had to patch (Ollama https://github.com/ollama/ollama/issues/7125, simonw/llm https://github.com/simonw/llm/issues/724, Home Assistant https://github.com/home-assistant/core/issues/137039, Spring AI https://github.com/spring-projects/spring-ai/issues/3300). **Rule: accept both on ingress, normalize internally to one limit value, emit whichever the upstream requires (per-provider table), never forward both.**7756. **`system_fingerprint`.** Optional in practice — OpenAI itself returns `null`/absent for many models (https://github.com/openai/openai-python/issues/1038, https://github.com/openai/openai-openapi/issues/167), and SDK type defs treat it as optional (https://github.com/openai/openai-node/issues/443). Gateways may safely omit it or set `null`; do not fabricate values (clients use it for determinism tracking with `seed`).7767. **`n > 1`.** Most non-OpenAI upstreams don't support multiple choices (Ollama: unsupported; Anthropic/Gemini: no direct equivalent). Options: reject with a clear 400, or fan out N upstream calls. LiteLLM/Ollama precedent: reject or ignore. `choices` must still always be an **array** with correct `index` fields even for n=1 (OpenRouter: "choices is always an array" — https://openrouter.ai/docs/api-reference/overview).7778. **Keep-alives, buffering, and timeouts.** Long prefills (big prompts, reasoning models) can be silent for 30s+; intermediaries and client idle timeouts kill the connection (e.g. https://github.com/microsoft/agent-framework/issues/6941). SSE **comment lines** (`: keep-alive`) every 15–30s are the only spec-compatible heartbeat; also send `Cache-Control: no-cache`, `Connection: keep-alive`, `X-Accel-Buffering: no`, and never gzip SSE (compression layers buffer the stream). Conversely, when *consuming*, tolerate comment lines from upstreams (OpenRouter emits `: OPENROUTER PROCESSING` — https://openrouter.ai/docs/api-reference/streaming). Streaming responses need effectively unlimited write timeouts; non-streaming needs a generous but bounded upstream timeout mapped to 504/408.7789. **CORS for browser clients.** A cross-origin `fetch` to the router fails before the first byte without correct `Access-Control-Allow-Origin` + preflight handling for `POST` with `Authorization`/`Content-Type: application/json` headers. OpenRouter even classifies CORS problems under 400 (https://openrouter.ai/docs/api-reference/errors). Zyquo Router's default-permissive-on-localhost CORS (with `OPTIONS` preflight support) is the right call for browser-based dev tools.77910. **Mid-stream failure surfacing.** After the first chunk, the status line is committed (200). The only clean options are OpenRouter's error-chunk-with-`finish_reason:"error"` (§2.2.3) followed by `[DONE]`, or an abrupt close (which strict SDKs report as a network error). Emit the error chunk.780781---782783### 2.5 Recommended pattern set for Zyquo Router784785Synthesis of the above into the concrete policy for our gateway:7867871. **Model namespacing** — `provider/model-id` (LiteLLM/OpenRouter convention), full catalog from Zyquo Cloud; also accept unambiguous bare upstream IDs (resolve via catalog; ambiguous → 400 listing candidates) and user aliases (LiteLLM `model_group_alias` pattern). `GET /v1/models` returns OpenAI list shape with `id` = namespaced ID, enriched per-model metadata (`context_length`, pricing as decimal strings, capability flags, `supported_parameters`) under an `x-zyquo` key, following OpenRouter's metadata precedent.7882. **Unknown/unsupported-param policy** — data-driven per-provider tables (LiteLLM style): *rename* where names differ (`max_tokens`/`max_completion_tokens`, `stop`→`stop_sequences`), *strip silently* what the upstream would reject (default `drop_params: true` behavior, log at debug in the request inspector), *reject with a helpful OpenAI-format 400* only when silently dropping would change semantics materially (e.g. `tools` on a no-tools model, `n>1`), and *pass through* unknown extra body keys to OpenAI-compatible upstreams (the `extra_body` idiom) — documented per provider in `docs/API.md`.7893. **finish_reason & usage normalization** — normalize `finish_reason` to `stop | length | tool_calls | content_filter` (+ `error` mid-stream only), preserving the raw upstream value as `native_finish_reason` (OpenRouter pattern). Usage: real upstream numbers when available (request usage-in-stream from upstreams that support it, per-provider flag); tokenizer-estimated otherwise, flagged (e.g. `"x-zyquo": {"usage_estimated": true}`); cost computed from the catalog's pricing at response time and exposed OpenRouter-style as extra `usage` fields. Streaming usage emitted **only** when the client sends `stream_options.include_usage`, as a final `"choices": []` chunk before `[DONE]` — byte-exact per §2.4.1–4.7904. **Upstream error surfacing** — map to OpenAI error JSON `{"error":{"message","type","param","code"}}` with LiteLLM's status taxonomy (401 provider-key invalid naming the provider, 429 with propagated `Retry-After`, 400 subtypes for context-window/content-policy, 502 "upstream returned an invalid response", 503 provider unavailable, 504/408 timeouts); include a canonical machine `code` plus the upstream's original code in the message (OpenRouter's `error_type`+`provider_code` idea) — never raw upstream payloads or key material. Mid-stream: OpenRouter-style error chunk with `finish_reason: "error"`, then `[DONE]`.7915. **Retry/fallback policy** — per-error-class retry policy (LiteLLM `RetryPolicy`): retries with exponential backoff + jitter on 429/5xx/timeouts (honoring `Retry-After`), zero retries on 400/401/403; then user-configured fallback chains (ordered model lists) triggered on retry exhaustion, context-window and content-policy errors; the response `model` field reports the model actually used and cost follows it (OpenRouter contract). Optional per-provider cooldown state feeding the dashboard's "degraded" indicator.7926. **Health & liveness** — `GET /health` (status, version, uptime, active streams) never touches upstreams; per-provider "Test key" in the UI does a minimal authenticated upstream call and reports latency; SSE keep-alive comments every ~20s of upstream silence; client disconnect cancels the upstream task immediately (guaranteed, since we own the upstream HTTP call).793794---795796*Primary sources:* https://docs.litellm.ai/docs/proxy/configs · https://docs.litellm.ai/docs/completion/drop_params · https://docs.litellm.ai/docs/exception_mapping · https://docs.litellm.ai/docs/routing · https://docs.litellm.ai/docs/proxy/reliability · https://docs.litellm.ai/docs/completion/token_usage · https://docs.litellm.ai/docs/completion/stream · https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json · https://openrouter.ai/docs/api-reference/streaming · https://openrouter.ai/docs/api-reference/errors · https://openrouter.ai/docs/api-reference/overview · https://openrouter.ai/docs/use-cases/usage-accounting · https://openrouter.ai/docs/guides/routing/model-fallbacks · https://openrouter.ai/docs/features/provider-routing · https://openrouter.ai/docs/models · https://docs.ollama.com/api/openai-compatibility · https://lmstudio.ai/docs/app/api/endpoints/openai · https://docs.vllm.ai/en/latest/serving/online_serving/ · GitHub issues linked inline (§2.4).797## 3. Translation matrices798799> Research date: 2026-07-30. Verified against live official documentation (URLs cited inline).800> This section is the contract for `Translate/AnthropicTranslator.swift`, `Translate/GeminiTranslator.swift`,801> and `Translate/CompatAdjuster.swift`. The router's canonical internal format IS the OpenAI802> `chat/completions` wire format (Section 1); every non-OpenAI upstream is mapped bidirectionally onto it.803804---805806### 3.1 Anthropic Messages API ⇄ OpenAI chat/completions807808Primary sources:809- Messages API reference: https://platform.claude.com/docs/en/api/messages (docs.anthropic.com 301-redirects here)810- Streaming: https://platform.claude.com/docs/en/docs/build-with-claude/streaming811- Tool use: https://platform.claude.com/docs/en/docs/agents-and-tools/tool-use/overview812- Errors: https://platform.claude.com/docs/en/api/errors813814#### 3.1.1 Endpoint & auth815816| | OpenAI (what our client sends us) | Anthropic (what we send upstream) |817|---|---|---|818| Endpoint | `POST /v1/chat/completions` | `POST https://api.anthropic.com/v1/messages` |819| Auth header | `Authorization: Bearer zyquo-sk-…` (local key) | `x-api-key: <ANTHROPIC_KEY>` |820| Version header | — | `anthropic-version: 2023-06-01` (required) |821| Content type | `application/json` | `application/json` |822823The `anthropic-version` header is **mandatory**; requests without it are rejected. Pin `2023-06-01` (the stable version used by all official SDKs).824825#### 3.1.2 Request translation (OpenAI → Anthropic)826827##### Parameter map828829| OpenAI request field | Anthropic field | Rule |830|---|---|---|831| `model` | `model` | Strip the `anthropic/` namespace prefix. |832| `messages[role=system]`, `messages[role=developer]` | top-level `system` | **Extract** all system/developer messages (in order), join text with `"\n\n"`. Anthropic has no `system` role inside `messages`. `developer` is treated identically to `system`. |833| `messages[role=user/assistant/tool]` | `messages` | See message-shape rules below. |834| `max_tokens` / `max_completion_tokens` | `max_tokens` | **REQUIRED by Anthropic.** If the client omits both, the router MUST synthesize a value. Strategy: use the model's catalog `max_output` (from Zyquo Cloud's `ModelCatalog`); fall back to `4096` if unknown. `max_completion_tokens` wins if both present. |835| `temperature` | `temperature` | OpenAI range **0–2**, Anthropic range **0–1** (default 1.0). Strategy: `min(temperature, 1.0)` (clamp). Do NOT divide by 2 — halving changes semantics for the common 0–1 sub-range that clients actually use. Log a warning when clamping. |836| `top_p` | `top_p` | Same 0–1 range, pass through. Anthropic advises using temperature OR top_p, not both — pass both if given (API accepts it). |837| — (extra body `top_k`) | `top_k` | Not an OpenAI param; accept as pass-through extra key. |838| `stop` (string or array) | `stop_sequences` (array) | Wrap a bare string in a 1-element array. |839| `n` | — | **Unsupported.** If `n > 1` → reject with OpenAI 400 error (`invalid_request_error`, param `n`). |840| `frequency_penalty`, `presence_penalty`, `logit_bias`, `seed`, `logprobs`, `top_logprobs` | — | **Unsupported → strip silently** (LiteLLM behavior); optionally record in the request log that params were dropped. |841| `stream` | `stream` | Pass through. |842| `stream_options` | — | Router-side only (controls our usage chunk emission). Never forwarded. |843| `user` | `metadata.user_id` | Direct map (Anthropic wants an opaque non-PII id — pass as-is). |844| `tools` | `tools` | See tools mapping. |845| `tool_choice` | `tool_choice` | See tool_choice mapping. |846| `parallel_tool_calls: false` | `tool_choice.disable_parallel_tool_use: true` | Set on whatever `tool_choice` object we send (`auto` if none was specified). `parallel_tool_calls: true` → omit (default). |847| `response_format` | `output_config.format` / workaround | See JSON-mode strategy. |848| `reasoning_effort` (OpenAI standard) | `output_config.effort` / `thinking` | See §3.4 (reasoning). |849| extra body `thinking` | `thinking` | Pass-through extra key for power users: `{"type":"enabled"\|"adaptive"\|"disabled","budget_tokens":≥1024,"display":"summarized"\|"omitted"}`. |850851##### Message-shape rules (the tricky part)852853Anthropic enforces constraints that OpenAI does not:8548551. **Roles are only `user` and `assistant`** inside `messages`.8562. **Turns must alternate.** Consecutive same-role messages must be **merged** into a single message whose `content` is an array of blocks, preserving order.8573. **The first message must be `user`.** If the client's first non-system message is `assistant`, prepend a placeholder user message (e.g. a single text block `"(continue)"`) or reject — the router merges/prepends (LiteLLM's approach) so real clients keep working.8584. **OpenAI `tool` role messages → `user` messages containing `tool_result` blocks.** Consecutive `tool` messages (parallel tool results) merge into ONE user message with multiple `tool_result` blocks. `tool_result` blocks must come FIRST in that user message's content array if user text follows.8595. **Assistant `tool_calls` → `tool_use` content blocks**, after any text content, with `input` as a **parsed JSON object** (OpenAI `arguments` is a JSON *string* — parse it; if unparseable, send `{}` and log).860861##### Content-part map862863| OpenAI content part | Anthropic content block |864|---|---|865| `{"type":"text","text":T}` | `{"type":"text","text":T}` |866| `{"type":"image_url","image_url":{"url":"data:image/png;base64,XXX"}}` | `{"type":"image","source":{"type":"base64","media_type":"image/png","data":"XXX"}}` — media_type parsed from the data URL; allowed: `image/jpeg`, `image/png`, `image/gif`, `image/webp`. |867| `{"type":"image_url","image_url":{"url":"https://…"}}` | `{"type":"image","source":{"type":"url","url":"https://…"}}` — Anthropic supports URL sources natively. |868| plain string `content` | plain string `content` (both APIs accept a bare string). |869870##### Side-by-side request example (tools + image + system)871872OpenAI request received by the router:873874```json875{876 "model": "anthropic/claude-sonnet-4-5",877 "max_tokens": 1024,878 "temperature": 1.4,879 "messages": [880 {"role": "system", "content": "You are terse."},881 {"role": "user", "content": [882 {"type": "text", "text": "What's in this image, and what's the weather there?"},883 {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ..."}}884 ]},885 {"role": "assistant", "content": null, "tool_calls": [886 {"id": "call_abc123", "type": "function",887 "function": {"name": "get_weather", "arguments": "{\"location\": \"Paris\"}"}}888 ]},889 {"role": "tool", "tool_call_id": "call_abc123", "content": "18°C, sunny"}890 ],891 "tools": [892 {"type": "function", "function": {893 "name": "get_weather",894 "description": "Get current weather",895 "parameters": {"type": "object",896 "properties": {"location": {"type": "string"}}, "required": ["location"]}}}897 ],898 "tool_choice": "auto",899 "parallel_tool_calls": false900}901```902903Anthropic request the router sends upstream:904905```json906{907 "model": "claude-sonnet-4-5",908 "max_tokens": 1024,909 "temperature": 1.0,910 "system": "You are terse.",911 "messages": [912 {"role": "user", "content": [913 {"type": "text", "text": "What's in this image, and what's the weather there?"},914 {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": "/9j/4AAQ..."}}915 ]},916 {"role": "assistant", "content": [917 {"type": "tool_use", "id": "call_abc123", "name": "get_weather",918 "input": {"location": "Paris"}}919 ]},920 {"role": "user", "content": [921 {"type": "tool_result", "tool_use_id": "call_abc123", "content": "18°C, sunny"}922 ]}923 ],924 "tools": [925 {"name": "get_weather", "description": "Get current weather",926 "input_schema": {"type": "object",927 "properties": {"location": {"type": "string"}}, "required": ["location"]}}928 ],929 "tool_choice": {"type": "auto", "disable_parallel_tool_use": true}930}931```932933Note the tool_use `id` is preserved verbatim in both directions so multi-turn tool loops round-trip.934935##### Tools & tool_choice map936937| OpenAI | Anthropic |938|---|---|939| `tools[].function.name` | `tools[].name` |940| `tools[].function.description` | `tools[].description` |941| `tools[].function.parameters` (JSON Schema) | `tools[].input_schema` (JSON Schema draft 2020-12) |942| `tools[].function.strict: true` | `tools[].strict: true` (now supported natively) |943| `tool_choice: "auto"` (or omitted with tools) | `{"type": "auto"}` |944| `tool_choice: "required"` | `{"type": "any"}` |945| `tool_choice: "none"` | `{"type": "none"}` |946| `tool_choice: {"type":"function","function":{"name":N}}` | `{"type": "tool", "name": N}` |947| `parallel_tool_calls: false` | `disable_parallel_tool_use: true` on the tool_choice object |948949##### JSON mode / `response_format` strategy950951Anthropic historically had **no native JSON mode**; the current API (2026) has **structured output** via952`output_config.format` (source: https://platform.claude.com/docs/en/api/messages — `output_config: { format: { type: "json_schema", schema: {...} } }`).953954Router strategy, in priority order:9559561. `response_format: {"type":"json_schema","json_schema":{"schema":S,...}}` → `output_config: {"format": {"type":"json_schema","schema": S}}` on models that support it (catalog capability flag).9572. On models WITHOUT structured-output support — **tool trick**: define a single synthetic tool (`name: "json_output"`, `input_schema: S`), force it with `tool_choice: {"type":"tool","name":"json_output"}`, and return the streamed/collected `input` object as the assistant `content` string (finish_reason `stop`, not `tool_calls`).9583. `response_format: {"type":"json_object"}` → append to `system`: `"You must respond with valid JSON only, no prose, no markdown fences."` and optionally **prefill** the assistant turn with `{` (append `{"role":"assistant","content":"{"}` and re-prepend `{` to the returned text). Prefill is Anthropic-sanctioned steering. Document in `docs/API.md` that json_object on Anthropic is best-effort.9594. `{"type":"text"}` → no-op.960961##### Extended thinking (request side)962963- `thinking: {"type":"enabled","budget_tokens":N}` (N ≥ 1024, must be < `max_tokens`) or `{"type":"adaptive"}`; `display: "summarized"|"omitted"` controls whether thinking text is streamed.964- Newer models also take `output_config.effort: "low"|"medium"|"high"|"xhigh"|"max"`.965- Router mapping for the standard OpenAI `reasoning_effort` param: `low → output_config.effort "low"` (or `thinking budget 1024`), `medium → "medium"` (8192), `high → "high"` (24576) — per-model capability gate from the catalog. Raw `thinking` extra-body always wins if provided.966- **Multi-turn constraint:** in tool-use loops with thinking enabled, Anthropic expects prior `thinking` blocks (with `signature`) to be passed back. See §3.4 for how the router preserves signatures via `reasoning_details`.967968#### 3.1.3 Response translation (Anthropic → OpenAI), non-streaming969970Anthropic response:971972```json973{974 "id": "msg_01XFDUDYJgAACzvnptvVoYEL",975 "type": "message",976 "role": "assistant",977 "model": "claude-sonnet-4-5",978 "content": [979 {"type": "text", "text": "It's 18°C and sunny in Paris."},980 {"type": "tool_use", "id": "toolu_01A09q90qw90lq917835lq9",981 "name": "get_weather", "input": {"location": "Paris"}}982 ],983 "stop_reason": "tool_use",984 "stop_sequence": null,985 "usage": {986 "input_tokens": 412, "output_tokens": 61,987 "cache_creation_input_tokens": 0, "cache_read_input_tokens": 128988 }989}990```991992Router emits:993994```json995{996 "id": "chatcmpl-9f3c1a2b7d4e",997 "object": "chat.completion",998 "created": 1753872000,999 "model": "anthropic/claude-sonnet-4-5",1000 "choices": [{1001 "index": 0,1002 "message": {1003 "role": "assistant",1004 "content": "It's 18°C and sunny in Paris.",1005 "tool_calls": [{1006 "id": "toolu_01A09q90qw90lq917835lq9",1007 "type": "function",1008 "function": {"name": "get_weather", "arguments": "{\"location\":\"Paris\"}"}1009 }]1010 },1011 "finish_reason": "tool_calls"1012 }],1013 "usage": {1014 "prompt_tokens": 540,1015 "completion_tokens": 61,1016 "total_tokens": 601,1017 "prompt_tokens_details": {"cached_tokens": 128}1018 }1019}1020```10211022Rules:10231024- **`id`**: generate a fresh `chatcmpl-<hex>` (keep the upstream `msg_…` id in the request log for tracing). `object: "chat.completion"`, `created`: gateway clock (Unix seconds). `model`: echo the **namespaced** router id.1025- **Content blocks →** concatenate all `text` block texts into `message.content` (`null` if none and tool_calls exist); each `tool_use` block → one `tool_calls[]` entry with `arguments` = `JSON.stringify(input)`; `thinking` blocks → `message.reasoning_content` (§3.4), `signature` → `reasoning_details`; `redacted_thinking` → `reasoning_details` only (opaque `data`).1026- **`stop_reason` → `finish_reason`:**10271028| Anthropic `stop_reason` | OpenAI `finish_reason` | Note |1029|---|---|---|1030| `end_turn` | `stop` | |1031| `max_tokens` | `length` | |1032| `stop_sequence` | `stop` | OpenAI has no separate value; log the matched `stop_sequence`. |1033| `tool_use` | `tool_calls` | |1034| `refusal` | `content_filter` | Closest OpenAI semantic; `stop_details` (category/explanation) goes to the request log only. |1035| `pause_turn` | `stop` | Long-running server-tool turns; router doesn't use server tools, treat as stop. |1036| `model_context_window_exceeded` | `length` | |10371038- **`usage`:** Anthropic's `input_tokens` EXCLUDES cache reads/writes. Normalize:1039 `prompt_tokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokens`;1040 `completion_tokens = output_tokens` (already includes thinking tokens);1041 `total_tokens = prompt + completion`;1042 `prompt_tokens_details.cached_tokens = cache_read_input_tokens`;1043 `completion_tokens_details.reasoning_tokens = usage.output_tokens_details.thinking_tokens` when present.10441045#### 3.1.4 SSE event model → OpenAI `chat.completion.chunk` (streaming)10461047Anthropic stream grammar (source: https://platform.claude.com/docs/en/docs/build-with-claude/streaming):10481049```1050message_start1051( content_block_start → content_block_delta* → content_block_stop )*1052message_delta+1053message_stop1054```1055with `ping` events anywhere and possible `error` events. Each SSE frame is1056`event: <name>\ndata: <json>\n\n`. `message_delta.usage.output_tokens` is **cumulative**.10571058Delta types inside `content_block_delta`: `text_delta` (`.text`), `input_json_delta` (`.partial_json`, a partial JSON **string** for `tool_use.input`), `thinking_delta` (`.thinking`), `signature_delta` (`.signature`, arrives just before the thinking block's `content_block_stop`).10591060##### Event → chunk mapping table10611062The translator keeps two counters: `toolIdx` = number of `tool_use` blocks seen so far (this is the OpenAI `tool_calls[].index`, 0-based, **independent of the Anthropic block `index`**), and cumulative usage.10631064| Anthropic event | Emitted OpenAI chunk delta | Notes |1065|---|---|---|1066| `message_start` | `{"delta":{"role":"assistant","content":""},"finish_reason":null}` | First chunk; role delta exactly once. Capture `message.usage.input_tokens` (+cache fields) for the final usage chunk. |1067| `ping` | *(nothing)* — or forward as SSE comment `: keep-alive` | Comments keep clients' sockets warm without confusing SDK parsers. |1068| `content_block_start` (`type:"text"`) | *(nothing)* | |1069| `content_block_delta` / `text_delta` | `{"delta":{"content": text}}` | |1070| `content_block_start` (`type:"tool_use"`) | `{"delta":{"tool_calls":[{"index": toolIdx, "id": block.id, "type": "function", "function": {"name": block.name, "arguments": ""}}]}}` | id + name announced once, `arguments:""` starts the accumulator — exactly the shape the OpenAI SDKs expect. |1071| `content_block_delta` / `input_json_delta` | `{"delta":{"tool_calls":[{"index": toolIdx, "function": {"arguments": partial_json}}]}}` | No id/name repetition. Empty `partial_json` frames may be skipped. |1072| `content_block_stop` (tool_use) | *(nothing)*; `toolIdx += 1` | |1073| `content_block_delta` / `thinking_delta` | `{"delta":{"reasoning_content": thinking}}` | §3.4 convention. |1074| `content_block_delta` / `signature_delta` | `{"delta":{"reasoning_details":[{"type":"anthropic.signature","signature":…,"index":blockIdx}]}}` — or drop if client didn't opt in | Needed only to round-trip thinking in tool loops. |1075| `content_block_start/stop` (`type:"thinking"`) | *(nothing)* | |1076| `message_delta` | `{"delta":{},"finish_reason": map(stop_reason)}` | finish_reason chunk (empty delta object, per OpenAI spec). Capture cumulative `usage.output_tokens`. |1077| `message_stop` | If `stream_options.include_usage`: `{"choices":[],"usage":{…}}` chunk; then `data: [DONE]` | Usage chunk has an EMPTY `choices` array per OpenAI spec. Then terminate. |1078| `error` | Emit `data: {"error":{"message":…,"type":"api_error","code":upstream_type}}` then close | OpenAI has no in-band stream-error spec; this LiteLLM-style error frame is the least-bad option — document it in `docs/API.md`. Map `overloaded_error` → our 529→503 semantics in logs. |10791080Every emitted chunk carries the constant envelope:1081`{"id":"chatcmpl-…","object":"chat.completion.chunk","created":C,"model":"anthropic/…","choices":[{"index":0,"delta":…,"finish_reason":…}]}` — same `id`/`created` for the whole stream.10821083##### Full example transcript (tool-use stream)10841085Anthropic events (left) → OpenAI chunks emitted by the router (right). Envelope fields elided for readability; every right-hand line is a full `chat.completion.chunk`.10861087```1088ANTHROPIC UPSTREAM → ZYQUO ROUTER EMITS (OpenAI SSE)10891090event: message_start1091data: {"type":"message_start","message":{"id":"msg_014p", → data: {"id":"chatcmpl-a1","object":"chat.completion.chunk",1092 "role":"assistant","content":[],"model":"claude-…", "created":1753872000,"model":"anthropic/claude-sonnet-4-5",1093 "usage":{"input_tokens":472,"output_tokens":2}, …}} "choices":[{"index":0,"delta":{"role":"assistant",1094 "content":""},"finish_reason":null}]}10951096event: content_block_start1097data: {"type":"content_block_start","index":0, → (nothing)1098 "content_block":{"type":"text","text":""}}10991100event: ping1101data: {"type":"ping"} → (nothing, or ": keep-alive" comment)11021103event: content_block_delta1104data: {…,"delta":{"type":"text_delta","text":"Okay,"}} → data: {…,"choices":[{"index":0,"delta":{"content":"Okay,"},1105 "finish_reason":null}]}11061107event: content_block_delta1108data: {…,"delta":{"type":"text_delta", → data: {…,"choices":[{"index":0,"delta":{"content":1109 "text":" checking the weather:"}} " checking the weather:"},"finish_reason":null}]}11101111event: content_block_stop1112data: {"type":"content_block_stop","index":0} → (nothing)11131114event: content_block_start1115data: {"type":"content_block_start","index":1, → data: {…,"choices":[{"index":0,"delta":{"tool_calls":[1116 "content_block":{"type":"tool_use", {"index":0,"id":"toolu_01T1x","type":"function",1117 "id":"toolu_01T1x","name":"get_weather","input":{}}} "function":{"name":"get_weather","arguments":""}}]},1118 "finish_reason":null}]}11191120event: content_block_delta1121data: {…,"delta":{"type":"input_json_delta", → (skipped — empty partial_json)1122 "partial_json":""}}11231124event: content_block_delta1125data: {…,"delta":{"type":"input_json_delta", → data: {…,"choices":[{"index":0,"delta":{"tool_calls":[1126 "partial_json":"{\"location\":"}} {"index":0,"function":{"arguments":"{\"location\":"}}]},1127 "finish_reason":null}]}11281129event: content_block_delta1130data: {…,"delta":{"type":"input_json_delta", → data: {…,"choices":[{"index":0,"delta":{"tool_calls":[1131 "partial_json":" \"San Francisco, CA\"}"}} {"index":0,"function":{"arguments":1132 " \"San Francisco, CA\"}"}}]},"finish_reason":null}]}11331134event: content_block_stop1135data: {"type":"content_block_stop","index":1} → (nothing; toolIdx→1)11361137event: message_delta1138data: {"type":"message_delta","delta":{"stop_reason": → data: {…,"choices":[{"index":0,"delta":{},1139 "tool_use","stop_sequence":null}, "finish_reason":"tool_calls"}]}1140 "usage":{"output_tokens":89}}11411142event: message_stop → data: {…,"choices":[],"usage":{"prompt_tokens":472,1143data: {"type":"message_stop"} "completion_tokens":89,"total_tokens":561}}1144 (only if stream_options.include_usage)1145 → data: [DONE]1146```11471148Thinking streams follow the same pattern: `content_block_start {type:"thinking"}` opens nothing, each `thinking_delta` → `{"delta":{"reasoning_content":"…"}}`, `signature_delta` → `reasoning_details` (or dropped), then the text block streams as normal `content` deltas.11491150---11511152### 3.2 Gemini `generateContent` / `streamGenerateContent` ⇄ OpenAI11531154Primary sources:1155- API reference: https://ai.google.dev/api/generate-content1156- Part/Content schema: https://ai.google.dev/api/caching#Part1157- Function calling: https://ai.google.dev/gemini-api/docs/function-calling1158- Structured output: https://ai.google.dev/gemini-api/docs/structured-output1159- Thinking: https://ai.google.dev/gemini-api/docs/thinking1160- Google's own OpenAI-compat layer (used as a mapping oracle): https://ai.google.dev/gemini-api/docs/openai11611162#### 3.2.1 Endpoint & auth11631164| | Form |1165|---|---|1166| Non-streaming | `POST https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent` |1167| Streaming (SSE) | `POST https://generativelanguage.googleapis.com/v1beta/models/{model}:streamGenerateContent?alt=sse` |1168| Auth | Header `x-goog-api-key: <GEMINI_KEY>` (preferred) or query `?key=<GEMINI_KEY>` |11691170**Always use `?alt=sse`** — without it, streamGenerateContent returns a chunked **JSON array**, not SSE. The model name is in the **path**, not the body. Use the header for auth so the key never appears in URLs/logs.11711172#### 3.2.2 Request translation (OpenAI → Gemini)11731174| OpenAI field | Gemini field | Rule |1175|---|---|---|1176| `model` | URL path | Strip `gemini/` prefix. |1177| system/developer messages | `systemInstruction: {"parts":[{"text": joined}]}` | Join multiple with `"\n\n"`. |1178| `messages[role=user]` | `contents[]` entry `role: "user"` | |1179| `messages[role=assistant]` | `contents[]` entry `role: "model"` | **Role rename user/assistant → user/model.** |1180| `messages[role=tool]` | `contents[]` entry `role: "user"` with `functionResponse` part(s) | See tool round-trip below. Consecutive `tool` messages merge into one user-role content with multiple `functionResponse` parts. |1181| `max_tokens`/`max_completion_tokens` | `generationConfig.maxOutputTokens` | Optional on Gemini (nice: no synthesis needed). |1182| `temperature` | `generationConfig.temperature` | Both 0–2. Pass through unchanged. |1183| `top_p` | `generationConfig.topP` | camelCase rename. |1184| extra `top_k` | `generationConfig.topK` | |1185| `stop` | `generationConfig.stopSequences` | Wrap string → array. |1186| `n` | `generationConfig.candidateCount` (1–8) | Router policy: support `n` here (one of the few upstreams that can) or clamp to 1 for uniformity — **decide once; recommend rejecting n>1 router-wide** for consistent behavior across providers. |1187| `seed` | `generationConfig.seed` | |1188| `presence_penalty` | `generationConfig.presencePenalty` | −2..2, same semantics. |1189| `frequency_penalty` | `generationConfig.frequencyPenalty` | |1190| `logit_bias`, `logprobs`, `user` | — | Strip. |1191| `response_format {"type":"json_object"}` | `generationConfig.responseMimeType: "application/json"` | |1192| `response_format {"type":"json_schema",…}` | `responseMimeType: "application/json"` + `generationConfig.responseJsonSchema` (standard JSON Schema; older models: `responseSchema` OpenAPI-subset) | Prefer `responseJsonSchema`; scrub unsupported keywords (`$schema`, `additionalProperties` on old models) defensively. |1193| `tools` | `tools: [{"functionDeclarations":[{name, description, parameters}]}]` | ALL functions go into ONE `functionDeclarations` array. `parameters` is JSON-Schema-like; scrub `strict`. |1194| `tool_choice` | `toolConfig.functionCallingConfig` | `"auto"`→`{"mode":"AUTO"}`; `"required"`→`{"mode":"ANY"}`; `"none"`→`{"mode":"NONE"}`; `{"function":{"name":N}}`→`{"mode":"ANY","allowedFunctionNames":[N]}`. (A `VALIDATED` mode also exists; unused by the router.) |1195| `parallel_tool_calls` | — | No Gemini equivalent; strip. Gemini decides parallelism itself (multiple `functionCall` parts in one candidate). |1196| `reasoning_effort` | `generationConfig.thinkingConfig` | Gemini 2.5: `low→thinkingBudget 1024`, `medium→8192`, `high→24576`; Gemini 3+: `thinkingLevel: "low"/"medium"/"high"`. (This is Google's own mapping in their OpenAI-compat layer.) Add `includeThoughts: true` when the client opted into reasoning output. |1197| `stream` | endpoint choice | `stream:true` → `:streamGenerateContent?alt=sse`. |11981199##### Content-part map12001201| OpenAI part | Gemini part |1202|---|---|1203| `{"type":"text","text":T}` | `{"text": T}` |1204| `image_url` with `data:` URL | `{"inlineData": {"mimeType": "image/png", "data": "<base64>"}}` |1205| `image_url` with `https://` URL | **Gemini cannot fetch arbitrary URLs** (`fileData.fileUri` requires the Google File API). Router policy: download the image itself (size-capped, e.g. 20 MB) and convert to `inlineData`; on failure return an OpenAI 400 error naming the URL. |12061207##### Tool round-trip (functionCall / functionResponse)12081209OpenAI assistant `tool_calls` → Gemini model turn:12101211```json1212{"role": "model", "parts": [1213 {"functionCall": {"id": "call_abc123", "name": "get_weather",1214 "args": {"location": "Paris"}}}1215]}1216```12171218(`args` is a parsed object — parse the OpenAI `arguments` string. `functionCall.id` / `functionResponse.id` exist in current v1beta for parallel-call matching; include them when the OpenAI ids are available.)12191220OpenAI `tool` message → Gemini user turn:12211222```json1223{"role": "user", "parts": [1224 {"functionResponse": {"id": "call_abc123", "name": "get_weather",1225 "response": {"result": "18°C, sunny"}}}1226]}1227```12281229Two traps:12301. **`functionResponse.response` must be a JSON OBJECT.** OpenAI tool content is a string → if it parses as a JSON object, pass it; otherwise wrap as `{"result": <string>}`.12312. **`name` is required**, but OpenAI `tool` messages carry only `tool_call_id`. The router must resolve `tool_call_id → name` from the preceding assistant message's `tool_calls` in the same request payload (always available in a well-formed OpenAI conversation).12323. **Thought signatures (Gemini 3+):** function-call parts may carry a `thoughtSignature` that should be echoed back on the following turn. Preserve it via `reasoning_details` (§3.4) and re-attach when translating the conversation back.12331234##### Side-by-side minimal request12351236```json1237// OpenAI in // Gemini out1238{ {1239 "model": "gemini/gemini-2.5-flash", // POST …/models/gemini-2.5-flash:generateContent1240 "messages": [ "systemInstruction": {"parts":[{"text":"Be brief."}]},1241 {"role":"system","content":"Be brief."}, "contents": [1242 {"role":"user","content":"Hi"}, {"role":"user","parts":[{"text":"Hi"}]},1243 {"role":"assistant","content":"Hello!"}, {"role":"model","parts":[{"text":"Hello!"}]},1244 {"role":"user","content":"Name a color"} {"role":"user","parts":[{"text":"Name a color"}]}1245 ], ],1246 "temperature": 0.7, "generationConfig": {1247 "max_tokens": 100, "temperature": 0.7,1248 "stop": ["\n\n"] "maxOutputTokens": 100,1249} "stopSequences": ["\n\n"]1250 }1251 }1252```12531254#### 3.2.3 Response translation (Gemini → OpenAI)12551256Gemini response:12571258```json1259{1260 "candidates": [{1261 "content": {"role": "model", "parts": [1262 {"functionCall": {"name": "get_weather", "args": {"location": "Paris"}}}1263 ]},1264 "finishReason": "STOP",1265 "index": 0,1266 "safetyRatings": [ … ]1267 }],1268 "usageMetadata": {1269 "promptTokenCount": 57, "candidatesTokenCount": 12,1270 "thoughtsTokenCount": 88, "totalTokenCount": 1571271 },1272 "modelVersion": "gemini-2.5-flash"1273}1274```12751276Router emits:12771278```json1279{1280 "id": "chatcmpl-7be2f0c4",1281 "object": "chat.completion",1282 "created": 1753872000,1283 "model": "gemini/gemini-2.5-flash",1284 "choices": [{1285 "index": 0,1286 "message": {"role": "assistant", "content": null,1287 "tool_calls": [{"id": "call_9d1e2f3a", "type": "function",1288 "function": {"name": "get_weather", "arguments": "{\"location\":\"Paris\"}"}}]},1289 "finish_reason": "tool_calls"1290 }],1291 "usage": {1292 "prompt_tokens": 57,1293 "completion_tokens": 100,1294 "total_tokens": 157,1295 "completion_tokens_details": {"reasoning_tokens": 88}1296 }1297}1298```12991300Rules:13011302- Concatenate `parts[].text` (where `part.thought != true`) → `content`; parts with `"thought": true` → `reasoning_content`; each `functionCall` part → a `tool_calls[]` entry. **Gemini often omits ids** → synthesize `call_<12-hex>` (and remember name↔id for the return trip).1303- **CRITICAL finish_reason rule:** Gemini reports `finishReason: "STOP"` even when the candidate contains `functionCall` parts. The router must emit `finish_reason: "tool_calls"` whenever any functionCall part is present, regardless of `finishReason`.13041305| Gemini `finishReason` | OpenAI `finish_reason` |1306|---|---|1307| `STOP` (with functionCall parts) | `tool_calls` |1308| `STOP` | `stop` |1309| `MAX_TOKENS` | `length` |1310| `SAFETY`, `PROHIBITED_CONTENT`, `BLOCKLIST`, `SPII`, `IMAGE_SAFETY` | `content_filter` |1311| `RECITATION` | `content_filter` (recitation = copyright block) |1312| `MALFORMED_FUNCTION_CALL` | map to a **502-style OpenAI error** on non-streaming (the candidate is unusable); on streaming, emit finish_reason `stop` + log |1313| `LANGUAGE`, `OTHER`, unknown | `stop` (+ log the raw value) |13141315- **Blocked prompts:** if `candidates` is empty and `promptFeedback.blockReason` is set (`SAFETY`, `BLOCKLIST`, `PROHIBITED_CONTENT`, `OTHER`, `IMAGE_SAFETY`), return an OpenAI 400 `invalid_request_error` with a clear message naming the block reason — never an empty 200.1316- **usage:** `prompt_tokens = promptTokenCount`; `completion_tokens = candidatesTokenCount + thoughtsTokenCount` (OpenAI counts reasoning inside completion tokens); `total_tokens = totalTokenCount`; `completion_tokens_details.reasoning_tokens = thoughtsTokenCount`; `prompt_tokens_details.cached_tokens = cachedContentTokenCount`.13171318#### 3.2.4 Streaming (`streamGenerateContent?alt=sse`) → OpenAI chunks13191320Each SSE `data:` line is a **complete `GenerateContentResponse`** whose `candidates[0].content.parts` holds the *increment* — there is no delta envelope and **no `[DONE]` terminator** (the stream simply ends after the chunk carrying the final `finishReason`). `usageMetadata` appears on chunks with cumulative counts; the last chunk has the authoritative totals.13211322```1323GEMINI SSE → ROUTER EMITS13241325data: {"candidates":[{"content":{"parts":[{"text":"The"}], → chunk 1 (synthesized role first):1326 "role":"model"},"index":0}], data: {…,"delta":{"role":"assistant","content":""},…}1327 "usageMetadata":{…},"modelVersion":"gemini-2.5-flash"} data: {…,"delta":{"content":"The"},"finish_reason":null}13281329data: {"candidates":[{"content":{"parts":[{"text": → data: {…,"delta":{"content":" sky is blue."},1330 " sky is blue."}],"role":"model"},"index":0}],…} "finish_reason":null}13311332data: {"candidates":[{"content":{"parts":[],"role": → data: {…,"delta":{},"finish_reason":"stop"}1333 "model"},"finishReason":"STOP","index":0}], → data: {…,"choices":[],"usage":{"prompt_tokens":8,1334 "usageMetadata":{"promptTokenCount":8, "completion_tokens":5,"total_tokens":13}}1335 "candidatesTokenCount":5,"totalTokenCount":13}} (if include_usage)1336(stream ends — no [DONE] from Gemini) → data: [DONE] (router ALWAYS adds it)1337```13381339Rules:1340- **Synthesize the role chunk**: Gemini has no role-only first frame; the router emits `{"role":"assistant","content":""}` before the first content delta.1341- **Tool calls are NOT argument-streamed**: a `functionCall` part arrives complete in one chunk → emit ONE `tool_calls` delta containing `index`, generated `id`, `name`, and the FULL `arguments` string; strict SDKs accept whole-argument single deltas fine.1342- Thought parts (`"thought": true`) → `reasoning_content` deltas.1343- `finishReason` on the final chunk → the finish_reason chunk (apply the functionCall→`tool_calls` override).1344- Router appends the OpenAI usage chunk (empty `choices`) and `data: [DONE]` itself.1345- If Gemini aborts mid-stream with `finishReason: SAFETY`, emit `finish_reason: "content_filter"` and terminate normally.13461347---13481349### 3.3 OpenAI-compatible providers — deviation table13501351All nine below speak the OpenAI chat/completions wire format closely enough for **near-pass-through**: the router's `CompatAdjuster` only needs a per-provider strip/rename/allow table plus finish/usage normalization. Auth is `Authorization: Bearer <key>` for all of them. (OpenAI itself, `api.openai.com/v1`, is the reference and needs no adjustment.)13521353#### Summary matrix13541355| Provider | Base URL | Strip / rename | Extra params to allow (pass-through) | Quirks |1356|---|---|---|---|---|1357| **xAI** | `https://api.x.ai/v1` | For Grok-4-family reasoning models: **strip `presence_penalty`, `frequency_penalty`, `stop`** (they 400, not ignore). Strip `reasoning_effort` on models that reject it. | `reasoning_effort` (model-gated: grok-3-mini; grok-4.3 `none/low/medium/high`; grok-4.5 `low/medium/high` only), `search_parameters` (Live Search), `deferred` | grok-3-mini returns `message.reasoning_content`; grok-4 does NOT expose reasoning content, only `usage.completion_tokens_details.reasoning_tokens`. Vision via standard `image_url` (jpeg/png, ≤20 MiB). Structured outputs supported. Chat Completions is now labeled a "legacy" endpoint (Responses API is primary) but remains fully supported. |1358| **Mistral** | `https://api.mistral.ai/v1` | **Rename `seed` → `random_seed`.** Strip `logit_bias`, `user`, `logprobs`. | `safe_prompt` (bool), `prompt_mode: "reasoning"`, `prediction`, `prompt_cache_key`, tool_choice value `"any"` | `tool_choice` accepts `auto/none/any/required` (`any`≈`required`). Temperature recommended 0–0.7. `response_format` supports `json_object` AND `json_schema`. Magistral reasoning models return `message.content` as an ARRAY of chunks: `{"type":"thinking","thinking":[{"type":"text","text":…}]}` + `{"type":"text","text":…}` — router must flatten: thinking chunks → `reasoning_content`, text chunks → `content` (same in streaming deltas, which shape-shift between array and string). SSE ends with `[DONE]`. |1359| **DashScope / Qwen (intl)** | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | Strip `logit_bias`. `n` forced to 1 when `tools` present. | `enable_thinking` (bool), `thinking_budget` (int) — both extra-body; `translation_options` | Hybrid-thinking models (qwen3/qwen-plus) emit `delta.reasoning_content` then `delta.content`. **Some open-source thinking models are streaming-only** (non-streaming call errors) → router should transparently stream-and-aggregate when a client asks non-streaming. `stream_options.include_usage` supported (usage in final chunk). Vision (qwen-vl) uses standard `image_url` parts. Newer qwen models enable thinking by default — set `enable_thinking:false` when the client didn't ask for reasoning. |1360| **DeepSeek** | `https://api.deepseek.com` (alias `…/v1`) | For reasoner/thinking mode: `temperature`, `top_p`, `presence_penalty`, `frequency_penalty` are **silently ignored** (no strip needed, but don't pretend they work); `logprobs`/`top_logprobs` error → strip. **Strip `reasoning_content` from incoming assistant messages** except in tool-call loops (see quirks). | `thinking: {"type":"enabled"/"disabled"}` (extra-body), `reasoning_effort` | `message.reasoning_content` + `delta.reasoning_content` (the convention we adopt, §3.4). Multi-turn: reasoning_content must NOT be resent in ordinary turns, but in TOOL-CALL loops it **must be passed back** or the API 400s (current docs). JSON mode: `response_format {"type":"json_object"}` requires the word "json" in the prompt — router auto-appends an instruction if missing. Tools supported. Usage includes `prompt_cache_hit_tokens` / `prompt_cache_miss_tokens` → map hit tokens to `prompt_tokens_details.cached_tokens`. |1361| **Kimi / Moonshot** | `https://api.moonshot.ai/v1` | **Clamp `temperature` to [0,1]** (out-of-range 400s; default 0.0). `stop` max 5 sequences × 32 bytes → truncate/reject. K3 reasoning model: strip `temperature`, `top_p`, penalties. | `partial: true` on last assistant message (prefill mode), `thinking: {"type":…}`, `reasoning_effort` (`low/high/max` on K3), `thinking.keep` | `n` 1–5. `response_format`: `json_object` and `json_schema` (strict). Vision + video via `image_url`/`video_url` (base64 or `ms://<file_id>`). Thinking models return `reasoning_content` (message + delta). `stream_options.include_usage` → usage in final pre-`[DONE]` chunk. |1362| **Perplexity** | `https://api.perplexity.ai` (`POST /chat/completions`; docs now also expose a gateway at `/router/v1/chat/completions`) | Strip `frequency_penalty`/`presence_penalty`/`top_k` if absent from current schema; **reject `tools`** on Sonar search models (no function calling) with a clear OpenAI error. Messages after `system` must **strictly alternate user/assistant** — merge consecutive same-role messages like the Anthropic path. | `search_mode` (`web/academic/sec`), `search_domain_filter`, `search_recency_filter`, `search_after_date_filter`/`search_before_date_filter`, `web_search_options` (`search_context_size`, `user_location`), `return_images`, `return_related_questions`, `disable_search`, `reasoning_effort`, `language_preference` | Response carries top-level `citations: [urls]` and `search_results: [{title,url,snippet,date}]` — the router passes these through verbatim as extension keys on the normalized response (and on the final stream chunk). `usage` extras: `citation_tokens`, `num_search_queries`, `reasoning_tokens`, `cost{…}` → keep under usage extension, fold `reasoning_tokens` into `completion_tokens_details`. sonar-reasoning models emit `<think>…</think>` inside `content` → router extracts to `reasoning_content`. |1363| **Together** | `https://api.together.xyz/v1` | Nothing mandatory to strip. | `top_k`, `min_p`, `repetition_penalty`, `safety_model`, `echo`, `context_length_exceeded_behavior`, `chat_template_kwargs` | **finish_reason may be `"eos"`** → normalize to `"stop"` (also normalize `function_call`→`tool_calls` if seen). Usage appears in the final chunk (and on some models in every chunk) — always take the LAST non-null usage. `json_schema` support is model-dependent → 400s surface as OpenAI errors. Open-weights models vary in tool-calling quality; nothing structural to translate. |1364| **DeepInfra** | `https://api.deepinfra.com/v1/openai` | Strip `logit_bias` (unsupported on most models). | `min_p`, `repetition_penalty`, `service_tier` (`priority`/`flex`), `fail_fast` | Usage object includes non-standard **`estimated_cost`** (USD) → feed it straight into `UsageMeter` as authoritative cost when present. Final stream chunk carries usage. Self-described as "not 100% compatible with all OpenAI parameters" — treat unknown-param 400s as strippable and retry once without extras. |1365| **Cerebras** | `https://api.cerebras.ai/v1` | **Vision: base64 data-URI images only — remote `image_url` http(s) URLs are rejected** → router inlines (download + base64) or rejects with a clear error. `response_format {"type":"json_object"}` is **incompatible with streaming** → reject that combination or fall back to json_schema. | `reasoning_effort` (`low/medium/high/none`), `clear_thinking` (Cerebras-specific: drop prior-turn reasoning), `service_tier`, `prediction`, `prompt_cache_key` | Extremely high tokens/s — the SSE writer must handle very fast chunk cadence (backpressure!). Usage included in stream. Extra response fields `time_info` (queue/prompt/completion latencies) and `service_tier_used` → log, don't forward. `max_completion_tokens` preferred name. Supports `logprobs`, penalties, `logit_bias` per current docs (verify in Phase 7). |13661367Sources: xAI — https://docs.x.ai/developers/model-capabilities/legacy/chat-completions, https://www.promptfoo.dev/docs/providers/xai/ ; Mistral — https://docs.mistral.ai/api/ , https://docs.mistral.ai/capabilities/reasoning/ ; DashScope — https://www.alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope , https://www.alibabacloud.com/help/en/model-studio/deep-thinking ; DeepSeek — https://api-docs.deepseek.com/guides/thinking_mode ; Kimi — https://platform.kimi.ai/docs/api/chat (platform.moonshot.ai redirects here; API host remains api.moonshot.ai) ; Perplexity — https://docs.perplexity.ai/api-reference/chat-completions-post , https://docs.perplexity.ai/getting-started/quickstart ; Together — https://docs.together.ai/reference/chat-completions-1 ; DeepInfra — https://docs.deepinfra.com/chat/overview ; Cerebras — https://inference-docs.cerebras.ai/api-reference/chat-completions .13681369#### Universal CompatAdjuster rules137013711. **Unknown-parameter resilience:** providers split between *ignore-unknown* (DeepSeek, Together) and *400-on-unknown* (xAI reasoning models, Moonshot on bad ranges). The per-provider strip table is authoritative; additionally, on a 400 whose message names a parameter, retry ONCE with that parameter removed, then surface the error.13722. **`max_tokens` vs `max_completion_tokens`:** accept both from clients; send whichever the provider documents (`max_completion_tokens` for Cerebras/Moonshot-K-series; `max_tokens` elsewhere). Never send both.13733. **Streaming usage:** request `stream_options: {"include_usage": true}` upstream wherever supported (DashScope, Moonshot, DeepSeek, Together, DeepInfra, Cerebras, xAI, Mistral); when the upstream can't provide usage, estimate tokens locally and flag `"x-zyquo": {"usage_estimated": true}`.13744. **finish_reason normalization set:** everything must land in `stop | length | tool_calls | content_filter`; map `eos→stop`, provider-specific refusal/safety values → `content_filter`, anything unknown → `stop` + log.13755. **Keep-alive noise:** some upstreams emit SSE comment lines (`: keep-alive`) or `ping`-ish frames — the SSE parser must skip comment lines and blank frames without erroring, and the router's own SSEWriter may emit comments to keep client sockets alive during long thinking phases.13761377---13781379### 3.4 Reasoning content — normalization decision13801381**Decision: normalize on DeepSeek's `reasoning_content` convention** — a sibling of `content` on both the non-streaming `message` and the streaming `delta`:13821383```json1384// non-streaming // streaming1385"message": { "delta": {1386 "role": "assistant", "reasoning_content": "Let me check…"1387 "reasoning_content": "Let me check…", }1388 "content": "The answer is 21."1389}1390```13911392Rationale: it is the oldest and most widely recognized wire convention (DeepSeek-R1 era) — Qwen/DashScope, Moonshot/Kimi, and xAI grok-3-mini already emit exactly this field, so most client tooling (chat UIs, LangChain, Continue, aider, etc.) knows to render it and to NOT confuse it with `content`. OpenRouter's richer `reasoning` + `reasoning_details` model (https://openrouter.ai/docs/use-cases/reasoning-tokens) is adopted only as a *supplement*: the router additionally emits `reasoning_details` (array of provider-native structured blocks) when signatures/opacity must round-trip.13931394#### Per-provider mapping into `reasoning_content`13951396| Provider | Native form | → Router normalization |1397|---|---|---|1398| DeepSeek | `message.reasoning_content` / `delta.reasoning_content` | Pass through unchanged. |1399| Qwen/DashScope | same field | Pass through. |1400| Kimi/Moonshot | same field | Pass through. |1401| xAI grok-3-mini | same field | Pass through. (grok-4: nothing exposed — only `reasoning_tokens` in usage.) |1402| Anthropic | `thinking` content blocks; streaming `thinking_delta` (+ `signature_delta`, `redacted_thinking`) | Thinking text → `reasoning_content`; signature + redacted blocks → `reasoning_details: [{"type":"anthropic.thinking_signature",…}]`. |1403| Gemini | parts with `"thought": true` (needs `thinkingConfig.includeThoughts`); `thoughtSignature` on parts | Thought text → `reasoning_content`; `thoughtSignature` → `reasoning_details: [{"type":"gemini.thought_signature",…}]`. |1404| Mistral Magistral | content chunk `{"type":"thinking",…}` inside the content array | Flatten to `reasoning_content`; text chunks → `content`. |1405| Perplexity sonar-reasoning | `<think>…</think>` prefix inside `content` | Extract tags → `reasoning_content`; strip from `content`. |1406| Cerebras (reasoning models) | model-dependent (`reasoning` field or `<think>` tags per hosted model) | Same extraction pipeline; verify per model in Phase 7. |14071408#### Usage normalization14091410All reasoning token counts land in the OpenAI-standard `usage.completion_tokens_details.reasoning_tokens` (Anthropic `thinking_tokens`, Gemini `thoughtsTokenCount`, xAI/Perplexity `reasoning_tokens`), and reasoning tokens are INCLUDED in `completion_tokens` (OpenAI semantics).14111412#### Request-side control14131414The router accepts the OpenAI-standard **`reasoning_effort`** (`"minimal"|"low"|"medium"|"high"`, plus provider extras like `"none"`/`"max"`) and translates per provider: Anthropic → `output_config.effort` / `thinking.budget_tokens`; Gemini → `thinkingConfig.thinkingBudget`/`thinkingLevel` (Google's own compat mapping: low=1024, medium=8192, high=24576 on 2.5-series); DeepSeek/Kimi/Cerebras/xAI/Perplexity → pass `reasoning_effort` through (gated by catalog capability); Qwen → `enable_thinking:true` (+ `thinking_budget`); Mistral → `prompt_mode:"reasoning"`. On models with no reasoning capability, `reasoning_effort` is stripped (never 400 the client for asking).14151416#### Echo-back rules (multi-turn)14171418Incoming assistant messages may contain `reasoning_content`/`reasoning_details` from prior router responses. Before forwarding:1419- **Strip `reasoning_content`** for all providers by default (DeepSeek 400s in plain turns if it leaks into context via unknown-field-strict paths; others ignore it but it wastes tokens),1420- **except**: DeepSeek tool-call loops (must be passed back per current docs), Anthropic thinking+tools loops (reconstruct `thinking` blocks with signatures from `reasoning_details`), Gemini 3 (re-attach `thoughtSignature`), Mistral Magistral (replay ThinkChunk to preserve the trace), Kimi with `thinking.keep`.1421This asymmetry is exactly why `reasoning_details` exists: it carries the provider-native, signed material that some upstreams demand back, while `reasoning_content` stays a clean display string.14221423---14241425### 3.5 Source index14261427- Anthropic Messages API: https://platform.claude.com/docs/en/api/messages1428- Anthropic streaming events: https://platform.claude.com/docs/en/docs/build-with-claude/streaming1429- Anthropic errors: https://platform.claude.com/docs/en/api/errors1430- Gemini generateContent reference: https://ai.google.dev/api/generate-content1431- Gemini Part/Content schema: https://ai.google.dev/api/caching#Part1432- Gemini function calling: https://ai.google.dev/gemini-api/docs/function-calling1433- Gemini OpenAI-compat layer (mapping oracle): https://ai.google.dev/gemini-api/docs/openai1434- xAI chat completions: https://docs.x.ai/developers/model-capabilities/legacy/chat-completions ; parameter-rejection field notes: https://www.promptfoo.dev/docs/providers/xai/ , https://github.com/vercel/ai/issues/128261435- Mistral API: https://docs.mistral.ai/api/ ; reasoning: https://docs.mistral.ai/capabilities/reasoning/1436- DashScope OpenAI compat: https://www.alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope ; deep thinking: https://www.alibabacloud.com/help/en/model-studio/deep-thinking1437- DeepSeek thinking mode: https://api-docs.deepseek.com/guides/thinking_mode1438- Kimi/Moonshot chat API: https://platform.kimi.ai/docs/api/chat1439- Perplexity chat completions: https://docs.perplexity.ai/api-reference/chat-completions-post ; quickstart: https://docs.perplexity.ai/getting-started/quickstart1440- Together chat completions: https://docs.together.ai/reference/chat-completions-11441- DeepInfra OpenAI API: https://docs.deepinfra.com/chat/overview1442- Cerebras chat completions: https://inference-docs.cerebras.ai/api-reference/chat-completions1443- OpenRouter reasoning normalization (prior art): https://openrouter.ai/docs/use-cases/reasoning-tokens1444## 4. HTTP server in Swift without heavyweight deps14451446Research date: 2026-07-30. Target: an embedded HTTP/1.1 server inside a SwiftUI macOS 13+ app (SPM, no Xcode IDE) serving an OpenAI-compatible API on `http://localhost:<port>`, with spec-exact SSE streaming, long-lived streams, client-disconnect → upstream-cancellation, and graceful shutdown.14471448### 4.1 Options evaluated14491450#### Option A — SwiftNIO directly (NIOCore + NIOPosix + NIOHTTP1 + NIOExtras)14511452- **State as of mid-2026:** swift-nio is at **2.101.3** (released ~2026-07-23), actively maintained by Apple, compatible with Swift **6.0–6.3** and strict concurrency. NIO 3 is expected "sometime around Swift 6" per the Swift.org server guidelines, with NIO 2 continuing to receive bug fixes afterwards — NIO 2.x is a safe multi-year foundation. Sources: [swift-nio releases](https://github.com/apple/swift-nio/releases), [Swift Package Index — swift-nio](https://swiftpackageindex.com/apple/swift-nio), [Swift.org concurrency adoption guidelines](https://www.swift.org/documentation/server/guides/libraries/concurrency-adoption-guidelines.html).1453- **Structured concurrency:** modern NIO exposes **`NIOAsyncChannel`**, which "abstracts the notion of a NIO `Channel` into something that can safely be used in a structured concurrency context". The recommended split: protocol-specific logic (HTTP parsing/encoding via `configureHTTPServerPipeline`) stays as `ChannelHandler`s; business logic consumes/produces via the `NIOAsyncChannel` inbound `AsyncSequence` / outbound writer. `executeThenClose` scopes the channel's lifetime to a closure — the channel closes when the closure returns, which maps perfectly onto "one inbound request = one cancellable `Task`". Sources: [NIOAsyncChannel docs](https://swiftinit.org/docs/swift-nio/niocore/nioasyncchannel), [NIO public async APIs](https://github.com/apple/swift-nio/blob/main/docs/public-async-nio-apis.md), [executeThenClose discussion](https://forums.swift.org/t/nioasyncchannel-executethenclose-is-too-restrictive/73460), [Using SwiftNIO — Channels](https://swiftonserver.com/using-swiftnio-channels/), [Building a web app with only SwiftNIO (2026)](https://blog.alexseifert.com/2026/06/29/building-a-web-app-in-swift-using-only-swiftnio/).1454- **Reference server:** Apple's `NIOHTTP1Server` example shows the canonical `ServerBootstrap` setup — `backlog: 256`, `so_reuseaddr` on server and child channels, `configureHTTPServerPipeline(withErrorHandling: true)`, explicit keep-alive state machine (idle → waiting-for-body → sending-response) and adding `Connection: close`/`keep-alive` headers for HTTP/1.0 or explicit-close requests. (The example itself is future-based; we use the NIOAsyncChannel equivalent.) Source: [NIOHTTP1Server main.swift](https://github.com/apple/swift-nio/blob/main/Sources/NIOHTTP1Server/main.swift).1455- **Graceful shutdown:** `swift-nio-extras` ships **`ServerQuiescingHelper`** — "helps to quiesce a server by notifying user code when all previously open connections have closed"; call `initiateShutdown(promise:)` to stop accepting and drain. There is a full demo (`HTTPServerWithQuiescingDemo`). Sources: [QuiescingHelper.swift](https://github.com/apple/swift-nio-extras/blob/main/Sources/NIOExtras/QuiescingHelper.swift), [HTTPServerWithQuiescingDemo](https://github.com/apple/swift-nio-extras/blob/main/Sources/HTTPServerWithQuiescingDemo/main.swift).1456- **Weight:** NIOCore/NIOPosix/NIOHTTP1 (+ optionally NIOExtras) — one Apple-maintained dependency tree, no routing framework, no ServiceLifecycle/Logging/Metrics transitive stack. Full control over every byte of the SSE wire format.1457- **Cost:** we write our own tiny router (we have ~5 routes), our own request-body accumulation with a size limit, and our own SSE writer. For this app that's a feature: the OpenAI chunk stream must be byte-exact, and owning `SSEWriter` end-to-end removes a framework abstraction between us and the wire.14581459#### Option B — Network.framework (`NWListener` + `NWProtocolFramer`)14601461`NWListener` replaces the BSD bind/listen/accept sequence, but HTTP itself must be brought along: either a custom `NWProtocolFramer` (real boilerplate via `NWProtocolFramerImplementation`) or hand-wiring a C parser (`http_parser.c`) onto raw connections — Helge Heß's NWHTTPProtocol does exactly that and its author notes that "for production use it's suggested to not use a protocol framer for HTTP" and to hook the parser up directly instead. No HTTP/1.1 pipeline, no chunked-encoding helpers, no keep-alive management, no quiescing utilities — all DIY. It buys nothing over NIO for a localhost server (its strengths are Wi-Fi/cellular path handling and Bonjour, irrelevant here). Sources: [NWHTTPProtocol](https://github.com/helje5/NWHTTPProtocol), [Intro to Network.framework servers](http://www.alwaysrightinstitute.com/network-framework/), [Apple Network framework docs](https://developer.apple.com/documentation/network). **Rejected.**14621463#### Option C — Hummingbird 214641465The strongest framework candidate. Built from scratch on SwiftNIO with Swift concurrency central; at **2.25.1** as of July 2026; Swift 6.1+; "designed to require the minimum number of dependencies". It has first-class pieces we'd otherwise hand-roll: a `ResponseBodyWriter` closure body with backpressure-aware `await writer.write(...)`, a `ServerSentEvent` type, `consumeWithInboundCloseHandler` for client-disconnect detection, and graceful shutdown via **swift-service-lifecycle** ("currently running requests continue being handled, while new connections and requests will not be accepted"), started with `Application.runService(gracefulShutdownSignals:)`. There is a dedicated `server-sent-events` example. Sources: [hummingbird repo](https://github.com/hummingbird-project/hummingbird), [SPI — hummingbird](https://swiftpackageindex.com/hummingbird-project/hummingbird), [What's new in Hummingbird 2](https://swiftonserver.com/whats-new-in-hummingbird-2/), [Hummingbird 2 announcement](https://hummingbird.codes/news/hummingbird-2/), [SSE example](https://github.com/hummingbird-project/hummingbird-examples), [swift-service-lifecycle](https://github.com/swift-server/swift-service-lifecycle).14661467Why not choose it: (1) it drags in ServiceLifecycle/Logging/Metrics/Tracing abstractions designed for long-running server binaries with signal-driven lifecycles, whereas our lifecycle is owned by a SwiftUI app's Start/Stop button — bridging `runService` into an app-owned start/stop adds friction rather than removing it; (2) its router/middleware/extractor machinery is overhead for ~5 fixed routes; (3) an extra abstraction layer sits between us and the SSE bytes, and byte-exact OpenAI chunk emission is the core deliverable. Its SSE example is nonetheless the best public reference for the disconnect/shutdown patterns we will reimplement on raw NIO (see 4.2).14681469#### Option D — Vapor14701471Batteries-included (HTTP/2, TLS, auth, validation, WebSockets, …) and correspondingly heavy: bulks the executable, increases compile time, ~20–30 MB idle memory vs Hummingbird's ~5–10 MB, and a large transitive dependency graph. Everything it adds over NIO is something this app doesn't need. Sources: [Hummingbird vs Vapor discussion](https://github.com/hummingbird-project/hummingbird/discussions/150), [Beginner's guide to Hummingbird](https://theswiftdev.com/beginners-guide-to-server-side-swift-using-the-hummingbird-framework/). **Rejected.**14721473#### Decision14741475**SwiftNIO directly** (`NIOCore`, `NIOPosix`, `NIOHTTP1`, plus `NIOExtras` for `ServerQuiescingHelper`), using the `NIOAsyncChannel` structured-concurrency APIs. Rationale: Apple-maintained, SPM-clean, one dependency tree, macOS 13+ fine, Swift 6 strict-concurrency ready, full control over SSE emission/flushing/backpressure, first-class quiescing for graceful shutdown, and a natural one-request-one-`Task` model so client disconnect cancels the upstream call structurally. Hummingbird 2 is the documented fallback if raw-NIO plumbing proves costlier than expected — the migration path is easy since both are NIO underneath.14761477### 4.2 Implementation specifics (SwiftNIO)14781479**Bootstrap and binding.**14801481```swift1482let group = MultiThreadedEventLoopGroup.singleton1483let quiesce = ServerQuiescingHelper(group: group)14841485let serverChannel = try await ServerBootstrap(group: group)1486 .serverChannelOption(ChannelOptions.backlog, value: 256)1487 .serverChannelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)1488 .serverChannelInitializer { channel in1489 channel.pipeline.addHandler(quiesce.makeServerChannelHandler(channel: channel))1490 }1491 .childChannelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)1492 .bind(host: bindHost, port: port) { channel in1493 channel.eventLoop.makeCompletedFuture {1494 try channel.pipeline.syncOperations.configureHTTPServerPipeline(withErrorHandling: true)1495 return try NIOAsyncChannel<HTTPServerRequestPart, HTTPServerResponsePart>(1496 wrappingChannelSynchronously: channel)1497 }1498 }1499```15001501- **Bind host:** `"127.0.0.1"` by default (localhost only, unreachable from the LAN); `"0.0.0.0"` only when the user explicitly enables LAN exposure (which per our security policy forces a local API key). This is a plain string parameter to `bind(host:port:)` — no extra API.1502- **Port-in-use:** `bind` throws; catch `IOError`/`NIOBSDSocket` errors and check `errno == EADDRINUSE` (48 on Darwin) → surface "Port 8787 is already in use" and probe upward (`try bind` on port+1, +2, …) to *suggest* the next free port (never silently switch). `EACCES` (ports < 1024 without privileges) gets its own message. `SO_REUSEADDR` on the server channel avoids spurious `EADDRINUSE` from sockets lingering in `TIME_WAIT` after a quick Stop→Start; note it does **not** let two live listeners share a port — a genuinely occupied port still fails, which is what we want. Sources: [ServerBootstrap docs](https://swiftinit.org/docs/swift-nio/nioposix/serverbootstrap), [Bind: address already in use](https://hea-www.harvard.edu/~fine/Tech/addrinuse.html), [NIOHTTP1Server example options](https://github.com/apple/swift-nio/blob/main/Sources/NIOHTTP1Server/main.swift).15031504**Concurrent connections with structured concurrency.** The async `bind` returns a `NIOAsyncChannel` of accepted-connection `NIOAsyncChannel`s. The serving loop:15051506```swift1507try await serverChannel.executeThenClose { acceptedConnections in1508 try await withThrowingDiscardingTaskGroup { group in1509 for try await connection in acceptedConnections {1510 group.addTask { await handleConnection(connection) } // one Task per connection1511 }1512 }1513}1514```15151516Inside `handleConnection`, `connection.executeThenClose { inbound, outbound in ... }` gives an `AsyncSequence` of `HTTPServerRequestPart` (`.head`, `.body` buffers, `.end`) and an outbound writer for `HTTPServerResponsePart`. Each request is parsed, dispatched to `Routes`, and answered; the loop iterates for keep-alive. Cancelling the connection's `Task` tears everything down cleanly — this is the backbone of both client-disconnect handling and graceful shutdown. Sources: [NIO public async APIs](https://github.com/apple/swift-nio/blob/main/docs/public-async-nio-apis.md), [NIOAsyncChannel docs](https://swiftinit.org/docs/swift-nio/niocore/nioasyncchannel).15171518**Request body size limit.** Accumulate `.body` parts into a `ByteBuffer` with a hard cap (default e.g. 20 MiB — base64 images are large; user-configurable in Settings). On overflow: respond `413` in OpenAI error format, drain remaining parts (or close), never buffer further.15191520**Timeouts.**1521- *Header/idle timeout:* add `IdleStateHandler` (NIOCore) ahead of the HTTP handlers, or a per-request `withTimeout` wrapper, to kill connections that never complete a request (~30 s read idle).1522- *Streaming:* once a response stream has started, the idle clock must apply to *write* progress, not total duration — chat completions can legitimately stream for many minutes. Practical policy: generous upstream time-to-first-byte timeout (e.g. 120 s, configurable), then no total cap while chunks keep flowing; abort if the upstream stalls (no chunk for N seconds, e.g. 300 s).1523- Remove/suspend the idle handler for the duration of an SSE response, restore for keep-alive reuse.15241525**Keep-alive.** `configureHTTPServerPipeline` parses `Connection` headers; our responder mirrors the NIOHTTP1Server example: honor `keepAlive` from the request head, set `Connection: keep-alive`/`close` explicitly for HTTP/1.0, close the channel after the response when keep-alive is false. After an SSE response we send the terminating `[DONE]` and `.end`; keeping the connection alive afterwards is legal (the response used chunked encoding with a proper terminator), but closing is also acceptable — OpenAI SDKs handle both. We keep it alive (SDKs reuse connections between calls). Source: [NIOHTTP1Server main.swift](https://github.com/apple/swift-nio/blob/main/Sources/NIOHTTP1Server/main.swift).15261527**SSE emission (`SSEWriter`).** Response head:15281529```1530HTTP/1.1 200 OK1531Content-Type: text/event-stream1532Cache-Control: no-cache1533Connection: keep-alive1534X-Accel-Buffering: no ← harmless; defeats proxy buffering if any intermediary appears1535Transfer-Encoding: chunked ← added automatically by HTTPResponseEncoder when no Content-Length1536```15371538Each OpenAI chunk is one SSE event: `data: {json}\n\n` (UTF-8; the OpenAI format is single-line JSON per `data:` line, terminated by `data: [DONE]\n\n`). **Flush per event:** with `NIOAsyncChannel`, every `try await outbound.write(.body(.byteBuffer(eventBuffer)))` is a `writeAndFlush` — each event leaves the process immediately; no coalescing layer may sit above it. `await`-ing each write is also the **backpressure** mechanism: a slow client suspends us, which suspends consumption of the upstream `AsyncSequence`, which propagates backpressure to the upstream HTTP read. Optionally set `TCP_NODELAY` on child channels so small event frames aren't Nagle-delayed. SSE format/caching rules: [MDN — Using server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events). (Note: OpenAI's stream is consumed by SDK parsers, not browser `EventSource`, so `event:`/`id:`/`retry:` fields are never used — `data:` lines only.)15391540**Client-disconnect → cancel upstream.** Two complementary signals:15411. Structured: the write to a closed socket throws (`NIOAsyncWriterError`/`channelInactive`-driven); catching it in the request `Task` must cancel the upstream streaming `Task` — with structured concurrency this is automatic if the upstream call is a child task (`withThrowingTaskGroup` per request: one child consumes upstream and writes SSE; error/cancel of either cancels the other).15422. Proactive: watch the inbound side for EOF/half-closure while streaming — the pattern Hummingbird's SSE example uses (`consumeWithInboundCloseHandler` yielding a cancel event merged with the data stream). On raw NIO: run a second child task iterating `inbound`; when the sequence ends (client closed), cancel the group. This detects disconnects *between* writes, not only on the next failed write.1543Cancellation must propagate into the provider client (`URLSession`/`AsyncHTTPClient` task cancelled) so upstream token spend stops — Phase 7 verifies "no orphaned upstream usage". Sources: [Hummingbird SSE example](https://github.com/hummingbird-project/hummingbird-examples), [NIOAsyncChannel docs](https://swiftinit.org/docs/swift-nio/niocore/nioasyncchannel).15441545**CORS.** Needed so browser-based tools can call the router. Implement in a small `CORS.swift`:1546- Preflight: `OPTIONS` with `Origin` + `Access-Control-Request-Method` → `204` with `Access-Control-Allow-Origin: *` (default; configurable to a specific origin), `Access-Control-Allow-Methods: GET, POST, OPTIONS`, `Access-Control-Allow-Headers: Authorization, Content-Type` (or echo the requested headers), `Access-Control-Max-Age: 600`.1547- Actual responses: add `Access-Control-Allow-Origin: *` (echo origin + `Access-Control-Allow-Credentials: true` only if credentials mode is ever needed — default permissive `*` for localhost tooling, per our spec).1548- SSE responses need the CORS headers too (the stream is fetched cross-origin by browser clients).15491550**Graceful shutdown.** On Stop:15511. `quiesce.initiateShutdown(promise:)` — the `ServerQuiescingHelper` closes the *listening* channel (no new connections) and signals when all child channels have closed. Sources: [QuiescingHelper.swift](https://github.com/apple/swift-nio-extras/blob/main/Sources/NIOExtras/QuiescingHelper.swift), [HTTPServerWithQuiescingDemo](https://github.com/apple/swift-nio-extras/blob/main/Sources/HTTPServerWithQuiescingDemo/main.swift).15522. Wait up to a drain deadline (e.g. 10 s) for in-flight non-streaming requests to finish.15533. Long-lived SSE streams won't drain on their own: after the deadline (or immediately if the user chose "stop now"), cancel the connection task group — cancellation unwinds each request task, cancels upstream calls, and closes channels.15544. Do **not** `shutdownGracefully()` the singleton `EventLoopGroup` (it's shared and Start may be pressed again); just release the server channel. Semantics mirror Hummingbird/ServiceLifecycle: "currently running requests continue being handled, while new connections and requests will not be accepted" ([swift-service-lifecycle](https://github.com/swift-server/swift-service-lifecycle), [What's new in Hummingbird 2](https://swiftonserver.com/whats-new-in-hummingbird-2/)).15551556### 4.3 macOS specifics15571558- **Local network privacy prompt.** Since macOS 15 Sequoia there is an iOS-style *Local Network* permission (System Settings → Privacy & Security → Local Network). Binding and serving on `127.0.0.1` does not involve the local network and triggers nothing. Interacting with LAN peers can trigger the prompt; users can later toggle it, and there are known Sequoia bugs where access silently breaks after a restart until re-toggled. When the user enables LAN mode (`0.0.0.0`) the app must explain the prompt and the Settings toggle in the UI. Also note the responsible-process rules: tools run from Terminal inherit Terminal's exemption; root daemons are auto-granted. Sources: [Apple forums — local network privacy on Sequoia](https://developer.apple.com/forums/thread/763484), [mjtsai — Local Network Privacy on Sequoia](https://mjtsai.com/blog/2024/10/02/local-network-privacy-on-sequoia/), [Foldr — macOS 15 local network privacy](https://foldr.com/foldr-support/foldr-for-macos/macos-15-sequoia-local-network-privacy/), [Panic — granting local network access](https://help.panic.com/prompt/prompt-local-network/), [access lost after restart](https://developer.apple.com/forums/thread/769037?page=2).1559- **App Sandbox vs Developer ID.** `com.apple.security.network.server` ("whether your app may listen for incoming network connections") is an **App Sandbox** entitlement; a **non-sandboxed, Hardened-Runtime Developer ID app needs no entitlement to listen on a socket** — "the sandbox was designed mainly for the App Store, while the hardened runtime was designed mainly for Developer ID". Decision (consistent with Phase 8 spec): ship non-sandboxed Developer ID + Hardened Runtime; document that if we ever adopt the sandbox we must add `com.apple.security.network.server` and `.client`. Sources: [Apple — com.apple.security.network.server](https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.security.network.server), [Hardened Runtime and Sandboxing (lapcatsoftware)](https://lapcatsoftware.com/articles/hardened-runtime-sandboxing.html).1560- **Terminal vs Finder launch.** From Finder the app gets the standard GUI context. From Terminal (during `make dev`), the *Terminal* process is the responsible process for TCC purposes, so privacy prompts may attribute to Terminal; also a bare executable launched from Terminal needs explicit `NSApplication` activation to come frontmost (already planned in Phase 1's entry point). Environment differs too (Terminal's shell env vs launchd's user session) — never rely on env vars for configuration of the shipped app. Source: [Apple forums — CLI tools and local network privacy](https://developer.apple.com/forums/thread/767391).15611562---15631564## 5. Gateway concerns15651566### 5.1 Request logging with redaction15671568Pattern (validated against LiteLLM's proxy design):1569- **Always log metadata:** timestamp, method/path, resolved model + provider, actually-used model (post-fallback), status, latency breakdown (queue → upstream TTFB → stream duration), tokens in/out, cost, local-key ID (its name/ID — never the token), stream flag, error class. This is safe and powers the whole dashboard.1570- **Bodies are opt-in.** LiteLLM's `turn_off_message_logging` redacts `messages`/`prompt` and `choices[].message.content` (and `reasoning_content`) while still tracking spend; its bug tracker shows the classic failure mode — one storage path redacted, another (raw `proxy_server_request`) not. Lesson: **redact at the single ingestion point** of `RequestLogStore`, not per-sink. Store `"[redacted]"` placeholders (plus sizes/counts, e.g. message count, image count) unless the per-session "reveal bodies" switch is on. Sources: [LiteLLM logging docs](https://docs.litellm.ai/docs/proxy/logging), [redaction bug #16336](https://github.com/BerriAI/litellm/issues/16336), [message redaction overview](https://deepwiki.com/BerriAI/litellm/6.3-message-redaction-and-privacy-controls).1571- **Keys never touch the log path:** strip/replace `Authorization`, `x-api-key`, `api-key`, and any `*key*`/`*token*` header before a request object is handed to the logger; scrub upstream error bodies (some providers echo the offending header) through the same filter. Verification (Phase 7): grep all logs/exports for known key material.1572- Ring buffer in memory (e.g. last 1–5k entries) + persisted store with retention setting; export honors the current redaction state.15731574### 5.2 Token usage extraction per provider15751576- **Prefer upstream-reported usage.** Non-streaming: OpenAI-compatible providers return `usage`; Anthropic returns `usage.input_tokens/output_tokens` (in `message_start` + `message_delta` when streaming); Gemini returns `usageMetadata`. Streaming with OpenAI-compatible upstreams: request `stream_options: {"include_usage": true}` upstream where supported so a final usage chunk arrives; some compatible providers (per-provider quirk table from Section 3) send usage on the last chunk regardless, others never do.1577- **Estimate only when the upstream gives nothing**, and **flag it** — e.g. `"usage": {..., "x_zyquo_estimated": true}` (or an `x-zyquo` extension block) so cost tiles can render "≈". Estimation options in Swift: [aespinilla/Tiktoken](https://github.com/aespinilla/Tiktoken) (pure-Swift tiktoken: cl100k_base etc.) and [narner/TiktokenSwift](https://github.com/narner/TiktokenSwift) (UniFFI bindings to the real tiktoken, incl. `o200k_base`). A dependency-free fallback — `chars/4` or `words × 4/3` — is acceptable for the *flagged-estimate* path given tokenizers differ per provider anyway; decide in Phase 3 whether pulling a tokenizer dep is worth it (recommendation: start with the heuristic, keep the field flagged, add Tiktoken later if users need tight estimates).1578- Cached-token counts: OpenAI reports `usage.prompt_tokens_details.cached_tokens`; Anthropic reports `cache_read_input_tokens`/`cache_creation_input_tokens`. Preserve these in the normalized usage when present (they change cost — see 5.3).15791580### 5.3 Cost calculation15811582- Source of truth: the ported Zyquo Cloud model catalog's per-model pricing (USD per 1M input tokens / per 1M output tokens). `cost = prompt_tokens × in_price/1e6 + completion_tokens × out_price/1e6`.1583- **Cached input tokens** are billed at a discount where reported (e.g. OpenAI cached input typically 50–90% off; Anthropic cache reads at 0.1× base input, cache writes at 1.25×): when the catalog has cached pricing and the upstream reports cached counts, split: `(prompt − cached) × in_price + cached × cached_price`. Where the catalog lacks a cached rate, fall back to full input price (over-estimate, never under).1584- Reasoning tokens (OpenAI `completion_tokens_details.reasoning_tokens`) are already included in `completion_tokens` — do not double-count.1585- Mark costs derived from estimated usage as estimated. Store per-request cost in `UsageRecord`; aggregate per key/model/provider/day for the dashboard. Settings allow a pricing override table (providers reprice frequently) and a display currency (store USD, convert for display only).1586- Precedent: OpenRouter prices "using the model that was ultimately used, which will be returned in the `model` attribute of the response body" — cost must always be computed against the *actually-used* model after fallbacks. Source: [OpenRouter model fallbacks](https://openrouter.ai/docs/guides/routing/model-fallbacks).15871588### 5.4 Rate limiting per local API key15891590- **Algorithm: token bucket**, the production default (AWS, Stripe) because "real traffic is bursty" — it allows short bursts up to bucket capacity while enforcing an average rate; sliding-window counters give smoother limits but punish legitimate bursts. For a local single-process gateway, in-memory is all we need (no distributed store). Sources: [Arcjet — rate limiting algorithms](https://blog.arcjet.com/rate-limiting-algorithms-token-bucket-vs-sliding-window-vs-fixed-window/), [token bucket vs sliding window](https://medium.com/@tihomir.manushev/token-bucket-vs-sliding-window-the-rate-limiting-choice-that-shapes-your-apis-behavior-e04fb2646ee5), [APISIX — gateway rate limiting](https://apisix.apache.org/learning-center/api-gateway-rate-limiting/).1591- Implementation: one `actor RateLimiter` holding `keyID → (tokens: Double, lastRefill: ContinuousClock.Instant)`. Lazy refill on each check: `tokens = min(capacity, tokens + elapsed × rate)`; admit iff `tokens ≥ 1` then decrement. Per-key config: requests/min (rate = rpm/60, capacity ≈ rpm burst allowance). No timers, O(1) per request, trivially Sendable.1592- On limit: `429` in OpenAI error format (`type: "rate_limit_error"`-style) with `Retry-After: ceil((1 − tokens)/rate)` seconds. Count rejections in per-key stats (Keys screen mini-chart).1593- Optional second dimension later: tokens-per-minute budget (LLM-style limits) using the same bucket with token-cost withdrawal after usage is known.15941595### 5.5 Retries with exponential backoff + jitter15961597Consensus best practice for LLM upstreams ([Zuplo 429 guide](https://zuplo.com/learning-center/http-429-too-many-requests-guide), [handling 429s in production LLM apps](https://www.getmaxim.ai/articles/handle-429-errors-in-production-llm-applications/), [retry strategies with backoff + jitter](https://callsphere.ai/blog/retry-strategies-llm-api-calls-exponential-backoff-jitter-tenacity)):1598- **Retry on:** 429, 500, 502, 503, 504, connection reset/refused, and TTFB timeout. **Never** on 400/401/403/404/422 (client/config errors — fail fast with the mapped OpenAI error).1599- **Schedule:** `delay = min(cap, base × 2^attempt) + random(0, jitter)` — e.g. base 1 s, cap 30 s, full jitter; **max 3 attempts** for these interactive, user-facing requests (total budget ≤ ~30 s before fallback/error).1600- **Respect `Retry-After`** when the provider sends it: `wait = max(retryAfter, computedBackoff)`; if `Retry-After` exceeds our remaining budget, skip retrying this provider and go straight to fallback/error (propagating `Retry-After` to our own 429 response).1601- **Idempotency / streaming rule:** a retry is only safe **before any response byte has been forwarded to the client**. Once the first SSE chunk has been written downstream, never retry or fall back — terminate the stream with an error event/close. (Chat completions are not idempotent upstream either: a "failed" request may still have consumed tokens; retrying after partial streaming double-bills and duplicates output.) So: retries apply to (a) whole non-streaming calls, (b) streaming calls that fail before the first upstream content delta.1602- Jitter exists to desynchronize concurrent clients; even locally, parallel requests from one SDK justify it.16031604### 5.6 Fallback chains16051606Modeled on LiteLLM and OpenRouter:1607- **Semantics:** an ordered model list tried in sequence. LiteLLM: "the router tries the primary model first; if it fails with a retry-able error (429, 5xx, context-limit, content-policy, timeout), it moves to the first fallback", in order. OpenRouter: a `models: [...]` array tried in order server-side. Sources: [LiteLLM reliability/fallbacks](https://docs.litellm.ai/docs/proxy/reliability), [LiteLLM router architecture](https://docs.litellm.ai/docs/router_architecture), [OpenRouter model fallbacks](https://openrouter.ai/docs/guides/routing/model-fallbacks).1608- **Trigger errors:** exhausted retries on 429/5xx/timeout; upstream auth failure (missing/invalid provider key — jumping to a provider the user *has* a key for is exactly the point); model-not-available (404 upstream). Optionally context-window errors (LiteLLM has a distinct `context_window_fallbacks` class). **Not** on content-policy 400s by default (surprising model swaps on policy errors are a footgun; make it opt-in like LiteLLM's `content_policy_fallbacks`).1609- **Same rule as retries:** no fallback after the first downstream byte.1610- **Honest reporting of the actually-used model** (our Phase 3 spec requirement): OpenRouter returns "the model that was ultimately used … in the `model` attribute of the response body"; LiteLLM exposes the concrete deployment via `x-litellm-model-id` header / `_hidden_params`. Zyquo Router does both: the response/chunks' `model` field carries the namespaced ID that actually served the request, plus an `x-zyquo-served-model` response header and the fallback hop count in the request log. Beware LiteLLM's documented pitfall of fallbacks resetting the retry cycle and re-running fallback models ([issue #19985](https://github.com/BerriAI/litellm/issues/19985)) — our loop: `for model in chain { retryPolicy(model) }`, each model getting one bounded retry budget, no restarts.1611- Per-chain config lives in the Models screen editor; a chain is addressable like a model/alias.16121613### 5.7 Health checks16141615- `GET /health` (no auth, no logging noise) returning:16161617```json1618{1619 "status": "ok",1620 "version": "1.0.0",1621 "uptime_seconds": 12345,1622 "server": { "host": "127.0.0.1", "port": 8787 },1623 "providers_configured": 7,1624 "active_streams": 21625}1626```16271628- `status` is `"ok"` if the server is accepting; no upstream probing on this path (it must be instant and side-effect-free — LiteLLM separates `/health` per-model probes, which cost real tokens, from cheap `/health/liveliness` liveness checks; ours is the cheap kind). Per-provider connectivity testing belongs to the Keys screen's explicit "Test" button, not the health endpoint. Source: [LiteLLM proxy docs](https://docs.litellm.ai/docs/proxy/reliability).1629- Suitable for `curl`-based readiness in scripts and the Phase 7 harness; also the phase-gate check for Phase 2.16301631### Sources (consolidated)16321633- SwiftNIO: https://github.com/apple/swift-nio · https://github.com/apple/swift-nio/releases · https://swiftpackageindex.com/apple/swift-nio · https://github.com/apple/swift-nio/blob/main/docs/public-async-nio-apis.md · https://swiftinit.org/docs/swift-nio/niocore/nioasyncchannel · https://github.com/apple/swift-nio/blob/main/Sources/NIOHTTP1Server/main.swift · https://swiftonserver.com/using-swiftnio-channels/ · https://forums.swift.org/t/nioasyncchannel-executethenclose-is-too-restrictive/73460 · https://blog.alexseifert.com/2026/06/29/building-a-web-app-in-swift-using-only-swiftnio/1634- NIOExtras quiescing: https://github.com/apple/swift-nio-extras/blob/main/Sources/NIOExtras/QuiescingHelper.swift · https://github.com/apple/swift-nio-extras/blob/main/Sources/HTTPServerWithQuiescingDemo/main.swift1635- Swift.org server guidelines (NIO3 timing): https://www.swift.org/documentation/server/guides/libraries/concurrency-adoption-guidelines.html1636- Hummingbird: https://github.com/hummingbird-project/hummingbird · https://swiftpackageindex.com/hummingbird-project/hummingbird · https://swiftonserver.com/whats-new-in-hummingbird-2/ · https://hummingbird.codes/news/hummingbird-2/ · https://github.com/hummingbird-project/hummingbird-examples (server-sent-events example) · https://github.com/swift-server/swift-service-lifecycle1637- Vapor comparison: https://github.com/hummingbird-project/hummingbird/discussions/150 · https://theswiftdev.com/beginners-guide-to-server-side-swift-using-the-hummingbird-framework/1638- Network.framework: https://github.com/helje5/NWHTTPProtocol · http://www.alwaysrightinstitute.com/network-framework/ · https://developer.apple.com/documentation/network1639- SSE format: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events1640- macOS local network privacy: https://developer.apple.com/forums/thread/763484 · https://developer.apple.com/forums/thread/767391 · https://mjtsai.com/blog/2024/10/02/local-network-privacy-on-sequoia/ · https://foldr.com/foldr-support/foldr-for-macos/macos-15-sequoia-local-network-privacy/ · https://help.panic.com/prompt/prompt-local-network/ · https://developer.apple.com/forums/thread/769037?page=21641- Entitlements/sandbox: https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.security.network.server · https://lapcatsoftware.com/articles/hardened-runtime-sandboxing.html1642- LiteLLM: https://docs.litellm.ai/docs/proxy/reliability · https://docs.litellm.ai/docs/router_architecture · https://docs.litellm.ai/docs/proxy/logging · https://github.com/BerriAI/litellm/issues/16336 · https://github.com/BerriAI/litellm/issues/19985 · https://deepwiki.com/BerriAI/litellm/6.3-message-redaction-and-privacy-controls1643- OpenRouter: https://openrouter.ai/docs/guides/routing/model-fallbacks · https://openrouter.ai/blog/insights/reliability-failover/1644- Tokenizers in Swift: https://github.com/aespinilla/Tiktoken · https://github.com/narner/TiktokenSwift1645- Rate limiting: https://blog.arcjet.com/rate-limiting-algorithms-token-bucket-vs-sliding-window-vs-fixed-window/ · https://medium.com/@tihomir.manushev/token-bucket-vs-sliding-window-the-rate-limiting-choice-that-shapes-your-apis-behavior-e04fb2646ee5 · https://apisix.apache.org/learning-center/api-gateway-rate-limiting/1646- Retries/backoff: https://zuplo.com/learning-center/http-429-too-many-requests-guide · https://www.getmaxim.ai/articles/handle-429-errors-in-production-llm-applications/ · https://callsphere.ai/blog/retry-strategies-llm-api-calls-exponential-backoff-jitter-tenacity1647