# ROUTER-RESEARCH.md — Zyquo Router

How to build a production-quality local LLM gateway, correctly. Compiled from intensive web research (2026-07-30) against current official documentation, SDK sources, and reference gateway implementations. Sections: (1) the OpenAI API specification the router implements, (2) how existing gateways do it, (3) translation matrices for non-OpenAI upstreams, (4) HTTP serving in Swift, (5) gateway concerns. Sources cited inline.

# Binding decisions (executive summary)

# 1. The OpenAI API specification

Research date: 2026-07-30. Primary sources: the OpenAI API reference (https://platform.openai.com/docs/api-reference/chat, mirrored at https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create), the streaming-events reference (https://developers.openai.com/api/reference/resources/chat/subresources/completions/streaming-events), and the official SDK type definitions, which are generated from OpenAI's OpenAPI spec and are therefore authoritative for the wire format (https://github.com/openai/openai-python/tree/main/src/openai/types/chat, https://github.com/openai/openai-node). This section is the implementation contract for Zyquo Router's public surface: whatever the router serves must match this, byte-shape for byte-shape.

# 1.1 POST /v1/chat/completions — request schema

Headers: Authorization: Bearer <key>, Content-Type: application/json. Only model and messages are required; every other parameter is optional and, when omitted, must be treated as "provider default" (the router must NOT inject its own defaults into upstream calls).

# 1.1.1 model (string, required)

Model ID, e.g. "gpt-4o". For Zyquo Router this is the namespaced provider/model-id, an unambiguous bare ID, or an alias. The response must echo a model string back (the router echoes the namespaced ID actually used).

# 1.1.2 messages (array, required)

Ordered conversation. Each element is an object with a role and role-specific fields. Current roles (per ChatCompletionMessageParam in openai-python, https://github.com/openai/openai-python/tree/main/src/openai/types/chat):

User content parts (array form of content):

• Text part: {"type": "text", "text": "..."}
• Image part (ChatCompletionContentPartImageParam, https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_content_part_image_param.py):

{
  "type": "image_url",
  "image_url": {
    "url": "https://example.com/cat.png",
    "detail": "auto"
  }
}

• image_url.url (required): "Either a URL of the image or the base64 encoded image data." The base64 form is a data URI: "data:image/jpeg;base64,/9j/4AAQ..." (data:<mime>;base64,<payload>; supported mimes: png, jpeg, webp, non-animated gif).
• image_url.detail (optional): "auto" (default) | "low" | "high" — "Specifies the detail level of the image."
• Audio part: {"type": "input_audio", "input_audio": {"data": "<base64>", "format": "wav"|"mp3"}} (audio-capable models only; the router may reject with a clean error for providers without audio-in).
• File part: {"type": "file", "file": {"file_id": "..."} } or {"file_data": "<base64 data URI>", "filename": "..."} (PDF input on OpenAI; per-provider support varies).

Full request example with roles + multimodal:

{
  "model": "gpt-4o",
  "messages": [
    {"role": "system", "content": "You are a terse assistant."},
    {"role": "user", "content": [
      {"type": "text", "text": "What is in this image?"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KG...", "detail": "high"}}
    ]},
    {"role": "assistant", "content": null, "tool_calls": [
      {"id": "call_abc123", "type": "function",
       "function": {"name": "lookup", "arguments": "{\"q\":\"cats\"}"}}
    ]},
    {"role": "tool", "tool_call_id": "call_abc123", "content": "{\"result\":\"a cat\"}"},
    {"role": "user", "content": "Thanks — summarize."}
  ]
}

# 1.1.3 Sampling & length parameters

Types/defaults/deprecations verified against CompletionCreateParams (https://github.com/openai/openai-python/blob/main/src/openai/types/chat/completion_create_params.py) and the API reference:

# 1.1.4 response_format

Three variants (see https://developers.openai.com/api/docs/guides/structured-outputs and shared_params/response_format_*.py in openai-python):

{"type": "text"}
{"type": "json_object"}
{
  "type": "json_schema",
  "json_schema": {
    "name": "weather_report",
    "description": "optional",
    "schema": {
      "type": "object",
      "properties": {"city": {"type": "string"}, "temp_c": {"type": "number"}},
      "required": ["city", "temp_c"],
      "additionalProperties": false
    },
    "strict": true
  }
}

• json_object = legacy JSON mode ("an older method of generating JSON responses"); the prompt must mention JSON or OpenAI errors.
• json_schema = Structured Outputs. json_schema.name is required ("Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64"); strict: true means "the model will always follow the exact schema defined" (subset of JSON Schema: all fields required, additionalProperties: false).
• Router: translate to each provider's equivalent (Gemini responseMimeType/responseSchema, provider-specific json modes) or reject with a helpful 400 where unsupported.

# 1.1.5 Tools / function calling

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"],
          "additionalProperties": false
        },
        "strict": true
      }
    }
  ],
  "tool_choice": "auto",
  "parallel_tool_calls": true
}

• tools[]: currently type: "function" for the public wire format (newer OpenAI additions include custom tools and hosted tools on the Responses API; a gateway needs only function). function.parameters is a JSON Schema object; function.strict optional.
• tool_choice — union per ChatCompletionToolChoiceOptionParam (https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_tool_choice_option_param.py):
  • "none" — never call tools ("default when no tools are present"),
  • "auto" — model decides (default when tools present),
  • "required" — model must call at least one tool,
  • named function: {"type": "function", "function": {"name": "get_weather"}},
  • (newer) allowed-tools form {"type": "allowed_tools", ...} — pass-through/optional for a gateway.
• parallel_tool_calls (boolean, default true): "Whether to enable parallel function calling during tool use."
• Deprecated legacy: functions ("Deprecated in favor of tools") and function_call ("Deprecated in favor of tool_choice") — accept, map to tools/tool_choice internally.

# 1.1.6 Streaming controls

• stream (boolean | null, default false): "If set to true, the model response data will be streamed to the client as it is generated using server-sent events."
• stream_options (object | null — "Only set this when you set stream: true"):
  • include_usage (boolean): "If set, an additional chunk will be streamed before the data: [DONE] message. The usage field on this chunk shows the token usage statistics for the entire request, and the choices field will always be an empty array. All other chunks will also include a usage field, but with a null value." (Source: https://community.openai.com/t/usage-stats-now-available-when-using-streaming-with-the-chat-completions-api-or-completions-api/738156, and the SDK type docstring.)
  • include_obfuscation (boolean, newer): adds random obfuscation padding fields to chunks; a local gateway should not emit it.

# 1.1.7 Identity, caching & misc parameters

Unknown-key policy: OpenAI itself returns 400 unrecognized argument for unknown top-level keys, but a gateway should follow LiteLLM/OpenRouter practice: accept unknown keys and pass them through to the upstream body (this is how provider-specific extras like Perplexity search_domain_filter or Qwen enable_thinking travel). Document accepted extras in docs/API.md.

# 1.2 Non-streaming response — chat.completion object

Fields verified against ChatCompletion (https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion.py) and CompletionUsage (https://github.com/openai/openai-python/blob/main/src/openai/types/completion_usage.py):

{
  "id": "chatcmpl-B9MBs8CjcvOU2jLn4n570S5qMJKcT",
  "object": "chat.completion",
  "created": 1741569952,
  "model": "gpt-4o-2024-08-06",
  "system_fingerprint": "fp_50cad350e4",
  "service_tier": "default",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I assist you today?",
        "refusal": null,
        "annotations": []
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 19,
    "completion_tokens": 10,
    "total_tokens": 29,
    "prompt_tokens_details": {"cached_tokens": 0, "audio_tokens": 0},
    "completion_tokens_details": {
      "reasoning_tokens": 0,
      "audio_tokens": 0,
      "accepted_prediction_tokens": 0,
      "rejected_prediction_tokens": 0
    }
  }
}

Field-by-field:

• id (string): "A unique identifier for the chat completion." Convention chatcmpl-<base62>; the router generates its own (chatcmpl- prefix keeps naive clients happy).
• object: literal "chat.completion".
• created (integer): Unix seconds.
• model (string): "The model used" — router echoes the namespaced ID actually served (incl. after fallback).
• system_fingerprint (string, optional, now marked deprecated in OpenAI docs): backend-config fingerprint for use with seed. Optional — the router may omit or emit a static value.
• service_tier (optional): echo only for OpenAI upstreams.
• choices[]:
  • index (integer),
  • message:
    • role: always "assistant",
    • content (string | null): null when the model only called tools,
    • refusal (string | null): structured-outputs refusal message,
    • tool_calls (array, optional): each {"id": "call_…", "type": "function", "function": {"name": "...", "arguments": "<JSON string>"}} — arguments is a string containing JSON, not an object,
    • annotations (array, optional): e.g. url_citation items from web search,
    • audio (optional, audio-out models),
    • reasoning models on other providers add reasoning_content (DeepSeek et al.) — not an OpenAI field, but the de-facto extension the router preserves (Phase 0 decision),
  • logprobs (object | null): {"content": [{token, logprob, bytes, top_logprobs: […]}], "refusal": […]} when requested,
  • finish_reason — exact literal set per the SDK: "stop" | "length" | "tool_calls" | "content_filter" | "function_call":
    • stop — natural stop or stop sequence hit,
    • length — token cap reached (max_completion_tokens or context limit),
    • tool_calls — the model called tools,
    • content_filter — content omitted by a filter,
    • function_call — deprecated legacy (only when using deprecated functions). Every upstream stop reason must be mapped into this set (e.g., Anthropic end_turn→stop, max_tokens→length, tool_use→tool_calls, stop_sequence→stop).
• usage:
  • prompt_tokens, completion_tokens, total_tokens (integers, total = prompt + completion),
  • prompt_tokens_details (optional): cached_tokens ("Cached tokens present in the prompt"), audio_tokens, and newer cache_write_tokens,
  • completion_tokens_details (optional): reasoning_tokens ("Tokens generated by the model for reasoning"), audio_tokens, accepted_prediction_tokens, rejected_prediction_tokens.
  • Gateway rule: use upstream-reported usage when present; when absent, estimate and flag (e.g. "x_zyquo": {"usage_estimated": true} — extension keys are tolerated by both SDKs).

Tool-call response example (non-streaming):

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1699896916,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_abc123",
            "type": "function",
            "function": {"name": "get_weather", "arguments": "{\n\"city\": \"Boston\"\n}"}
          }
        ]
      },
      "logprobs": null,
      "finish_reason": "tool_calls"
    }
  ],
  "usage": {"prompt_tokens": 82, "completion_tokens": 17, "total_tokens": 99}
}

# 1.3 The SSE streaming format — byte-level contract

Sources: streaming reference (https://developers.openai.com/api/reference/resources/chat/subresources/completions/streaming-events, https://platform.openai.com/docs/api-reference/chat-streaming/streaming), the cookbook (https://cookbook.openai.com/examples/how_to_stream_completions), and ChatCompletionChunk SDK types.

Transport. Response headers: Content-Type: text/event-stream; charset=utf-8, Cache-Control: no-cache, chunked transfer (no Content-Length), keep connection open. Each event is exactly:

data: <one-line JSON>\n
\n

i.e. the 6 bytes data: , the JSON serialized without newlines, then \n\n. OpenAI emits only data: lines — no event:, id:, or retry: fields, no SSE comments. The stream terminates with the sentinel:

data: [DONE]\n
\n

([DONE] is not JSON; both official SDKs special-case this exact string.) An HTTP-level error that occurs before streaming starts is a plain JSON error body with a proper status code; the status is sent before any chunk, so a request that fails validation must NOT return 200 + SSE.

Chunk object (object: "chat.completion.chunk"): same id ("Each chunk has the same ID"), created, and model across all chunks of one completion; choices[] with {index, delta, logprobs, finish_reason}; usage null/absent except the final usage chunk. Delta fields: role, content, refusal, tool_calls[] (each with index, optional id, optional type: "function", optional function.name, optional function.arguments fragment), deprecated function_call.

Chunk sequence rules:

1. First chunk carries the role delta: "delta": {"role": "assistant", "content": ""} (OpenAI includes the empty content string; emit it — some clients concatenate blindly). May also carry refusal: null — harmless.
2. Content chunks: "delta": {"content": "<fragment>"}, finish_reason: null.
3. Final content chunk: "delta": {} (empty object) with "finish_reason": "stop" (or length/tool_calls/content_filter). The finish_reason travels on a chunk whose delta is empty — not alongside content.
4. Optional usage chunk (only when stream_options.include_usage is true): "choices": [] (empty array — quoted from the SDK: choices "can also be empty for the last chunk if you set stream_options: {\"include_usage\": true}") and a populated usage object. "All other chunks will also include a usage field, but with a null value" when include_usage is set.
5. data: [DONE].

# (a) Plain-text transcript (stream: true, stream_options: {"include_usage": true})

data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"usage":null}

data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"content":"Hello"},"logprobs":null,"finish_reason":null}],"usage":null}

data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"content":" there"},"logprobs":null,"finish_reason":null}],"usage":null}

data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{"content":"!"},"logprobs":null,"finish_reason":null}],"usage":null}

data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null}

data: {"id":"chatcmpl-9rORw7dbcqvUv0entIFewX1LLQBmy","object":"chat.completion.chunk","created":1722525116,"model":"gpt-4o-mini-2024-07-18","system_fingerprint":"fp_611b667b19","choices":[],"usage":{"prompt_tokens":19,"completion_tokens":3,"total_tokens":22,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}}}

data: [DONE]

(Without include_usage, the usage chunk is absent and no usage key appears on chunks.)

# (b) Streamed tool call transcript

Tool-call arguments stream as string fragments. The first tool_call delta for a given index carries id, type, and function.name (with "arguments":""); every subsequent delta for that index carries ONLY index and function.arguments fragments — no id, no name. Parallel tool calls interleave via index 0,1,…; clients accumulate by index and concatenate arguments.

data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"role":"assistant","content":null},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_KSEnFnucOtZNQqEZF9wvfIWm","type":"function","function":{"name":"get_weather","arguments":""}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"ci"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ty\": \"Bos"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"ton\"}"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9s0Ab","object":"chat.completion.chunk","created":1722530000,"model":"gpt-4o-2024-08-06","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"tool_calls"}]}

data: [DONE]

# What strict SDK clients require

• openai-python parses SSE by splitting on blank lines, reads only data: payloads, stops at the exact string [DONE], and constructs a pydantic ChatCompletionChunk. Pydantic will fail the whole stream if id, object, created, model, or choices is missing/wrong-typed, if object isn't exactly "chat.completion.chunk", or if finish_reason is a value outside the literal set. Extra unknown fields are tolerated (kept as extras) — so reasoning_content in deltas and x_zyquo extensions are safe. (See https://github.com/openai/openai-python; a real-world failure mode of non-conforming servers: https://github.com/janhq/jan/issues/8280 — "the chat.completion.chunk schema … requires choices[]; type validation failed" halts the stream.)
• openai-node is looser at runtime (TypeScript types are compile-time), but its stream accumulator (stream.finalChatCompletion(), and the Vercel AI SDK on top of it) indexes choices[0].delta, accumulates tool_calls strictly by index, and expects id/function.name on the first delta of each tool call; missing index or re-sending name fragments corrupts accumulation (see https://ai-sdk.dev/providers/openai-compatible-providers on buffering unreliable tool-call deltas).
• Both SDKs ignore SSE event: fields only if the line isn't a data: line; do not emit named events or comments — emit only data: lines exactly as above.
• Keep every JSON chunk on a single line; UTF-8; never split a multibyte character across chunks inside one JSON string (JSON-escape or buffer to codepoint boundaries).

# 1.4 GET /v1/models and GET /v1/models/{id}

Source: https://developers.openai.com/api/reference/resources/models (mirrors https://platform.openai.com/docs/api-reference/models).

GET /v1/models →

{
  "object": "list",
  "data": [
    {"id": "gpt-4o", "object": "model", "created": 1686935002, "owned_by": "openai"},
    {"id": "gpt-4o-mini", "object": "model", "created": 1686935002, "owned_by": "openai"}
  ]
}

Model object: id (string — "The model identifier, which can be referenced in the API endpoints"), object (always "model"), created (Unix seconds), owned_by (string — "The organization that owns the model").

GET /v1/models/{model} → a single model object:

{"id": "gpt-4o", "object": "model", "created": 1686935002, "owned_by": "openai"}

Unknown ID → 404 with the model-not-found error (§1.5). Router mapping: id = namespaced provider/model-id (aliases listed too), owned_by = provider name; enriched metadata (context window, pricing, capabilities) goes under an x_zyquo extension key on each entry — both SDKs tolerate extra fields.

# 1.5 Error response format

Every error is JSON with a single error object (see https://developers.openai.com/api/docs/guides/error-codes and https://community.openai.com/t/openai-chat-list-of-error-codes-and-types/357791):

{
  "error": {
    "message": "<human-readable description>",
    "type": "<error family>",
    "param": "<offending request parameter or null>",
    "code": "<machine-readable code or null>"
  }
}

All four keys are always present (param/code may be null). The official SDKs map HTTP status → typed exceptions (BadRequestError 400, AuthenticationError 401, PermissionDeniedError 403, NotFoundError 404, UnprocessableEntityError 422, RateLimitError 429, InternalServerError ≥500, per https://github.com/openai/openai-python#handling-errors).

The exact model-not-found error (HTTP 404) as OpenAI returns it (sources: https://community.openai.com/t/openai-error-invalidrequesterror-the-model-gpt-4-does-not-exist-or-you-do-not-have-access-to-it/376230, https://community.openai.com/t/api-returning-404-model-not-found-all-of-a-sudden-why-and-how-to-fix/679777):

{
  "error": {
    "message": "The model `gpt-5-nonexistent` does not exist or you do not have access to it.",
    "type": "invalid_request_error",
    "param": null,
    "code": "model_not_found"
  }
}

The router must reproduce this shape verbatim (with its namespaced ID in backticks) for unknown/disabled models — SDK error-handling paths and agent frameworks string-match parts of it.

Streaming errors: if the failure happens before any chunk, return the JSON error with the real status code (no SSE). If the upstream dies mid-stream, OpenAI's own behavior is to emit an error payload as a data: line ({"error": {...}}) and close without [DONE]; a gateway should do the same — strict clients surface it as a stream error rather than hanging.

Gateway error-mapping rules (Phase 3 contract): upstream 401 (provider key bad) → 401 with "provider key for <provider> was rejected" (never echo the key); upstream 429 → 429 + Retry-After when given; upstream timeout → 504 timeout; upstream 5xx/unparseable → 502 bad_gateway with sanitized detail; never leak raw provider error shapes or key material.

# 1.6 POST /v1/responses — evaluate and decide

Summary (sources: https://platform.openai.com/docs/guides/migrate-to-responses, https://developers.openai.com/api/docs/guides/migrate-to-responses):

• Shape: flatter request — input (string or item array) + top-level instructions instead of a messages array; response is a typed output array of items (message, reasoning with encrypted_content, function_call, function_call_output, hosted-tool items) plus an output_text convenience; server-side state via store: true / previous_response_id; built-in hosted tools (web search, file search, code interpreter, computer use).
• Streaming: semantic events, not chunk deltas — named SSE events like response.created, response.output_item.added, response.output_text.delta, response.completed — a completely different event model from chat.completion.chunk.
• Adoption (as of mid-2026): OpenAI recommends Responses "for all new projects" and newest OpenAI-native features (encrypted reasoning, hosted tools) land there first, but they state Chat Completions "remains supported" indefinitely as the industry standard. Critically for a multi-provider gateway: the entire compatible-provider ecosystem (DeepSeek, Qwen, Mistral, xAI, Together, Cerebras, Ollama, LM Studio, vLLM…) standardized on chat/completions, and gateways (LiteLLM, OpenRouter) still treat it as the lingua franca (OpenRouter exposes chat/completions; LiteLLM added a Responses bridge that internally converts to chat/completions).

DECISION for Zyquo Router: do NOT expose /v1/responses in v1. Rationale:

1. Zero translation leverage — none of our 12 upstream providers speak Responses natively except OpenAI itself; we'd be building a second full bidirectional translation layer (items + semantic streaming events) purely as a front-end conversion to the same canonical internal request.
2. Statefulness (store, previous_response_id, encrypted_content) implies server-side conversation storage — out of scope and against the router's privacy posture.
3. Client compatibility target is met without it: every OpenAI-SDK-based tool can (and with third-party base URLs, typically does) use chat.completions.
4. Cheap to add later as a stateless subset (input/instructions → messages; emit response.output_text.delta events) once the chat/completions core is green — track as a post-v1 enhancement in docs/PLAN.md. /health should not advertise it; requests to /v1/responses return a 404 OpenAI-format error with a message pointing to /v1/chat/completions.

# 1.7 POST /v1/embeddings (optional gateway endpoint)

Source: https://platform.openai.com/docs/api-reference/embeddings and embedding_create_params.py / create_embedding_response.py in openai-python.

Request:

{
  "model": "text-embedding-3-small",
  "input": "The food was delicious and the waiter...",
  "encoding_format": "float",
  "dimensions": 512,
  "user": "optional-end-user-id"
}

• input (required): "string or array of tokens. To embed multiple inputs in a single request, pass an array of strings or array of token arrays." (Max ~2048 inputs per request; each within the model's token limit.)
• model (required); encoding_format: "float" (default) or "base64"; dimensions: "Only supported in text-embedding-3 and later models"; user: abuse-monitoring ID.

Response:

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [0.0023064255, -0.009327292, -0.0028842222]
    }
  ],
  "model": "text-embedding-3-small",
  "usage": {"prompt_tokens": 8, "total_tokens": 8}
}

Notes for the router: data[] is ordered by index matching the input array; usage has only prompt_tokens + total_tokens (no completion tokens); encoding_format: "base64" returns each embedding as a base64 string of little-endian float32 — support it, since openai-python requests base64 by default when numpy is available. Route only to providers that offer embeddings; others 404 with model_not_found.

# 1.8 Contract checklist for Zyquo Router (derived from this section)

• Accept full request schema §1.1 incl. both max_tokens and max_completion_tokens, developer role, multimodal parts, all three response_format variants, all tool_choice forms, deprecated functions/function_call.
• Emit spec-exact chat.completion (§1.2) with mapped finish_reason from the closed literal set and real-or-flagged usage incl. details sub-objects when upstreams provide them.
• Emit byte-exact SSE (§1.3): role-first chunk, single-line JSON data: events, empty-delta finish chunk, empty-choices usage chunk only under include_usage, terminating data: [DONE], correct tool-call delta id/name/index rules.
• GET /v1/models + /v1/models/{id} per §1.4 with namespaced IDs.
• All errors per §1.5 incl. verbatim model-not-found shape and gateway 502/504 conventions.
• /v1/responses: not exposed in v1 (documented 404 with pointer); /v1/embeddings: optional, per §1.7.

# 2. How existing gateways do it

Research date: 2026-07-30. Sources fetched live from docs.litellm.ai, openrouter.ai/docs, docs.ollama.com, lmstudio.ai/docs, docs.vllm.ai, and GitHub issue trackers. This section extracts the concrete, battle-tested patterns from the reference gateways that Zyquo Router should copy — and the compatibility landmines it must avoid.

# 2.1 LiteLLM proxy — the reference for config, translation, and error mapping

LiteLLM is the most complete open-source implementation of exactly what Zyquo Router is: an OpenAI-compatible front for ~100 providers. Its patterns are the closest prior art.

# 2.1.1 Model naming: provider/model

LiteLLM routes on a provider prefix in the model string: openai/gpt-4o, azure/gpt-4o, anthropic/claude-..., bedrock/anthropic.claude-instant-v1, ollama/mistral, gemini/gemini-2.5-pro. The prefix selects the client implementation; the remainder is the upstream model ID sent to the provider. (Source: https://docs.litellm.ai/docs/proxy/configs)

The proxy adds a second layer of indirection: a user-facing model_name alias mapped to one or more concrete deployments in config.yaml:

model_list:
  - model_name: gpt-4o                # what clients send in "model"
    litellm_params:
      model: azure/gpt-4o-eu          # provider/upstream-id actually called
      api_base: https://endpoint-europe.openai.azure.com/
      api_key: "os.environ/AZURE_API_KEY_EU"   # env-var indirection, secrets never in config
      rpm: 6                          # per-deployment rate limit
    model_info:                       # optional metadata (pricing/context overrides)
      max_input_tokens: 128000

litellm_settings:
  drop_params: true
  num_retries: 3
  request_timeout: 10
  fallbacks: [{"gpt-4o": ["claude-sonnet"]}]

router_settings:
  routing_strategy: simple-shuffle
  model_group_alias: {"gpt-4": "gpt-4o"}   # request-time alias remapping

general_settings:
  master_key: sk-1234                  # local bearer key gating the proxy

Key ideas to steal (https://docs.litellm.ai/docs/proxy/configs):

• Two-level naming: public alias (model_name) → concrete provider/model deployment. Zyquo Router's aliases (fast → cerebras/...) are exactly this.
• Multiple entries with the same model_name = load balancing group (Zyquo Router doesn't need multi-deployment balancing, but the alias→target indirection is the same shape).
• model_group_alias maps well-known client names (gpt-4) onto configured groups — useful for tools that hardcode OpenAI model names.
• Wildcards (model_name: "*", model: openai/*) allow pass-through of any model given credentials — Zyquo Router's "accept unambiguous bare IDs" is a constrained version of this.
• os.environ/VAR indirection keeps keys out of the config file (Zyquo Router: keys live only in the vault; config export never includes them).

# 2.1.2 Param translation and drop tables

LiteLLM maintains, per provider, a mapping table of which OpenAI params the provider supports (queryable via litellm.get_supported_openai_params(model)), and translates names where they differ. Handling of unsupported params is explicit policy, not accident (https://docs.litellm.ai/docs/completion/drop_params):

• Default: raise an exception if a param is sent to a model that doesn't support it — loud failure over silent behavior change.
• drop_params: true (global, per-deployment, or per-request): silently strip unsupported params instead of erroring. Most proxies run with this on.
• additional_drop_params: ["response_format"] — per-deployment list of specific params to strip even if nominally supported; supports JSONPath-ish nested syntax (tools[*].input_examples, parent.child, array[0]).
• allowed_openai_params: ["tools"] — the inverse escape hatch: force-forward a param LiteLLM believes is unsupported (settable in config or per-request via extra_body).
• Provider-specific extras ride through extra_body on the OpenAI SDK and are passed to the upstream unchanged.

Lesson for Zyquo Router's CompatAdjuster: implement a per-provider param table (supported / rename / strip / pass-through-extras) as data, not scattered ifs, and make the strip-vs-error policy explicit and configurable.

# 2.1.3 Error mapping to OpenAI exceptions

LiteLLM maps every upstream failure onto exception types that inherit from the OpenAI SDK's own exceptions, so client code catching openai.RateLimitError works against any provider (https://docs.litellm.ai/docs/exception_mapping). Status-code taxonomy:

Every mapped exception carries status_code, message, and llm_provider (which upstream failed) — Zyquo Router should likewise name the provider in error messages ("Anthropic key invalid") without leaking payloads. Note that ContextWindowExceededError and ContentPolicyViolationError are distinguished subtypes of 400: this is what makes context-window fallbacks and content-policy fallbacks possible. A _should_retry(status_code) helper centralizes the retryability decision (429, 5xx, timeouts → retry; 4xx auth/validation → don't).

# 2.1.4 Retries, fallbacks, cooldowns

(Sources: https://docs.litellm.ai/docs/routing, https://docs.litellm.ai/docs/proxy/reliability)

• num_retries: 3 with exponential backoff for RateLimitError, immediate retry for transient errors; retry_after sets a minimum wait. A RetryPolicy can set retry counts per exception class (e.g., AuthenticationErrorRetries=0, RateLimitErrorRetries=3, TimeoutErrorRetries=2) — never retry auth failures.
• Three fallback kinds, configured as ordered maps {"primary": ["fallback1", "fallback2"]}:
  • fallbacks — general retryable errors (429/5xx) after retries exhaust;
  • context_window_fallbacks — prompt too big → reroute to a bigger-context model;
  • content_policy_fallbacks — content filter tripped → reroute to a laxer model;
  • default_fallbacks: ["claude-opus"] — catch-all.
• Per-request fallbacks via a fallbacks: [...] array in the request body, and disable_fallbacks: true to opt out per request.
• Cooldowns: allowed_fails: 3 failures per minute puts a deployment on a cooldown_time: 30s bench so the router stops hammering a failing upstream. For a single-deployment-per-model local router this maps to "mark provider degraded, surface in dashboard, fail fast or fall back."
• Execution order: retries on the primary first, then fallbacks in order until success or exhaustion. The actually-used deployment is reported via an x-litellm-model-id response header — Zyquo Router should report the actually-used model in the response model field (OpenRouter's approach, §2.2.4) and/or a header.

# 2.1.5 Usage & cost tracking

(Source: https://docs.litellm.ai/docs/completion/token_usage)

• Pricing lives in one community-maintained JSON file, model_prices_and_context_window.json (https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) — the de-facto industry pricing database. Entry shape:

{
  "gpt-4o": {
    "max_tokens": 4000,
    "input_cost_per_token": 1.5e-06,
    "output_cost_per_token": 2e-06,
    "litellm_provider": "openai",
    "mode": "chat"
  }
}

• completion_cost(response) computes USD from the usage in a response + this table; cost_per_token(model, prompt_tokens, completion_tokens) is the primitive. Every response carries response_cost in hidden params.
• register_model({...}) lets users override/add pricing — Zyquo Router's Settings ▸ Usage & Pricing "pricing override" is the same feature.
• When upstream doesn't return usage (some streams), tokens are estimated with a tokenizer (tiktoken default, provider-specific where known) — and Zyquo Router should flag estimated usage as such.

Pattern: pricing is data keyed by model ID, cost computed at response time from usage, estimation as fallback. Zyquo Router already has per-model pricing in the Zyquo Cloud catalog; reuse it as the single pricing source.

# 2.1.6 Streaming normalization

LiteLLM wraps every provider stream in a CustomStreamWrapper that re-emits uniform OpenAI-shaped chunk objects (choices[0].delta.content, etc.) regardless of upstream wire format, plus a stream_chunk_builder(chunks) helper that reassembles a full chat.completion from chunks (useful for logging/cost of streamed requests — Zyquo Router's request log needs exactly this) (https://docs.litellm.ai/docs/completion/stream). It also guards against pathological streams: REPEATED_STREAMING_CHUNK_LIMIT = 100 aborts with an InternalServerError if the same chunk repeats endlessly.

Cautionary tales from LiteLLM's own tracker — even the reference implementation gets chunk shape wrong:

• Its synthetic usage chunk violated the OpenAI spec by carrying a non-empty choices array (spec: the include_usage final chunk has "choices": []) — https://github.com/BerriAI/litellm/issues/28735
• It lost vLLM's usage because vLLM sends usage in a separate empty-choices chunk after the finish_reason chunk and LiteLLM stopped reading at finish_reason — https://github.com/BerriAI/litellm/issues/25389 . Lesson: when consuming upstreams, read until the stream actually ends, not until finish_reason.
• Grok returned usage in the wrong chunk (extra empty final chunk) — https://github.com/BerriAI/litellm/issues/17136 ; and some providers reject stream_options as an unknown param, so the gateway must know per provider whether it can request usage-in-stream — https://github.com/BerriAI/litellm/issues/23847

# 2.2 OpenRouter — the reference for unified IDs, streaming discipline, and honest accounting

# 2.2.1 Unified model IDs and variants

• IDs are vendor/model-name (anthropic/claude-3.5-sonnet, openai/gpt-4o), plus a permanent canonical_slug that survives renames. (https://openrouter.ai/docs/models)
• Variant suffixes append behavior to a slug: :free (free tier), :thinking (reasoning mode), :nitro (= provider.sort: "throughput"), :floor (sort by price). A suffix-on-the-ID is a very ergonomic way to encode routing preferences without extra params (https://openrouter.ai/docs/models, https://openrouter.ai/docs/features/provider-routing). Zyquo Router could reserve this pattern for future use (e.g., model:nostore to skip logging).
• GET /api/v1/models returns rich metadata per model: id, canonical_slug, name, context_length, architecture (input/output modalities, tokenizer), pricing (USD per token as strings — "0" means free; string avoids float precision issues), supported_parameters (array of OpenAI params this model accepts — clients can pre-check!), top_provider. Zyquo Router's plan to enrich /v1/models under an x-zyquo extension key mirrors this; adopting context_length, pricing, and supported_parameters fields is directly useful for the Playground and Docs UI.

# 2.2.2 Provider-specific params & headers

• Default posture: unsupported params are silently ignored by the receiving provider; setting "provider": {"require_parameters": true} restricts routing to providers that support every param in the request (https://openrouter.ai/docs/features/provider-routing). For a single-upstream-per-model router, the equivalent decision is strip-vs-reject per param (see §2.5).
• Extra attribution headers HTTP-Referer and X-Title are optional and additive — the API remains pure OpenAI otherwise (https://openrouter.ai/docs/api-reference/overview).
• Provider-specific features arrive as extra top-level body keys (e.g., models, provider, plugins, transforms) that OpenAI SDKs send via extra_body — the standard pass-through idiom Zyquo Router should adopt for things like Perplexity search options.

# 2.2.3 Streaming normalization — the details that matter

(Source: https://openrouter.ai/docs/api-reference/streaming)

• Keep-alive SSE comments: OpenRouter periodically emits : OPENROUTER PROCESSING comment lines to hold connections open during long prefill/queue waits. Per the SSE spec, lines starting with : are comments and must be ignored — but naive client loops that JSON.parse every line crash on them. Zyquo Router should (a) emit its own : keep-alive comments during long upstream silences (Anthropic thinking, queued requests), and (b) tolerate/strip comments when consuming upstream SSE.
• Termination is always data: [DONE].
• Mid-stream errors: once tokens have flowed you can't change the HTTP status, so errors arrive as a final SSE event that is still a valid chat.completion.chunk with an added error object and finish_reason: "error":

data: {"id":"gen-abc123","object":"chat.completion.chunk","created":1730000000,
       "model":"...","error":{"code":429,"message":"Rate limit exceeded",
       "metadata":{"error_type":"rate_limit_exceeded"}},
       "choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}]}

(https://openrouter.ai/docs/api-reference/errors) This is the pattern Zyquo Router needs for "upstream died mid-stream": emit a well-formed error chunk, then [DONE], then close — never a bare connection reset.

• Cancellation: aborting the connection cancels the upstream only for streaming requests on providers that support cancellation; otherwise the upstream finishes and bills anyway. Zyquo Router must cancel the upstream URLSession/NIO task on client disconnect (its clients are direct HTTP calls, so cancellation is always possible) — this is Phase 7 test 3.
• Usage stats ride in the final chunk (chunk.usage).

# 2.2.4 Usage accounting

(Source: https://openrouter.ai/docs/use-cases/usage-accounting)

• Usage (with cost) is now included in every response automatically (the old usage: {include: true} opt-in is deprecated). Shape extends OpenAI's usage:

"usage": {
  "prompt_tokens": 194,
  "completion_tokens": 2,
  "total_tokens": 196,
  "cost": 0.95,
  "cost_details": {"upstream_inference_cost": 19},
  "prompt_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 100},
  "completion_tokens_details": {"reasoning_tokens": 0}
}

Putting cost inside usage is a compatible extension (SDKs ignore unknown fields on responses) — a good idea for Zyquo Router's estimated-cost surfacing.

• For streams, usage appears in the last SSE message; there's also a post-hoc GET /api/v1/generation?id=... stats endpoint keyed by the response id for auditing. Zyquo Router's request-log detail view is the local equivalent.

# 2.2.5 Error format

(Source: https://openrouter.ai/docs/api-reference/errors)

• Error shape: {"error": {"code": <number>, "message": "...", "metadata": {...}}} with HTTP status = error.code for pre-stream failures. Status vocabulary: 400 bad params/CORS, 401 bad key, 402 out of credits, 403 moderation/guardrail, 408 timeout, 429 rate limited, 502 "your chosen model is down or we received an invalid response", 503 "no available provider meets your routing requirements".
• metadata carries structured context without leaking raw payloads: moderation errors include reasons, a truncated flagged_input (max 100 chars), provider_name, model_slug; provider errors include a canonical error_type plus the original provider_code. This "canonical code + original provider code" pair is exactly the honest-but-normalized surfacing Zyquo Router wants.
• 429/503 responses include a standard Retry-After header — Zyquo Router should propagate upstream Retry-After to its own clients.
• Note: OpenRouter uses a numeric error.code; the strict OpenAI format is {"error": {"message", "type", "param", "code"}} with string-ish code. Zyquo Router should keep the OpenAI field set (per §1) and put router-specific context in the message and/or an extension key, since OpenAI SDKs construct exceptions from type/code.

# 2.2.6 Fallbacks & routing preferences

(Sources: https://openrouter.ai/docs/guides/routing/model-fallbacks, https://openrouter.ai/docs/features/provider-routing)

• Model fallbacks: an extra models: ["primary", "fallback1", ...] array in the body (the model field is the first attempt; via OpenAI SDK it goes in extra_body). Fallback triggers on any error: context-length validation, moderation flags, rate limits, downtime. The response's model field always reports the model actually used, and pricing follows the actually-used model. This "honest model echo" is the contract Zyquo Router's CLAUDE.md already mandates for its fallback chains.
• Provider preferences (provider object): order (try providers in this order), allow_fallbacks (default true), require_parameters, ignore/only allow/deny lists, sort by price/throughput/latency, max_price. Default load balancing prefers providers without recent outages, weighted by inverse square of price. Mostly N/A for a local single-key-per-provider router, but the cooldown-on-recent-outage idea maps to LiteLLM cooldowns.
• finish_reason is normalized to exactly five values — stop, length, tool_calls, content_filter, error — with the raw upstream value preserved in a separate native_finish_reason field (https://openrouter.ai/docs/api-reference/overview). Recommended verbatim for Zyquo Router (OpenAI's own set is the first four plus function_call legacy; error only ever appears mid-stream).

# 2.3 Local OpenAI-compatible servers — what "compatible enough" looks like

These show which subset of the spec real clients actually depend on, and which deviations are tolerated.

# 2.3.1 Ollama (http://localhost:11434/v1)

(Source: https://docs.ollama.com/api/openai-compatibility)

• Endpoints: /v1/chat/completions, /v1/completions, /v1/models, /v1/models/{model}, /v1/embeddings, /v1/responses.
• Supported on chat: model, messages, temperature, top_p, max_tokens, frequency_penalty, presence_penalty, seed, stop, stream, stream_options.include_usage, response_format (JSON mode), tools, reasoning_effort/reasoning, vision via base64 images only.
• Not supported: logprobs, user, n, tool_choice, logit_bias, image URLs. Unsupported params are ignored rather than erroring.
• Auth: any/no API key accepted (pure localhost trust). Zyquo Router improves on this with optional local keys.
• Notable: Ollama had to add max_completion_tokens support because OpenAI deprecated max_tokens (https://github.com/ollama/ollama/issues/7125) — see §2.4.5.

Takeaways: even a hugely popular compat layer omits n, logprobs, logit_bias and (long) omitted tool_choice — clients broadly tolerate missing niche params, but tools, response_format, stream_options.include_usage, and vision are table stakes in 2026.

# 2.3.2 LM Studio (http://localhost:1234/v1)

(Source: https://lmstudio.ai/docs/app/api/endpoints/openai)

• Endpoints: /v1/models, /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/responses (added specifically so OpenAI Codex CLI works against it).
• Compatibility story is purely "change base_url on the official SDK" — the same acceptance test Zyquo Router's Phase 3 gate uses.
• Signal: local servers are converging on also exposing /v1/responses because new OpenAI tooling (Codex) speaks only the Responses API. Relevant to the Phase 0 "is /v1/responses worth it" decision: optional now, trending toward expected.

# 2.3.3 vLLM OpenAI-compatible server

(Source: https://docs.vllm.ai/en/latest/serving/online_serving/)

• Implements /v1/chat/completions (+ batch), /v1/completions (no suffix), /v1/responses, /v1/embeddings, transcription/translation.
• Known deviations: user is ignored; parallel_tool_calls defaults to true; extra sampling params (top_k, best_of, guided decoding) accepted via extra_body — the standard pass-through idiom again.
• vLLM's habit of sending usage in a separate empty-choices chunk after the finish_reason chunk is spec-correct but broke LiteLLM's consumer (https://github.com/BerriAI/litellm/issues/25389) — Zyquo Router's normalizer must handle both orderings when consuming OpenAI-compatible upstreams.

# 2.4 Compatibility pitfalls that trip up real clients

Strict SDKs (openai-python uses Pydantic models; openai-node/Vercel AI SDK use zod-like validation) parse every chunk. These are the documented, real-world failure modes a gateway must design around:

1. Usage chunk shape. With stream_options: {"include_usage": true}, OpenAI's contract is: every content chunk has "usage": null, and one final extra chunk before [DONE] has "choices": [] and the full usage object. Emitting usage with non-empty choices violates the spec (LiteLLM bug https://github.com/BerriAI/litellm/issues/28735); conversely, clients that stop at finish_reason miss the usage chunk (https://github.com/BerriAI/litellm/issues/25389); AutoGen crashed on the empty-choices chunk itself (https://github.com/microsoft/autogen/issues/5078); llama.cpp put usage in a slightly different chunk than OpenAI and broke clients (https://github.com/ggml-org/llama.cpp/issues/15443). Rule: emit usage exactly like OpenAI (separate final empty-choices chunk, only when requested), and when consuming, read to true end-of-stream and tolerate usage in any late chunk.
2. Streamed tool-call deltas. The first tool_calls delta must carry index, id, type: "function", and function.name; subsequent deltas carry only index + function.arguments fragments. Real breakage: Gemini-behind-a-compat-shim omitting index (https://github.com/anomalyco/opencode/issues/17902); providers sending function.name only in a later chunk (https://github.com/anomalyco/opencode/issues/24137, https://github.com/anomalyco/opencode/issues/26412); vLLM omitting "type":"function" under forced tool_choice (https://github.com/vllm-project/vllm/issues/16340). Rule: the translator owns tool-call chunk assembly — always emit index/id/type/name complete in the first delta for each call, arguments-only after.
3. Role delta discipline. The first chunk of each choice must have delta: {"role": "assistant"} (optionally with content: ""); an empty-string role instead of "assistant" breaks strict parsers (https://github.com/anomalyco/opencode/issues/28427). Later deltas must omit role entirely rather than repeat it as "".
4. All-or-nothing chunk validation. Some client stacks silently drop an entire chunk if any field fails validation — provider quirks then surface as silently missing content, which is undebuggable (https://github.com/Effect-TS/effect-smol/issues/2337, https://github.com/pydantic/pydantic-ai/issues/3658 — OpenRouter reasoning_details variant missing a field broke pydantic-ai). Rule: every field Zyquo Router emits must be exactly typed (created as integer epoch seconds, object exactly "chat.completion.chunk", index present on every choice/tool_call); when adding fields (e.g. reasoning_content), add only well-formed, consistently shaped ones. Also: null-valued token-detail fields inside usage broke the OpenAI Agents SDK (https://github.com/openai/openai-agents-python/issues/1179) — omit detail objects rather than sending them with null members.
5. max_tokens vs max_completion_tokens. OpenAI deprecated max_tokens in favor of max_completion_tokens; o-series/reasoning models hard-reject max_tokens ("Unsupported parameter"). Every ecosystem project had to patch (Ollama https://github.com/ollama/ollama/issues/7125, simonw/llm https://github.com/simonw/llm/issues/724, Home Assistant https://github.com/home-assistant/core/issues/137039, Spring AI https://github.com/spring-projects/spring-ai/issues/3300). Rule: accept both on ingress, normalize internally to one limit value, emit whichever the upstream requires (per-provider table), never forward both.
6. system_fingerprint. Optional in practice — OpenAI itself returns null/absent for many models (https://github.com/openai/openai-python/issues/1038, https://github.com/openai/openai-openapi/issues/167), and SDK type defs treat it as optional (https://github.com/openai/openai-node/issues/443). Gateways may safely omit it or set null; do not fabricate values (clients use it for determinism tracking with seed).
7. n > 1. Most non-OpenAI upstreams don't support multiple choices (Ollama: unsupported; Anthropic/Gemini: no direct equivalent). Options: reject with a clear 400, or fan out N upstream calls. LiteLLM/Ollama precedent: reject or ignore. choices must still always be an array with correct index fields even for n=1 (OpenRouter: "choices is always an array" — https://openrouter.ai/docs/api-reference/overview).
8. Keep-alives, buffering, and timeouts. Long prefills (big prompts, reasoning models) can be silent for 30s+; intermediaries and client idle timeouts kill the connection (e.g. https://github.com/microsoft/agent-framework/issues/6941). SSE comment lines (: keep-alive) every 15–30s are the only spec-compatible heartbeat; also send Cache-Control: no-cache, Connection: keep-alive, X-Accel-Buffering: no, and never gzip SSE (compression layers buffer the stream). Conversely, when consuming, tolerate comment lines from upstreams (OpenRouter emits : OPENROUTER PROCESSING — https://openrouter.ai/docs/api-reference/streaming). Streaming responses need effectively unlimited write timeouts; non-streaming needs a generous but bounded upstream timeout mapped to 504/408.
9. CORS for browser clients. A cross-origin fetch to the router fails before the first byte without correct Access-Control-Allow-Origin + preflight handling for POST with Authorization/Content-Type: application/json headers. OpenRouter even classifies CORS problems under 400 (https://openrouter.ai/docs/api-reference/errors). Zyquo Router's default-permissive-on-localhost CORS (with OPTIONS preflight support) is the right call for browser-based dev tools.
10. Mid-stream failure surfacing. After the first chunk, the status line is committed (200). The only clean options are OpenRouter's error-chunk-with-finish_reason:"error" (§2.2.3) followed by [DONE], or an abrupt close (which strict SDKs report as a network error). Emit the error chunk.

# 2.5 Recommended pattern set for Zyquo Router

Synthesis of the above into the concrete policy for our gateway:

1. Model namespacing — provider/model-id (LiteLLM/OpenRouter convention), full catalog from Zyquo Cloud; also accept unambiguous bare upstream IDs (resolve via catalog; ambiguous → 400 listing candidates) and user aliases (LiteLLM model_group_alias pattern). GET /v1/models returns OpenAI list shape with id = namespaced ID, enriched per-model metadata (context_length, pricing as decimal strings, capability flags, supported_parameters) under an x-zyquo key, following OpenRouter's metadata precedent.
2. Unknown/unsupported-param policy — data-driven per-provider tables (LiteLLM style): rename where names differ (max_tokens/max_completion_tokens, stop→stop_sequences), strip silently what the upstream would reject (default drop_params: true behavior, log at debug in the request inspector), reject with a helpful OpenAI-format 400 only when silently dropping would change semantics materially (e.g. tools on a no-tools model, n>1), and pass through unknown extra body keys to OpenAI-compatible upstreams (the extra_body idiom) — documented per provider in docs/API.md.
3. finish_reason & usage normalization — normalize finish_reason to stop | length | tool_calls | content_filter (+ error mid-stream only), preserving the raw upstream value as native_finish_reason (OpenRouter pattern). Usage: real upstream numbers when available (request usage-in-stream from upstreams that support it, per-provider flag); tokenizer-estimated otherwise, flagged (e.g. "x-zyquo": {"usage_estimated": true}); cost computed from the catalog's pricing at response time and exposed OpenRouter-style as extra usage fields. Streaming usage emitted only when the client sends stream_options.include_usage, as a final "choices": [] chunk before [DONE] — byte-exact per §2.4.1–4.
4. Upstream error surfacing — map to OpenAI error JSON {"error":{"message","type","param","code"}} with LiteLLM's status taxonomy (401 provider-key invalid naming the provider, 429 with propagated Retry-After, 400 subtypes for context-window/content-policy, 502 "upstream returned an invalid response", 503 provider unavailable, 504/408 timeouts); include a canonical machine code plus the upstream's original code in the message (OpenRouter's error_type+provider_code idea) — never raw upstream payloads or key material. Mid-stream: OpenRouter-style error chunk with finish_reason: "error", then [DONE].
5. Retry/fallback policy — per-error-class retry policy (LiteLLM RetryPolicy): retries with exponential backoff + jitter on 429/5xx/timeouts (honoring Retry-After), zero retries on 400/401/403; then user-configured fallback chains (ordered model lists) triggered on retry exhaustion, context-window and content-policy errors; the response model field reports the model actually used and cost follows it (OpenRouter contract). Optional per-provider cooldown state feeding the dashboard's "degraded" indicator.
6. Health & liveness — GET /health (status, version, uptime, active streams) never touches upstreams; per-provider "Test key" in the UI does a minimal authenticated upstream call and reports latency; SSE keep-alive comments every ~20s of upstream silence; client disconnect cancels the upstream task immediately (guaranteed, since we own the upstream HTTP call).

Primary sources: https://docs.litellm.ai/docs/proxy/configs · https://docs.litellm.ai/docs/completion/drop_params · https://docs.litellm.ai/docs/exception_mapping · https://docs.litellm.ai/docs/routing · https://docs.litellm.ai/docs/proxy/reliability · https://docs.litellm.ai/docs/completion/token_usage · https://docs.litellm.ai/docs/completion/stream · https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json · https://openrouter.ai/docs/api-reference/streaming · https://openrouter.ai/docs/api-reference/errors · https://openrouter.ai/docs/api-reference/overview · https://openrouter.ai/docs/use-cases/usage-accounting · https://openrouter.ai/docs/guides/routing/model-fallbacks · https://openrouter.ai/docs/features/provider-routing · https://openrouter.ai/docs/models · https://docs.ollama.com/api/openai-compatibility · https://lmstudio.ai/docs/app/api/endpoints/openai · https://docs.vllm.ai/en/latest/serving/online_serving/ · GitHub issues linked inline (§2.4).

# 3. Translation matrices

Research date: 2026-07-30. Verified against live official documentation (URLs cited inline). This section is the contract for Translate/AnthropicTranslator.swift, Translate/GeminiTranslator.swift, and Translate/CompatAdjuster.swift. The router's canonical internal format IS the OpenAI chat/completions wire format (Section 1); every non-OpenAI upstream is mapped bidirectionally onto it.

# 3.1 Anthropic Messages API ⇄ OpenAI chat/completions

Primary sources:

• Messages API reference: https://platform.claude.com/docs/en/api/messages (docs.anthropic.com 301-redirects here)
• Streaming: https://platform.claude.com/docs/en/docs/build-with-claude/streaming
• Tool use: https://platform.claude.com/docs/en/docs/agents-and-tools/tool-use/overview
• Errors: https://platform.claude.com/docs/en/api/errors

# 3.1.1 Endpoint & auth

The anthropic-version header is mandatory; requests without it are rejected. Pin 2023-06-01 (the stable version used by all official SDKs).

# 3.1.2 Request translation (OpenAI → Anthropic)

# Parameter map

# Message-shape rules (the tricky part)

Anthropic enforces constraints that OpenAI does not:

1. Roles are only user and assistant inside messages.
2. Turns must alternate. Consecutive same-role messages must be merged into a single message whose content is an array of blocks, preserving order.
3. The first message must be user. If the client's first non-system message is assistant, prepend a placeholder user message (e.g. a single text block "(continue)") or reject — the router merges/prepends (LiteLLM's approach) so real clients keep working.
4. OpenAI tool role messages → user messages containing tool_result blocks. Consecutive tool messages (parallel tool results) merge into ONE user message with multiple tool_result blocks. tool_result blocks must come FIRST in that user message's content array if user text follows.
5. Assistant tool_calls → tool_use content blocks, after any text content, with input as a parsed JSON object (OpenAI arguments is a JSON string — parse it; if unparseable, send {} and log).

# Content-part map

# Side-by-side request example (tools + image + system)

OpenAI request received by the router:

{
  "model": "anthropic/claude-sonnet-4-5",
  "max_tokens": 1024,
  "temperature": 1.4,
  "messages": [
    {"role": "system", "content": "You are terse."},
    {"role": "user", "content": [
      {"type": "text", "text": "What's in this image, and what's the weather there?"},
      {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ..."}}
    ]},
    {"role": "assistant", "content": null, "tool_calls": [
      {"id": "call_abc123", "type": "function",
       "function": {"name": "get_weather", "arguments": "{\"location\": \"Paris\"}"}}
    ]},
    {"role": "tool", "tool_call_id": "call_abc123", "content": "18°C, sunny"}
  ],
  "tools": [
    {"type": "function", "function": {
      "name": "get_weather",
      "description": "Get current weather",
      "parameters": {"type": "object",
        "properties": {"location": {"type": "string"}}, "required": ["location"]}}}
  ],
  "tool_choice": "auto",
  "parallel_tool_calls": false
}

Anthropic request the router sends upstream:

{
  "model": "claude-sonnet-4-5",
  "max_tokens": 1024,
  "temperature": 1.0,
  "system": "You are terse.",
  "messages": [
    {"role": "user", "content": [
      {"type": "text", "text": "What's in this image, and what's the weather there?"},
      {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": "/9j/4AAQ..."}}
    ]},
    {"role": "assistant", "content": [
      {"type": "tool_use", "id": "call_abc123", "name": "get_weather",
       "input": {"location": "Paris"}}
    ]},
    {"role": "user", "content": [
      {"type": "tool_result", "tool_use_id": "call_abc123", "content": "18°C, sunny"}
    ]}
  ],
  "tools": [
    {"name": "get_weather", "description": "Get current weather",
     "input_schema": {"type": "object",
       "properties": {"location": {"type": "string"}}, "required": ["location"]}}
  ],
  "tool_choice": {"type": "auto", "disable_parallel_tool_use": true}
}

Note the tool_use id is preserved verbatim in both directions so multi-turn tool loops round-trip.

# Tools & tool_choice map

# JSON mode / response_format strategy

Anthropic historically had no native JSON mode; the current API (2026) has structured output via output_config.format (source: https://platform.claude.com/docs/en/api/messages — output_config: { format: { type: "json_schema", schema: {...} } }).

Router strategy, in priority order:

1. response_format: {"type":"json_schema","json_schema":{"schema":S,...}} → output_config: {"format": {"type":"json_schema","schema": S}} on models that support it (catalog capability flag).
2. On models WITHOUT structured-output support — tool trick: define a single synthetic tool (name: "json_output", input_schema: S), force it with tool_choice: {"type":"tool","name":"json_output"}, and return the streamed/collected input object as the assistant content string (finish_reason stop, not tool_calls).
3. response_format: {"type":"json_object"} → append to system: "You must respond with valid JSON only, no prose, no markdown fences." and optionally prefill the assistant turn with { (append {"role":"assistant","content":"{"} and re-prepend { to the returned text). Prefill is Anthropic-sanctioned steering. Document in docs/API.md that json_object on Anthropic is best-effort.
4. {"type":"text"} → no-op.

# Extended thinking (request side)

• thinking: {"type":"enabled","budget_tokens":N} (N ≥ 1024, must be < max_tokens) or {"type":"adaptive"}; display: "summarized"|"omitted" controls whether thinking text is streamed.
• Newer models also take output_config.effort: "low"|"medium"|"high"|"xhigh"|"max".
• Router mapping for the standard OpenAI reasoning_effort param: low → output_config.effort "low" (or thinking budget 1024), medium → "medium" (8192), high → "high" (24576) — per-model capability gate from the catalog. Raw thinking extra-body always wins if provided.
• Multi-turn constraint: in tool-use loops with thinking enabled, Anthropic expects prior thinking blocks (with signature) to be passed back. See §3.4 for how the router preserves signatures via reasoning_details.

# 3.1.3 Response translation (Anthropic → OpenAI), non-streaming

Anthropic response:

{
  "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
  "type": "message",
  "role": "assistant",
  "model": "claude-sonnet-4-5",
  "content": [
    {"type": "text", "text": "It's 18°C and sunny in Paris."},
    {"type": "tool_use", "id": "toolu_01A09q90qw90lq917835lq9",
     "name": "get_weather", "input": {"location": "Paris"}}
  ],
  "stop_reason": "tool_use",
  "stop_sequence": null,
  "usage": {
    "input_tokens": 412, "output_tokens": 61,
    "cache_creation_input_tokens": 0, "cache_read_input_tokens": 128
  }
}

Router emits:

{
  "id": "chatcmpl-9f3c1a2b7d4e",
  "object": "chat.completion",
  "created": 1753872000,
  "model": "anthropic/claude-sonnet-4-5",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "It's 18°C and sunny in Paris.",
      "tool_calls": [{
        "id": "toolu_01A09q90qw90lq917835lq9",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"location\":\"Paris\"}"}
      }]
    },
    "finish_reason": "tool_calls"
  }],
  "usage": {
    "prompt_tokens": 540,
    "completion_tokens": 61,
    "total_tokens": 601,
    "prompt_tokens_details": {"cached_tokens": 128}
  }
}

Rules:

• id: generate a fresh chatcmpl-<hex> (keep the upstream msg_… id in the request log for tracing). object: "chat.completion", created: gateway clock (Unix seconds). model: echo the namespaced router id.
• Content blocks → concatenate all text block texts into message.content (null if none and tool_calls exist); each tool_use block → one tool_calls[] entry with arguments = JSON.stringify(input); thinking blocks → message.reasoning_content (§3.4), signature → reasoning_details; redacted_thinking → reasoning_details only (opaque data).
• stop_reason → finish_reason:

• usage: Anthropic's input_tokens EXCLUDES cache reads/writes. Normalize: prompt_tokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokens; completion_tokens = output_tokens (already includes thinking tokens); total_tokens = prompt + completion; prompt_tokens_details.cached_tokens = cache_read_input_tokens; completion_tokens_details.reasoning_tokens = usage.output_tokens_details.thinking_tokens when present.

# 3.1.4 SSE event model → OpenAI chat.completion.chunk (streaming)

Anthropic stream grammar (source: https://platform.claude.com/docs/en/docs/build-with-claude/streaming):

message_start
( content_block_start → content_block_delta* → content_block_stop )*
message_delta+
message_stop

with ping events anywhere and possible error events. Each SSE frame is event: <name>\ndata: <json>\n\n. message_delta.usage.output_tokens is cumulative.

Delta types inside content_block_delta: text_delta (.text), input_json_delta (.partial_json, a partial JSON string for tool_use.input), thinking_delta (.thinking), signature_delta (.signature, arrives just before the thinking block's content_block_stop).

# Event → chunk mapping table

The translator keeps two counters: toolIdx = number of tool_use blocks seen so far (this is the OpenAI tool_calls[].index, 0-based, independent of the Anthropic block index), and cumulative usage.

Every emitted chunk carries the constant envelope: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":C,"model":"anthropic/…","choices":[{"index":0,"delta":…,"finish_reason":…}]} — same id/created for the whole stream.

# Full example transcript (tool-use stream)

Anthropic events (left) → OpenAI chunks emitted by the router (right). Envelope fields elided for readability; every right-hand line is a full chat.completion.chunk.

ANTHROPIC UPSTREAM                                          → ZYQUO ROUTER EMITS (OpenAI SSE)

event: message_start
data: {"type":"message_start","message":{"id":"msg_014p",  → data: {"id":"chatcmpl-a1","object":"chat.completion.chunk",
  "role":"assistant","content":[],"model":"claude-…",          "created":1753872000,"model":"anthropic/claude-sonnet-4-5",
  "usage":{"input_tokens":472,"output_tokens":2}, …}}          "choices":[{"index":0,"delta":{"role":"assistant",
                                                               "content":""},"finish_reason":null}]}

event: content_block_start
data: {"type":"content_block_start","index":0,             → (nothing)
  "content_block":{"type":"text","text":""}}

event: ping
data: {"type":"ping"}                                      → (nothing, or ": keep-alive" comment)

event: content_block_delta
data: {…,"delta":{"type":"text_delta","text":"Okay,"}}     → data: {…,"choices":[{"index":0,"delta":{"content":"Okay,"},
                                                               "finish_reason":null}]}

event: content_block_delta
data: {…,"delta":{"type":"text_delta",                     → data: {…,"choices":[{"index":0,"delta":{"content":
  "text":" checking the weather:"}}                            " checking the weather:"},"finish_reason":null}]}

event: content_block_stop
data: {"type":"content_block_stop","index":0}              → (nothing)

event: content_block_start
data: {"type":"content_block_start","index":1,             → data: {…,"choices":[{"index":0,"delta":{"tool_calls":[
  "content_block":{"type":"tool_use",                          {"index":0,"id":"toolu_01T1x","type":"function",
  "id":"toolu_01T1x","name":"get_weather","input":{}}}         "function":{"name":"get_weather","arguments":""}}]},
                                                               "finish_reason":null}]}

event: content_block_delta
data: {…,"delta":{"type":"input_json_delta",               → (skipped — empty partial_json)
  "partial_json":""}}

event: content_block_delta
data: {…,"delta":{"type":"input_json_delta",               → data: {…,"choices":[{"index":0,"delta":{"tool_calls":[
  "partial_json":"{\"location\":"}}                            {"index":0,"function":{"arguments":"{\"location\":"}}]},
                                                               "finish_reason":null}]}

event: content_block_delta
data: {…,"delta":{"type":"input_json_delta",               → data: {…,"choices":[{"index":0,"delta":{"tool_calls":[
  "partial_json":" \"San Francisco, CA\"}"}}                   {"index":0,"function":{"arguments":
                                                               " \"San Francisco, CA\"}"}}]},"finish_reason":null}]}

event: content_block_stop
data: {"type":"content_block_stop","index":1}              → (nothing; toolIdx→1)

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":      → data: {…,"choices":[{"index":0,"delta":{},
  "tool_use","stop_sequence":null},                            "finish_reason":"tool_calls"}]}
  "usage":{"output_tokens":89}}

event: message_stop                                        → data: {…,"choices":[],"usage":{"prompt_tokens":472,
data: {"type":"message_stop"}                                  "completion_tokens":89,"total_tokens":561}}
                                                               (only if stream_options.include_usage)
                                                           → data: [DONE]

Thinking streams follow the same pattern: content_block_start {type:"thinking"} opens nothing, each thinking_delta → {"delta":{"reasoning_content":"…"}}, signature_delta → reasoning_details (or dropped), then the text block streams as normal content deltas.

# 3.2 Gemini generateContent / streamGenerateContent ⇄ OpenAI

Primary sources:

• API reference: https://ai.google.dev/api/generate-content
• Part/Content schema: https://ai.google.dev/api/caching#Part
• Function calling: https://ai.google.dev/gemini-api/docs/function-calling
• Structured output: https://ai.google.dev/gemini-api/docs/structured-output
• Thinking: https://ai.google.dev/gemini-api/docs/thinking
• Google's own OpenAI-compat layer (used as a mapping oracle): https://ai.google.dev/gemini-api/docs/openai

# 3.2.1 Endpoint & auth

Always use ?alt=sse — without it, streamGenerateContent returns a chunked JSON array, not SSE. The model name is in the path, not the body. Use the header for auth so the key never appears in URLs/logs.

# 3.2.2 Request translation (OpenAI → Gemini)

# Content-part map

# Tool round-trip (functionCall / functionResponse)

OpenAI assistant tool_calls → Gemini model turn:

{"role": "model", "parts": [
  {"functionCall": {"id": "call_abc123", "name": "get_weather",
    "args": {"location": "Paris"}}}
]}

(args is a parsed object — parse the OpenAI arguments string. functionCall.id / functionResponse.id exist in current v1beta for parallel-call matching; include them when the OpenAI ids are available.)

OpenAI tool message → Gemini user turn:

{"role": "user", "parts": [
  {"functionResponse": {"id": "call_abc123", "name": "get_weather",
    "response": {"result": "18°C, sunny"}}}
]}

Two traps:

1. functionResponse.response must be a JSON OBJECT. OpenAI tool content is a string → if it parses as a JSON object, pass it; otherwise wrap as {"result": <string>}.
2. name is required, but OpenAI tool messages carry only tool_call_id. The router must resolve tool_call_id → name from the preceding assistant message's tool_calls in the same request payload (always available in a well-formed OpenAI conversation).
3. Thought signatures (Gemini 3+): function-call parts may carry a thoughtSignature that should be echoed back on the following turn. Preserve it via reasoning_details (§3.4) and re-attach when translating the conversation back.

# Side-by-side minimal request

// OpenAI in                                    // Gemini out
{                                               {
  "model": "gemini/gemini-2.5-flash",             // POST …/models/gemini-2.5-flash:generateContent
  "messages": [                                   "systemInstruction": {"parts":[{"text":"Be brief."}]},
    {"role":"system","content":"Be brief."},      "contents": [
    {"role":"user","content":"Hi"},                 {"role":"user","parts":[{"text":"Hi"}]},
    {"role":"assistant","content":"Hello!"},        {"role":"model","parts":[{"text":"Hello!"}]},
    {"role":"user","content":"Name a color"}        {"role":"user","parts":[{"text":"Name a color"}]}
  ],                                              ],
  "temperature": 0.7,                             "generationConfig": {
  "max_tokens": 100,                                "temperature": 0.7,
  "stop": ["\n\n"]                                  "maxOutputTokens": 100,
}                                                   "stopSequences": ["\n\n"]
                                                  }
                                                }

# 3.2.3 Response translation (Gemini → OpenAI)

Gemini response:

{
  "candidates": [{
    "content": {"role": "model", "parts": [
      {"functionCall": {"name": "get_weather", "args": {"location": "Paris"}}}
    ]},
    "finishReason": "STOP",
    "index": 0,
    "safetyRatings": [ … ]
  }],
  "usageMetadata": {
    "promptTokenCount": 57, "candidatesTokenCount": 12,
    "thoughtsTokenCount": 88, "totalTokenCount": 157
  },
  "modelVersion": "gemini-2.5-flash"
}

Router emits:

{
  "id": "chatcmpl-7be2f0c4",
  "object": "chat.completion",
  "created": 1753872000,
  "model": "gemini/gemini-2.5-flash",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": null,
      "tool_calls": [{"id": "call_9d1e2f3a", "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"location\":\"Paris\"}"}}]},
    "finish_reason": "tool_calls"
  }],
  "usage": {
    "prompt_tokens": 57,
    "completion_tokens": 100,
    "total_tokens": 157,
    "completion_tokens_details": {"reasoning_tokens": 88}
  }
}

Rules:

• Concatenate parts[].text (where part.thought != true) → content; parts with "thought": true → reasoning_content; each functionCall part → a tool_calls[] entry. Gemini often omits ids → synthesize call_<12-hex> (and remember name↔id for the return trip).
• CRITICAL finish_reason rule: Gemini reports finishReason: "STOP" even when the candidate contains functionCall parts. The router must emit finish_reason: "tool_calls" whenever any functionCall part is present, regardless of finishReason.

• Blocked prompts: if candidates is empty and promptFeedback.blockReason is set (SAFETY, BLOCKLIST, PROHIBITED_CONTENT, OTHER, IMAGE_SAFETY), return an OpenAI 400 invalid_request_error with a clear message naming the block reason — never an empty 200.
• usage: prompt_tokens = promptTokenCount; completion_tokens = candidatesTokenCount + thoughtsTokenCount (OpenAI counts reasoning inside completion tokens); total_tokens = totalTokenCount; completion_tokens_details.reasoning_tokens = thoughtsTokenCount; prompt_tokens_details.cached_tokens = cachedContentTokenCount.

# 3.2.4 Streaming (streamGenerateContent?alt=sse) → OpenAI chunks

Each SSE data: line is a complete GenerateContentResponse whose candidates[0].content.parts holds the increment — there is no delta envelope and no [DONE] terminator (the stream simply ends after the chunk carrying the final finishReason). usageMetadata appears on chunks with cumulative counts; the last chunk has the authoritative totals.

GEMINI SSE                                                 → ROUTER EMITS

data: {"candidates":[{"content":{"parts":[{"text":"The"}], → chunk 1 (synthesized role first):
  "role":"model"},"index":0}],                               data: {…,"delta":{"role":"assistant","content":""},…}
  "usageMetadata":{…},"modelVersion":"gemini-2.5-flash"}     data: {…,"delta":{"content":"The"},"finish_reason":null}

data: {"candidates":[{"content":{"parts":[{"text":          → data: {…,"delta":{"content":" sky is blue."},
  " sky is blue."}],"role":"model"},"index":0}],…}             "finish_reason":null}

data: {"candidates":[{"content":{"parts":[],"role":        → data: {…,"delta":{},"finish_reason":"stop"}
  "model"},"finishReason":"STOP","index":0}],              → data: {…,"choices":[],"usage":{"prompt_tokens":8,
  "usageMetadata":{"promptTokenCount":8,                       "completion_tokens":5,"total_tokens":13}}
  "candidatesTokenCount":5,"totalTokenCount":13}}              (if include_usage)
(stream ends — no [DONE] from Gemini)                      → data: [DONE]   (router ALWAYS adds it)

Rules:

• Synthesize the role chunk: Gemini has no role-only first frame; the router emits {"role":"assistant","content":""} before the first content delta.
• Tool calls are NOT argument-streamed: a functionCall part arrives complete in one chunk → emit ONE tool_calls delta containing index, generated id, name, and the FULL arguments string; strict SDKs accept whole-argument single deltas fine.
• Thought parts ("thought": true) → reasoning_content deltas.
• finishReason on the final chunk → the finish_reason chunk (apply the functionCall→tool_calls override).
• Router appends the OpenAI usage chunk (empty choices) and data: [DONE] itself.
• If Gemini aborts mid-stream with finishReason: SAFETY, emit finish_reason: "content_filter" and terminate normally.

# 3.3 OpenAI-compatible providers — deviation table

All nine below speak the OpenAI chat/completions wire format closely enough for near-pass-through: the router's CompatAdjuster only needs a per-provider strip/rename/allow table plus finish/usage normalization. Auth is Authorization: Bearer <key> for all of them. (OpenAI itself, api.openai.com/v1, is the reference and needs no adjustment.)

# Summary matrix

Sources: xAI — https://docs.x.ai/developers/model-capabilities/legacy/chat-completions, https://www.promptfoo.dev/docs/providers/xai/ ; Mistral — https://docs.mistral.ai/api/ , https://docs.mistral.ai/capabilities/reasoning/ ; DashScope — https://www.alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope , https://www.alibabacloud.com/help/en/model-studio/deep-thinking ; DeepSeek — https://api-docs.deepseek.com/guides/thinking_mode ; Kimi — https://platform.kimi.ai/docs/api/chat (platform.moonshot.ai redirects here; API host remains api.moonshot.ai) ; Perplexity — https://docs.perplexity.ai/api-reference/chat-completions-post , https://docs.perplexity.ai/getting-started/quickstart ; Together — https://docs.together.ai/reference/chat-completions-1 ; DeepInfra — https://docs.deepinfra.com/chat/overview ; Cerebras — https://inference-docs.cerebras.ai/api-reference/chat-completions .

# Universal CompatAdjuster rules

1. Unknown-parameter resilience: providers split between ignore-unknown (DeepSeek, Together) and 400-on-unknown (xAI reasoning models, Moonshot on bad ranges). The per-provider strip table is authoritative; additionally, on a 400 whose message names a parameter, retry ONCE with that parameter removed, then surface the error.
2. max_tokens vs max_completion_tokens: accept both from clients; send whichever the provider documents (max_completion_tokens for Cerebras/Moonshot-K-series; max_tokens elsewhere). Never send both.
3. Streaming usage: request stream_options: {"include_usage": true} upstream wherever supported (DashScope, Moonshot, DeepSeek, Together, DeepInfra, Cerebras, xAI, Mistral); when the upstream can't provide usage, estimate tokens locally and flag "x-zyquo": {"usage_estimated": true}.
4. finish_reason normalization set: everything must land in stop | length | tool_calls | content_filter; map eos→stop, provider-specific refusal/safety values → content_filter, anything unknown → stop + log.
5. Keep-alive noise: some upstreams emit SSE comment lines (: keep-alive) or ping-ish frames — the SSE parser must skip comment lines and blank frames without erroring, and the router's own SSEWriter may emit comments to keep client sockets alive during long thinking phases.

# 3.4 Reasoning content — normalization decision

Decision: normalize on DeepSeek's reasoning_content convention — a sibling of content on both the non-streaming message and the streaming delta:

// non-streaming                              // streaming
"message": {                                  "delta": {
  "role": "assistant",                          "reasoning_content": "Let me check…"
  "reasoning_content": "Let me check…",       }
  "content": "The answer is 21."
}

Rationale: it is the oldest and most widely recognized wire convention (DeepSeek-R1 era) — Qwen/DashScope, Moonshot/Kimi, and xAI grok-3-mini already emit exactly this field, so most client tooling (chat UIs, LangChain, Continue, aider, etc.) knows to render it and to NOT confuse it with content. OpenRouter's richer reasoning + reasoning_details model (https://openrouter.ai/docs/use-cases/reasoning-tokens) is adopted only as a supplement: the router additionally emits reasoning_details (array of provider-native structured blocks) when signatures/opacity must round-trip.

# Per-provider mapping into reasoning_content

# Usage normalization

All reasoning token counts land in the OpenAI-standard usage.completion_tokens_details.reasoning_tokens (Anthropic thinking_tokens, Gemini thoughtsTokenCount, xAI/Perplexity reasoning_tokens), and reasoning tokens are INCLUDED in completion_tokens (OpenAI semantics).

# Request-side control

The router accepts the OpenAI-standard reasoning_effort ("minimal"|"low"|"medium"|"high", plus provider extras like "none"/"max") and translates per provider: Anthropic → output_config.effort / thinking.budget_tokens; Gemini → thinkingConfig.thinkingBudget/thinkingLevel (Google's own compat mapping: low=1024, medium=8192, high=24576 on 2.5-series); DeepSeek/Kimi/Cerebras/xAI/Perplexity → pass reasoning_effort through (gated by catalog capability); Qwen → enable_thinking:true (+ thinking_budget); Mistral → prompt_mode:"reasoning". On models with no reasoning capability, reasoning_effort is stripped (never 400 the client for asking).

# Echo-back rules (multi-turn)

Incoming assistant messages may contain reasoning_content/reasoning_details from prior router responses. Before forwarding:

• Strip reasoning_content for all providers by default (DeepSeek 400s in plain turns if it leaks into context via unknown-field-strict paths; others ignore it but it wastes tokens),
• except: DeepSeek tool-call loops (must be passed back per current docs), Anthropic thinking+tools loops (reconstruct thinking blocks with signatures from reasoning_details), Gemini 3 (re-attach thoughtSignature), Mistral Magistral (replay ThinkChunk to preserve the trace), Kimi with thinking.keep. This asymmetry is exactly why reasoning_details exists: it carries the provider-native, signed material that some upstreams demand back, while reasoning_content stays a clean display string.

# 3.5 Source index

• Anthropic Messages API: https://platform.claude.com/docs/en/api/messages
• Anthropic streaming events: https://platform.claude.com/docs/en/docs/build-with-claude/streaming
• Anthropic errors: https://platform.claude.com/docs/en/api/errors
• Gemini generateContent reference: https://ai.google.dev/api/generate-content
• Gemini Part/Content schema: https://ai.google.dev/api/caching#Part
• Gemini function calling: https://ai.google.dev/gemini-api/docs/function-calling
• Gemini OpenAI-compat layer (mapping oracle): https://ai.google.dev/gemini-api/docs/openai
• xAI chat completions: https://docs.x.ai/developers/model-capabilities/legacy/chat-completions ; parameter-rejection field notes: https://www.promptfoo.dev/docs/providers/xai/ , https://github.com/vercel/ai/issues/12826
• Mistral API: https://docs.mistral.ai/api/ ; reasoning: https://docs.mistral.ai/capabilities/reasoning/
• DashScope OpenAI compat: https://www.alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope ; deep thinking: https://www.alibabacloud.com/help/en/model-studio/deep-thinking
• DeepSeek thinking mode: https://api-docs.deepseek.com/guides/thinking_mode
• Kimi/Moonshot chat API: https://platform.kimi.ai/docs/api/chat
• Perplexity chat completions: https://docs.perplexity.ai/api-reference/chat-completions-post ; quickstart: https://docs.perplexity.ai/getting-started/quickstart
• Together chat completions: https://docs.together.ai/reference/chat-completions-1
• DeepInfra OpenAI API: https://docs.deepinfra.com/chat/overview
• Cerebras chat completions: https://inference-docs.cerebras.ai/api-reference/chat-completions
• OpenRouter reasoning normalization (prior art): https://openrouter.ai/docs/use-cases/reasoning-tokens

# 4. HTTP server in Swift without heavyweight deps

Research date: 2026-07-30. Target: an embedded HTTP/1.1 server inside a SwiftUI macOS 13+ app (SPM, no Xcode IDE) serving an OpenAI-compatible API on http://localhost:<port>, with spec-exact SSE streaming, long-lived streams, client-disconnect → upstream-cancellation, and graceful shutdown.

# 4.1 Options evaluated

# Option A — SwiftNIO directly (NIOCore + NIOPosix + NIOHTTP1 + NIOExtras)

• State as of mid-2026: swift-nio is at 2.101.3 (released ~2026-07-23), actively maintained by Apple, compatible with Swift 6.0–6.3 and strict concurrency. NIO 3 is expected "sometime around Swift 6" per the Swift.org server guidelines, with NIO 2 continuing to receive bug fixes afterwards — NIO 2.x is a safe multi-year foundation. Sources: swift-nio releases, Swift Package Index — swift-nio, Swift.org concurrency adoption guidelines.
• Structured concurrency: modern NIO exposes NIOAsyncChannel, which "abstracts the notion of a NIO Channel into something that can safely be used in a structured concurrency context". The recommended split: protocol-specific logic (HTTP parsing/encoding via configureHTTPServerPipeline) stays as ChannelHandlers; business logic consumes/produces via the NIOAsyncChannel inbound AsyncSequence / outbound writer. executeThenClose scopes the channel's lifetime to a closure — the channel closes when the closure returns, which maps perfectly onto "one inbound request = one cancellable Task". Sources: NIOAsyncChannel docs, NIO public async APIs, executeThenClose discussion, Using SwiftNIO — Channels, Building a web app with only SwiftNIO (2026).
• Reference server: Apple's NIOHTTP1Server example shows the canonical ServerBootstrap setup — backlog: 256, so_reuseaddr on server and child channels, configureHTTPServerPipeline(withErrorHandling: true), explicit keep-alive state machine (idle → waiting-for-body → sending-response) and adding Connection: close/keep-alive headers for HTTP/1.0 or explicit-close requests. (The example itself is future-based; we use the NIOAsyncChannel equivalent.) Source: NIOHTTP1Server main.swift.
• Graceful shutdown: swift-nio-extras ships ServerQuiescingHelper — "helps to quiesce a server by notifying user code when all previously open connections have closed"; call initiateShutdown(promise:) to stop accepting and drain. There is a full demo (HTTPServerWithQuiescingDemo). Sources: QuiescingHelper.swift, HTTPServerWithQuiescingDemo.
• Weight: NIOCore/NIOPosix/NIOHTTP1 (+ optionally NIOExtras) — one Apple-maintained dependency tree, no routing framework, no ServiceLifecycle/Logging/Metrics transitive stack. Full control over every byte of the SSE wire format.
• Cost: we write our own tiny router (we have ~5 routes), our own request-body accumulation with a size limit, and our own SSE writer. For this app that's a feature: the OpenAI chunk stream must be byte-exact, and owning SSEWriter end-to-end removes a framework abstraction between us and the wire.

# Option B — Network.framework (NWListener + NWProtocolFramer)

NWListener replaces the BSD bind/listen/accept sequence, but HTTP itself must be brought along: either a custom NWProtocolFramer (real boilerplate via NWProtocolFramerImplementation) or hand-wiring a C parser (http_parser.c) onto raw connections — Helge Heß's NWHTTPProtocol does exactly that and its author notes that "for production use it's suggested to not use a protocol framer for HTTP" and to hook the parser up directly instead. No HTTP/1.1 pipeline, no chunked-encoding helpers, no keep-alive management, no quiescing utilities — all DIY. It buys nothing over NIO for a localhost server (its strengths are Wi-Fi/cellular path handling and Bonjour, irrelevant here). Sources: NWHTTPProtocol, Intro to Network.framework servers, Apple Network framework docs. Rejected.

# Option C — Hummingbird 2

The strongest framework candidate. Built from scratch on SwiftNIO with Swift concurrency central; at 2.25.1 as of July 2026; Swift 6.1+; "designed to require the minimum number of dependencies". It has first-class pieces we'd otherwise hand-roll: a ResponseBodyWriter closure body with backpressure-aware await writer.write(...), a ServerSentEvent type, consumeWithInboundCloseHandler for client-disconnect detection, and graceful shutdown via swift-service-lifecycle ("currently running requests continue being handled, while new connections and requests will not be accepted"), started with Application.runService(gracefulShutdownSignals:). There is a dedicated server-sent-events example. Sources: hummingbird repo, SPI — hummingbird, What's new in Hummingbird 2, Hummingbird 2 announcement, SSE example, swift-service-lifecycle.

Why not choose it: (1) it drags in ServiceLifecycle/Logging/Metrics/Tracing abstractions designed for long-running server binaries with signal-driven lifecycles, whereas our lifecycle is owned by a SwiftUI app's Start/Stop button — bridging runService into an app-owned start/stop adds friction rather than removing it; (2) its router/middleware/extractor machinery is overhead for ~5 fixed routes; (3) an extra abstraction layer sits between us and the SSE bytes, and byte-exact OpenAI chunk emission is the core deliverable. Its SSE example is nonetheless the best public reference for the disconnect/shutdown patterns we will reimplement on raw NIO (see 4.2).

# Option D — Vapor

Batteries-included (HTTP/2, TLS, auth, validation, WebSockets, …) and correspondingly heavy: bulks the executable, increases compile time, ~20–30 MB idle memory vs Hummingbird's ~5–10 MB, and a large transitive dependency graph. Everything it adds over NIO is something this app doesn't need. Sources: Hummingbird vs Vapor discussion, Beginner's guide to Hummingbird. Rejected.

# Decision

SwiftNIO directly (NIOCore, NIOPosix, NIOHTTP1, plus NIOExtras for ServerQuiescingHelper), using the NIOAsyncChannel structured-concurrency APIs. Rationale: Apple-maintained, SPM-clean, one dependency tree, macOS 13+ fine, Swift 6 strict-concurrency ready, full control over SSE emission/flushing/backpressure, first-class quiescing for graceful shutdown, and a natural one-request-one-Task model so client disconnect cancels the upstream call structurally. Hummingbird 2 is the documented fallback if raw-NIO plumbing proves costlier than expected — the migration path is easy since both are NIO underneath.

# 4.2 Implementation specifics (SwiftNIO)

Bootstrap and binding.

let group = MultiThreadedEventLoopGroup.singleton
let quiesce = ServerQuiescingHelper(group: group)

let serverChannel = try await ServerBootstrap(group: group)
    .serverChannelOption(ChannelOptions.backlog, value: 256)
    .serverChannelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)
    .serverChannelInitializer { channel in
        channel.pipeline.addHandler(quiesce.makeServerChannelHandler(channel: channel))
    }
    .childChannelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)
    .bind(host: bindHost, port: port) { channel in
        channel.eventLoop.makeCompletedFuture {
            try channel.pipeline.syncOperations.configureHTTPServerPipeline(withErrorHandling: true)
            return try NIOAsyncChannel<HTTPServerRequestPart, HTTPServerResponsePart>(
                wrappingChannelSynchronously: channel)
        }
    }

• Bind host: "127.0.0.1" by default (localhost only, unreachable from the LAN); "0.0.0.0" only when the user explicitly enables LAN exposure (which per our security policy forces a local API key). This is a plain string parameter to bind(host:port:) — no extra API.
• Port-in-use: bind throws; catch IOError/NIOBSDSocket errors and check errno == EADDRINUSE (48 on Darwin) → surface "Port 8787 is already in use" and probe upward (try bind on port+1, +2, …) to suggest the next free port (never silently switch). EACCES (ports < 1024 without privileges) gets its own message. SO_REUSEADDR on the server channel avoids spurious EADDRINUSE from sockets lingering in TIME_WAIT after a quick Stop→Start; note it does not let two live listeners share a port — a genuinely occupied port still fails, which is what we want. Sources: ServerBootstrap docs, Bind: address already in use, NIOHTTP1Server example options.

Concurrent connections with structured concurrency. The async bind returns a NIOAsyncChannel of accepted-connection NIOAsyncChannels. The serving loop:

try await serverChannel.executeThenClose { acceptedConnections in
    try await withThrowingDiscardingTaskGroup { group in
        for try await connection in acceptedConnections {
            group.addTask { await handleConnection(connection) }   // one Task per connection
        }
    }
}

Inside handleConnection, connection.executeThenClose { inbound, outbound in ... } gives an AsyncSequence of HTTPServerRequestPart (.head, .body buffers, .end) and an outbound writer for HTTPServerResponsePart. Each request is parsed, dispatched to Routes, and answered; the loop iterates for keep-alive. Cancelling the connection's Task tears everything down cleanly — this is the backbone of both client-disconnect handling and graceful shutdown. Sources: NIO public async APIs, NIOAsyncChannel docs.

Request body size limit. Accumulate .body parts into a ByteBuffer with a hard cap (default e.g. 20 MiB — base64 images are large; user-configurable in Settings). On overflow: respond 413 in OpenAI error format, drain remaining parts (or close), never buffer further.

Timeouts.

• Header/idle timeout: add IdleStateHandler (NIOCore) ahead of the HTTP handlers, or a per-request withTimeout wrapper, to kill connections that never complete a request (~30 s read idle).
• Streaming: once a response stream has started, the idle clock must apply to write progress, not total duration — chat completions can legitimately stream for many minutes. Practical policy: generous upstream time-to-first-byte timeout (e.g. 120 s, configurable), then no total cap while chunks keep flowing; abort if the upstream stalls (no chunk for N seconds, e.g. 300 s).
• Remove/suspend the idle handler for the duration of an SSE response, restore for keep-alive reuse.

Keep-alive. configureHTTPServerPipeline parses Connection headers; our responder mirrors the NIOHTTP1Server example: honor keepAlive from the request head, set Connection: keep-alive/close explicitly for HTTP/1.0, close the channel after the response when keep-alive is false. After an SSE response we send the terminating [DONE] and .end; keeping the connection alive afterwards is legal (the response used chunked encoding with a proper terminator), but closing is also acceptable — OpenAI SDKs handle both. We keep it alive (SDKs reuse connections between calls). Source: NIOHTTP1Server main.swift.

SSE emission (SSEWriter). Response head:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
X-Accel-Buffering: no          ← harmless; defeats proxy buffering if any intermediary appears
Transfer-Encoding: chunked     ← added automatically by HTTPResponseEncoder when no Content-Length

Each OpenAI chunk is one SSE event: data: {json}\n\n (UTF-8; the OpenAI format is single-line JSON per data: line, terminated by data: [DONE]\n\n). Flush per event: with NIOAsyncChannel, every try await outbound.write(.body(.byteBuffer(eventBuffer))) is a writeAndFlush — each event leaves the process immediately; no coalescing layer may sit above it. await-ing each write is also the backpressure mechanism: a slow client suspends us, which suspends consumption of the upstream AsyncSequence, which propagates backpressure to the upstream HTTP read. Optionally set TCP_NODELAY on child channels so small event frames aren't Nagle-delayed. SSE format/caching rules: MDN — Using server-sent events. (Note: OpenAI's stream is consumed by SDK parsers, not browser EventSource, so event:/id:/retry: fields are never used — data: lines only.)

Client-disconnect → cancel upstream. Two complementary signals:

1. Structured: the write to a closed socket throws (NIOAsyncWriterError/channelInactive-driven); catching it in the request Task must cancel the upstream streaming Task — with structured concurrency this is automatic if the upstream call is a child task (withThrowingTaskGroup per request: one child consumes upstream and writes SSE; error/cancel of either cancels the other).
2. Proactive: watch the inbound side for EOF/half-closure while streaming — the pattern Hummingbird's SSE example uses (consumeWithInboundCloseHandler yielding a cancel event merged with the data stream). On raw NIO: run a second child task iterating inbound; when the sequence ends (client closed), cancel the group. This detects disconnects between writes, not only on the next failed write. Cancellation must propagate into the provider client (URLSession/AsyncHTTPClient task cancelled) so upstream token spend stops — Phase 7 verifies "no orphaned upstream usage". Sources: Hummingbird SSE example, NIOAsyncChannel docs.

CORS. Needed so browser-based tools can call the router. Implement in a small CORS.swift:

• Preflight: OPTIONS with Origin + Access-Control-Request-Method → 204 with Access-Control-Allow-Origin: * (default; configurable to a specific origin), Access-Control-Allow-Methods: GET, POST, OPTIONS, Access-Control-Allow-Headers: Authorization, Content-Type (or echo the requested headers), Access-Control-Max-Age: 600.
• Actual responses: add Access-Control-Allow-Origin: * (echo origin + Access-Control-Allow-Credentials: true only if credentials mode is ever needed — default permissive * for localhost tooling, per our spec).
• SSE responses need the CORS headers too (the stream is fetched cross-origin by browser clients).

Graceful shutdown. On Stop:

1. quiesce.initiateShutdown(promise:) — the ServerQuiescingHelper closes the listening channel (no new connections) and signals when all child channels have closed. Sources: QuiescingHelper.swift, HTTPServerWithQuiescingDemo.
2. Wait up to a drain deadline (e.g. 10 s) for in-flight non-streaming requests to finish.
3. Long-lived SSE streams won't drain on their own: after the deadline (or immediately if the user chose "stop now"), cancel the connection task group — cancellation unwinds each request task, cancels upstream calls, and closes channels.
4. Do not shutdownGracefully() the singleton EventLoopGroup (it's shared and Start may be pressed again); just release the server channel. Semantics mirror Hummingbird/ServiceLifecycle: "currently running requests continue being handled, while new connections and requests will not be accepted" (swift-service-lifecycle, What's new in Hummingbird 2).

# 4.3 macOS specifics

• Local network privacy prompt. Since macOS 15 Sequoia there is an iOS-style Local Network permission (System Settings → Privacy & Security → Local Network). Binding and serving on 127.0.0.1 does not involve the local network and triggers nothing. Interacting with LAN peers can trigger the prompt; users can later toggle it, and there are known Sequoia bugs where access silently breaks after a restart until re-toggled. When the user enables LAN mode (0.0.0.0) the app must explain the prompt and the Settings toggle in the UI. Also note the responsible-process rules: tools run from Terminal inherit Terminal's exemption; root daemons are auto-granted. Sources: Apple forums — local network privacy on Sequoia, mjtsai — Local Network Privacy on Sequoia, Foldr — macOS 15 local network privacy, Panic — granting local network access, access lost after restart.
• App Sandbox vs Developer ID. com.apple.security.network.server ("whether your app may listen for incoming network connections") is an App Sandbox entitlement; a non-sandboxed, Hardened-Runtime Developer ID app needs no entitlement to listen on a socket — "the sandbox was designed mainly for the App Store, while the hardened runtime was designed mainly for Developer ID". Decision (consistent with Phase 8 spec): ship non-sandboxed Developer ID + Hardened Runtime; document that if we ever adopt the sandbox we must add com.apple.security.network.server and .client. Sources: Apple — com.apple.security.network.server, Hardened Runtime and Sandboxing (lapcatsoftware).
• Terminal vs Finder launch. From Finder the app gets the standard GUI context. From Terminal (during make dev), the Terminal process is the responsible process for TCC purposes, so privacy prompts may attribute to Terminal; also a bare executable launched from Terminal needs explicit NSApplication activation to come frontmost (already planned in Phase 1's entry point). Environment differs too (Terminal's shell env vs launchd's user session) — never rely on env vars for configuration of the shipped app. Source: Apple forums — CLI tools and local network privacy.

# 5. Gateway concerns

# 5.1 Request logging with redaction

Pattern (validated against LiteLLM's proxy design):

• Always log metadata: timestamp, method/path, resolved model + provider, actually-used model (post-fallback), status, latency breakdown (queue → upstream TTFB → stream duration), tokens in/out, cost, local-key ID (its name/ID — never the token), stream flag, error class. This is safe and powers the whole dashboard.
• Bodies are opt-in. LiteLLM's turn_off_message_logging redacts messages/prompt and choices[].message.content (and reasoning_content) while still tracking spend; its bug tracker shows the classic failure mode — one storage path redacted, another (raw proxy_server_request) not. Lesson: redact at the single ingestion point of RequestLogStore, not per-sink. Store "[redacted]" placeholders (plus sizes/counts, e.g. message count, image count) unless the per-session "reveal bodies" switch is on. Sources: LiteLLM logging docs, redaction bug #16336, message redaction overview.
• Keys never touch the log path: strip/replace Authorization, x-api-key, api-key, and any *key*/*token* header before a request object is handed to the logger; scrub upstream error bodies (some providers echo the offending header) through the same filter. Verification (Phase 7): grep all logs/exports for known key material.
• Ring buffer in memory (e.g. last 1–5k entries) + persisted store with retention setting; export honors the current redaction state.

# 5.2 Token usage extraction per provider

• Prefer upstream-reported usage. Non-streaming: OpenAI-compatible providers return usage; Anthropic returns usage.input_tokens/output_tokens (in message_start + message_delta when streaming); Gemini returns usageMetadata. Streaming with OpenAI-compatible upstreams: request stream_options: {"include_usage": true} upstream where supported so a final usage chunk arrives; some compatible providers (per-provider quirk table from Section 3) send usage on the last chunk regardless, others never do.
• Estimate only when the upstream gives nothing, and flag it — e.g. "usage": {..., "x_zyquo_estimated": true} (or an x-zyquo extension block) so cost tiles can render "≈". Estimation options in Swift: aespinilla/Tiktoken (pure-Swift tiktoken: cl100k_base etc.) and narner/TiktokenSwift (UniFFI bindings to the real tiktoken, incl. o200k_base). A dependency-free fallback — chars/4 or words × 4/3 — is acceptable for the flagged-estimate path given tokenizers differ per provider anyway; decide in Phase 3 whether pulling a tokenizer dep is worth it (recommendation: start with the heuristic, keep the field flagged, add Tiktoken later if users need tight estimates).
• Cached-token counts: OpenAI reports usage.prompt_tokens_details.cached_tokens; Anthropic reports cache_read_input_tokens/cache_creation_input_tokens. Preserve these in the normalized usage when present (they change cost — see 5.3).

# 5.3 Cost calculation

• Source of truth: the ported Zyquo Cloud model catalog's per-model pricing (USD per 1M input tokens / per 1M output tokens). cost = prompt_tokens × in_price/1e6 + completion_tokens × out_price/1e6.
• Cached input tokens are billed at a discount where reported (e.g. OpenAI cached input typically 50–90% off; Anthropic cache reads at 0.1× base input, cache writes at 1.25×): when the catalog has cached pricing and the upstream reports cached counts, split: (prompt − cached) × in_price + cached × cached_price. Where the catalog lacks a cached rate, fall back to full input price (over-estimate, never under).
• Reasoning tokens (OpenAI completion_tokens_details.reasoning_tokens) are already included in completion_tokens — do not double-count.
• Mark costs derived from estimated usage as estimated. Store per-request cost in UsageRecord; aggregate per key/model/provider/day for the dashboard. Settings allow a pricing override table (providers reprice frequently) and a display currency (store USD, convert for display only).
• Precedent: OpenRouter prices "using the model that was ultimately used, which will be returned in the model attribute of the response body" — cost must always be computed against the actually-used model after fallbacks. Source: OpenRouter model fallbacks.

# 5.4 Rate limiting per local API key

• Algorithm: token bucket, the production default (AWS, Stripe) because "real traffic is bursty" — it allows short bursts up to bucket capacity while enforcing an average rate; sliding-window counters give smoother limits but punish legitimate bursts. For a local single-process gateway, in-memory is all we need (no distributed store). Sources: Arcjet — rate limiting algorithms, token bucket vs sliding window, APISIX — gateway rate limiting.
• Implementation: one actor RateLimiter holding keyID → (tokens: Double, lastRefill: ContinuousClock.Instant). Lazy refill on each check: tokens = min(capacity, tokens + elapsed × rate); admit iff tokens ≥ 1 then decrement. Per-key config: requests/min (rate = rpm/60, capacity ≈ rpm burst allowance). No timers, O(1) per request, trivially Sendable.
• On limit: 429 in OpenAI error format (type: "rate_limit_error"-style) with Retry-After: ceil((1 − tokens)/rate) seconds. Count rejections in per-key stats (Keys screen mini-chart).
• Optional second dimension later: tokens-per-minute budget (LLM-style limits) using the same bucket with token-cost withdrawal after usage is known.

# 5.5 Retries with exponential backoff + jitter

Consensus best practice for LLM upstreams (Zuplo 429 guide, handling 429s in production LLM apps, retry strategies with backoff + jitter):

• Retry on: 429, 500, 502, 503, 504, connection reset/refused, and TTFB timeout. Never on 400/401/403/404/422 (client/config errors — fail fast with the mapped OpenAI error).
• Schedule: delay = min(cap, base × 2^attempt) + random(0, jitter) — e.g. base 1 s, cap 30 s, full jitter; max 3 attempts for these interactive, user-facing requests (total budget ≤ ~30 s before fallback/error).
• Respect Retry-After when the provider sends it: wait = max(retryAfter, computedBackoff); if Retry-After exceeds our remaining budget, skip retrying this provider and go straight to fallback/error (propagating Retry-After to our own 429 response).
• Idempotency / streaming rule: a retry is only safe before any response byte has been forwarded to the client. Once the first SSE chunk has been written downstream, never retry or fall back — terminate the stream with an error event/close. (Chat completions are not idempotent upstream either: a "failed" request may still have consumed tokens; retrying after partial streaming double-bills and duplicates output.) So: retries apply to (a) whole non-streaming calls, (b) streaming calls that fail before the first upstream content delta.
• Jitter exists to desynchronize concurrent clients; even locally, parallel requests from one SDK justify it.

# 5.6 Fallback chains

Modeled on LiteLLM and OpenRouter:

• Semantics: an ordered model list tried in sequence. LiteLLM: "the router tries the primary model first; if it fails with a retry-able error (429, 5xx, context-limit, content-policy, timeout), it moves to the first fallback", in order. OpenRouter: a models: [...] array tried in order server-side. Sources: LiteLLM reliability/fallbacks, LiteLLM router architecture, OpenRouter model fallbacks.
• Trigger errors: exhausted retries on 429/5xx/timeout; upstream auth failure (missing/invalid provider key — jumping to a provider the user has a key for is exactly the point); model-not-available (404 upstream). Optionally context-window errors (LiteLLM has a distinct context_window_fallbacks class). Not on content-policy 400s by default (surprising model swaps on policy errors are a footgun; make it opt-in like LiteLLM's content_policy_fallbacks).
• Same rule as retries: no fallback after the first downstream byte.
• Honest reporting of the actually-used model (our Phase 3 spec requirement): OpenRouter returns "the model that was ultimately used … in the model attribute of the response body"; LiteLLM exposes the concrete deployment via x-litellm-model-id header / _hidden_params. Zyquo Router does both: the response/chunks' model field carries the namespaced ID that actually served the request, plus an x-zyquo-served-model response header and the fallback hop count in the request log. Beware LiteLLM's documented pitfall of fallbacks resetting the retry cycle and re-running fallback models (issue #19985) — our loop: for model in chain { retryPolicy(model) }, each model getting one bounded retry budget, no restarts.
• Per-chain config lives in the Models screen editor; a chain is addressable like a model/alias.

# 5.7 Health checks

• GET /health (no auth, no logging noise) returning:

{
  "status": "ok",
  "version": "1.0.0",
  "uptime_seconds": 12345,
  "server": { "host": "127.0.0.1", "port": 8787 },
  "providers_configured": 7,
  "active_streams": 2
}

• status is "ok" if the server is accepting; no upstream probing on this path (it must be instant and side-effect-free — LiteLLM separates /health per-model probes, which cost real tokens, from cheap /health/liveliness liveness checks; ours is the cheap kind). Per-provider connectivity testing belongs to the Keys screen's explicit "Test" button, not the health endpoint. Source: LiteLLM proxy docs.
• Suitable for curl-based readiness in scripts and the Phase 7 harness; also the phase-gate check for Phase 2.

# Sources (consolidated)

• SwiftNIO: https://github.com/apple/swift-nio · https://github.com/apple/swift-nio/releases · https://swiftpackageindex.com/apple/swift-nio · https://github.com/apple/swift-nio/blob/main/docs/public-async-nio-apis.md · https://swiftinit.org/docs/swift-nio/niocore/nioasyncchannel · https://github.com/apple/swift-nio/blob/main/Sources/NIOHTTP1Server/main.swift · https://swiftonserver.com/using-swiftnio-channels/ · https://forums.swift.org/t/nioasyncchannel-executethenclose-is-too-restrictive/73460 · https://blog.alexseifert.com/2026/06/29/building-a-web-app-in-swift-using-only-swiftnio/
• NIOExtras quiescing: https://github.com/apple/swift-nio-extras/blob/main/Sources/NIOExtras/QuiescingHelper.swift · https://github.com/apple/swift-nio-extras/blob/main/Sources/HTTPServerWithQuiescingDemo/main.swift
• Swift.org server guidelines (NIO3 timing): https://www.swift.org/documentation/server/guides/libraries/concurrency-adoption-guidelines.html
• Hummingbird: https://github.com/hummingbird-project/hummingbird · https://swiftpackageindex.com/hummingbird-project/hummingbird · https://swiftonserver.com/whats-new-in-hummingbird-2/ · https://hummingbird.codes/news/hummingbird-2/ · https://github.com/hummingbird-project/hummingbird-examples (server-sent-events example) · https://github.com/swift-server/swift-service-lifecycle
• Vapor comparison: https://github.com/hummingbird-project/hummingbird/discussions/150 · https://theswiftdev.com/beginners-guide-to-server-side-swift-using-the-hummingbird-framework/
• Network.framework: https://github.com/helje5/NWHTTPProtocol · http://www.alwaysrightinstitute.com/network-framework/ · https://developer.apple.com/documentation/network
• SSE format: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
• macOS local network privacy: https://developer.apple.com/forums/thread/763484 · https://developer.apple.com/forums/thread/767391 · https://mjtsai.com/blog/2024/10/02/local-network-privacy-on-sequoia/ · https://foldr.com/foldr-support/foldr-for-macos/macos-15-sequoia-local-network-privacy/ · https://help.panic.com/prompt/prompt-local-network/ · https://developer.apple.com/forums/thread/769037?page=2
• Entitlements/sandbox: https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.security.network.server · https://lapcatsoftware.com/articles/hardened-runtime-sandboxing.html
• LiteLLM: https://docs.litellm.ai/docs/proxy/reliability · https://docs.litellm.ai/docs/router_architecture · https://docs.litellm.ai/docs/proxy/logging · https://github.com/BerriAI/litellm/issues/16336 · https://github.com/BerriAI/litellm/issues/19985 · https://deepwiki.com/BerriAI/litellm/6.3-message-redaction-and-privacy-controls
• OpenRouter: https://openrouter.ai/docs/guides/routing/model-fallbacks · https://openrouter.ai/blog/insights/reliability-failover/
• Tokenizers in Swift: https://github.com/aespinilla/Tiktoken · https://github.com/narner/TiktokenSwift
• Rate limiting: https://blog.arcjet.com/rate-limiting-algorithms-token-bucket-vs-sliding-window-vs-fixed-window/ · https://medium.com/@tihomir.manushev/token-bucket-vs-sliding-window-the-rate-limiting-choice-that-shapes-your-apis-behavior-e04fb2646ee5 · https://apisix.apache.org/learning-center/api-gateway-rate-limiting/
• Retries/backoff: https://zuplo.com/learning-center/http-429-too-many-requests-guide · https://www.getmaxim.ai/articles/handle-429-errors-in-production-llm-applications/ · https://callsphere.ai/blog/retry-strategies-llm-api-calls-exponential-backoff-jitter-tenacity