1# Architecture23```4Client (OpenAI SDK, curl, console)5 │ HTTPS www.llm-api.io6 ▼7MacLustr Tunnel gateway (BHS64, Caddy TLS) ──WireGuard──▶ M1M64 wg1 10.67.0.x8 ▼9FastAPI gateway 127.0.0.1:8300 (server/llm_api)10 ├─ /v1/* OpenAI-compatible routes ── auth (API key or admin session) ── proxy to worker11 ├─ /api/* management API ── admin session / admin-scoped key, CSRF header12 ├─ /health public status13 └─ /* reverse proxy → Next.js console 127.0.0.1:830114 │15 ├─ Registry (SQLite) models, aliases, benchmarks, events, requests, jobs, keys, settings, audit16 ├─ Scanner safetensors + GGUF headers → architecture, params, quant, KV bytes/token17 ├─ Compatibility engine estimate(weights, KV, overhead) vs budget → status + recommended context18 ├─ Model Manager switch lock, eviction (LRU / pinned), spawn, wait-ready, warm-up, monitor, idle19 ├─ Job runner downloads (serialized), scans, benchmarks, harvests20 ├─ Downloader / Harvester Hugging Face inspection, verified downloads, manifests, discovery21 ├─ Metrics collector psutil + ioreg + vm_stat + pmset → SSE + SQLite history22 └─ Event bus SSE fan-out to the console23 │24 ▼ one process per loaded model, 127.0.0.1:8310-839925 ┌──────────────────────┐ ┌──────────────────────────┐26 │ mlx_worker (python) │ │ llama-server (llama.cpp) │27 │ mlx-lm / mlx-vlm │ │ Metal, --jinja, slots │28 │ /v1/chat/completions │ │ /v1/chat/completions │29 │ /v1/embeddings … │ │ /v1/embeddings … │30 └──────────────────────┘ └──────────────────────────┘31```3233## Request path (load-on-demand)34351. `POST /v1/chat/completions` → auth → body validation → model resolution (id, alias, name, repository, `auto`).362. `ModelManager.ensure_loaded(id)`:37 - fast path if the worker is `ready`;38 - otherwise take the **switch lock** (one switch at a time; other requests wait on it, so no thrashing);39 - policy: enabled, compatible (or `force`), estimate at the chosen context ≤ budget;40 - eviction: keep at most `MAX_SIMULTANEOUS_MODELS` large models and `resident + new ≤ budget`; victims are LRU, pinned last; small embedding/reranker workers (≤ 3 GB) may stay resident;41 - real free-memory check after eviction (waits for the OS to reclaim);42 - spawn the worker, poll `/health` until `ready` (or crash/timeout), run a **warm-up** inference, measure memory, mark `ready`, record load time.433. The request is proxied to the worker (streamed for SSE). Usage + `timings` are recorded per request and per model.4445## Worker protocol4647Every runtime is a local HTTP server speaking the OpenAI API. Adapters only differ in how they spawn and how they detect readiness:4849- **MLXAdapter** → `python -m llm_api.worker.mlx_worker --model-path … --port … --max-context …`. All MLX work (import, load, generate, embed) runs on a single dedicated thread inside the worker (MLX streams are thread-affine). Features: chat templates (tools, `enable_thinking`), `<think>` → `reasoning_content`, tool-call parsing, stop sequences with hold-back, prompt-cache reuse across turns, sampling (temperature, top_p, top_k, min_p, penalties, logit_bias, seed), last-token-pooled embeddings, yes/no rerank scoring, vision via mlx-vlm.50- **LlamaCppAdapter** → `llama-server -m … --port … -ngl 999 -c ctx --jinja --no-webui -np 1 -fa auto --cache-reuse 256` (+ `--embeddings`, `--reranking`, `--mmproj`, `--reasoning-format deepseek`).5152Memory release = process exit (SIGTERM → SIGKILL). The manager waits for `vm_stat` available memory to come back before the next load.5354## Memory model5556```57total = weights × 1.03–1.0458 + KV bytes/token × context (2 · layers · kv_heads · head_dim · 2 bytes; sliding-window aware)59 + runtime overhead (1.0–1.2 GB) + activations (0.6–0.8 GB) + vision (0.8 GB)60 + long-context scratch (≤ 2 GB)61```6263Recommended context = largest of 2K…32K that fits the budget (larger contexts can be requested explicitly). Budget default 45 GB on 64 GB; absolute limit 50 GB; both editable in Settings, which re-evaluates every model.6465## Crash recovery6667`data/workers.json` lists live worker PIDs. On startup, any leftover worker is killed and registry state reset; jobs left `running` are marked failed. The monitor loop (5 s) detects a dead worker and publishes `worker_crashed`.6869## Future cluster7071Nothing binds the gateway to one node: the registry row carries `path`/`runtime`, the manager talks HTTP to workers, telemetry is per host. A central router could hold several `ModelManager`s (one per node, over the private WireGuard network) and score `model already loaded · free RAM · GPU · queue · tokens/s · thermal`. Model replication would reuse the verified-manifest download path over the private network only.72