# Architecture

Client (OpenAI SDK, curl, console)
   │  HTTPS  www.llm-api.io
   ▼
MacLustr Tunnel gateway (BHS64, Caddy TLS) ──WireGuard──▶ M1M64 wg1 10.67.0.x
   ▼
FastAPI gateway  127.0.0.1:8300   (server/llm_api)
   ├─ /v1/*      OpenAI-compatible routes  ── auth (API key or admin session) ── proxy to worker
   ├─ /api/*     management API           ── admin session / admin-scoped key, CSRF header
   ├─ /health    public status
   └─ /*         reverse proxy → Next.js console 127.0.0.1:8301
        │
        ├─ Registry (SQLite)        models, aliases, benchmarks, events, requests, jobs, keys, settings, audit
        ├─ Scanner                  safetensors + GGUF headers → architecture, params, quant, KV bytes/token
        ├─ Compatibility engine     estimate(weights, KV, overhead) vs budget → status + recommended context
        ├─ Model Manager            switch lock, eviction (LRU / pinned), spawn, wait-ready, warm-up, monitor, idle
        ├─ Job runner               downloads (serialized), scans, benchmarks, harvests
        ├─ Downloader / Harvester   Hugging Face inspection, verified downloads, manifests, discovery
        ├─ Metrics collector        psutil + ioreg + vm_stat + pmset → SSE + SQLite history
        └─ Event bus                SSE fan-out to the console
              │
              ▼  one process per loaded model, 127.0.0.1:8310-8399
        ┌──────────────────────┐   ┌──────────────────────────┐
        │ mlx_worker (python)  │   │ llama-server (llama.cpp) │
        │ mlx-lm / mlx-vlm     │   │ Metal, --jinja, slots    │
        │ /v1/chat/completions │   │ /v1/chat/completions     │
        │ /v1/embeddings …     │   │ /v1/embeddings …         │
        └──────────────────────┘   └──────────────────────────┘

# Request path (load-on-demand)

1. POST /v1/chat/completions → auth → body validation → model resolution (id, alias, name, repository, auto).
2. ModelManager.ensure_loaded(id):
   • fast path if the worker is ready;
   • otherwise take the switch lock (one switch at a time; other requests wait on it, so no thrashing);
   • policy: enabled, compatible (or force), estimate at the chosen context ≤ budget;
   • eviction: keep at most MAX_SIMULTANEOUS_MODELS large models and resident + new ≤ budget; victims are LRU, pinned last; small embedding/reranker workers (≤ 3 GB) may stay resident;
   • real free-memory check after eviction (waits for the OS to reclaim);
   • spawn the worker, poll /health until ready (or crash/timeout), run a warm-up inference, measure memory, mark ready, record load time.
3. The request is proxied to the worker (streamed for SSE). Usage + timings are recorded per request and per model.

# Worker protocol

Every runtime is a local HTTP server speaking the OpenAI API. Adapters only differ in how they spawn and how they detect readiness:

• MLXAdapter → python -m llm_api.worker.mlx_worker --model-path … --port … --max-context …. All MLX work (import, load, generate, embed) runs on a single dedicated thread inside the worker (MLX streams are thread-affine). Features: chat templates (tools, enable_thinking), <think> → reasoning_content, tool-call parsing, stop sequences with hold-back, prompt-cache reuse across turns, sampling (temperature, top_p, top_k, min_p, penalties, logit_bias, seed), last-token-pooled embeddings, yes/no rerank scoring, vision via mlx-vlm.
• LlamaCppAdapter → llama-server -m … --port … -ngl 999 -c ctx --jinja --no-webui -np 1 -fa auto --cache-reuse 256 (+ --embeddings, --reranking, --mmproj, --reasoning-format deepseek).

Memory release = process exit (SIGTERM → SIGKILL). The manager waits for vm_stat available memory to come back before the next load.

# Memory model

total = weights × 1.03–1.04
      + KV bytes/token × context          (2 · layers · kv_heads · head_dim · 2 bytes; sliding-window aware)
      + runtime overhead (1.0–1.2 GB) + activations (0.6–0.8 GB) + vision (0.8 GB)
      + long-context scratch (≤ 2 GB)

Recommended context = largest of 2K…32K that fits the budget (larger contexts can be requested explicitly). Budget default 45 GB on 64 GB; absolute limit 50 GB; both editable in Settings, which re-evaluates every model.

# Crash recovery

data/workers.json lists live worker PIDs. On startup, any leftover worker is killed and registry state reset; jobs left running are marked failed. The monitor loop (5 s) detects a dead worker and publishes worker_crashed.

# Future cluster

Nothing binds the gateway to one node: the registry row carries path/runtime, the manager talks HTTP to workers, telemetry is per host. A central router could hold several ModelManagers (one per node, over the private WireGuard network) and score model already loaded · free RAM · GPU · queue · tokens/s · thermal. Model replication would reuse the verified-manifest download path over the private network only.