LLM API v0.1.0 — private OpenAI-compatible local model server for Apple Silicon · a48486c · llm-api

LLM API v0.1.0 — private OpenAI-compatible local model server for Apple Silicon

FastAPI gateway + SQLite registry, MLX-LM / mlx-vlm and llama.cpp workers, load-on-demand
with memory policy and eviction, OpenAI endpoints (chat/completions/embeddings/rerank, streaming),
management API, Model Harvester, benchmarks, API keys, Next.js 16 console, tests and docs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 1 mo ago (Sep 11, 2026) Browse files

70 changed files +13,830 −0

added .env.example +52 −0

		`@@ -0,0 +1,52 @@`
	1	+# LLM API — environment (copy to <LLM_API_ROOT>/.env or export before `llm-api serve`)
	2	+
	3	+APP_ENV=production
	4	+DOMAIN=www.llm-api.io
	5	+PUBLIC_URL=https://www.llm-api.io
	6	+
	7	+# Bind address of the API gateway (the public reverse proxy talks to this)
	8	+HOST=127.0.0.1
	9	+PORT=8300
	10	+# Where the Next.js dashboard runs (proxied by the API for non-/api, non-/v1 paths)
	11	+WEB_URL=http://127.0.0.1:8301
	12	+
	13	+# Data root: models/, data/ (SQLite + secrets), logs/
	14	+LLM_API_ROOT=/Users/USERNAME/llm-api
	15	+# Override the model directory (e.g. an external SSD)
	16	+# MODEL_ROOT=/Volumes/LLM/models
	17	+
	18	+# Memory policy (GB). 64 GB machine: 8 macOS + 3 app + 4 KV + 4 safety = 45 GB for models
	19	+MAX_MODEL_MEMORY_GB=45
	20	+ABSOLUTE_MAX_MEMORY_GB=50
	21	+MACOS_RESERVE_GB=8
	22	+MIN_FREE_DISK_GB=100
	23	+
	24	+MAX_SIMULTANEOUS_MODELS=1
	25	+MODEL_IDLE_TIMEOUT_MINUTES=30
	26	+SMALL_MODEL_RESIDENT_GB=3
	27	+LOAD_TIMEOUT_SECONDS=900
	28	+GENERATION_TIMEOUT_SECONDS=1800
	29	+PRELOAD_MODEL=none
	30	+
	31	+ENABLE_MLX=true
	32	+ENABLE_GGUF=true
	33	+ALLOW_DOWNLOADS=true
	34	+LOG_PROMPTS=false
	35	+
	36	+DEFAULT_MAX_TOKENS=2048
	37	+DEFAULT_CONTEXT=16384
	38	+
	39	+# Hugging Face token for gated repositories (never exposed to the frontend)
	40	+HF_TOKEN=
	41	+LLAMA_SERVER_BIN=llama-server
	42	+
	43	+# Sessions/CSRF secret (auto-generated into data/.secret when empty)
	44	+SECRET_KEY=
	45	+# First-run admin (used only when no user exists)
	46	+ADMIN_EMAIL=
	47	+ADMIN_PASSWORD=
	48	+SESSION_HOURS=336
	49	+SECURE_COOKIES=true
	50	+
	51	+METRICS_INTERVAL_SECONDS=15
	52	+METRICS_RETENTION_DAYS=30

added .gitignore +22 −0

		`@@ -0,0 +1,22 @@`
	1	+# python
	2	+__pycache__/
	3	+*.pyc
	4	+server/.venv/
	5	+.venv/
	6	+*.egg-info/
	7	+.pytest_cache/
	8	+.ruff_cache/
	9	+# node
	10	+web/node_modules/
	11	+web/.next/
	12	+web/next-env.d.ts
	13	+web/*.tsbuildinfo
	14	+# secrets / data
	15	+.env
	16	+.env.*
	17	+!.env.example
	18	+data/
	19	+logs/
	20	+models/
	21	+# os
	22	+.DS_Store

added CLAUDE.md +25 −0

		`@@ -0,0 +1,25 @@`
	1	+# CLAUDE.md — LLM API (www.llm-api.io)
	2	+
	3	+Private OpenAI-compatible local model server for Apple Silicon. Full product spec: see `docs/` and README.
	4	+Runs on M1M64 (Mac Studio M1 Max 64 GB, rented, user `simon`), deployed with `mld deploy llm-api --node M1M64`
	5	+(manifest `M1M32:~/dispatch/apps/llm-api.json`), public via MacLustr Tunnel (BHS64) at https://www.llm-api.io.
	6	+
	7	+## Layout
	8	+- `server/` Python 3.13 FastAPI (`llm_api`): gateway, registry, manager, workers (`llm_api/worker/mlx_worker.py`, llama-server), tests (`pytest`, fake workers, no MLX needed).
	9	+- `web/` Next.js 16 console (proxied by the API in production; `pnpm dev` proxies /api,/v1 to `API_URL`).
	10	+- Data root on the node: `~/llm-api/` (`models/` library, `data/` SQLite + `.secret` + `.env`, `logs/`). Code: `~/apps/llm-api/`.
	11	+
	12	+## Rules (non-negotiable)
	13	+1. Never load a model whose estimate exceeds the budget; never rely on swap; never fake metrics.
	14	+2. Workers only on 127.0.0.1; memory release = kill the worker process.
	15	+3. Never mark ready before warm-up; never delete model files automatically.
	16	+4. Verify MLX / mlx-lm / mlx-vlm / llama.cpp / huggingface-hub APIs against the installed versions before changing runtime code.
	17	+5. Always test load → inference → unload → memory recovery after touching the manager or workers.
	18	+
	19	+## Dev loop
	20	+```bash
	21	+cd server && python -m compileall -q llm_api && rsync -az --delete --exclude __pycache__ --exclude .venv ./ M1M64:~/apps/llm-api/server/
	22	+ssh M1M64 'source ~/llm-api/.venv/bin/activate; cd ~/apps/llm-api/server && python -m pytest -q --basetemp=/tmp/llmapi-tests; rm -rf /tmp/llmapi-tests'
	23	+cd web && pnpm exec tsc --noEmit && pnpm build
	24	+```
	25	+Gotchas: pytest temp dirs must stay sparse (`--basetemp` + sparse fixtures — a real 400-layer fixture once filled the 1.8 TB SSD); `socket.getfqdn` hangs on the rented Mac (never use stock `HTTPServer.server_bind`); MLX work must stay on one thread per worker.

added README.md +85 −0

		`@@ -0,0 +1,85 @@`
	1	+# LLM API — `https://www.llm-api.io`
	2	+
	3	+A private, production-grade, OpenAI-compatible local LLM API for Apple Silicon. Store a large library of models on SSD, load only the one that is requested into unified memory, unload it when something else is asked for, and expose everything behind one clean, authenticated API plus a management console.
	4	+
	5	+```
	6	+OpenAI API — but backed by my own Apple Silicon machine and my own locally stored models.
	7	+```
	8	+
	9	+First deployment: Mac Studio M1 Max, 10-core CPU, 32-core GPU, 64 GB unified memory, 1.8 TB SSD (node `M1M64`).
	10	+
	11	+## What it does
	12	+
	13	+- Load-on-demand: `{"model": "qwen3.8-27b-4bit"}` loads that model from SSD if it is not resident, evicting the previous one (LRU + idle timeout), then serves the request. Switches are serialized; requests queue while a model loads.
	14	+- Two runtimes: MLX / MLX-LM (preferred, safetensors) and llama.cpp `llama-server` (GGUF, Metal). Each model runs in its own worker process on `127.0.0.1` — killing the process is how memory is guaranteed to come back.
	15	+- Memory policy: weights + KV cache + runtime overhead are estimated before loading; the default budget is 45 GB of 64 GB (never all of it). Too-large models are refused with a structured error. Swap is treated as a warning, never a feature.
	16	+- Compatibility engine: `compatible` / `compatible_with_restrictions` / `experimental` / `not_recommended` / `incompatible`, with a recommended context per model.
	17	+- OpenAI endpoints: `/v1/models`, `/v1/chat/completions` (streaming, tools, reasoning content), `/v1/completions`, `/v1/embeddings`, `/v1/rerank`. Works with the official OpenAI SDKs.
	18	+- Management API + console: model library, per-model pages, playground, downloads from Hugging Face (inspected first: size, RAM, disk reserve), Model Harvester (explores HF for models that truly fit, dedupes quantizations, proposes a download queue), benchmarks, API keys (hashed), settings, real-time telemetry (RAM, GPU, CPU, thermal, disk) via SSE.
	19	+- Aliases and `auto`: `fast`, `coder`, `reasoning`, `vision`, `embedding`, `default`… `model: "auto"` routes by prompt content (never when a model is named explicitly).
	20	+
	21	+## Layout
	22	+
	23	+```
	24	+server/ Python 3.13 · FastAPI · SQLite — API gateway, registry, model manager, workers
	25	+web/ Next.js 16 · React 19 · Tailwind 4 — console (proxied by the API in production)
	26	+scripts/ install.sh
	27	+docs/ architecture, api, models, security, deployment, troubleshooting
	28	+```
	29	+
	30	+## Quick start (clean Mac)
	31	+
	32	+```bash
	33	+git clone <repo> llm-api && cd llm-api
	34	+./scripts/install.sh --test # checks the Mac, installs uv/Node/llama.cpp, Python deps, builds the web app
	35	+$EDITOR ~/llm-api/.env # ADMIN_EMAIL, ADMIN_PASSWORD, HF_TOKEN, PUBLIC_URL
	36	+(cd ~/llm-api && ../path/to/server/.venv/bin/llm-api serve) # API on 127.0.0.1:8300
	37	+(cd web && pnpm start -p 8301 -H 127.0.0.1) # console
	38	+```
	39	+
	40	+Then open `http://127.0.0.1:8300`, sign in, create an API key, and:
	41	+
	42	+```python
	43	+from openai import OpenAI
	44	+client = OpenAI(base_url="https://www.llm-api.io/v1", api_key="llm_live_xxxxx")
	45	+r = client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hello"}])
	46	+print(r.choices[0].message.content)
	47	+```
	48	+
	49	+```bash
	50	+curl https://www.llm-api.io/v1/chat/completions \
	51	+ -H "Authorization: Bearer llm_live_xxxxx" -H "Content-Type: application/json" \
	52	+ -d '{"model": "qwen3.8-27b-4bit", "messages": [{"role": "user", "content": "Explain monetary policy."}], "stream": true}'
	53	+```
	54	+
	55	+## Verified on the M1 Max (2026-09-10)
	56	+
	57	+\| Step \| Result \|
	58	+\|---\|---\|
	59	+\| Cold load Qwen3-4B-Instruct-2507 (MLX 4-bit) \| 1.7 s, ~100 tok/s generation \|
	60	+\| Cold load Llama-3.2-3B (MLX 4-bit) \| 1.4 s, ~170 tok/s \|
	61	+\| Gemma-3-1B GGUF via llama.cpp \| 1.1 s, ~150 tok/s, streaming with `timings` \|
	62	+\| Switch A → B → A \| previous worker killed, memory returned to the OS before the next load \|
	63	+\| OpenAI Python SDK \| models / chat / stream / embeddings OK \|
	64	+\| Memory rejection \| budget lowered → model refused with `MODEL_TOO_LARGE`, compatibility re-evaluated \|
	65	+
	66	+## Tests
	67	+
	68	+```bash
	69	+cd server && .venv/bin/python -m pytest -q
	70	+```
	71	+
	72	+The suite runs without MLX (fake workers) and covers: authentication, API keys (creation, rejection, revocation), registry scanning, missing-file detection, load-on-demand, streaming, model switching, concurrent requests + switch lock, memory rejection, worker crash, load timeout, deletion confirmation, low-disk refusal, benchmarks, restart cleanup, settings validation.
	73	+
	74	+## Docs
	75	+
	76	+- [docs/architecture.md](docs/architecture.md) — components, data flow, worker protocol, memory model
	77	+- [docs/api.md](docs/api.md) — OpenAI endpoints, extensions, management API, errors
	78	+- [docs/models.md](docs/models.md) — storage layout, discovery, compatibility, quantization policy, Harvester, starter library
	79	+- [docs/security.md](docs/security.md) — auth, keys, CSRF, sandboxing, what is never logged
	80	+- [docs/deployment.md](docs/deployment.md) — M1M64 deployment with `mld`, PM2, MacLustr Tunnel, DNS
	81	+- [docs/troubleshooting.md](docs/troubleshooting.md) — failure modes and what to look at
	82	+
	83	+## Non-negotiables (from CLAUDE.md)
	84	+
	85	+Never crash the host with an oversized model · never rely on swap · never expose secrets or raw workers · never mark a model ready before a successful warm-up · never delete model files automatically · never fake metrics · verify runtime APIs against the installed versions · always test load → inference → unload → memory recovery · keep the design ready for multiple Apple Silicon nodes.

added docs/api.md +77 −0

		`@@ -0,0 +1,77 @@`
	1	+# API
	2	+
	3	+Base URL: `https://www.llm-api.io/v1` (also reachable at `http://127.0.0.1:8300/v1` on the node). Interactive OpenAPI schema: `/openapi`.
	4	+
	5	+## Authentication
	6	+
	7	+`Authorization: Bearer llm_live_…` (or `X-API-Key`). Keys are created in the console (Keys page) or `llm-api create-key <name> [--admin]`. Scopes: `inference` (OpenAI endpoints), `admin` (management API). The console session cookie also works for `/v1` (playground).
	8	+
	9	+## OpenAI-compatible endpoints
	10	+
	11	+\| Method \| Path \| Notes \|
	12	+\|---\|---\|---\|
	13	+\| GET \| `/v1/models` \| installed models + aliases; extra fields: `runtime`, `quantization`, `estimated_ram_gb`, `context`, `compatibility`, `status`, `capabilities` \|
	14	+\| GET \| `/v1/models/{id}` \| \|
	15	+\| POST \| `/v1/chat/completions` \| `stream`, `temperature`, `top_p`, `max_tokens`/`max_completion_tokens`, `stop` (≤ 8), `seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `tools`, `stream_options.include_usage` (always on) \|
	16	+\| POST \| `/v1/completions` \| `prompt`, `echo`, same sampling params \|
	17	+\| POST \| `/v1/embeddings` \| `input` (str or ≤ 256 strings), `dimensions`, `encoding_format` \|
	18	+\| POST \| `/v1/rerank` \| `query`, `documents` (≤ 200), `top_n`, `return_documents`, `instruction` \|
	19	+
	20	+Extensions (ignored by SDKs):
	21	+
	22	+- `timings`: `{ttft_ms, prompt_ms, generation_ms, total_ms, prompt_tps, generation_tps, peak_memory_gb}` on responses and on the final stream chunk.
	23	+- `usage.prompt_tokens_details.cached_tokens` when the prompt cache was reused.
	24	+- `reasoning_content` on `message` / `delta` for thinking models. Control with `chat_template_kwargs: {"enable_thinking": false}`, `reasoning: {"effort": "none"}` or `reasoning_effort`.
	25	+- `top_k`, `min_p`, `repetition_penalty` (MLX).
	26	+- `model: "auto"` — routed by prompt (code → `coder`, long/analytical → `reasoning`, images → `vision`, embeddings → `embedding`), else current model / `default` alias / default model setting.
	27	+
	28	+### Model resolution
	29	+
	30	+`model` may be a registry id (`qwen3.8-27b-4bit`), an alias (`coder`), the directory name, or the Hugging Face repository (`mlx-community/Qwen3.8-27B-4bit`).
	31	+
	32	+### Errors
	33	+
	34	+```json
	35	+{"error": {"message": "Model requires approximately 52.0 GB at a 32768 context but the safe limit is 45 GB.",
	36	+ "type": "model_memory_error", "code": "MODEL_TOO_LARGE", "param": null, "estimate": {...}}}
	37	+```
	38	+
	39	+\| HTTP \| code \| when \|
	40	+\|---\|---\|---\|
	41	+\| 400 \| `INVALID_REQUEST`, `CONTEXT_TOO_LARGE`, `TEMPLATE_ERROR`, `VISION_UNSUPPORTED` \| bad body, prompt longer than the loaded context \|
	42	+\| 401 \| `INVALID_API_KEY`, `UNAUTHENTICATED` \| \|
	43	+\| 403 \| `FORBIDDEN`, `CSRF`, `DOWNLOADS_DISABLED`, `PATH_NOT_ALLOWED` \| \|
	44	+\| 404 \| `MODEL_NOT_FOUND` \| \|
	45	+\| 409 \| `CONFLICT`, `ALREADY_INSTALLED` \| \|
	46	+\| 413 \| `BODY_TOO_LARGE` \| > 20 MB \|
	47	+\| 422 \| `MODEL_INCOMPATIBLE`, `RUNTIME_UNSUPPORTED`, `WRONG_MODEL_TYPE` \| \|
	48	+\| 429 \| `RATE_LIMITED` \| 600 req/min per IP; 8 logins/min \|
	49	+\| 502 \| `WORKER_UNREACHABLE`, `DOWNLOAD_FAILED` \| \|
	50	+\| 503 \| `MODEL_LOAD_FAILED`, `MODEL_LOAD_TIMEOUT`, `WORKER_CRASHED`, `WORKER_BUSY` \| \|
	51	+\| 504 \| `GENERATION_TIMEOUT` \| \|
	52	+\| 507 \| `MODEL_TOO_LARGE`, `INSUFFICIENT_DISK` \| \|
	53	+
	54	+## Management API (`/api`, admin)
	55	+
	56	+Session mutations need header `X-LLM-CSRF: 1`.
	57	+
	58	+- Auth: `GET /api/auth/status`, `POST /api/auth/setup\|login\|logout\|password`, `GET /api/auth/me`
	59	+- Models: `GET /api/models[?include_missing]`, `GET /api/models/{id}` (benchmarks, events, memory curve), `GET /api/models/{id}/files`, `POST /api/models/{id}/load {context?, force?}`, `POST /api/models/{id}/unload`, `PATCH /api/models/{id}` (favorite, pinned, enabled, notes, tags, name, overrides), `POST /api/models/{id}/pin?pinned=`, `POST /api/models/{id}/favorite?favorite=`, `DELETE /api/models/{id} {confirm: id, keep_benchmarks}`, `POST /api/models/rescan`, `POST /api/models/{id}/benchmark {max_tokens, runs, long_prompt}`, `GET /api/models/{id}/benchmarks`, `POST /api/models/{id}/tokenize`
	60	+- Aliases: `GET /api/aliases`, `PUT /api/aliases {alias, model_id}`, `DELETE /api/aliases/{alias}`
	61	+- Downloads: `POST /api/models/inspect {repository, quant?}`, `POST /api/models/download {repository, quant?, force?}`, `GET /api/downloads`, `POST /api/downloads/{job}/retry`, `GET /api/jobs[?kind]`, `POST /api/jobs/{id}/cancel`
	62	+- Harvester: `POST /api/harvest/scan {runtimes, authors, limit_per_author, min_downloads, max_ram_gb, families, tasks, search}`, `GET /api/harvest/candidates[?task&runtime&family&size_class&q&include_duplicates]` (+ starter slots), `POST /api/harvest/select\|dismiss {repo_id, selected}`, `POST /api/harvest/queue`
	63	+- System: `GET /api/system`, `/api/system/memory`, `/api/system/gpu`, `/api/system/storage`, `/api/system/processes`, `/api/system/health`, `/api/system/metrics?minutes&hours`, `GET /api/runtime/status`, `GET /api/runtime/current-model`
	64	+- Keys: `GET/POST /api/keys`, `PATCH /api/keys/{id} {name}`, `DELETE /api/keys/{id}`
	65	+- Settings: `GET/PATCH /api/settings`
	66	+- Logs: `GET /api/logs/audit\|events\|requests`, `GET /api/logs/worker/{model_id}`
	67	+- Live: `GET /api/events` — SSE stream: `snapshot`, `manager`, `model` (load progress), `metrics` (every 3 s), `job`, `request`, `alert`, `settings`.
	68	+
	69	+## Status
	70	+
	71	+`GET /health` (public):
	72	+
	73	+```json
	74	+{"status": "ok", "hardware": {"chip": "Apple M1 Max", "memory_gb": 64.0, "gpu_cores": 32},
	75	+ "model": {"loaded": true, "id": "qwen3-4b-instruct-2507-4bit", "loaded_models": ["…"]},
	76	+ "memory": {"used_gb": 23.1, "available_gb": 40.9, "pressure": "normal"}, "uptime_seconds": 1234, "version": "0.1.0"}
	77	+```

added docs/architecture.md +71 −0

		`@@ -0,0 +1,71 @@`
	1	+# Architecture
	2	+
	3	+```
	4	+Client (OpenAI SDK, curl, console)
	5	+ │ HTTPS www.llm-api.io
	6	+ ▼
	7	+MacLustr Tunnel gateway (BHS64, Caddy TLS) ──WireGuard──▶ M1M64 wg1 10.67.0.x
	8	+ ▼
	9	+FastAPI gateway 127.0.0.1:8300 (server/llm_api)
	10	+ ├─ /v1/* OpenAI-compatible routes ── auth (API key or admin session) ── proxy to worker
	11	+ ├─ /api/* management API ── admin session / admin-scoped key, CSRF header
	12	+ ├─ /health public status
	13	+ └─ /* reverse proxy → Next.js console 127.0.0.1:8301
	14	+ │
	15	+ ├─ Registry (SQLite) models, aliases, benchmarks, events, requests, jobs, keys, settings, audit
	16	+ ├─ Scanner safetensors + GGUF headers → architecture, params, quant, KV bytes/token
	17	+ ├─ Compatibility engine estimate(weights, KV, overhead) vs budget → status + recommended context
	18	+ ├─ Model Manager switch lock, eviction (LRU / pinned), spawn, wait-ready, warm-up, monitor, idle
	19	+ ├─ Job runner downloads (serialized), scans, benchmarks, harvests
	20	+ ├─ Downloader / Harvester Hugging Face inspection, verified downloads, manifests, discovery
	21	+ ├─ Metrics collector psutil + ioreg + vm_stat + pmset → SSE + SQLite history
	22	+ └─ Event bus SSE fan-out to the console
	23	+ │
	24	+ ▼ one process per loaded model, 127.0.0.1:8310-8399
	25	+ ┌──────────────────────┐ ┌──────────────────────────┐
	26	+ │ mlx_worker (python) │ │ llama-server (llama.cpp) │
	27	+ │ mlx-lm / mlx-vlm │ │ Metal, --jinja, slots │
	28	+ │ /v1/chat/completions │ │ /v1/chat/completions │
	29	+ │ /v1/embeddings … │ │ /v1/embeddings … │
	30	+ └──────────────────────┘ └──────────────────────────┘
	31	+```
	32	+
	33	+## Request path (load-on-demand)
	34	+
	35	+1. `POST /v1/chat/completions` → auth → body validation → model resolution (id, alias, name, repository, `auto`).
	36	+2. `ModelManager.ensure_loaded(id)`:
	37	+ - fast path if the worker is `ready`;
	38	+ - otherwise take the switch lock (one switch at a time; other requests wait on it, so no thrashing);
	39	+ - policy: enabled, compatible (or `force`), estimate at the chosen context ≤ budget;
	40	+ - eviction: keep at most `MAX_SIMULTANEOUS_MODELS` large models and `resident + new ≤ budget`; victims are LRU, pinned last; small embedding/reranker workers (≤ 3 GB) may stay resident;
	41	+ - real free-memory check after eviction (waits for the OS to reclaim);
	42	+ - spawn the worker, poll `/health` until `ready` (or crash/timeout), run a warm-up inference, measure memory, mark `ready`, record load time.
	43	+3. The request is proxied to the worker (streamed for SSE). Usage + `timings` are recorded per request and per model.
	44	+
	45	+## Worker protocol
	46	+
	47	+Every runtime is a local HTTP server speaking the OpenAI API. Adapters only differ in how they spawn and how they detect readiness:
	48	+
	49	+- MLXAdapter → `python -m llm_api.worker.mlx_worker --model-path … --port … --max-context …`. All MLX work (import, load, generate, embed) runs on a single dedicated thread inside the worker (MLX streams are thread-affine). Features: chat templates (tools, `enable_thinking`), `<think>` → `reasoning_content`, tool-call parsing, stop sequences with hold-back, prompt-cache reuse across turns, sampling (temperature, top_p, top_k, min_p, penalties, logit_bias, seed), last-token-pooled embeddings, yes/no rerank scoring, vision via mlx-vlm.
	50	+- LlamaCppAdapter → `llama-server -m … --port … -ngl 999 -c ctx --jinja --no-webui -np 1 -fa auto --cache-reuse 256` (+ `--embeddings`, `--reranking`, `--mmproj`, `--reasoning-format deepseek`).
	51	+
	52	+Memory release = process exit (SIGTERM → SIGKILL). The manager waits for `vm_stat` available memory to come back before the next load.
	53	+
	54	+## Memory model
	55	+
	56	+```
	57	+total = weights × 1.03–1.04
	58	+ + KV bytes/token × context (2 · layers · kv_heads · head_dim · 2 bytes; sliding-window aware)
	59	+ + runtime overhead (1.0–1.2 GB) + activations (0.6–0.8 GB) + vision (0.8 GB)
	60	+ + long-context scratch (≤ 2 GB)
	61	+```
	62	+
	63	+Recommended context = largest of 2K…32K that fits the budget (larger contexts can be requested explicitly). Budget default 45 GB on 64 GB; absolute limit 50 GB; both editable in Settings, which re-evaluates every model.
	64	+
	65	+## Crash recovery
	66	+
	67	+`data/workers.json` lists live worker PIDs. On startup, any leftover worker is killed and registry state reset; jobs left `running` are marked failed. The monitor loop (5 s) detects a dead worker and publishes `worker_crashed`.
	68	+
	69	+## Future cluster
	70	+
	71	+Nothing binds the gateway to one node: the registry row carries `path`/`runtime`, the manager talks HTTP to workers, telemetry is per host. A central router could hold several `ModelManager`s (one per node, over the private WireGuard network) and score `model already loaded · free RAM · GPU · queue · tokens/s · thermal`. Model replication would reuse the verified-manifest download path over the private network only.

added docs/deployment.md +57 −0

		`@@ -0,0 +1,57 @@`
	1	+# Deployment
	2	+
	3	+## Production topology (2026-09-10)
	4	+
	5	+```
	6	+GoDaddy DNS llm-api.io A www → 51.161.112.61 (BHS64)
	7	+BHS64 (OVH Beauharnois) Caddy TLS https://www.llm-api.io → 10.67.0.x:8300 over WireGuard
	8	+M1M64 (Mac Studio M1 Max 64 GB, rented, public IP 45.74.241.221, user simon)
	9	+ wg1 10.67.0.x
	10	+ PM2 (LaunchDaemon) llm-api-server → .venv/bin/llm-api serve 127.0.0.1:8300
	11	+ llm-api-web → next start -p 8301 -H 127.0.0.1
	12	+ ~/llm-api/ models/ (SSD library) data/ (SQLite, .secret, .env) logs/
	13	+ ~/apps/llm-api/ code (rsync'd by mld: server/ + web/)
	14	+```
	15	+
	16	+The whole app is deployed and routed by the cluster orchestrator mld (gateway M1M32):
	17	+
	18	+```bash
	19	+mld stage ~/Desktop/Cluster/llm-api llm-api # laptop → M1M32 staging (excludes node_modules, .next, .venv)
	20	+mld deploy llm-api --node M1M64 # rsync → post_sync hooks (uv venv + pip, pnpm build) → PM2 → health → tunnel route → registry
	21	+mld status --live \| grep llm-api
	22	+mld logs llm-api
	23	+```
	24	+
	25	+Manifest: `M1M32:~/dispatch/apps/llm-api.json` (dir `~/apps/llm-api`, port 8300, health `/health`, tunnel `www.llm-api.io` on BHS64, pinned to `M1M64`). Secrets (`ADMIN_PASSWORD`, `HF_TOKEN`) live in the manifest's `env` on M1M32 only and are written into the PM2 environment; the app also reads `~/llm-api/.env`.
	26	+
	27	+M1M64 is a reserved node: always deploy with `--node M1M64`. It has no graphical session guaranteed at boot, so PM2 runs as a system LaunchDaemon (`mld prepare` does this).
	28	+
	29	+## Manual (without mld)
	30	+
	31	+```bash
	32	+./scripts/install.sh
	33	+cd ~/llm-api && /path/to/server/.venv/bin/llm-api serve # API
	34	+cd /path/to/web && pnpm start -p 8301 -H 127.0.0.1 # console
	35	+```
	36	+
	37	+PM2:
	38	+
	39	+```bash
	40	+pm2 start /path/to/server/.venv/bin/llm-api --name llm-api-server --cwd ~/llm-api -- serve
	41	+pm2 start node --name llm-api-web --cwd /path/to/web -- node_modules/next/dist/bin/next start -p 8301 -H 127.0.0.1
	42	+pm2 save && pm2 startup
	43	+```
	44	+
	45	+launchd alternative: a `LaunchDaemon` plist running `.venv/bin/llm-api serve` with `WorkingDirectory=~/llm-api`, `KeepAlive=true`, `RunAtLoad=true` (see `mld` render for the pattern).
	46	+
	47	+## Reverse proxy
	48	+
	49	+Any TLS terminator works (Caddy, nginx). Forward everything to `127.0.0.1:8300`; the API serves `/v1`, `/api`, `/health`, `/openapi` itself and proxies the rest to Next. SSE needs buffering off (`X-Accel-Buffering: no` is already set). `api.llm-api.io` can point at the same upstream if a separate API host is wanted.
	50	+
	51	+## Updating
	52	+
	53	+```bash
	54	+mld stage ~/Desktop/Cluster/llm-api llm-api && mld deploy llm-api --node M1M64
	55	+```
	56	+
	57	+Deploy = rsync + rebuild + `pm2 restart`; the manager unloads the loaded model on shutdown and clears stale state on start.

added docs/models.md +55 −0

		`@@ -0,0 +1,55 @@`
	1	+# Models
	2	+
	3	+## Storage
	4	+
	5	+```
	6	+<LLM_API_ROOT>/models/
	7	+├── mlx/<family>/<Repo-Name>/ safetensors + config.json + tokenizer (+ llm-api.json manifest)
	8	+├── gguf/<family>/<Repo-Name-GGUF>/ one chosen quantization (+ mmproj for vision)
	9	+├── embeddings/ rerankers/ vision/ same layout, by role
	10	+└── manifests/<org--repo>.json copy of every download manifest
	11	+```
	12	+
	13	+One recommended quantization per model; the downloader picks it (`Q4_K_M` → `Q5_K_M` → `Q6_K` → `Q8_0` → `MXFP4`… for GGUF) unless a `quant` is given. Nothing is ever deleted by a scan.
	14	+
	15	+## Discovery (`POST /api/models/rescan`, startup)
	16	+
	17	+For each directory: `config.json` + `.safetensors` → MLX model (architecture, layers, heads, quant bits from `quantization`, params from safetensors headers — packed `uint32` ×32/bits); `.gguf` → llama.cpp model (GGUF header: architecture, `block_count`, `head_count_kv`, `key_length`, `context_length`, `file_type`, tensor shapes). Type detection: embedding (name / pipeline / pooling), reranker, vision (`vision_config`, mmproj, model type), thinking / tools (chat template). Registry ids are stable slugs (`qwen3-4b-instruct-2507-4bit`); a directory keeps its id across rescans. Manual overrides (task, context, KV bits, pooling, extra llama args) live in `overrides` and survive rescans.
	18	+
	19	+## Size classes (estimated RAM at the recommended context)
	20	+
	21	+TINY < 5 GB · SMALL 5–10 · MEDIUM 10–20 · LARGE 20–35 · XL 35–45 · TOO_LARGE > budget.
	22	+
	23	+## Compatibility
	24	+
	25	+\| status \| meaning \|
	26	+\|---\|---\|
	27	+\| `compatible` \| fits with ≥ 16K context in the budget \|
	28	+\| `compatible_with_restrictions` \| fits, but context must stay below 16K (the recommended context is enforced at load) \|
	29	+\| `experimental` \| runs but unverified: architecture not in the llama.cpp known list, VLM type falling back to text, very low quantization on a large model \|
	30	+\| `not_recommended` \| fits only through heavy swap (≤ absolute limit but > budget) — not loadable without `force` \|
	31	+\| `incompatible` \| runtime missing, architecture unsupported by the installed mlx-lm/mlx-vlm, or > absolute limit \|
	32	+
	33	+## Quantization policy (Harvester scoring)
	34	+
	35	+≤ 8B prefer 8/6-bit · 10–20B Q6/Q5 · 20–40B Q5/Q4 · 40–80B Q4 · larger Q3 only when genuinely useful. Lower than that is penalized (`too_low`).
	36	+
	37	+## Model Harvester
	38	+
	39	+`POST /api/harvest/scan` lists trusted authors (MLX: `mlx-community`; GGUF: `unsloth`, `bartowski`, `ggml-org`, `lmstudio-community`), excludes ASR/TTS/image/video/NSFW/base/draft repos, estimates RAM from the listing config + safetensors metadata (MLX) or from the chosen GGUF file (one repo-tree call), evaluates compatibility, scores (popularity, recency, size, quantization policy), and dedupes by base model + runtime (only the best quantization per base model is proposed). Candidates land in the Harvester page with a suggested starter library (small/medium/large general, coding, reasoning, vision, embedding, reranker). Selected candidates are queued through the normal inspected download path.
	40	+
	41	+## Starter library on M1M64 (downloaded 2026-09-10, all mlx-community unless noted)
	42	+
	43	+\| slot \| model \| RAM est. \|
	44	+\|---\|---\|---\|
	45	+\| tiny test \| Qwen3-4B-Instruct-2507-4bit, Llama-3.2-3B-Instruct-4bit, gemma-3-1b-it Q4_K_M (GGUF) \| 4–9 GB \|
	46	+\| small general / vision \| Qwen3.5-9B-MLX-4bit \| ~8 GB \|
	47	+\| medium general \| gemma-4-12B-it-qat-4bit \| ~10 GB \|
	48	+\| reasoning \| gpt-oss-20b-MXFP4-Q8 \| ~14 GB \|
	49	+\| coding \| Devstral-Small-2-24B-Instruct-2512-4bit, Qwen3-Coder-30B-A3B-Instruct-4bit \| 15–20 GB \|
	50	+\| large general \| Qwen3.8-27B-4bit, Qwen3.6-35B-A3B-4bit (MoE), gemma-4-26b-a4b-it-4bit (MoE) \| 17–24 GB \|
	51	+\| XL \| Llama-3.3-70B-Instruct-4bit \| ~42 GB (context ≤ 8K) \|
	52	+\| embedding \| Qwen3-Embedding-0.6B-8bit (MLX), embeddinggemma-300M (GGUF) \| < 1 GB, can stay resident \|
	53	+\| reranker \| Qwen3-Reranker-0.6B-4bit \| < 1 GB \|
	54	+
	55	+Versions verified against the installed runtimes: mlx 0.32.2, mlx-lm 0.31.3, mlx-vlm 0.7.0, llama.cpp 0.4.0 (Homebrew), huggingface-hub 1.31.0.

added docs/security.md +14 −0

		`@@ -0,0 +1,14 @@`
	1	+# Security
	2	+
	3	+- Private by default. The API binds `127.0.0.1:8300`; workers bind `127.0.0.1:8310-8399`; the console binds `127.0.0.1:8301`. Only the WireGuard-connected gateway (Caddy on BHS64) reaches port 8300 over the tunnel; nothing else is exposed.
	4	+- Admin auth: single admin user, argon2 password hashes, signed session cookie (`itsdangerous`, HttpOnly, Secure behind HTTPS, SameSite=Lax, 14 days). First user is seeded from `ADMIN_EMAIL`/`ADMIN_PASSWORD` or created once via `/api/auth/setup` (only when no user exists). Login rate limit 8/min/IP. Password change requires the current password.
	5	+- CSRF: session-authenticated mutations require header `X-LLM-CSRF: 1` and, when an `Origin` is present, it must match the public URL / host.
	6	+- API keys: `llm_live_` + 40 random chars, stored as SHA-256; only the prefix is displayed after creation. Scopes `inference` / `admin`. Revocation is immediate. Per-key last-used and request counters.
	7	+- Request validation: JSON bodies ≤ 20 MB (middleware + parser), sampling ranges checked, ≤ 8 stop sequences, ≤ 256 embedding inputs, ≤ 200 rerank documents, `n=1` only.
	8	+- Path safety: repository ids validated (`org/name`, no `..`); model deletion refuses paths outside the model root; scanned paths are only read.
	9	+- No shell from the API: workers are started with `subprocess.Popen(list)` (no shell); user-supplied extra llama args are filtered (`\|;&$\`` rejected) and only settable by an admin.
	10	+- Secrets: `.env` is `600` and outside the repo; `HF_TOKEN` only reaches the downloader; the session secret is generated into `data/.secret` (600). Keys, passwords and tokens are never logged.
	11	+- Prompt privacy: `LOG_PROMPTS=false` by default; when enabled, prompts/completions are stored truncated in `inference_requests`.
	12	+- Audit log: login/failed login/password change, key create/revoke, model load/unload/update/delete, downloads, harvest scans, settings changes, server start/stop — with actor and IP.
	13	+- Headers: `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: same-origin`, `Cache-Control: no-store` on API paths. Console is `noindex`.
	14	+- Memory safety is part of security here: a model that would exhaust unified memory is refused before any allocation; swap use raises a warning alert; workers are killed on unload so memory always returns.

added docs/troubleshooting.md +32 −0

		`@@ -0,0 +1,32 @@`
	1	+# Troubleshooting
	2	+
	3	+\| Symptom \| Where to look \| Likely cause / fix \|
	4	+\|---\|---\|---\|
	5	+\| `MODEL_NOT_FOUND` \| `GET /v1/models`, Models page \| id/alias typo; model files missing (`installed=false` after rescan) \|
	6	+\| `MODEL_TOO_LARGE` / `not_recommended` \| model page → Memory by context \| budget 45 GB exceeded; lower context (`overrides.context`), use a smaller quant, or raise the budget in Settings (never above RAM − 8 GB) \|
	7	+\| `MODEL_INCOMPATIBLE` (architecture) \| `pip show mlx-lm`, `llama-server --version` \| runtime too old for the architecture: upgrade mlx-lm / llama.cpp, rescan \|
	8	+\| `MODEL_LOAD_FAILED` with log tail \| `~/llm-api/logs/workers/worker-<id>.log` \| corrupted download (re-download with force), missing tokenizer file, unsupported quant \|
	9	+\| `MODEL_LOAD_TIMEOUT` \| same log + Activity Monitor \| very large model on a cold SSD cache; raise `LOAD_TIMEOUT_SECONDS` \|
	10	+\| `WORKER_CRASHED` \| worker log, `dmesg`-style Metal errors \| out-of-memory at the Metal level → lower context / KV bits (`overrides.kv_bits=8`) \|
	11	+\| `CONTEXT_TOO_LARGE` \| request \| prompt longer than the loaded context; reload with a larger `context` (model page) \|
	12	+\| `GENERATION_TIMEOUT` \| Settings \| `GENERATION_TIMEOUT_SECONDS`; check thermal throttling in System \|
	13	+\| Swap warning / memory pressure `critical` \| Dashboard alert, System \| model exceeds the envelope; unload, lower budget; never leave it swapping \|
	14	+\| Download fails 401/403 \| Settings → HF token \| gated repo: set `HF_TOKEN`, accept the license on Hugging Face \|
	15	+\| Download fails 429 \| Downloads history \| HF rate limit: retry later \|
	16	+\| `INSUFFICIENT_DISK` \| System → Storage \| `MIN_FREE_DISK_GB` reserve; delete unused models \|
	17	+\| Dashboard 503 "not reachable" \| `pm2 logs llm-api-web`, port 8301 \| Next not running / build failed (`pnpm build`) \|
	18	+\| Login loop \| cookies \| behind HTTPS set `SECURE_COOKIES=true`; ensure the proxy passes `X-Forwarded-Proto` \|
	19	+\| Model shows loaded after a crash \| restart server \| stale state is cleared at startup (`workers.json`) \|
	20	+\| GPU shows `—` \| `ioreg -r -c IOAccelerator` \| metric unavailable on this macOS; degrades gracefully \|
	21	+
	22	+Logs: `~/llm-api/logs/llm-api.log` (server), `~/llm-api/logs/workers/*.log` (per model), `pm2 logs`. Events: `/api/logs/events`, audit `/api/logs/audit`.
	23	+
	24	+Handy commands on the node:
	25	+
	26	+```bash
	27	+source ~/apps/llm-api/server/.venv/bin/activate && cd ~/llm-api
	28	+llm-api status # hardware + telemetry + model count
	29	+llm-api scan # rescan and list registry with compatibility
	30	+pgrep -fl "mlx_worker\|llama-server"
	31	+curl -s localhost:8300/health
	32	+```

added scripts/install.sh +98 −0

		`@@ -0,0 +1,98 @@`
	1	+#!/usr/bin/env bash
	2	+# LLM API — install on a clean Apple Silicon Mac.
	3	+# Usage: ./scripts/install.sh [--root ~/llm-api] [--no-llama] [--no-web] [--test]
	4	+set -euo pipefail
	5	+
	6	+ROOT="${LLM_API_ROOT:-$HOME/llm-api}"
	7	+WITH_LLAMA=1
	8	+WITH_WEB=1
	9	+RUN_TEST=0
	10	+for a in "$@"; do
	11	+ case "$a" in
	12	+ --root) shift; ROOT="$1";;
	13	+ --root=*) ROOT="${a#--root=}";;
	14	+ --no-llama) WITH_LLAMA=0;;
	15	+ --no-web) WITH_WEB=0;;
	16	+ --test) RUN_TEST=1;;
	17	+ esac
	18	+done
	19	+HERE="$(cd "$(dirname "$0")/.." && pwd)"
	20	+say() { printf "\033[1;34m==>\033[0m %s\n" "$*"; }
	21	+warn() { printf "\033[1;33m!!\033[0m %s\n" "$*"; }
	22	+
	23	+# 1. platform checks -------------------------------------------------------
	24	+[ "$(uname -s)" = "Darwin" ] \|\| { echo "macOS required"; exit 1; }
	25	+[ "$(uname -m)" = "arm64" ] \|\| { echo "Apple Silicon (arm64) required"; exit 1; }
	26	+CHIP=$(sysctl -n machdep.cpu.brand_string)
	27	+MEM_GB=$(( $(sysctl -n hw.memsize) / 1024 / 1024 / 1024 ))
	28	+FREE_GB=$(df -g "$HOME" \| awk 'NR==2 {print $4}')
	29	+say "Detected $CHIP, ${MEM_GB} GB unified memory, ${FREE_GB} GB free on $HOME"
	30	+[ "$MEM_GB" -ge 16 ] \|\| warn "Less than 16 GB of memory: only small models will run."
	31	+[ "$FREE_GB" -ge 120 ] \|\| warn "Less than 120 GB free: downloads stop at the MIN_FREE_DISK_GB reserve."
	32	+export PATH="/opt/homebrew/bin:$HOME/.local/bin:$PATH"
	33	+
	34	+# 2. toolchain ---------------------------------------------------------------
	35	+xcode-select -p >/dev/null 2>&1 \|\| { say "Installing Xcode Command Line Tools (a dialog may appear)"; xcode-select --install \|\| true; }
	36	+command -v brew >/dev/null \|\| { say "Installing Homebrew"; /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"; eval "$(/opt/homebrew/bin/brew shellenv)"; }
	37	+command -v uv >/dev/null \|\| { say "Installing uv"; brew install uv; }
	38	+if [ "$WITH_WEB" = 1 ]; then
	39	+ command -v node >/dev/null \|\| { say "Installing Node"; brew install node; }
	40	+ command -v pnpm >/dev/null \|\| { say "Installing pnpm"; brew install pnpm; }
	41	+fi
	42	+if [ "$WITH_LLAMA" = 1 ]; then
	43	+ command -v llama-server >/dev/null \|\| { say "Installing llama.cpp (Metal)"; brew install llama.cpp; }
	44	+fi
	45	+
	46	+# 3. directories ---------------------------------------------------------------
	47	+say "Creating $ROOT"
	48	+mkdir -p "$ROOT"/{models/{mlx,gguf,embeddings,rerankers,vision,manifests},data,logs}
	49	+chmod 700 "$ROOT/data"
	50	+
	51	+# 4. python env -----------------------------------------------------------------
	52	+cd "$HERE/server"
	53	+if [ ! -x .venv/bin/python ]; then
	54	+ say "Creating Python 3.13 virtualenv"
	55	+ uv venv --python 3.13 .venv
	56	+fi
	57	+say "Installing Python dependencies (MLX, MLX-LM, mlx-vlm, FastAPI…)"
	58	+uv pip install --python .venv/bin/python -q -e ".[mlx,test]" "mlx-vlm>=0.7" asgi-lifespan
	59	+.venv/bin/python - <<'PY'
	60	+import mlx.core as mx, mlx_lm
	61	+print(f" mlx {mx.__version__} · mlx-lm {mlx_lm.__version__} · Metal device: {mx.default_device()}")
	62	+PY
	63	+
	64	+# 5. web ------------------------------------------------------------------------
	65	+if [ "$WITH_WEB" = 1 ]; then
	66	+ say "Installing and building the dashboard"
	67	+ (cd "$HERE/web" && pnpm install --frozen-lockfile --silent && NEXT_TELEMETRY_DISABLED=1 pnpm build \| tail -3)
	68	+fi
	69	+
	70	+# 6. env file (never overwrite) ----------------------------------------------------
	71	+if [ ! -f "$ROOT/.env" ]; then
	72	+ say "Writing $ROOT/.env from .env.example (edit ADMIN_EMAIL / ADMIN_PASSWORD / HF_TOKEN)"
	73	+ sed "s#/Users/USERNAME/llm-api#$ROOT#" "$HERE/.env.example" > "$ROOT/.env"
	74	+ chmod 600 "$ROOT/.env"
	75	+else
	76	+ say "$ROOT/.env already exists — left untouched"
	77	+fi
	78	+
	79	+# 7. database + scan ------------------------------------------------------------------
	80	+say "Initializing database and scanning $ROOT/models"
	81	+(cd "$ROOT" && "$HERE/server/.venv/bin/llm-api" scan \| tail -5)
	82	+
	83	+# 8. optional local inference test -------------------------------------------------------
	84	+if [ "$RUN_TEST" = 1 ]; then
	85	+ say "Downloading a tiny test model and running one generation"
	86	+ "$HERE/server/.venv/bin/hf" download mlx-community/Qwen3-0.6B-4bit --local-dir "$ROOT/models/mlx/qwen/Qwen3-0.6B-4bit" >/dev/null
	87	+ "$HERE/server/.venv/bin/python" -m mlx_lm generate --model "$ROOT/models/mlx/qwen/Qwen3-0.6B-4bit" --prompt "Say hello in one word." --max-tokens 8 \| tail -3
	88	+fi
	89	+
	90	+cat <<EOF
	91	+
	92	+Done. Next steps:
	93	+ 1. Edit $ROOT/.env (ADMIN_EMAIL, ADMIN_PASSWORD, HF_TOKEN, PUBLIC_URL)
	94	+ 2. Start the API: cd $ROOT && $HERE/server/.venv/bin/llm-api serve
	95	+ 3. Start the dashboard: cd $HERE/web && pnpm start -p 8301 -H 127.0.0.1
	96	+ 4. Open http://127.0.0.1:8300 — or put a TLS reverse proxy in front for https://$(grep -E '^DOMAIN=' "$ROOT/.env" \| cut -d= -f2)
	97	+ Process management: see docs/deployment.md (PM2 / launchd examples).
	98	+EOF

added server/llm_api/__init__.py +3 −0

		`@@ -0,0 +1,3 @@`
	1	+"""LLM API — private OpenAI-compatible local model server for Apple Silicon."""
	2	+
	3	+__version__ = "0.1.0"

added server/llm_api/api/__init__.py +0 −0

added server/llm_api/api/admin_routes.py +809 −0

		`@@ -0,0 +1,809 @@`
	1	+"""Private management API (/api/*): auth, models, system, runtime, downloads, harvest, keys, settings, events."""
	2	+
	3	+from __future__ import annotations
	4	+
	5	+import asyncio
	6	+import json
	7	+import time
	8	+from pathlib import Path
	9	+from typing import Any
	10	+
	11	+from fastapi import APIRouter, Depends, Query, Request, Response
	12	+from fastapi.responses import StreamingResponse
	13	+from pydantic import BaseModel, Field
	14	+
	15	+from ..auth import SESSION_COOKIE, Principal, get_auth, login_limiter, require_admin
	16	+from ..errors import APIError, AuthError, Conflict, ModelNotFound, RateLimited
	17	+from ..events import bus
	18	+from ..hardware import GB, detect_hardware
	19	+
	20	+router = APIRouter(prefix="/api")
	21	+
	22	+SETTINGS_KEYS = {
	23	+ "max_model_memory_gb": float, "absolute_max_memory_gb": float, "min_free_disk_gb": float,
	24	+ "model_idle_timeout_minutes": int, "max_simultaneous_models": int, "preferred_runtime": str,
	25	+ "default_model": str, "preload_model": str, "log_prompts": bool, "allow_downloads": bool,
	26	+ "allow_gguf": bool, "allow_mlx": bool, "default_context": int, "default_max_tokens": int,
	27	+ "benchmark_max_tokens": int, "benchmark_runs": int,
	28	+}
	29	+
	30	+
	31	+def _ip(request: Request) -> str:
	32	+ fwd = request.headers.get("x-forwarded-for")
	33	+ return fwd.split(",")[0].strip() if fwd else (request.client.host if request.client else "?")
	34	+
	35	+
	36	+def _actor(request: Request) -> str:
	37	+ p = getattr(request.state, "principal", None)
	38	+ return f"{p.kind}:{p.name}" if p else "anonymous"
	39	+
	40	+
	41	+# ---------------------------------------------------------------------------
	42	+# Auth
	43	+# ---------------------------------------------------------------------------
	44	+
	45	+
	46	+class LoginBody(BaseModel):
	47	+ email: str
	48	+ password: str
	49	+
	50	+
	51	+class SetupBody(BaseModel):
	52	+ email: str
	53	+ password: str = Field(min_length=10)
	54	+
	55	+
	56	+class PasswordBody(BaseModel):
	57	+ current_password: str
	58	+ new_password: str = Field(min_length=10)
	59	+
	60	+
	61	+def _set_cookie(response: Response, request: Request, token: str) -> None:
	62	+ settings = request.app.state.settings
	63	+ secure = settings.secure_cookies or request.url.scheme == "https" or request.headers.get("x-forwarded-proto") == "https"
	64	+ response.set_cookie(SESSION_COOKIE, token, httponly=True, secure=secure, samesite="lax",
	65	+ max_age=settings.session_hours * 3600, path="/")
	66	+
	67	+
	68	+@router.get("/auth/status")
	69	+async def auth_status(request: Request):
	70	+ auth = get_auth(request)
	71	+ from ..auth import principal_from_request
	72	+ p = await principal_from_request(request, auth)
	73	+ return {"needs_setup": (await auth.user_count()) == 0, "authenticated": bool(p and p.has("admin")),
	74	+ "principal": {"kind": p.kind, "name": p.name} if p else None}
	75	+
	76	+
	77	+@router.post("/auth/setup")
	78	+async def auth_setup(body: SetupBody, request: Request, response: Response):
	79	+ auth = get_auth(request)
	80	+ if await auth.user_count() > 0:
	81	+ raise Conflict("Setup already completed.")
	82	+ uid = await auth.create_user(body.email, body.password)
	83	+ token = auth.make_session(uid, body.email.lower())
	84	+ _set_cookie(response, request, token)
	85	+ await request.app.state.db.audit("auth.setup", actor=body.email, ip=_ip(request))
	86	+ return {"ok": True, "email": body.email.lower()}
	87	+
	88	+
	89	+@router.post("/auth/login")
	90	+async def auth_login(body: LoginBody, request: Request, response: Response):
	91	+ auth = get_auth(request)
	92	+ if not login_limiter.check(_ip(request)):
	93	+ raise RateLimited("Too many login attempts. Try again in a minute.")
	94	+ user = await auth.authenticate(body.email, body.password)
	95	+ if not user:
	96	+ await request.app.state.db.audit("auth.login_failed", actor=body.email, ip=_ip(request))
	97	+ raise AuthError("Invalid email or password.", code="INVALID_CREDENTIALS")
	98	+ token = auth.make_session(user["id"], user["email"])
	99	+ _set_cookie(response, request, token)
	100	+ await request.app.state.db.audit("auth.login", actor=user["email"], ip=_ip(request))
	101	+ return {"ok": True, "email": user["email"]}
	102	+
	103	+
	104	+@router.post("/auth/logout")
	105	+async def auth_logout(request: Request, response: Response):
	106	+ response.delete_cookie(SESSION_COOKIE, path="/")
	107	+ return {"ok": True}
	108	+
	109	+
	110	+@router.get("/auth/me")
	111	+async def auth_me(request: Request, p: Principal = Depends(require_admin)):
	112	+ return {"kind": p.kind, "name": p.name, "scopes": sorted(p.scopes)}
	113	+
	114	+
	115	+@router.post("/auth/password")
	116	+async def auth_password(body: PasswordBody, request: Request, p: Principal = Depends(require_admin)):
	117	+ auth = get_auth(request)
	118	+ if p.kind != "session":
	119	+ raise APIError("Password changes require a dashboard session.")
	120	+ user = await auth.authenticate(p.name, body.current_password)
	121	+ if not user:
	122	+ raise AuthError("Current password is incorrect.", code="INVALID_CREDENTIALS")
	123	+ await auth.change_password(user["id"], body.new_password)
	124	+ await request.app.state.db.audit("auth.password_changed", actor=p.name, ip=_ip(request))
	125	+ return {"ok": True}
	126	+
	127	+
	128	+# ---------------------------------------------------------------------------
	129	+# Models
	130	+# ---------------------------------------------------------------------------
	131	+
	132	+
	133	+def _decorate(manager, m: dict) -> dict:
	134	+ st = manager.status_of(m["id"])
	135	+ m["status"] = st
	136	+ m["loaded"] = st == "ready"
	137	+ lm = manager.loaded.get(m["id"])
	138	+ if lm:
	139	+ m["worker"] = lm.to_dict()
	140	+ p = manager.progress.get(m["id"])
	141	+ if p:
	142	+ m["progress"] = p
	143	+ return m
	144	+
	145	+
	146	+@router.get("/models")
	147	+async def list_models(request: Request, p: Principal = Depends(require_admin), include_missing: bool = False):
	148	+ manager = request.app.state.manager
	149	+ models = await manager.registry.list_models(include_missing=include_missing)
	150	+ aliases = await manager.registry.aliases()
	151	+ by_model: dict[str, list[str]] = {}
	152	+ for a, mid in aliases.items():
	153	+ by_model.setdefault(mid, []).append(a)
	154	+ for m in models:
	155	+ _decorate(manager, m)
	156	+ m["aliases"] = by_model.get(m["id"], [])
	157	+ return {"models": models, "aliases": aliases, "count": len(models)}
	158	+
	159	+
	160	+@router.get("/models/{model_id}")
	161	+async def get_model(model_id: str, request: Request, p: Principal = Depends(require_admin)):
	162	+ manager = request.app.state.manager
	163	+ m = await manager.registry.resolve(model_id)
	164	+ if not m:
	165	+ raise ModelNotFound(f"Model '{model_id}' not found.")
	166	+ _decorate(manager, m)
	167	+ db = request.app.state.db
	168	+ m["benchmarks"] = await db.fetchall("SELECT * FROM model_benchmarks WHERE model_id=? ORDER BY created_at DESC LIMIT 30", (m["id"],))
	169	+ for b in m["benchmarks"]:
	170	+ for k in ("params", "notes"):
	171	+ if b.get(k):
	172	+ try:
	173	+ b[k] = json.loads(b[k])
	174	+ except Exception:
	175	+ pass
	176	+ m["events"] = await db.fetchall("SELECT * FROM model_events WHERE model_id=? ORDER BY created_at DESC LIMIT 50", (m["id"],))
	177	+ m["aliases"] = [a for a, mid in (await manager.registry.aliases()).items() if mid == m["id"]]
	178	+ m["recent_requests"] = await db.fetchall(
	179	+ "SELECT created_at, endpoint, prompt_tokens, completion_tokens, ttft_ms, total_ms, tps, status FROM inference_requests "
	180	+ "WHERE model_id=? ORDER BY created_at DESC LIMIT 20", (m["id"],))
	181	+ from ..models.estimator import CONTEXT_STEPS, estimate
	182	+ rt = m["runtime"]
	183	+ m["memory_curve"] = [estimate(m["weights_bytes"], rt, m["kv_bytes_per_token"] or 0, c, m["vision"]).to_dict()
	184	+ for c in CONTEXT_STEPS if not m["max_context"] or c <= max(m["max_context"], 2048)]
	185	+ return m
	186	+
	187	+
	188	+@router.get("/models/{model_id}/files")
	189	+async def model_files(model_id: str, request: Request, p: Principal = Depends(require_admin)):
	190	+ manager = request.app.state.manager
	191	+ m = await manager.registry.resolve(model_id)
	192	+ if not m:
	193	+ raise ModelNotFound(f"Model '{model_id}' not found.")
	194	+ return {"path": m["path"], "files": await manager.registry.files(m["id"])}
	195	+
	196	+
	197	+class LoadBody(BaseModel):
	198	+ context: int \| None = None
	199	+ force: bool = False
	200	+
	201	+
	202	+@router.post("/models/{model_id}/load")
	203	+async def load_model(model_id: str, request: Request, body: LoadBody \| None = None, p: Principal = Depends(require_admin)):
	204	+ manager = request.app.state.manager
	205	+ body = body or LoadBody()
	206	+ lm = await manager.load(model_id, context=body.context, force=body.force)
	207	+ await request.app.state.db.audit("model.load", actor=_actor(request), target=lm.model["id"], ip=_ip(request))
	208	+ return {"ok": True, "worker": lm.to_dict()}
	209	+
	210	+
	211	+@router.post("/models/{model_id}/unload")
	212	+async def unload_model(model_id: str, request: Request, p: Principal = Depends(require_admin)):
	213	+ manager = request.app.state.manager
	214	+ m = await manager.registry.resolve(model_id)
	215	+ if not m:
	216	+ raise ModelNotFound(f"Model '{model_id}' not found.")
	217	+ ok = await manager.unload(m["id"], reason="manual")
	218	+ await request.app.state.db.audit("model.unload", actor=_actor(request), target=m["id"], ip=_ip(request))
	219	+ return {"ok": ok, "status": manager.status_of(m["id"])}
	220	+
	221	+
	222	+class PatchModel(BaseModel):
	223	+ favorite: bool \| None = None
	224	+ pinned: bool \| None = None
	225	+ enabled: bool \| None = None
	226	+ notes: str \| None = None
	227	+ tags: list[str] \| None = None
	228	+ name: str \| None = None
	229	+ overrides: dict[str, Any] \| None = None
	230	+
	231	+
	232	+@router.patch("/models/{model_id}")
	233	+async def patch_model(model_id: str, body: PatchModel, request: Request, p: Principal = Depends(require_admin)):
	234	+ manager = request.app.state.manager
	235	+ m = await manager.registry.resolve(model_id)
	236	+ if not m:
	237	+ raise ModelNotFound(f"Model '{model_id}' not found.")
	238	+ fields = {k: v for k, v in body.model_dump().items() if v is not None}
	239	+ if "overrides" in fields:
	240	+ ov = fields["overrides"]
	241	+ allowed = {"task", "vision", "embedding", "reranker", "family", "quantization", "max_context", "context", "kv_bits",
	242	+ "parameter_count", "pooling", "llama_args", "thinking", "tools", "name"}
	243	+ fields["overrides"] = {k: v for k, v in ov.items() if k in allowed}
	244	+ if fields:
	245	+ await manager.registry.update(m["id"], **fields)
	246	+ if "overrides" in fields:
	247	+ await manager.registry.rescan()
	248	+ await manager.refresh_model(m["id"])
	249	+ await request.app.state.db.audit("model.update", actor=_actor(request), target=m["id"], detail=fields, ip=_ip(request))
	250	+ return _decorate(manager, await manager.registry.get(m["id"])) # type: ignore[arg-type]
	251	+
	252	+
	253	+@router.post("/models/{model_id}/pin")
	254	+async def pin_model(model_id: str, request: Request, p: Principal = Depends(require_admin), pinned: bool = True):
	255	+ manager = request.app.state.manager
	256	+ m = await manager.registry.resolve(model_id)
	257	+ if not m:
	258	+ raise ModelNotFound(f"Model '{model_id}' not found.")
	259	+ await manager.registry.update(m["id"], pinned=pinned)
	260	+ await manager.refresh_model(m["id"])
	261	+ return {"ok": True, "pinned": pinned}
	262	+
	263	+
	264	+@router.post("/models/{model_id}/favorite")
	265	+async def favorite_model(model_id: str, request: Request, p: Principal = Depends(require_admin), favorite: bool = True):
	266	+ manager = request.app.state.manager
	267	+ m = await manager.registry.resolve(model_id)
	268	+ if not m:
	269	+ raise ModelNotFound(f"Model '{model_id}' not found.")
	270	+ await manager.registry.update(m["id"], favorite=favorite)
	271	+ return {"ok": True, "favorite": favorite}
	272	+
	273	+
	274	+class DeleteBody(BaseModel):
	275	+ confirm: str
	276	+ keep_benchmarks: bool = True
	277	+
	278	+
	279	+@router.delete("/models/{model_id}")
	280	+async def delete_model(model_id: str, body: DeleteBody, request: Request, p: Principal = Depends(require_admin)):
	281	+ manager = request.app.state.manager
	282	+ m = await manager.registry.resolve(model_id)
	283	+ if not m:
	284	+ raise ModelNotFound(f"Model '{model_id}' not found.")
	285	+ if body.confirm != m["id"]:
	286	+ raise APIError("Type the exact model id in 'confirm' to delete it.", code="CONFIRMATION_REQUIRED")
	287	+ if manager.loaded.get(m["id"]):
	288	+ raise Conflict("Unload the model before deleting it.")
	289	+ return await request.app.state.downloader.delete_model(m["id"], actor=_actor(request), keep_benchmarks=body.keep_benchmarks)
	290	+
	291	+
	292	+@router.post("/models/rescan")
	293	+async def rescan(request: Request, p: Principal = Depends(require_admin)):
	294	+ state = request.app.state
	295	+
	296	+ async def run(job):
	297	+ def prog(frac, name):
	298	+ state.jobs.update(job, progress=frac, current=name)
	299	+ summary = await state.manager.registry.rescan(progress=prog)
	300	+ bus.publish("models", {"event": "rescan", **summary})
	301	+ return summary
	302	+
	303	+ job = state.jobs.submit("scan", "Rescan model directory", {}, run)
	304	+ # Scans are quick: wait a bit so callers get the result directly when possible
	305	+ for _ in range(100):
	306	+ await asyncio.sleep(0.1)
	307	+ if job.status in ("completed", "failed"):
	308	+ break
	309	+ return {"job": job.to_dict()}
	310	+
	311	+
	312	+class BenchBody(BaseModel):
	313	+ max_tokens: int = 256
	314	+ runs: int = 2
	315	+ long_prompt: bool = True
	316	+
	317	+
	318	+@router.post("/models/{model_id}/benchmark")
	319	+async def benchmark(model_id: str, request: Request, body: BenchBody \| None = None, p: Principal = Depends(require_admin)):
	320	+ state = request.app.state
	321	+ m = await state.manager.registry.resolve(model_id)
	322	+ if not m:
	323	+ raise ModelNotFound(f"Model '{model_id}' not found.")
	324	+ body = body or BenchBody()
	325	+ from ..bench import run_benchmark
	326	+
	327	+ async def run(job):
	328	+ return await run_benchmark(state, m["id"], job, body.model_dump())
	329	+
	330	+ job = state.jobs.submit("benchmark", f"Benchmark {m['name']}", {"model_id": m["id"], **body.model_dump()}, run)
	331	+ return {"job": job.to_dict()}
	332	+
	333	+
	334	+@router.get("/models/{model_id}/benchmarks")
	335	+async def benchmarks(model_id: str, request: Request, p: Principal = Depends(require_admin)):
	336	+ rows = await request.app.state.db.fetchall("SELECT * FROM model_benchmarks WHERE model_id=? ORDER BY created_at DESC LIMIT 100", (model_id,))
	337	+ for b in rows:
	338	+ for k in ("params", "notes"):
	339	+ if b.get(k):
	340	+ try:
	341	+ b[k] = json.loads(b[k])
	342	+ except Exception:
	343	+ pass
	344	+ return {"benchmarks": rows}
	345	+
	346	+
	347	+# ---------------------------------------------------------------------------
	348	+# Aliases
	349	+# ---------------------------------------------------------------------------
	350	+
	351	+
	352	+class AliasBody(BaseModel):
	353	+ alias: str = Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
	354	+ model_id: str
	355	+
	356	+
	357	+@router.get("/aliases")
	358	+async def list_aliases(request: Request, p: Principal = Depends(require_admin)):
	359	+ return {"aliases": await request.app.state.manager.registry.aliases()}
	360	+
	361	+
	362	+@router.put("/aliases")
	363	+async def put_alias(body: AliasBody, request: Request, p: Principal = Depends(require_admin)):
	364	+ reg = request.app.state.manager.registry
	365	+ m = await reg.get(body.model_id)
	366	+ if not m:
	367	+ raise ModelNotFound(f"Model '{body.model_id}' not found.")
	368	+ if await reg.get(body.alias):
	369	+ raise Conflict("An installed model already has this id.")
	370	+ if body.alias == "auto":
	371	+ raise APIError("'auto' is reserved.")
	372	+ await reg.set_alias(body.alias, m["id"])
	373	+ await request.app.state.db.audit("alias.set", actor=_actor(request), target=body.alias, detail=body.model_id)
	374	+ return {"aliases": await reg.aliases()}
	375	+
	376	+
	377	+@router.delete("/aliases/{alias}")
	378	+async def delete_alias(alias: str, request: Request, p: Principal = Depends(require_admin)):
	379	+ reg = request.app.state.manager.registry
	380	+ await reg.delete_alias(alias)
	381	+ return {"aliases": await reg.aliases()}
	382	+
	383	+
	384	+# ---------------------------------------------------------------------------
	385	+# System / runtime
	386	+# ---------------------------------------------------------------------------
	387	+
	388	+
	389	+@router.get("/system")
	390	+async def system(request: Request, p: Principal = Depends(require_admin)):
	391	+ state = request.app.state
	392	+ hw = detect_hardware(state.settings.models_dir).to_dict()
	393	+ tel = state.metrics.last or await state.metrics.sample()
	394	+ budget, absolute, max_models = await state.manager.budgets()
	395	+ return {"hardware": hw, "telemetry": tel, "manager": state.manager.snapshot(),
	396	+ "policy": {"max_model_memory_gb": budget, "absolute_max_memory_gb": absolute, "max_simultaneous_models": max_models,
	397	+ "macos_reserve_gb": state.settings.macos_reserve_gb,
	398	+ "min_free_disk_gb": float(await state.db.get_setting("min_free_disk_gb", state.settings.min_free_disk_gb))},
	399	+ "version": request.app.version, "started_at": state.metrics.started_at,
	400	+ "runtimes": await _runtime_versions()}
	401	+
	402	+
	403	+async def _runtime_versions() -> dict:
	404	+ out: dict[str, Any] = {}
	405	+ try:
	406	+ import mlx.core as mx
	407	+ import mlx_lm
	408	+ out["mlx"] = mx.__version__
	409	+ out["mlx_lm"] = mlx_lm.__version__
	410	+ except Exception:
	411	+ out["mlx"] = None
	412	+ try:
	413	+ import mlx_vlm
	414	+ out["mlx_vlm"] = mlx_vlm.__version__
	415	+ except Exception:
	416	+ out["mlx_vlm"] = None
	417	+ try:
	418	+ import shutil
	419	+ import subprocess
	420	+ b = shutil.which("llama-server") or "/opt/homebrew/bin/llama-server"
	421	+ r = subprocess.run([b, "--version"], capture_output=True, text=True, timeout=5)
	422	+ out["llama_cpp"] = (r.stdout + r.stderr).strip().splitlines()[0][:80] if (r.stdout or r.stderr) else None
	423	+ except Exception:
	424	+ out["llama_cpp"] = None
	425	+ return out
	426	+
	427	+
	428	+@router.get("/system/memory")
	429	+async def system_memory(request: Request, p: Principal = Depends(require_admin)):
	430	+ tel = await request.app.state.metrics.sample()
	431	+ return {k: tel[k] for k in tel if k.startswith(("mem_", "swap_", "worker_", "loaded_"))}
	432	+
	433	+
	434	+@router.get("/system/gpu")
	435	+async def system_gpu(request: Request, p: Principal = Depends(require_admin)):
	436	+ tel = await request.app.state.metrics.sample()
	437	+ hw = detect_hardware(request.app.state.settings.models_dir)
	438	+ return {"gpu_cores": hw.gpu_cores, "chip": hw.chip, "gpu_percent": tel["gpu_percent"],
	439	+ "gpu_renderer_percent": tel["gpu_renderer_percent"], "gpu_memory_gb": tel["gpu_memory_gb"],
	440	+ "thermal_state": tel["thermal_state"]}
	441	+
	442	+
	443	+@router.get("/system/storage")
	444	+async def system_storage(request: Request, p: Principal = Depends(require_admin)):
	445	+ state = request.app.state
	446	+ settings = state.settings
	447	+ import shutil
	448	+
	449	+ def du(path: Path) -> int:
	450	+ total = 0
	451	+ if path.exists():
	452	+ for f in path.rglob("*"):
	453	+ if f.is_file():
	454	+ try:
	455	+ total += f.stat().st_size
	456	+ except OSError:
	457	+ pass
	458	+ return total
	459	+
	460	+ usage = shutil.disk_usage(settings.models_dir)
	461	+ models_bytes, logs_bytes, db_bytes = await asyncio.gather(
	462	+ asyncio.to_thread(du, settings.models_dir), asyncio.to_thread(du, settings.logs_path),
	463	+ asyncio.to_thread(lambda: sum(du(p) for p in [settings.db_path] if p.exists()) + du(settings.data_path)))
	464	+ hf_cache = Path.home() / ".cache" / "huggingface"
	465	+ cache_bytes = await asyncio.to_thread(du, hf_cache)
	466	+ by_model = await state.db.fetchall("SELECT id, name, disk_size_bytes, runtime, last_used_at FROM models WHERE installed=1 ORDER BY disk_size_bytes DESC")
	467	+ return {"total_gb": round(usage.total / GB, 1), "used_gb": round(usage.used / GB, 1), "free_gb": round(usage.free / GB, 1),
	468	+ "models_gb": round(models_bytes / GB, 2), "logs_gb": round(logs_bytes / GB, 3), "database_gb": round(db_bytes / GB, 3),
	469	+ "cache_gb": round(cache_bytes / GB, 2), "other_gb": round(max(0, usage.used - models_bytes - logs_bytes - db_bytes - cache_bytes) / GB, 1),
	470	+ "model_root": str(settings.models_dir), "models": by_model,
	471	+ "min_free_gb": float(await state.db.get_setting("min_free_disk_gb", settings.min_free_disk_gb))}
	472	+
	473	+
	474	+@router.get("/system/processes")
	475	+async def system_processes(request: Request, p: Principal = Depends(require_admin)):
	476	+ import psutil
	477	+ manager = request.app.state.manager
	478	+ procs = []
	479	+ for lm in manager.loaded.values():
	480	+ try:
	481	+ pr = psutil.Process(lm.handle.pid)
	482	+ procs.append({"model_id": lm.model["id"], "pid": lm.handle.pid, "port": lm.handle.port, "runtime": lm.handle.runtime,
	483	+ "rss_gb": round(lm.handle.memory_bytes() / GB, 2), "cpu_percent": pr.cpu_percent(interval=None),
	484	+ "status": lm.status, "threads": pr.num_threads(), "created": pr.create_time()})
	485	+ except psutil.Error:
	486	+ pass
	487	+ me = psutil.Process()
	488	+ return {"workers": procs, "server": {"pid": me.pid, "rss_gb": round(me.memory_info().rss / GB, 3),
	489	+ "cpu_percent": me.cpu_percent(interval=None), "threads": me.num_threads()}}
	490	+
	491	+
	492	+@router.get("/system/health")
	493	+async def system_health_admin(request: Request):
	494	+ return await public_health(request)
	495	+
	496	+
	497	+async def public_health(request: Request) -> dict:
	498	+ state = request.app.state
	499	+ hw = detect_hardware(state.settings.models_dir)
	500	+ tel = state.metrics.last or await state.metrics.sample()
	501	+ cur = state.manager.current_model()
	502	+ return {"status": "ok", "hardware": {"chip": hw.chip, "memory_gb": hw.memory_gb, "gpu_cores": hw.gpu_cores},
	503	+ "model": {"loaded": cur is not None, "id": cur.model["id"] if cur else None,
	504	+ "loaded_models": [lm.model["id"] for lm in state.manager.loaded.values() if lm.status == "ready"]},
	505	+ "memory": {"used_gb": tel["mem_used_gb"], "available_gb": tel["mem_available_gb"], "pressure": tel["mem_pressure_level"]},
	506	+ "uptime_seconds": round(time.time() - state.metrics.started_at), "version": request.app.version}
	507	+
	508	+
	509	+@router.get("/system/metrics")
	510	+async def system_metrics(request: Request, p: Principal = Depends(require_admin), minutes: int = Query(60, ge=5, le=60 * 24 * 7),
	511	+ hours: int = Query(24, ge=1, le=24 * 30)):
	512	+ state = request.app.state
	513	+ return {"history": await state.metrics.history(minutes), "requests": await state.metrics.request_stats(hours),
	514	+ "live": state.metrics.last}
	515	+
	516	+
	517	+@router.get("/runtime/status")
	518	+async def runtime_status(request: Request, p: Principal = Depends(require_admin)):
	519	+ return request.app.state.manager.snapshot()
	520	+
	521	+
	522	+@router.get("/runtime/current-model")
	523	+async def runtime_current(request: Request, p: Principal = Depends(require_admin)):
	524	+ cur = request.app.state.manager.current_model()
	525	+ return {"model": cur.to_dict() if cur else None}
	526	+
	527	+
	528	+# ---------------------------------------------------------------------------
	529	+# Jobs / downloads / harvest
	530	+# ---------------------------------------------------------------------------
	531	+
	532	+
	533	+@router.get("/jobs")
	534	+async def jobs(request: Request, p: Principal = Depends(require_admin), kind: str \| None = None, limit: int = 100):
	535	+ kinds = {kind} if kind else None
	536	+ return {"jobs": await request.app.state.jobs.history(kinds, limit)}
	537	+
	538	+
	539	+@router.post("/jobs/{job_id}/cancel")
	540	+async def cancel_job(job_id: str, request: Request, p: Principal = Depends(require_admin)):
	541	+ ok = await request.app.state.jobs.cancel(job_id)
	542	+ return {"ok": ok}
	543	+
	544	+
	545	+class InspectBody(BaseModel):
	546	+ repository: str
	547	+ quant: str \| None = None
	548	+
	549	+
	550	+class DownloadBody(BaseModel):
	551	+ repository: str
	552	+ quant: str \| None = None
	553	+ force: bool = False
	554	+
	555	+
	556	+@router.post("/models/inspect")
	557	+async def inspect_repo(body: InspectBody, request: Request, p: Principal = Depends(require_admin)):
	558	+ return await request.app.state.downloader.inspect(body.repository, body.quant)
	559	+
	560	+
	561	+@router.post("/models/download")
	562	+async def download(body: DownloadBody, request: Request, p: Principal = Depends(require_admin)):
	563	+ job = await request.app.state.downloader.start_download(body.repository, body.quant, force=body.force, actor=_actor(request))
	564	+ return {"job": job.to_dict()}
	565	+
	566	+
	567	+@router.get("/downloads")
	568	+async def downloads(request: Request, p: Principal = Depends(require_admin)):
	569	+ return {"downloads": await request.app.state.jobs.history({"download"}, 100)}
	570	+
	571	+
	572	+@router.post("/downloads/{job_id}/retry")
	573	+async def retry_download(job_id: str, request: Request, p: Principal = Depends(require_admin)):
	574	+ rows = await request.app.state.db.fetchone("SELECT payload FROM jobs WHERE id=?", (job_id,))
	575	+ if not rows:
	576	+ raise APIError("Job not found.", status_code=404, code="JOB_NOT_FOUND")
	577	+ payload = json.loads(rows["payload"] or "{}")
	578	+ job = await request.app.state.downloader.start_download(payload["repository"], payload.get("quant"), force=True, actor=_actor(request))
	579	+ return {"job": job.to_dict()}
	580	+
	581	+
	582	+class HarvestBody(BaseModel):
	583	+ runtimes: list[str] \| None = None
	584	+ authors: list[str] \| None = None
	585	+ limit_per_author: int = 150
	586	+ min_downloads: int = 500
	587	+ max_ram_gb: float \| None = None
	588	+ families: list[str] \| None = None
	589	+ tasks: list[str] \| None = None
	590	+ search: str \| None = None
	591	+
	592	+
	593	+@router.post("/harvest/scan")
	594	+async def harvest_scan(body: HarvestBody, request: Request, p: Principal = Depends(require_admin)):
	595	+ job = await request.app.state.harvester.start_scan(body.model_dump(), actor=_actor(request))
	596	+ return {"job": job.to_dict()}
	597	+
	598	+
	599	+@router.get("/harvest/candidates")
	600	+async def harvest_candidates(request: Request, p: Principal = Depends(require_admin), include_duplicates: bool = False,
	601	+ task: str \| None = None, runtime: str \| None = None, family: str \| None = None,
	602	+ size_class: str \| None = None, q: str \| None = None, limit: int = 300):
	603	+ h = request.app.state.harvester
	604	+ rows = await h.candidates(include_duplicates=include_duplicates, task=task, runtime=runtime, family=family,
	605	+ size_class=size_class, q=q, limit=limit)
	606	+ last = await request.app.state.db.scalar("SELECT MAX(scanned_at) FROM harvest_candidates")
	607	+ return {"candidates": rows, "last_scan": last, "starter": await h.suggest_starter()}
	608	+
	609	+
	610	+class SelectBody(BaseModel):
	611	+ repo_id: str
	612	+ selected: bool = True
	613	+
	614	+
	615	+@router.post("/harvest/select")
	616	+async def harvest_select(body: SelectBody, request: Request, p: Principal = Depends(require_admin)):
	617	+ await request.app.state.harvester.select(body.repo_id, body.selected)
	618	+ return {"ok": True}
	619	+
	620	+
	621	+@router.post("/harvest/dismiss")
	622	+async def harvest_dismiss(body: SelectBody, request: Request, p: Principal = Depends(require_admin)):
	623	+ await request.app.state.harvester.dismiss(body.repo_id)
	624	+ return {"ok": True}
	625	+
	626	+
	627	+@router.post("/harvest/queue")
	628	+async def harvest_queue(request: Request, p: Principal = Depends(require_admin)):
	629	+ return {"queued": await request.app.state.harvester.queue_selected(actor=_actor(request))}
	630	+
	631	+
	632	+# ---------------------------------------------------------------------------
	633	+# API keys
	634	+# ---------------------------------------------------------------------------
	635	+
	636	+
	637	+class KeyBody(BaseModel):
	638	+ name: str = Field(min_length=1, max_length=64)
	639	+ scopes: list[str] = ["inference"]
	640	+
	641	+
	642	+@router.get("/keys")
	643	+async def list_keys(request: Request, p: Principal = Depends(require_admin)):
	644	+ rows = await request.app.state.db.fetchall(
	645	+ "SELECT id, name, prefix, scopes, created_at, last_used_at, request_count, revoked_at FROM api_keys ORDER BY created_at DESC")
	646	+ for r in rows:
	647	+ r["scopes"] = r["scopes"].split(",")
	648	+ return {"keys": rows}
	649	+
	650	+
	651	+@router.post("/keys")
	652	+async def create_key(body: KeyBody, request: Request, p: Principal = Depends(require_admin)):
	653	+ scopes = [s for s in body.scopes if s in ("inference", "admin")] or ["inference"]
	654	+ raw, row = await get_auth(request).create_key(body.name, scopes)
	655	+ await request.app.state.db.audit("key.create", actor=_actor(request), target=body.name, ip=_ip(request))
	656	+ row = dict(row)
	657	+ row["scopes"] = row["scopes"].split(",")
	658	+ return {"key": raw, "record": row}
	659	+
	660	+
	661	+class KeyPatch(BaseModel):
	662	+ name: str \| None = None
	663	+
	664	+
	665	+@router.patch("/keys/{key_id}")
	666	+async def rename_key(key_id: int, body: KeyPatch, request: Request, p: Principal = Depends(require_admin)):
	667	+ if body.name:
	668	+ await request.app.state.db.execute("UPDATE api_keys SET name=? WHERE id=?", (body.name, key_id))
	669	+ return {"ok": True}
	670	+
	671	+
	672	+@router.delete("/keys/{key_id}")
	673	+async def revoke_key(key_id: int, request: Request, p: Principal = Depends(require_admin)):
	674	+ await request.app.state.db.execute("UPDATE api_keys SET revoked_at=? WHERE id=? AND revoked_at IS NULL", (time.time(), key_id))
	675	+ await request.app.state.db.audit("key.revoke", actor=_actor(request), target=str(key_id), ip=_ip(request))
	676	+ return {"ok": True}
	677	+
	678	+
	679	+# ---------------------------------------------------------------------------
	680	+# Settings / logs / events
	681	+# ---------------------------------------------------------------------------
	682	+
	683	+
	684	+@router.get("/settings")
	685	+async def get_settings_(request: Request, p: Principal = Depends(require_admin)):
	686	+ state = request.app.state
	687	+ s = state.settings
	688	+ defaults = {
	689	+ "max_model_memory_gb": s.max_model_memory_gb, "absolute_max_memory_gb": s.absolute_max_memory_gb,
	690	+ "min_free_disk_gb": s.min_free_disk_gb, "model_idle_timeout_minutes": s.model_idle_timeout_minutes,
	691	+ "max_simultaneous_models": s.max_simultaneous_models, "preferred_runtime": "mlx", "default_model": None,
	692	+ "preload_model": s.preload_model, "log_prompts": s.log_prompts, "allow_downloads": s.allow_downloads,
	693	+ "allow_gguf": s.enable_gguf, "allow_mlx": s.enable_mlx, "default_context": s.default_context,
	694	+ "default_max_tokens": s.default_max_tokens, "benchmark_max_tokens": 256, "benchmark_runs": 2,
	695	+ }
	696	+ stored = await state.db.all_settings()
	697	+ merged = {defaults, {k: v for k, v in stored.items() if k in defaults}}
	698	+ return {"settings": merged, "defaults": defaults, "paths": {"root": str(s.root), "models": str(s.models_dir),
	699	+ "data": str(s.data_path), "logs": str(s.logs_path), "db": str(s.db_path)},
	700	+ "hf_token_set": bool(s.hf_token), "public_url": s.public_url}
	701	+
	702	+
	703	+@router.patch("/settings")
	704	+async def patch_settings(request: Request, p: Principal = Depends(require_admin)):
	705	+ state = request.app.state
	706	+ body = await request.json()
	707	+ if not isinstance(body, dict):
	708	+ raise APIError("Body must be an object.")
	709	+ changed = {}
	710	+ hw = detect_hardware(state.settings.models_dir)
	711	+ for k, v in body.items():
	712	+ if k not in SETTINGS_KEYS:
	713	+ continue
	714	+ typ = SETTINGS_KEYS[k]
	715	+ try:
	716	+ if typ is bool:
	717	+ v = bool(v)
	718	+ elif v is None or v == "":
	719	+ v = None
	720	+ else:
	721	+ v = typ(v)
	722	+ except (TypeError, ValueError):
	723	+ raise APIError(f"Invalid value for {k}.", param=k)
	724	+ if k in ("max_model_memory_gb", "absolute_max_memory_gb") and v is not None:
	725	+ if v < 1 or v > hw.memory_gb - 4:
	726	+ raise APIError(f"{k} must be between 1 and {hw.memory_gb - 4:.0f} GB on this machine.", param=k)
	727	+ if k == "max_simultaneous_models" and v is not None and (v < 1 or v > 4):
	728	+ raise APIError("max_simultaneous_models must be 1-4.", param=k)
	729	+ changed[k] = v
	730	+ await state.db.set_setting(k, v)
	731	+ if changed:
	732	+ await state.db.audit("settings.update", actor=_actor(request), detail=changed, ip=_ip(request))
	733	+ if any(k in changed for k in ("max_model_memory_gb", "absolute_max_memory_gb")):
	734	+ await state.manager.registry.reevaluate_all()
	735	+ bus.publish("settings", changed)
	736	+ return {"changed": changed}
	737	+
	738	+
	739	+@router.get("/logs/audit")
	740	+async def audit_logs(request: Request, p: Principal = Depends(require_admin), limit: int = 200):
	741	+ rows = await request.app.state.db.fetchall("SELECT * FROM audit_logs ORDER BY created_at DESC LIMIT ?", (limit,))
	742	+ return {"logs": rows}
	743	+
	744	+
	745	+@router.get("/logs/events")
	746	+async def model_events(request: Request, p: Principal = Depends(require_admin), limit: int = 200):
	747	+ rows = await request.app.state.db.fetchall("SELECT * FROM model_events ORDER BY created_at DESC LIMIT ?", (limit,))
	748	+ return {"events": rows}
	749	+
	750	+
	751	+@router.get("/logs/requests")
	752	+async def request_logs(request: Request, p: Principal = Depends(require_admin), limit: int = 200):
	753	+ rows = await request.app.state.db.fetchall(
	754	+ "SELECT id, created_at, model_id, requested_model, endpoint, api_key_id, stream, prompt_tokens, completion_tokens, ttft_ms, "
	755	+ "total_ms, tps, load_wait_ms, status, error_code FROM inference_requests ORDER BY created_at DESC LIMIT ?", (limit,))
	756	+ return {"requests": rows}
	757	+
	758	+
	759	+@router.get("/logs/worker/{model_id}")
	760	+async def worker_log(model_id: str, request: Request, p: Principal = Depends(require_admin), lines: int = 200):
	761	+ path = request.app.state.settings.logs_path / "workers" / f"worker-{model_id}.log"
	762	+ if not path.exists():
	763	+ return {"lines": []}
	764	+ txt = path.read_text(errors="replace").splitlines()
	765	+ return {"lines": txt[-lines:]}
	766	+
	767	+
	768	+@router.get("/events")
	769	+async def events(request: Request, p: Principal = Depends(require_admin)):
	770	+ q = bus.subscribe()
	771	+ state = request.app.state
	772	+
	773	+ async def gen():
	774	+ try:
	775	+ # initial snapshot
	776	+ snap = {"seq": 0, "ts": time.time(), "type": "snapshot",
	777	+ "data": {"manager": state.manager.snapshot(), "metrics": state.metrics.last,
	778	+ "jobs": state.jobs.list(limit=30)}}
	779	+ yield bus.format_sse(snap)
	780	+ while True:
	781	+ try:
	782	+ ev = await asyncio.wait_for(q.get(), timeout=15)
	783	+ yield bus.format_sse(ev)
	784	+ except asyncio.TimeoutError:
	785	+ yield ": ping\n\n"
	786	+ if await request.is_disconnected():
	787	+ break
	788	+ finally:
	789	+ bus.unsubscribe(q)
	790	+
	791	+ return StreamingResponse(gen(), media_type="text/event-stream",
	792	+ headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})
	793	+
	794	+
	795	+@router.post("/models/{model_id}/tokenize")
	796	+async def tokenize(model_id: str, request: Request, p: Principal = Depends(require_admin)):
	797	+ manager = request.app.state.manager
	798	+ lm = manager.get_ready(model_id)
	799	+ if not lm:
	800	+ raise Conflict("Model is not loaded.")
	801	+ body = await request.json()
	802	+ if lm.handle.runtime == "mlx":
	803	+ r = await manager.client.post(f"{lm.handle.base_url}/tokenize", json=body, timeout=30)
	804	+ else:
	805	+ text = body.get("text") or " ".join(m.get("content", "") for m in body.get("messages", []) if isinstance(m.get("content"), str))
	806	+ r = await manager.client.post(f"{lm.handle.base_url}/tokenize", json={"content": text}, timeout=30)
	807	+ d = r.json()
	808	+ return {"tokens": len(d.get("tokens", [])), "max_context": lm.handle.context}
	809	+ return r.json()

added server/llm_api/api/openai_routes.py +369 −0

		`@@ -0,0 +1,369 @@`
	1	+"""OpenAI-compatible endpoints. Requests are validated, the model is loaded on demand, then the
	2	+request is proxied to the local worker. Usage/timings are recorded from the worker's response."""
	3	+
	4	+from __future__ import annotations
	5	+
	6	+import asyncio
	7	+import json
	8	+import logging
	9	+import time
	10	+from typing import Any
	11	+
	12	+import httpx
	13	+from fastapi import APIRouter, Depends, Request
	14	+from fastapi.responses import JSONResponse, StreamingResponse
	15	+
	16	+from ..auth import Principal, require_inference
	17	+from ..errors import APIError, GenerationTimeout, WorkerCrashed
	18	+from ..events import bus
	19	+from ..manager import ModelManager
	20	+
	21	+log = logging.getLogger("llm_api.openai")
	22	+router = APIRouter()
	23	+
	24	+
	25	+def _manager(request: Request) -> ModelManager:
	26	+ return request.app.state.manager
	27	+
	28	+
	29	+async def _parse_body(request: Request) -> dict:
	30	+ settings = request.app.state.settings
	31	+ raw = await request.body()
	32	+ if len(raw) > settings.max_body_bytes:
	33	+ raise APIError("Request body too large.", status_code=413, code="BODY_TOO_LARGE")
	34	+ try:
	35	+ body = json.loads(raw or b"{}")
	36	+ except json.JSONDecodeError:
	37	+ raise APIError("Request body is not valid JSON.")
	38	+ if not isinstance(body, dict):
	39	+ raise APIError("Request body must be a JSON object.")
	40	+ return body
	41	+
	42	+
	43	+def _validate_sampling(body: dict) -> None:
	44	+ def num(k, lo, hi):
	45	+ v = body.get(k)
	46	+ if v is None:
	47	+ return
	48	+ if not isinstance(v, (int, float)) or v < lo or v > hi:
	49	+ raise APIError(f"{k} must be a number between {lo} and {hi}.", param=k)
	50	+ num("temperature", 0, 2)
	51	+ num("top_p", 0, 1)
	52	+ num("presence_penalty", -2, 2)
	53	+ num("frequency_penalty", -2, 2)
	54	+ mt = body.get("max_tokens", body.get("max_completion_tokens"))
	55	+ if mt is not None and (not isinstance(mt, int) or mt < 1 or mt > 200000):
	56	+ raise APIError("max_tokens must be a positive integer.", param="max_tokens")
	57	+ n = body.get("n")
	58	+ if n not in (None, 1):
	59	+ raise APIError("Only n=1 is supported.", param="n")
	60	+ stop = body.get("stop")
	61	+ if stop is not None and not isinstance(stop, (str, list)):
	62	+ raise APIError("stop must be a string or an array of strings.", param="stop")
	63	+ if isinstance(stop, list) and len(stop) > 8:
	64	+ raise APIError("At most 8 stop sequences are supported.", param="stop")
	65	+
	66	+
	67	+async def _pick_model(request: Request, body: dict, endpoint: str) -> tuple[str, dict]:
	68	+ """Resolve model name (alias/auto) -> registry row."""
	69	+ manager = _manager(request)
	70	+ name = body.get("model")
	71	+ if not name or not isinstance(name, str):
	72	+ # default model setting
	73	+ name = await request.app.state.db.get_setting("default_model")
	74	+ if not name:
	75	+ cur = manager.current_model()
	76	+ if cur:
	77	+ name = cur.model["id"]
	78	+ if not name:
	79	+ raise APIError("model is required.", param="model")
	80	+ if name == "auto":
	81	+ from ..routing import choose_auto
	82	+ name = await choose_auto(request.app.state, body, endpoint)
	83	+ model = await manager.registry.resolve(name)
	84	+ if not model:
	85	+ from ..errors import ModelNotFound
	86	+ raise ModelNotFound(f"The model '{name}' does not exist. Use GET /v1/models to list available models.")
	87	+ return name, model
	88	+
	89	+
	90	+async def _record(app_state, *, model_id: str \| None, requested: str, endpoint: str, principal: Principal,
	91	+ stream: bool, usage: dict \| None, timings: dict \| None, status: int, error_code: str \| None,
	92	+ started: float, load_wait_ms: float, prompt: Any = None, completion: str \| None = None) -> None:
	93	+ db = app_state.db
	94	+ settings = app_state.settings
	95	+ log_prompts = bool(await db.get_setting("log_prompts", settings.log_prompts))
	96	+ usage = usage or {}
	97	+ timings = timings or {}
	98	+ tps = timings.get("generation_tps") or timings.get("predicted_per_second")
	99	+ ttft = timings.get("ttft_ms") or timings.get("prompt_ms")
	100	+ total_ms = (time.time() - started) * 1000
	101	+ ctoks = usage.get("completion_tokens") or timings.get("predicted_n")
	102	+ ptoks = usage.get("prompt_tokens") or timings.get("prompt_n")
	103	+ await db.execute(
	104	+ "INSERT INTO inference_requests(created_at, model_id, requested_model, endpoint, api_key_id, stream, prompt_tokens, "
	105	+ "completion_tokens, ttft_ms, total_ms, tps, load_wait_ms, status, error_code, prompt, completion) "
	106	+ "VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
	107	+ (started, model_id, requested, endpoint, principal.id if principal.kind == "api_key" else None, int(stream),
	108	+ ptoks, ctoks, ttft, round(total_ms, 1), tps, round(load_wait_ms, 1), status, error_code,
	109	+ json.dumps(prompt, ensure_ascii=False)[:20000] if (log_prompts and prompt is not None) else None,
	110	+ (completion or "")[:20000] if log_prompts else None))
	111	+ if model_id and status < 400:
	112	+ await app_state.manager.registry.record_usage(model_id, int(ctoks or 0), tps, ttft)
	113	+ app_state.manager.stats["tokens"] += int(ctoks or 0)
	114	+ bus.publish("request", {"model_id": model_id, "endpoint": endpoint, "status": status, "tps": tps, "ttft_ms": ttft,
	115	+ "completion_tokens": ctoks, "prompt_tokens": ptoks, "total_ms": round(total_ms), "stream": stream})
	116	+
	117	+
	118	+def _extract_timings(data: dict) -> dict:
	119	+ t = data.get("timings") or {}
	120	+ if "predicted_per_second" in t and "generation_tps" not in t: # llama.cpp
	121	+ t = {"generation_tps": t.get("predicted_per_second"), "prompt_tps": t.get("prompt_per_second"),
	122	+ "ttft_ms": t.get("prompt_ms"), "prompt_ms": t.get("prompt_ms"), "generation_ms": t.get("predicted_ms"),
	123	+ "predicted_n": t.get("predicted_n"), "prompt_n": t.get("prompt_n"), "cached_tokens": t.get("cache_n")}
	124	+ return t
	125	+
	126	+
	127	+async def _proxy(request: Request, endpoint: str, body: dict, principal: Principal, *, kind: str):
	128	+ """Common path for chat/completions/embeddings/rerank."""
	129	+ app_state = request.app.state
	130	+ manager: ModelManager = app_state.manager
	131	+ started = time.time()
	132	+ requested, model = await _pick_model(request, body, endpoint)
	133	+ if kind == "chat" or kind == "completion":
	134	+ if model.get("embedding") or model.get("reranker"):
	135	+ raise APIError(f"Model '{model['id']}' is an {'embedding' if model.get('embedding') else 'reranking'} model and cannot generate text.",
	136	+ code="WRONG_MODEL_TYPE")
	137	+ if kind == "embeddings" and not model.get("embedding"):
	138	+ # Allow causal LMs served by MLX to embed (last-token pooling) only if explicitly tagged; otherwise reject
	139	+ if model["runtime"] != "mlx":
	140	+ raise APIError(f"Model '{model['id']}' is not an embedding model.", code="WRONG_MODEL_TYPE")
	141	+ if kind == "rerank" and not model.get("reranker"):
	142	+ raise APIError(f"Model '{model['id']}' is not a reranking model.", code="WRONG_MODEL_TYPE")
	143	+ stream = bool(body.get("stream")) and kind in ("chat", "completion")
	144	+ t_load = time.time()
	145	+ try:
	146	+ lm = await manager.ensure_loaded(model["id"], reason=f"{endpoint}")
	147	+ except APIError as e:
	148	+ await _record(app_state, model_id=model["id"], requested=requested, endpoint=endpoint, principal=principal,
	149	+ stream=stream, usage=None, timings=None, status=e.status_code, error_code=e.code, started=started,
	150	+ load_wait_ms=(time.time() - t_load) * 1000)
	151	+ raise
	152	+ load_wait_ms = (time.time() - t_load) * 1000
	153	+ body = dict(body)
	154	+ body["model"] = model["id"]
	155	+ if stream and kind == "chat":
	156	+ so = dict(body.get("stream_options") or {})
	157	+ so["include_usage"] = True
	158	+ body["stream_options"] = so
	159	+ url = f"{lm.handle.base_url}{endpoint}"
	160	+ timeout = httpx.Timeout(connect=10.0, read=float(app_state.settings.generation_timeout_seconds), write=60.0, pool=10.0)
	161	+ prompt_for_log = body.get("messages") or body.get("prompt") or body.get("input")
	162	+
	163	+ async with manager.use(lm):
	164	+ if not stream:
	165	+ try:
	166	+ r = await manager.client.post(url, json=body, timeout=timeout)
	167	+ except httpx.ReadTimeout:
	168	+ raise GenerationTimeout("The model did not finish generating in time.")
	169	+ except httpx.HTTPError as e:
	170	+ if not lm.handle.alive():
	171	+ raise WorkerCrashed(f"The inference worker for '{model['id']}' crashed during the request.")
	172	+ raise APIError(f"Worker connection error: {e}", status_code=502, code="WORKER_UNREACHABLE", error_type="runtime_error")
	173	+ try:
	174	+ data = r.json()
	175	+ except Exception:
	176	+ data = {"error": {"message": r.text[:500], "type": "runtime_error", "code": "WORKER_BAD_RESPONSE"}}
	177	+ if r.status_code >= 400:
	178	+ err = data.get("error") if isinstance(data, dict) else None
	179	+ code = (err or {}).get("code") if isinstance(err, dict) else None
	180	+ msg = (err or {}).get("message") if isinstance(err, dict) else str(err)
	181	+ await _record(app_state, model_id=model["id"], requested=requested, endpoint=endpoint, principal=principal,
	182	+ stream=False, usage=None, timings=None, status=r.status_code, error_code=str(code or "WORKER_ERROR"),
	183	+ started=started, load_wait_ms=load_wait_ms, prompt=prompt_for_log)
	184	+ # normalise llama.cpp error shape
	185	+ return JSONResponse(status_code=r.status_code, content={"error": {"message": msg or "worker error",
	186	+ "type": (err or {}).get("type", "runtime_error") if isinstance(err, dict) else "runtime_error",
	187	+ "code": code or "WORKER_ERROR"}})
	188	+ data["model"] = requested if requested != "auto" else model["id"]
	189	+ timings = _extract_timings(data)
	190	+ if timings:
	191	+ data["timings"] = timings
	192	+ completion = None
	193	+ if kind == "chat":
	194	+ completion = ((data.get("choices") or [{}])[0].get("message") or {}).get("content")
	195	+ elif kind == "completion":
	196	+ completion = (data.get("choices") or [{}])[0].get("text")
	197	+ await _record(app_state, model_id=model["id"], requested=requested, endpoint=endpoint, principal=principal,
	198	+ stream=False, usage=data.get("usage"), timings=timings, status=200, error_code=None,
	199	+ started=started, load_wait_ms=load_wait_ms, prompt=prompt_for_log, completion=completion)
	200	+ return JSONResponse(content=data)
	201	+
	202	+ # ---- streaming ----------------------------------------------------
	203	+ req = manager.client.build_request("POST", url, json=body, timeout=timeout)
	204	+ try:
	205	+ resp = await manager.client.send(req, stream=True)
	206	+ except httpx.HTTPError as e:
	207	+ raise APIError(f"Worker connection error: {e}", status_code=502, code="WORKER_UNREACHABLE", error_type="runtime_error")
	208	+ if resp.status_code >= 400:
	209	+ raw = await resp.aread()
	210	+ await resp.aclose()
	211	+ try:
	212	+ data = json.loads(raw)
	213	+ except Exception:
	214	+ data = {"error": {"message": raw.decode(errors="replace")[:500], "type": "runtime_error", "code": "WORKER_ERROR"}}
	215	+ err = data.get("error") or {}
	216	+ await _record(app_state, model_id=model["id"], requested=requested, endpoint=endpoint, principal=principal,
	217	+ stream=True, usage=None, timings=None, status=resp.status_code, error_code=str(err.get("code") or "WORKER_ERROR"),
	218	+ started=started, load_wait_ms=load_wait_ms, prompt=prompt_for_log)
	219	+ return JSONResponse(status_code=resp.status_code, content={"error": {"message": err.get("message", "worker error"),
	220	+ "type": err.get("type", "runtime_error"),
	221	+ "code": err.get("code", "WORKER_ERROR")}})
	222	+
	223	+ lm.in_flight += 1 # held until the generator finishes
	224	+
	225	+ async def gen():
	226	+ usage = None
	227	+ timings: dict = {}
	228	+ completion_parts: list[str] = []
	229	+ status = 200
	230	+ error_code = None
	231	+ try:
	232	+ async for line in resp.aiter_lines():
	233	+ if not line:
	234	+ continue
	235	+ if line.startswith("data: "):
	236	+ payload = line[6:]
	237	+ if payload.strip() == "[DONE]":
	238	+ yield b"data: [DONE]\n\n"
	239	+ continue
	240	+ try:
	241	+ obj = json.loads(payload)
	242	+ except json.JSONDecodeError:
	243	+ yield (line + "\n\n").encode()
	244	+ continue
	245	+ if "error" in obj and "choices" not in obj:
	246	+ status = 500
	247	+ error_code = (obj["error"] or {}).get("code", "GENERATION_FAILED")
	248	+ yield (f"data: {json.dumps(obj)}\n\n").encode()
	249	+ continue
	250	+ obj["model"] = requested if requested != "auto" else model["id"]
	251	+ if obj.get("usage"):
	252	+ usage = obj["usage"]
	253	+ if obj.get("timings"):
	254	+ timings = _extract_timings(obj)
	255	+ obj["timings"] = timings
	256	+ for ch in obj.get("choices") or []:
	257	+ d = ch.get("delta") or {}
	258	+ if d.get("content"):
	259	+ completion_parts.append(d["content"])
	260	+ if ch.get("text"):
	261	+ completion_parts.append(ch["text"])
	262	+ yield (f"data: {json.dumps(obj, ensure_ascii=False)}\n\n").encode()
	263	+ else:
	264	+ yield (line + "\n").encode()
	265	+ except (httpx.ReadTimeout, asyncio.TimeoutError):
	266	+ status, error_code = 504, "GENERATION_TIMEOUT"
	267	+ yield (f"data: {json.dumps({'error': {'message': 'generation timed out', 'type': 'runtime_error', 'code': 'GENERATION_TIMEOUT'}})}\n\n").encode()
	268	+ except httpx.HTTPError:
	269	+ status, error_code = 502, "WORKER_CRASHED" if not lm.handle.alive() else "WORKER_STREAM_ERROR"
	270	+ yield (f"data: {json.dumps({'error': {'message': 'worker stream interrupted', 'type': 'runtime_error', 'code': error_code}})}\n\n").encode()
	271	+ finally:
	272	+ await resp.aclose()
	273	+ lm.in_flight = max(0, lm.in_flight - 1)
	274	+ lm.last_used = time.time()
	275	+ try:
	276	+ await _record(app_state, model_id=model["id"], requested=requested, endpoint=endpoint, principal=principal,
	277	+ stream=True, usage=usage, timings=timings, status=status, error_code=error_code,
	278	+ started=started, load_wait_ms=load_wait_ms, prompt=prompt_for_log,
	279	+ completion="".join(completion_parts))
	280	+ except Exception:
	281	+ log.exception("record failed")
	282	+
	283	+ return StreamingResponse(gen(), media_type="text/event-stream",
	284	+ headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "Connection": "keep-alive"})
	285	+
	286	+
	287	+# ---------------------------------------------------------------------------
	288	+
	289	+
	290	+@router.get("/v1/models")
	291	+async def list_models(request: Request, principal: Principal = Depends(require_inference)):
	292	+ manager = _manager(request)
	293	+ models = await manager.registry.list_models()
	294	+ aliases = await manager.registry.aliases()
	295	+ out = []
	296	+ for m in models:
	297	+ if not m.get("enabled"):
	298	+ continue
	299	+ st = manager.status_of(m["id"])
	300	+ out.append({
	301	+ "id": m["id"], "object": "model", "created": int(m["created_at"]), "owned_by": m.get("provider") or "local",
	302	+ "root": m["id"], "parent": None,
	303	+ # llm-api extensions (ignored by SDKs)
	304	+ "name": m["name"], "family": m["family"], "runtime": m["runtime"], "quantization": m["quantization"],
	305	+ "parameter_count": m["parameter_count"], "estimated_ram_gb": m["estimated_ram_gb"],
	306	+ "context": m["recommended_context"], "max_context": m["max_context"], "task": m["task"],
	307	+ "compatibility": m["compatibility_status"], "status": st, "loaded": st == "ready",
	308	+ "capabilities": {"vision": m["vision"], "embedding": m["embedding"], "reranking": m["reranker"],
	309	+ "tools": m["tools"], "thinking": m["thinking"]},
	310	+ })
	311	+ for alias, mid in aliases.items():
	312	+ out.append({"id": alias, "object": "model", "created": 0, "owned_by": "alias", "root": mid, "parent": mid,
	313	+ "alias_of": mid})
	314	+ return {"object": "list", "data": out}
	315	+
	316	+
	317	+@router.get("/v1/models/{model_id:path}")
	318	+async def get_model(model_id: str, request: Request, principal: Principal = Depends(require_inference)):
	319	+ manager = _manager(request)
	320	+ m = await manager.registry.resolve(model_id)
	321	+ if not m:
	322	+ from ..errors import ModelNotFound
	323	+ raise ModelNotFound(f"The model '{model_id}' does not exist.")
	324	+ return {"id": m["id"], "object": "model", "created": int(m["created_at"]), "owned_by": m.get("provider") or "local",
	325	+ "status": manager.status_of(m["id"])}
	326	+
	327	+
	328	+@router.post("/v1/chat/completions")
	329	+async def chat_completions(request: Request, principal: Principal = Depends(require_inference)):
	330	+ body = await _parse_body(request)
	331	+ msgs = body.get("messages")
	332	+ if not isinstance(msgs, list) or not msgs:
	333	+ raise APIError("messages must be a non-empty array.", param="messages")
	334	+ for m in msgs:
	335	+ if not isinstance(m, dict) or "role" not in m:
	336	+ raise APIError("Each message needs a role.", param="messages")
	337	+ _validate_sampling(body)
	338	+ return await _proxy(request, "/v1/chat/completions", body, principal, kind="chat")
	339	+
	340	+
	341	+@router.post("/v1/completions")
	342	+async def completions(request: Request, principal: Principal = Depends(require_inference)):
	343	+ body = await _parse_body(request)
	344	+ if "prompt" not in body:
	345	+ raise APIError("prompt is required.", param="prompt")
	346	+ _validate_sampling(body)
	347	+ return await _proxy(request, "/v1/completions", body, principal, kind="completion")
	348	+
	349	+
	350	+@router.post("/v1/embeddings")
	351	+async def embeddings(request: Request, principal: Principal = Depends(require_inference)):
	352	+ body = await _parse_body(request)
	353	+ if "input" not in body:
	354	+ raise APIError("input is required.", param="input")
	355	+ inp = body["input"]
	356	+ n = len(inp) if isinstance(inp, list) else 1
	357	+ if n > 256:
	358	+ raise APIError("At most 256 inputs per request.", param="input")
	359	+ return await _proxy(request, "/v1/embeddings", body, principal, kind="embeddings")
	360	+
	361	+
	362	+@router.post("/v1/rerank")
	363	+async def rerank(request: Request, principal: Principal = Depends(require_inference)):
	364	+ body = await _parse_body(request)
	365	+ if not body.get("query") or not isinstance(body.get("documents"), list):
	366	+ raise APIError("query and documents are required.")
	367	+ if len(body["documents"]) > 200:
	368	+ raise APIError("At most 200 documents per request.", param="documents")
	369	+ return await _proxy(request, "/v1/rerank", body, principal, kind="rerank")

added server/llm_api/auth.py +210 −0

Large diff (210 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/bench.py +158 −0

Large diff (158 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/cli.py +72 −0

Large diff (72 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/config.py +122 −0

Large diff (122 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/db.py +300 −0

Large diff (300 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/downloads.py +341 −0

Large diff (341 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/errors.py +122 −0

Large diff (122 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/events.py +48 −0

Large diff (48 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/hardware.py +254 −0

Large diff (254 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/harvester.py +397 −0

Large diff (397 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/jobs.py +162 −0

Large diff (162 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/main.py +166 −0

Large diff (166 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/manager.py +506 −0

Large diff (506 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/metrics.py +98 −0

Large diff (98 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/proxy_ui.py +50 −0

Large diff (50 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/routing.py +70 −0

Large diff (70 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/runtimes/__init__.py +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/runtimes/base.py +149 −0

Large diff (149 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/runtimes/llamacpp_adapter.py +69 −0

Large diff (69 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/runtimes/mlx_adapter.py +58 −0

Large diff (58 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/worker/__init__.py +0 −0

Large diff (0 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/worker/mlx_worker.py +1117 −0

Large diff (1,117 lines) is not loaded by default to keep the page fast.Load diff

added server/llm_api/worker/openai_types.py +239 −0

Large diff (239 lines) is not loaded by default to keep the page fast.Load diff

added server/pyproject.toml +48 −0

Large diff (48 lines) is not loaded by default to keep the page fast.Load diff

added server/tests/__init__.py +0 −0

Large diff (0 lines) is not loaded by default to keep the page fast.Load diff

added server/tests/conftest.py +294 −0

Large diff (294 lines) is not loaded by default to keep the page fast.Load diff

added server/tests/test_api.py +298 −0

Large diff (298 lines) is not loaded by default to keep the page fast.Load diff

added server/tests/test_units.py +140 −0

Large diff (140 lines) is not loaded by default to keep the page fast.Load diff

added web/.gitignore +41 −0

Large diff (41 lines) is not loaded by default to keep the page fast.Load diff

added web/README.md +36 −0

Large diff (36 lines) is not loaded by default to keep the page fast.Load diff

added web/eslint.config.mjs +18 −0

Large diff (18 lines) is not loaded by default to keep the page fast.Load diff

added web/next.config.ts +20 −0

Large diff (20 lines) is not loaded by default to keep the page fast.Load diff

added web/package.json +27 −0

Large diff (27 lines) is not loaded by default to keep the page fast.Load diff

added web/pnpm-lock.yaml +4184 −0

Large diff (4,184 lines) is not loaded by default to keep the page fast.Load diff

added web/pnpm-workspace.yaml +3 −0

Large diff (3 lines) is not loaded by default to keep the page fast.Load diff

added web/postcss.config.mjs +7 −0

Large diff (7 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/docs/page.tsx +64 −0

Large diff (64 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/downloads/page.tsx +149 −0

Large diff (149 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/favicon.ico +0 −0

Binary file not shown.

added web/src/app/globals.css +101 −0

Large diff (101 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/harvester/page.tsx +159 −0

Large diff (159 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/keys/page.tsx +89 −0

Large diff (89 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/layout.tsx +21 −0

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/login/page.tsx +73 −0

Large diff (73 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/page.tsx +170 −0

Large diff (170 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/playground/page.tsx +199 −0

Large diff (199 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/settings/page.tsx +134 −0

Large diff (134 lines) is not loaded by default to keep the page fast.Load diff

added web/src/app/system/page.tsx +132 −0

Large diff (132 lines) is not loaded by default to keep the page fast.Load diff

added web/src/components/shell.tsx +157 −0

Large diff (157 lines) is not loaded by default to keep the page fast.Load diff

added web/src/components/ui.tsx +233 −0

Large diff (233 lines) is not loaded by default to keep the page fast.Load diff

added web/src/lib/api.ts +89 −0

Large diff (89 lines) is not loaded by default to keep the page fast.Load diff

added web/src/lib/events.tsx +130 −0

Large diff (130 lines) is not loaded by default to keep the page fast.Load diff

added web/src/lib/format.ts +73 −0

Large diff (73 lines) is not loaded by default to keep the page fast.Load diff

added web/src/lib/types.ts +225 −0

Large diff (225 lines) is not loaded by default to keep the page fast.Load diff

added web/tsconfig.json +34 −0

Large diff (35 lines) is not loaded by default to keep the page fast.Load diff

		`@@ -0,0 +1,14 @@`
	1	+# Security
	2	+
	3	+- Private by default. The API binds `127.0.0.1:8300`; workers bind `127.0.0.1:8310-8399`; the console binds `127.0.0.1:8301`. Only the WireGuard-connected gateway (Caddy on BHS64) reaches port 8300 over the tunnel; nothing else is exposed.
	4	+- Admin auth: single admin user, argon2 password hashes, signed session cookie (`itsdangerous`, HttpOnly, Secure behind HTTPS, SameSite=Lax, 14 days). First user is seeded from `ADMIN_EMAIL`/`ADMIN_PASSWORD` or created once via `/api/auth/setup` (only when no user exists). Login rate limit 8/min/IP. Password change requires the current password.
	5	+- CSRF: session-authenticated mutations require header `X-LLM-CSRF: 1` and, when an `Origin` is present, it must match the public URL / host.
	6	+- API keys: `llm_live_` + 40 random chars, stored as SHA-256; only the prefix is displayed after creation. Scopes `inference` / `admin`. Revocation is immediate. Per-key last-used and request counters.
	7	+- Request validation: JSON bodies ≤ 20 MB (middleware + parser), sampling ranges checked, ≤ 8 stop sequences, ≤ 256 embedding inputs, ≤ 200 rerank documents, `n=1` only.
	8	+- Path safety: repository ids validated (`org/name`, no `..`); model deletion refuses paths outside the model root; scanned paths are only read.
	9	+- No shell from the API: workers are started with `subprocess.Popen(list)` (no shell); user-supplied extra llama args are filtered (`\|;&$\`` rejected) and only settable by an admin.
	10	+- Secrets: `.env` is `600` and outside the repo; `HF_TOKEN` only reaches the downloader; the session secret is generated into `data/.secret` (600). Keys, passwords and tokens are never logged.
	11	+- Prompt privacy: `LOG_PROMPTS=false` by default; when enabled, prompts/completions are stored truncated in `inference_requests`.
	12	+- Audit log: login/failed login/password change, key create/revoke, model load/unload/update/delete, downloads, harvest scans, settings changes, server start/stop — with actor and IP.
	13	+- Headers: `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: same-origin`, `Cache-Control: no-store` on API paths. Console is `noindex`.
	14	+- Memory safety is part of security here: a model that would exhaust unified memory is refused before any allocation; swap use raises a warning alert; workers are killed on unload so memory always returns.

spb/llm-api

LLM API v0.1.0 — private OpenAI-compatible local model server for Apple Silicon

Theme