SPB Git
spboucher.ai

spb/llm-api

Public

LLM API v0.1.0 — private OpenAI-compatible local model server for Apple Silicon

FastAPI gateway + SQLite registry, MLX-LM / mlx-vlm and llama.cpp workers, load-on-demand
with memory policy and eviction, OpenAI endpoints (chat/completions/embeddings/rerank, streaming),
management API, Model Harvester, benchmarks, API keys, Next.js 16 console, tests and docs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 11, 2026) Browse files

70 changed files +13,830 −0

70 changed files
Load all 70 files Unified Split
added .env.example +52 −0
@@ -0,0 +1,52 @@
1+# LLM API — environment (copy to <LLM_API_ROOT>/.env or export before `llm-api serve`)
2+
3+APP_ENV=production
4+DOMAIN=www.llm-api.io
5+PUBLIC_URL=https://www.llm-api.io
6+
7+# Bind address of the API gateway (the public reverse proxy talks to this)
8+HOST=127.0.0.1
9+PORT=8300
10+# Where the Next.js dashboard runs (proxied by the API for non-/api, non-/v1 paths)
11+WEB_URL=http://127.0.0.1:8301
12+
13+# Data root: models/, data/ (SQLite + secrets), logs/
14+LLM_API_ROOT=/Users/USERNAME/llm-api
15+# Override the model directory (e.g. an external SSD)
16+# MODEL_ROOT=/Volumes/LLM/models
17+
18+# Memory policy (GB). 64 GB machine: 8 macOS + 3 app + 4 KV + 4 safety = 45 GB for models
19+MAX_MODEL_MEMORY_GB=45
20+ABSOLUTE_MAX_MEMORY_GB=50
21+MACOS_RESERVE_GB=8
22+MIN_FREE_DISK_GB=100
23+
24+MAX_SIMULTANEOUS_MODELS=1
25+MODEL_IDLE_TIMEOUT_MINUTES=30
26+SMALL_MODEL_RESIDENT_GB=3
27+LOAD_TIMEOUT_SECONDS=900
28+GENERATION_TIMEOUT_SECONDS=1800
29+PRELOAD_MODEL=none
30+
31+ENABLE_MLX=true
32+ENABLE_GGUF=true
33+ALLOW_DOWNLOADS=true
34+LOG_PROMPTS=false
35+
36+DEFAULT_MAX_TOKENS=2048
37+DEFAULT_CONTEXT=16384
38+
39+# Hugging Face token for gated repositories (never exposed to the frontend)
40+HF_TOKEN=
41+LLAMA_SERVER_BIN=llama-server
42+
43+# Sessions/CSRF secret (auto-generated into data/.secret when empty)
44+SECRET_KEY=
45+# First-run admin (used only when no user exists)
46+ADMIN_EMAIL=
47+ADMIN_PASSWORD=
48+SESSION_HOURS=336
49+SECURE_COOKIES=true
50+
51+METRICS_INTERVAL_SECONDS=15
52+METRICS_RETENTION_DAYS=30
added .gitignore +22 −0
@@ -0,0 +1,22 @@
1+# python
2+__pycache__/
3+*.pyc
4+server/.venv/
5+.venv/
6+*.egg-info/
7+.pytest_cache/
8+.ruff_cache/
9+# node
10+web/node_modules/
11+web/.next/
12+web/next-env.d.ts
13+web/*.tsbuildinfo
14+# secrets / data
15+.env
16+.env.*
17+!.env.example
18+data/
19+logs/
20+models/
21+# os
22+.DS_Store
added CLAUDE.md +25 −0
@@ -0,0 +1,25 @@
1+# CLAUDE.md — LLM API (www.llm-api.io)
2+
3+Private OpenAI-compatible local model server for Apple Silicon. Full product spec: see `docs/` and README.
4+Runs on **M1M64** (Mac Studio M1 Max 64 GB, rented, user `simon`), deployed with `mld deploy llm-api --node M1M64`
5+(manifest `M1M32:~/dispatch/apps/llm-api.json`), public via MacLustr Tunnel (BHS64) at https://www.llm-api.io.
6+
7+## Layout
8+- `server/` Python 3.13 FastAPI (`llm_api`): gateway, registry, manager, workers (`llm_api/worker/mlx_worker.py`, llama-server), tests (`pytest`, fake workers, no MLX needed).
9+- `web/` Next.js 16 console (proxied by the API in production; `pnpm dev` proxies /api,/v1 to `API_URL`).
10+- Data root on the node: `~/llm-api/` (`models/` library, `data/` SQLite + `.secret` + `.env`, `logs/`). Code: `~/apps/llm-api/`.
11+
12+## Rules (non-negotiable)
13+1. Never load a model whose estimate exceeds the budget; never rely on swap; never fake metrics.
14+2. Workers only on 127.0.0.1; memory release = kill the worker process.
15+3. Never mark ready before warm-up; never delete model files automatically.
16+4. Verify MLX / mlx-lm / mlx-vlm / llama.cpp / huggingface-hub APIs against the installed versions before changing runtime code.
17+5. Always test load → inference → unload → memory recovery after touching the manager or workers.
18+
19+## Dev loop
20+```bash
21+cd server && python -m compileall -q llm_api && rsync -az --delete --exclude __pycache__ --exclude .venv ./ M1M64:~/apps/llm-api/server/
22+ssh M1M64 'source ~/llm-api/.venv/bin/activate; cd ~/apps/llm-api/server && python -m pytest -q --basetemp=/tmp/llmapi-tests; rm -rf /tmp/llmapi-tests'
23+cd web && pnpm exec tsc --noEmit && pnpm build
24+```
25+Gotchas: pytest temp dirs must stay sparse (`--basetemp` + sparse fixtures — a real 400-layer fixture once filled the 1.8 TB SSD); `socket.getfqdn` hangs on the rented Mac (never use stock `HTTPServer.server_bind`); MLX work must stay on one thread per worker.
added README.md +85 −0
@@ -0,0 +1,85 @@
1+# LLM API — `https://www.llm-api.io`
2+
3+A private, production-grade, **OpenAI-compatible local LLM API for Apple Silicon**. Store a large library of models on SSD, load only the one that is requested into unified memory, unload it when something else is asked for, and expose everything behind one clean, authenticated API plus a management console.
4+
5+```
6+OpenAI API — but backed by my own Apple Silicon machine and my own locally stored models.
7+```
8+
9+First deployment: Mac Studio M1 Max, 10-core CPU, 32-core GPU, 64 GB unified memory, 1.8 TB SSD (node `M1M64`).
10+
11+## What it does
12+
13+- **Load-on-demand**: `{"model": "qwen3.8-27b-4bit"}` loads that model from SSD if it is not resident, evicting the previous one (LRU + idle timeout), then serves the request. Switches are serialized; requests queue while a model loads.
14+- **Two runtimes**: MLX / MLX-LM (preferred, safetensors) and llama.cpp `llama-server` (GGUF, Metal). Each model runs in its **own worker process** on `127.0.0.1` — killing the process is how memory is guaranteed to come back.
15+- **Memory policy**: weights + KV cache + runtime overhead are estimated *before* loading; the default budget is 45 GB of 64 GB (never all of it). Too-large models are refused with a structured error. Swap is treated as a warning, never a feature.
16+- **Compatibility engine**: `compatible` / `compatible_with_restrictions` / `experimental` / `not_recommended` / `incompatible`, with a recommended context per model.
17+- **OpenAI endpoints**: `/v1/models`, `/v1/chat/completions` (streaming, tools, reasoning content), `/v1/completions`, `/v1/embeddings`, `/v1/rerank`. Works with the official OpenAI SDKs.
18+- **Management API + console**: model library, per-model pages, playground, downloads from Hugging Face (inspected first: size, RAM, disk reserve), **Model Harvester** (explores HF for models that truly fit, dedupes quantizations, proposes a download queue), benchmarks, API keys (hashed), settings, real-time telemetry (RAM, GPU, CPU, thermal, disk) via SSE.
19+- **Aliases and `auto`**: `fast`, `coder`, `reasoning`, `vision`, `embedding`, `default`… `model: "auto"` routes by prompt content (never when a model is named explicitly).
20+
21+## Layout
22+
23+```
24+server/ Python 3.13 · FastAPI · SQLite — API gateway, registry, model manager, workers
25+web/ Next.js 16 · React 19 · Tailwind 4 — console (proxied by the API in production)
26+scripts/ install.sh
27+docs/ architecture, api, models, security, deployment, troubleshooting
28+```
29+
30+## Quick start (clean Mac)
31+
32+```bash
33+git clone <repo> llm-api && cd llm-api
34+./scripts/install.sh --test # checks the Mac, installs uv/Node/llama.cpp, Python deps, builds the web app
35+$EDITOR ~/llm-api/.env # ADMIN_EMAIL, ADMIN_PASSWORD, HF_TOKEN, PUBLIC_URL
36+(cd ~/llm-api && ../path/to/server/.venv/bin/llm-api serve) # API on 127.0.0.1:8300
37+(cd web && pnpm start -p 8301 -H 127.0.0.1) # console
38+```
39+
40+Then open `http://127.0.0.1:8300`, sign in, create an API key, and:
41+
42+```python
43+from openai import OpenAI
44+client = OpenAI(base_url="https://www.llm-api.io/v1", api_key="llm_live_xxxxx")
45+r = client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hello"}])
46+print(r.choices[0].message.content)
47+```
48+
49+```bash
50+curl https://www.llm-api.io/v1/chat/completions \
51+ -H "Authorization: Bearer llm_live_xxxxx" -H "Content-Type: application/json" \
52+ -d '{"model": "qwen3.8-27b-4bit", "messages": [{"role": "user", "content": "Explain monetary policy."}], "stream": true}'
53+```
54+
55+## Verified on the M1 Max (2026-09-10)
56+
57+| Step | Result |
58+|---|---|
59+| Cold load Qwen3-4B-Instruct-2507 (MLX 4-bit) | 1.7 s, ~100 tok/s generation |
60+| Cold load Llama-3.2-3B (MLX 4-bit) | 1.4 s, ~170 tok/s |
61+| Gemma-3-1B GGUF via llama.cpp | 1.1 s, ~150 tok/s, streaming with `timings` |
62+| Switch A → B → A | previous worker killed, memory returned to the OS before the next load |
63+| OpenAI Python SDK | models / chat / stream / embeddings OK |
64+| Memory rejection | budget lowered → model refused with `MODEL_TOO_LARGE`, compatibility re-evaluated |
65+
66+## Tests
67+
68+```bash
69+cd server && .venv/bin/python -m pytest -q
70+```
71+
72+The suite runs without MLX (fake workers) and covers: authentication, API keys (creation, rejection, revocation), registry scanning, missing-file detection, load-on-demand, streaming, model switching, concurrent requests + switch lock, memory rejection, worker crash, load timeout, deletion confirmation, low-disk refusal, benchmarks, restart cleanup, settings validation.
73+
74+## Docs
75+
76+- [docs/architecture.md](docs/architecture.md) — components, data flow, worker protocol, memory model
77+- [docs/api.md](docs/api.md) — OpenAI endpoints, extensions, management API, errors
78+- [docs/models.md](docs/models.md) — storage layout, discovery, compatibility, quantization policy, Harvester, starter library
79+- [docs/security.md](docs/security.md) — auth, keys, CSRF, sandboxing, what is never logged
80+- [docs/deployment.md](docs/deployment.md) — M1M64 deployment with `mld`, PM2, MacLustr Tunnel, DNS
81+- [docs/troubleshooting.md](docs/troubleshooting.md) — failure modes and what to look at
82+
83+## Non-negotiables (from CLAUDE.md)
84+
85+Never crash the host with an oversized model · never rely on swap · never expose secrets or raw workers · never mark a model ready before a successful warm-up · never delete model files automatically · never fake metrics · verify runtime APIs against the installed versions · always test load → inference → unload → memory recovery · keep the design ready for multiple Apple Silicon nodes.
added docs/api.md +77 −0
@@ -0,0 +1,77 @@
1+# API
2+
3+Base URL: `https://www.llm-api.io/v1` (also reachable at `http://127.0.0.1:8300/v1` on the node). Interactive OpenAPI schema: `/openapi`.
4+
5+## Authentication
6+
7+`Authorization: Bearer llm_live_…` (or `X-API-Key`). Keys are created in the console (Keys page) or `llm-api create-key <name> [--admin]`. Scopes: `inference` (OpenAI endpoints), `admin` (management API). The console session cookie also works for `/v1` (playground).
8+
9+## OpenAI-compatible endpoints
10+
11+| Method | Path | Notes |
12+|---|---|---|
13+| GET | `/v1/models` | installed models + aliases; extra fields: `runtime`, `quantization`, `estimated_ram_gb`, `context`, `compatibility`, `status`, `capabilities` |
14+| GET | `/v1/models/{id}` | |
15+| POST | `/v1/chat/completions` | `stream`, `temperature`, `top_p`, `max_tokens`/`max_completion_tokens`, `stop` (≤ 8), `seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `tools`, `stream_options.include_usage` (always on) |
16+| POST | `/v1/completions` | `prompt`, `echo`, same sampling params |
17+| POST | `/v1/embeddings` | `input` (str or ≤ 256 strings), `dimensions`, `encoding_format` |
18+| POST | `/v1/rerank` | `query`, `documents` (≤ 200), `top_n`, `return_documents`, `instruction` |
19+
20+Extensions (ignored by SDKs):
21+
22+- `timings`: `{ttft_ms, prompt_ms, generation_ms, total_ms, prompt_tps, generation_tps, peak_memory_gb}` on responses and on the final stream chunk.
23+- `usage.prompt_tokens_details.cached_tokens` when the prompt cache was reused.
24+- `reasoning_content` on `message` / `delta` for thinking models. Control with `chat_template_kwargs: {"enable_thinking": false}`, `reasoning: {"effort": "none"}` or `reasoning_effort`.
25+- `top_k`, `min_p`, `repetition_penalty` (MLX).
26+- `model: "auto"` — routed by prompt (code → `coder`, long/analytical → `reasoning`, images → `vision`, embeddings → `embedding`), else current model / `default` alias / default model setting.
27+
28+### Model resolution
29+
30+`model` may be a registry id (`qwen3.8-27b-4bit`), an alias (`coder`), the directory name, or the Hugging Face repository (`mlx-community/Qwen3.8-27B-4bit`).
31+
32+### Errors
33+
34+```json
35+{"error": {"message": "Model requires approximately 52.0 GB at a 32768 context but the safe limit is 45 GB.",
36+ "type": "model_memory_error", "code": "MODEL_TOO_LARGE", "param": null, "estimate": {...}}}
37+```
38+
39+| HTTP | code | when |
40+|---|---|---|
41+| 400 | `INVALID_REQUEST`, `CONTEXT_TOO_LARGE`, `TEMPLATE_ERROR`, `VISION_UNSUPPORTED` | bad body, prompt longer than the loaded context |
42+| 401 | `INVALID_API_KEY`, `UNAUTHENTICATED` | |
43+| 403 | `FORBIDDEN`, `CSRF`, `DOWNLOADS_DISABLED`, `PATH_NOT_ALLOWED` | |
44+| 404 | `MODEL_NOT_FOUND` | |
45+| 409 | `CONFLICT`, `ALREADY_INSTALLED` | |
46+| 413 | `BODY_TOO_LARGE` | > 20 MB |
47+| 422 | `MODEL_INCOMPATIBLE`, `RUNTIME_UNSUPPORTED`, `WRONG_MODEL_TYPE` | |
48+| 429 | `RATE_LIMITED` | 600 req/min per IP; 8 logins/min |
49+| 502 | `WORKER_UNREACHABLE`, `DOWNLOAD_FAILED` | |
50+| 503 | `MODEL_LOAD_FAILED`, `MODEL_LOAD_TIMEOUT`, `WORKER_CRASHED`, `WORKER_BUSY` | |
51+| 504 | `GENERATION_TIMEOUT` | |
52+| 507 | `MODEL_TOO_LARGE`, `INSUFFICIENT_DISK` | |
53+
54+## Management API (`/api`, admin)
55+
56+Session mutations need header `X-LLM-CSRF: 1`.
57+
58+- Auth: `GET /api/auth/status`, `POST /api/auth/setup|login|logout|password`, `GET /api/auth/me`
59+- Models: `GET /api/models[?include_missing]`, `GET /api/models/{id}` (benchmarks, events, memory curve), `GET /api/models/{id}/files`, `POST /api/models/{id}/load {context?, force?}`, `POST /api/models/{id}/unload`, `PATCH /api/models/{id}` (favorite, pinned, enabled, notes, tags, name, overrides), `POST /api/models/{id}/pin?pinned=`, `POST /api/models/{id}/favorite?favorite=`, `DELETE /api/models/{id} {confirm: id, keep_benchmarks}`, `POST /api/models/rescan`, `POST /api/models/{id}/benchmark {max_tokens, runs, long_prompt}`, `GET /api/models/{id}/benchmarks`, `POST /api/models/{id}/tokenize`
60+- Aliases: `GET /api/aliases`, `PUT /api/aliases {alias, model_id}`, `DELETE /api/aliases/{alias}`
61+- Downloads: `POST /api/models/inspect {repository, quant?}`, `POST /api/models/download {repository, quant?, force?}`, `GET /api/downloads`, `POST /api/downloads/{job}/retry`, `GET /api/jobs[?kind]`, `POST /api/jobs/{id}/cancel`
62+- Harvester: `POST /api/harvest/scan {runtimes, authors, limit_per_author, min_downloads, max_ram_gb, families, tasks, search}`, `GET /api/harvest/candidates[?task&runtime&family&size_class&q&include_duplicates]` (+ starter slots), `POST /api/harvest/select|dismiss {repo_id, selected}`, `POST /api/harvest/queue`
63+- System: `GET /api/system`, `/api/system/memory`, `/api/system/gpu`, `/api/system/storage`, `/api/system/processes`, `/api/system/health`, `/api/system/metrics?minutes&hours`, `GET /api/runtime/status`, `GET /api/runtime/current-model`
64+- Keys: `GET/POST /api/keys`, `PATCH /api/keys/{id} {name}`, `DELETE /api/keys/{id}`
65+- Settings: `GET/PATCH /api/settings`
66+- Logs: `GET /api/logs/audit|events|requests`, `GET /api/logs/worker/{model_id}`
67+- Live: `GET /api/events` — SSE stream: `snapshot`, `manager`, `model` (load progress), `metrics` (every 3 s), `job`, `request`, `alert`, `settings`.
68+
69+## Status
70+
71+`GET /health` (public):
72+
73+```json
74+{"status": "ok", "hardware": {"chip": "Apple M1 Max", "memory_gb": 64.0, "gpu_cores": 32},
75+ "model": {"loaded": true, "id": "qwen3-4b-instruct-2507-4bit", "loaded_models": ["…"]},
76+ "memory": {"used_gb": 23.1, "available_gb": 40.9, "pressure": "normal"}, "uptime_seconds": 1234, "version": "0.1.0"}
77+```
added docs/architecture.md +71 −0
@@ -0,0 +1,71 @@
1+# Architecture
2+
3+```
4+Client (OpenAI SDK, curl, console)
5+ │ HTTPS www.llm-api.io
6+ ▼
7+MacLustr Tunnel gateway (BHS64, Caddy TLS) ──WireGuard──▶ M1M64 wg1 10.67.0.x
8+ ▼
9+FastAPI gateway 127.0.0.1:8300 (server/llm_api)
10+ ├─ /v1/* OpenAI-compatible routes ── auth (API key or admin session) ── proxy to worker
11+ ├─ /api/* management API ── admin session / admin-scoped key, CSRF header
12+ ├─ /health public status
13+ └─ /* reverse proxy → Next.js console 127.0.0.1:8301
14+ │
15+ ├─ Registry (SQLite) models, aliases, benchmarks, events, requests, jobs, keys, settings, audit
16+ ├─ Scanner safetensors + GGUF headers → architecture, params, quant, KV bytes/token
17+ ├─ Compatibility engine estimate(weights, KV, overhead) vs budget → status + recommended context
18+ ├─ Model Manager switch lock, eviction (LRU / pinned), spawn, wait-ready, warm-up, monitor, idle
19+ ├─ Job runner downloads (serialized), scans, benchmarks, harvests
20+ ├─ Downloader / Harvester Hugging Face inspection, verified downloads, manifests, discovery
21+ ├─ Metrics collector psutil + ioreg + vm_stat + pmset → SSE + SQLite history
22+ └─ Event bus SSE fan-out to the console
23+ │
24+ ▼ one process per loaded model, 127.0.0.1:8310-8399
25+ ┌──────────────────────┐ ┌──────────────────────────┐
26+ │ mlx_worker (python) │ │ llama-server (llama.cpp) │
27+ │ mlx-lm / mlx-vlm │ │ Metal, --jinja, slots │
28+ │ /v1/chat/completions │ │ /v1/chat/completions │
29+ │ /v1/embeddings … │ │ /v1/embeddings … │
30+ └──────────────────────┘ └──────────────────────────┘
31+```
32+
33+## Request path (load-on-demand)
34+
35+1. `POST /v1/chat/completions` → auth → body validation → model resolution (id, alias, name, repository, `auto`).
36+2. `ModelManager.ensure_loaded(id)`:
37+ - fast path if the worker is `ready`;
38+ - otherwise take the **switch lock** (one switch at a time; other requests wait on it, so no thrashing);
39+ - policy: enabled, compatible (or `force`), estimate at the chosen context ≤ budget;
40+ - eviction: keep at most `MAX_SIMULTANEOUS_MODELS` large models and `resident + new ≤ budget`; victims are LRU, pinned last; small embedding/reranker workers (≤ 3 GB) may stay resident;
41+ - real free-memory check after eviction (waits for the OS to reclaim);
42+ - spawn the worker, poll `/health` until `ready` (or crash/timeout), run a **warm-up** inference, measure memory, mark `ready`, record load time.
43+3. The request is proxied to the worker (streamed for SSE). Usage + `timings` are recorded per request and per model.
44+
45+## Worker protocol
46+
47+Every runtime is a local HTTP server speaking the OpenAI API. Adapters only differ in how they spawn and how they detect readiness:
48+
49+- **MLXAdapter** → `python -m llm_api.worker.mlx_worker --model-path … --port … --max-context …`. All MLX work (import, load, generate, embed) runs on a single dedicated thread inside the worker (MLX streams are thread-affine). Features: chat templates (tools, `enable_thinking`), `<think>` → `reasoning_content`, tool-call parsing, stop sequences with hold-back, prompt-cache reuse across turns, sampling (temperature, top_p, top_k, min_p, penalties, logit_bias, seed), last-token-pooled embeddings, yes/no rerank scoring, vision via mlx-vlm.
50+- **LlamaCppAdapter** → `llama-server -m … --port … -ngl 999 -c ctx --jinja --no-webui -np 1 -fa auto --cache-reuse 256` (+ `--embeddings`, `--reranking`, `--mmproj`, `--reasoning-format deepseek`).
51+
52+Memory release = process exit (SIGTERM → SIGKILL). The manager waits for `vm_stat` available memory to come back before the next load.
53+
54+## Memory model
55+
56+```
57+total = weights × 1.03–1.04
58+ + KV bytes/token × context (2 · layers · kv_heads · head_dim · 2 bytes; sliding-window aware)
59+ + runtime overhead (1.0–1.2 GB) + activations (0.6–0.8 GB) + vision (0.8 GB)
60+ + long-context scratch (≤ 2 GB)
61+```
62+
63+Recommended context = largest of 2K…32K that fits the budget (larger contexts can be requested explicitly). Budget default 45 GB on 64 GB; absolute limit 50 GB; both editable in Settings, which re-evaluates every model.
64+
65+## Crash recovery
66+
67+`data/workers.json` lists live worker PIDs. On startup, any leftover worker is killed and registry state reset; jobs left `running` are marked failed. The monitor loop (5 s) detects a dead worker and publishes `worker_crashed`.
68+
69+## Future cluster
70+
71+Nothing binds the gateway to one node: the registry row carries `path`/`runtime`, the manager talks HTTP to workers, telemetry is per host. A central router could hold several `ModelManager`s (one per node, over the private WireGuard network) and score `model already loaded · free RAM · GPU · queue · tokens/s · thermal`. Model replication would reuse the verified-manifest download path over the private network only.
added docs/deployment.md +57 −0
@@ -0,0 +1,57 @@
1+# Deployment
2+
3+## Production topology (2026-09-10)
4+
5+```
6+GoDaddy DNS llm-api.io A www → 51.161.112.61 (BHS64)
7+BHS64 (OVH Beauharnois) Caddy TLS https://www.llm-api.io → 10.67.0.x:8300 over WireGuard
8+M1M64 (Mac Studio M1 Max 64 GB, rented, public IP 45.74.241.221, user simon)
9+ wg1 10.67.0.x
10+ PM2 (LaunchDaemon) llm-api-server → .venv/bin/llm-api serve 127.0.0.1:8300
11+ llm-api-web → next start -p 8301 -H 127.0.0.1
12+ ~/llm-api/ models/ (SSD library) data/ (SQLite, .secret, .env) logs/
13+ ~/apps/llm-api/ code (rsync'd by mld: server/ + web/)
14+```
15+
16+The whole app is deployed and routed by the cluster orchestrator **mld** (gateway M1M32):
17+
18+```bash
19+mld stage ~/Desktop/Cluster/llm-api llm-api # laptop → M1M32 staging (excludes node_modules, .next, .venv)
20+mld deploy llm-api --node M1M64 # rsync → post_sync hooks (uv venv + pip, pnpm build) → PM2 → health → tunnel route → registry
21+mld status --live | grep llm-api
22+mld logs llm-api
23+```
24+
25+Manifest: `M1M32:~/dispatch/apps/llm-api.json` (dir `~/apps/llm-api`, port 8300, health `/health`, tunnel `www.llm-api.io` on BHS64, pinned to `M1M64`). Secrets (`ADMIN_PASSWORD`, `HF_TOKEN`) live in the manifest's `env` on M1M32 only and are written into the PM2 environment; the app also reads `~/llm-api/.env`.
26+
27+M1M64 is a *reserved* node: always deploy with `--node M1M64`. It has no graphical session guaranteed at boot, so PM2 runs as a system LaunchDaemon (`mld prepare` does this).
28+
29+## Manual (without mld)
30+
31+```bash
32+./scripts/install.sh
33+cd ~/llm-api && /path/to/server/.venv/bin/llm-api serve # API
34+cd /path/to/web && pnpm start -p 8301 -H 127.0.0.1 # console
35+```
36+
37+PM2:
38+
39+```bash
40+pm2 start /path/to/server/.venv/bin/llm-api --name llm-api-server --cwd ~/llm-api -- serve
41+pm2 start node --name llm-api-web --cwd /path/to/web -- node_modules/next/dist/bin/next start -p 8301 -H 127.0.0.1
42+pm2 save && pm2 startup
43+```
44+
45+launchd alternative: a `LaunchDaemon` plist running `.venv/bin/llm-api serve` with `WorkingDirectory=~/llm-api`, `KeepAlive=true`, `RunAtLoad=true` (see `mld` render for the pattern).
46+
47+## Reverse proxy
48+
49+Any TLS terminator works (Caddy, nginx). Forward everything to `127.0.0.1:8300`; the API serves `/v1`, `/api`, `/health`, `/openapi` itself and proxies the rest to Next. SSE needs buffering off (`X-Accel-Buffering: no` is already set). `api.llm-api.io` can point at the same upstream if a separate API host is wanted.
50+
51+## Updating
52+
53+```bash
54+mld stage ~/Desktop/Cluster/llm-api llm-api && mld deploy llm-api --node M1M64
55+```
56+
57+Deploy = rsync + rebuild + `pm2 restart`; the manager unloads the loaded model on shutdown and clears stale state on start.
added docs/models.md +55 −0
@@ -0,0 +1,55 @@
1+# Models
2+
3+## Storage
4+
5+```
6+<LLM_API_ROOT>/models/
7+├── mlx/<family>/<Repo-Name>/ safetensors + config.json + tokenizer (+ llm-api.json manifest)
8+├── gguf/<family>/<Repo-Name-GGUF>/ one chosen quantization (+ mmproj for vision)
9+├── embeddings/ rerankers/ vision/ same layout, by role
10+└── manifests/<org--repo>.json copy of every download manifest
11+```
12+
13+One recommended quantization per model; the downloader picks it (`Q4_K_M` → `Q5_K_M` → `Q6_K` → `Q8_0` → `MXFP4`… for GGUF) unless a `quant` is given. Nothing is ever deleted by a scan.
14+
15+## Discovery (`POST /api/models/rescan`, startup)
16+
17+For each directory: `config.json` + `*.safetensors` → MLX model (architecture, layers, heads, quant bits from `quantization`, params from safetensors headers — packed `uint32` ×32/bits); `*.gguf` → llama.cpp model (GGUF header: architecture, `block_count`, `head_count_kv`, `key_length`, `context_length`, `file_type`, tensor shapes). Type detection: embedding (name / pipeline / pooling), reranker, vision (`vision_config`, mmproj, model type), thinking / tools (chat template). Registry ids are stable slugs (`qwen3-4b-instruct-2507-4bit`); a directory keeps its id across rescans. Manual overrides (task, context, KV bits, pooling, extra llama args) live in `overrides` and survive rescans.
18+
19+## Size classes (estimated RAM at the recommended context)
20+
21+TINY < 5 GB · SMALL 5–10 · MEDIUM 10–20 · LARGE 20–35 · XL 35–45 · TOO_LARGE > budget.
22+
23+## Compatibility
24+
25+| status | meaning |
26+|---|---|
27+| `compatible` | fits with ≥ 16K context in the budget |
28+| `compatible_with_restrictions` | fits, but context must stay below 16K (the recommended context is enforced at load) |
29+| `experimental` | runs but unverified: architecture not in the llama.cpp known list, VLM type falling back to text, very low quantization on a large model |
30+| `not_recommended` | fits only through heavy swap (≤ absolute limit but > budget) — not loadable without `force` |
31+| `incompatible` | runtime missing, architecture unsupported by the installed mlx-lm/mlx-vlm, or > absolute limit |
32+
33+## Quantization policy (Harvester scoring)
34+
35+≤ 8B prefer 8/6-bit · 10–20B Q6/Q5 · 20–40B Q5/Q4 · 40–80B Q4 · larger Q3 only when genuinely useful. Lower than that is penalized (`too_low`).
36+
37+## Model Harvester
38+
39+`POST /api/harvest/scan` lists trusted authors (MLX: `mlx-community`; GGUF: `unsloth`, `bartowski`, `ggml-org`, `lmstudio-community`), excludes ASR/TTS/image/video/NSFW/base/draft repos, estimates RAM from the listing config + safetensors metadata (MLX) or from the chosen GGUF file (one repo-tree call), evaluates compatibility, scores (popularity, recency, size, quantization policy), and **dedupes by base model + runtime** (only the best quantization per base model is proposed). Candidates land in the Harvester page with a suggested starter library (small/medium/large general, coding, reasoning, vision, embedding, reranker). Selected candidates are queued through the normal inspected download path.
40+
41+## Starter library on M1M64 (downloaded 2026-09-10, all mlx-community unless noted)
42+
43+| slot | model | RAM est. |
44+|---|---|---|
45+| tiny test | Qwen3-4B-Instruct-2507-4bit, Llama-3.2-3B-Instruct-4bit, gemma-3-1b-it Q4_K_M (GGUF) | 4–9 GB |
46+| small general / vision | Qwen3.5-9B-MLX-4bit | ~8 GB |
47+| medium general | gemma-4-12B-it-qat-4bit | ~10 GB |
48+| reasoning | gpt-oss-20b-MXFP4-Q8 | ~14 GB |
49+| coding | Devstral-Small-2-24B-Instruct-2512-4bit, Qwen3-Coder-30B-A3B-Instruct-4bit | 15–20 GB |
50+| large general | Qwen3.8-27B-4bit, Qwen3.6-35B-A3B-4bit (MoE), gemma-4-26b-a4b-it-4bit (MoE) | 17–24 GB |
51+| XL | Llama-3.3-70B-Instruct-4bit | ~42 GB (context ≤ 8K) |
52+| embedding | Qwen3-Embedding-0.6B-8bit (MLX), embeddinggemma-300M (GGUF) | < 1 GB, can stay resident |
53+| reranker | Qwen3-Reranker-0.6B-4bit | < 1 GB |
54+
55+Versions verified against the installed runtimes: mlx 0.32.2, mlx-lm 0.31.3, mlx-vlm 0.7.0, llama.cpp 0.4.0 (Homebrew), huggingface-hub 1.31.0.
added docs/security.md +14 −0
@@ -0,0 +1,14 @@
1+# Security
2+
3+- **Private by default.** The API binds `127.0.0.1:8300`; workers bind `127.0.0.1:8310-8399`; the console binds `127.0.0.1:8301`. Only the WireGuard-connected gateway (Caddy on BHS64) reaches port 8300 over the tunnel; nothing else is exposed.
4+- **Admin auth**: single admin user, argon2 password hashes, signed session cookie (`itsdangerous`, HttpOnly, Secure behind HTTPS, SameSite=Lax, 14 days). First user is seeded from `ADMIN_EMAIL`/`ADMIN_PASSWORD` or created once via `/api/auth/setup` (only when no user exists). Login rate limit 8/min/IP. Password change requires the current password.
5+- **CSRF**: session-authenticated mutations require header `X-LLM-CSRF: 1` and, when an `Origin` is present, it must match the public URL / host.
6+- **API keys**: `llm_live_` + 40 random chars, stored as SHA-256; only the prefix is displayed after creation. Scopes `inference` / `admin`. Revocation is immediate. Per-key last-used and request counters.
7+- **Request validation**: JSON bodies ≤ 20 MB (middleware + parser), sampling ranges checked, ≤ 8 stop sequences, ≤ 256 embedding inputs, ≤ 200 rerank documents, `n=1` only.
8+- **Path safety**: repository ids validated (`org/name`, no `..`); model deletion refuses paths outside the model root; scanned paths are only read.
9+- **No shell from the API**: workers are started with `subprocess.Popen(list)` (no shell); user-supplied extra llama args are filtered (`|;&$\`` rejected) and only settable by an admin.
10+- **Secrets**: `.env` is `600` and outside the repo; `HF_TOKEN` only reaches the downloader; the session secret is generated into `data/.secret` (600). Keys, passwords and tokens are never logged.
11+- **Prompt privacy**: `LOG_PROMPTS=false` by default; when enabled, prompts/completions are stored truncated in `inference_requests`.
12+- **Audit log**: login/failed login/password change, key create/revoke, model load/unload/update/delete, downloads, harvest scans, settings changes, server start/stop — with actor and IP.
13+- **Headers**: `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: same-origin`, `Cache-Control: no-store` on API paths. Console is `noindex`.
14+- **Memory safety** is part of security here: a model that would exhaust unified memory is refused before any allocation; swap use raises a warning alert; workers are killed on unload so memory always returns.
added docs/troubleshooting.md +32 −0
@@ -0,0 +1,32 @@
1+# Troubleshooting
2+
3+| Symptom | Where to look | Likely cause / fix |
4+|---|---|---|
5+| `MODEL_NOT_FOUND` | `GET /v1/models`, Models page | id/alias typo; model files missing (`installed=false` after rescan) |
6+| `MODEL_TOO_LARGE` / `not_recommended` | model page → *Memory by context* | budget 45 GB exceeded; lower context (`overrides.context`), use a smaller quant, or raise the budget in Settings (never above RAM − 8 GB) |
7+| `MODEL_INCOMPATIBLE` (architecture) | `pip show mlx-lm`, `llama-server --version` | runtime too old for the architecture: upgrade mlx-lm / llama.cpp, rescan |
8+| `MODEL_LOAD_FAILED` with log tail | `~/llm-api/logs/workers/worker-<id>.log` | corrupted download (re-download with force), missing tokenizer file, unsupported quant |
9+| `MODEL_LOAD_TIMEOUT` | same log + Activity Monitor | very large model on a cold SSD cache; raise `LOAD_TIMEOUT_SECONDS` |
10+| `WORKER_CRASHED` | worker log, `dmesg`-style Metal errors | out-of-memory at the Metal level → lower context / KV bits (`overrides.kv_bits=8`) |
11+| `CONTEXT_TOO_LARGE` | request | prompt longer than the loaded context; reload with a larger `context` (model page) |
12+| `GENERATION_TIMEOUT` | Settings | `GENERATION_TIMEOUT_SECONDS`; check thermal throttling in System |
13+| Swap warning / memory pressure `critical` | Dashboard alert, System | model exceeds the envelope; unload, lower budget; never leave it swapping |
14+| Download fails 401/403 | Settings → HF token | gated repo: set `HF_TOKEN`, accept the license on Hugging Face |
15+| Download fails 429 | Downloads history | HF rate limit: retry later |
16+| `INSUFFICIENT_DISK` | System → Storage | `MIN_FREE_DISK_GB` reserve; delete unused models |
17+| Dashboard 503 "not reachable" | `pm2 logs llm-api-web`, port 8301 | Next not running / build failed (`pnpm build`) |
18+| Login loop | cookies | behind HTTPS set `SECURE_COOKIES=true`; ensure the proxy passes `X-Forwarded-Proto` |
19+| Model shows loaded after a crash | restart server | stale state is cleared at startup (`workers.json`) |
20+| GPU shows `—` | `ioreg -r -c IOAccelerator` | metric unavailable on this macOS; degrades gracefully |
21+
22+Logs: `~/llm-api/logs/llm-api.log` (server), `~/llm-api/logs/workers/*.log` (per model), `pm2 logs`. Events: `/api/logs/events`, audit `/api/logs/audit`.
23+
24+Handy commands on the node:
25+
26+```bash
27+source ~/apps/llm-api/server/.venv/bin/activate && cd ~/llm-api
28+llm-api status # hardware + telemetry + model count
29+llm-api scan # rescan and list registry with compatibility
30+pgrep -fl "mlx_worker|llama-server"
31+curl -s localhost:8300/health
32+```
added scripts/install.sh +98 −0
@@ -0,0 +1,98 @@
1+#!/usr/bin/env bash
2+# LLM API — install on a clean Apple Silicon Mac.
3+# Usage: ./scripts/install.sh [--root ~/llm-api] [--no-llama] [--no-web] [--test]
4+set -euo pipefail
5+
6+ROOT="${LLM_API_ROOT:-$HOME/llm-api}"
7+WITH_LLAMA=1
8+WITH_WEB=1
9+RUN_TEST=0
10+for a in "$@"; do
11+ case "$a" in
12+ --root) shift; ROOT="$1";;
13+ --root=*) ROOT="${a#--root=}";;
14+ --no-llama) WITH_LLAMA=0;;
15+ --no-web) WITH_WEB=0;;
16+ --test) RUN_TEST=1;;
17+ esac
18+done
19+HERE="$(cd "$(dirname "$0")/.." && pwd)"
20+say() { printf "\033[1;34m==>\033[0m %s\n" "$*"; }
21+warn() { printf "\033[1;33m!!\033[0m %s\n" "$*"; }
22+
23+# 1. platform checks -------------------------------------------------------
24+[ "$(uname -s)" = "Darwin" ] || { echo "macOS required"; exit 1; }
25+[ "$(uname -m)" = "arm64" ] || { echo "Apple Silicon (arm64) required"; exit 1; }
26+CHIP=$(sysctl -n machdep.cpu.brand_string)
27+MEM_GB=$(( $(sysctl -n hw.memsize) / 1024 / 1024 / 1024 ))
28+FREE_GB=$(df -g "$HOME" | awk 'NR==2 {print $4}')
29+say "Detected $CHIP, ${MEM_GB} GB unified memory, ${FREE_GB} GB free on $HOME"
30+[ "$MEM_GB" -ge 16 ] || warn "Less than 16 GB of memory: only small models will run."
31+[ "$FREE_GB" -ge 120 ] || warn "Less than 120 GB free: downloads stop at the MIN_FREE_DISK_GB reserve."
32+export PATH="/opt/homebrew/bin:$HOME/.local/bin:$PATH"
33+
34+# 2. toolchain ---------------------------------------------------------------
35+xcode-select -p >/dev/null 2>&1 || { say "Installing Xcode Command Line Tools (a dialog may appear)"; xcode-select --install || true; }
36+command -v brew >/dev/null || { say "Installing Homebrew"; /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"; eval "$(/opt/homebrew/bin/brew shellenv)"; }
37+command -v uv >/dev/null || { say "Installing uv"; brew install uv; }
38+if [ "$WITH_WEB" = 1 ]; then
39+ command -v node >/dev/null || { say "Installing Node"; brew install node; }
40+ command -v pnpm >/dev/null || { say "Installing pnpm"; brew install pnpm; }
41+fi
42+if [ "$WITH_LLAMA" = 1 ]; then
43+ command -v llama-server >/dev/null || { say "Installing llama.cpp (Metal)"; brew install llama.cpp; }
44+fi
45+
46+# 3. directories ---------------------------------------------------------------
47+say "Creating $ROOT"
48+mkdir -p "$ROOT"/{models/{mlx,gguf,embeddings,rerankers,vision,manifests},data,logs}
49+chmod 700 "$ROOT/data"
50+
51+# 4. python env -----------------------------------------------------------------
52+cd "$HERE/server"
53+if [ ! -x .venv/bin/python ]; then
54+ say "Creating Python 3.13 virtualenv"
55+ uv venv --python 3.13 .venv
56+fi
57+say "Installing Python dependencies (MLX, MLX-LM, mlx-vlm, FastAPI…)"
58+uv pip install --python .venv/bin/python -q -e ".[mlx,test]" "mlx-vlm>=0.7" asgi-lifespan
59+.venv/bin/python - <<'PY'
60+import mlx.core as mx, mlx_lm
61+print(f" mlx {mx.__version__} · mlx-lm {mlx_lm.__version__} · Metal device: {mx.default_device()}")
62+PY
63+
64+# 5. web ------------------------------------------------------------------------
65+if [ "$WITH_WEB" = 1 ]; then
66+ say "Installing and building the dashboard"
67+ (cd "$HERE/web" && pnpm install --frozen-lockfile --silent && NEXT_TELEMETRY_DISABLED=1 pnpm build | tail -3)
68+fi
69+
70+# 6. env file (never overwrite) ----------------------------------------------------
71+if [ ! -f "$ROOT/.env" ]; then
72+ say "Writing $ROOT/.env from .env.example (edit ADMIN_EMAIL / ADMIN_PASSWORD / HF_TOKEN)"
73+ sed "s#/Users/USERNAME/llm-api#$ROOT#" "$HERE/.env.example" > "$ROOT/.env"
74+ chmod 600 "$ROOT/.env"
75+else
76+ say "$ROOT/.env already exists — left untouched"
77+fi
78+
79+# 7. database + scan ------------------------------------------------------------------
80+say "Initializing database and scanning $ROOT/models"
81+(cd "$ROOT" && "$HERE/server/.venv/bin/llm-api" scan | tail -5)
82+
83+# 8. optional local inference test -------------------------------------------------------
84+if [ "$RUN_TEST" = 1 ]; then
85+ say "Downloading a tiny test model and running one generation"
86+ "$HERE/server/.venv/bin/hf" download mlx-community/Qwen3-0.6B-4bit --local-dir "$ROOT/models/mlx/qwen/Qwen3-0.6B-4bit" >/dev/null
87+ "$HERE/server/.venv/bin/python" -m mlx_lm generate --model "$ROOT/models/mlx/qwen/Qwen3-0.6B-4bit" --prompt "Say hello in one word." --max-tokens 8 | tail -3
88+fi
89+
90+cat <<EOF
91+
92+Done. Next steps:
93+ 1. Edit $ROOT/.env (ADMIN_EMAIL, ADMIN_PASSWORD, HF_TOKEN, PUBLIC_URL)
94+ 2. Start the API: cd $ROOT && $HERE/server/.venv/bin/llm-api serve
95+ 3. Start the dashboard: cd $HERE/web && pnpm start -p 8301 -H 127.0.0.1
96+ 4. Open http://127.0.0.1:8300 — or put a TLS reverse proxy in front for https://$(grep -E '^DOMAIN=' "$ROOT/.env" | cut -d= -f2)
97+ Process management: see docs/deployment.md (PM2 / launchd examples).
98+EOF
added server/llm_api/__init__.py +3 −0
@@ -0,0 +1,3 @@
1+"""LLM API — private OpenAI-compatible local model server for Apple Silicon."""
2+
3+__version__ = "0.1.0"
added server/llm_api/api/__init__.py +0 −0
added server/llm_api/api/admin_routes.py +809 −0
@@ -0,0 +1,809 @@
1+"""Private management API (/api/*): auth, models, system, runtime, downloads, harvest, keys, settings, events."""
2+
3+from __future__ import annotations
4+
5+import asyncio
6+import json
7+import time
8+from pathlib import Path
9+from typing import Any
10+
11+from fastapi import APIRouter, Depends, Query, Request, Response
12+from fastapi.responses import StreamingResponse
13+from pydantic import BaseModel, Field
14+
15+from ..auth import SESSION_COOKIE, Principal, get_auth, login_limiter, require_admin
16+from ..errors import APIError, AuthError, Conflict, ModelNotFound, RateLimited
17+from ..events import bus
18+from ..hardware import GB, detect_hardware
19+
20+router = APIRouter(prefix="/api")
21+
22+SETTINGS_KEYS = {
23+ "max_model_memory_gb": float, "absolute_max_memory_gb": float, "min_free_disk_gb": float,
24+ "model_idle_timeout_minutes": int, "max_simultaneous_models": int, "preferred_runtime": str,
25+ "default_model": str, "preload_model": str, "log_prompts": bool, "allow_downloads": bool,
26+ "allow_gguf": bool, "allow_mlx": bool, "default_context": int, "default_max_tokens": int,
27+ "benchmark_max_tokens": int, "benchmark_runs": int,
28+}
29+
30+
31+def _ip(request: Request) -> str:
32+ fwd = request.headers.get("x-forwarded-for")
33+ return fwd.split(",")[0].strip() if fwd else (request.client.host if request.client else "?")
34+
35+
36+def _actor(request: Request) -> str:
37+ p = getattr(request.state, "principal", None)
38+ return f"{p.kind}:{p.name}" if p else "anonymous"
39+
40+
41+# ---------------------------------------------------------------------------
42+# Auth
43+# ---------------------------------------------------------------------------
44+
45+
46+class LoginBody(BaseModel):
47+ email: str
48+ password: str
49+
50+
51+class SetupBody(BaseModel):
52+ email: str
53+ password: str = Field(min_length=10)
54+
55+
56+class PasswordBody(BaseModel):
57+ current_password: str
58+ new_password: str = Field(min_length=10)
59+
60+
61+def _set_cookie(response: Response, request: Request, token: str) -> None:
62+ settings = request.app.state.settings
63+ secure = settings.secure_cookies or request.url.scheme == "https" or request.headers.get("x-forwarded-proto") == "https"
64+ response.set_cookie(SESSION_COOKIE, token, httponly=True, secure=secure, samesite="lax",
65+ max_age=settings.session_hours * 3600, path="/")
66+
67+
68+@router.get("/auth/status")
69+async def auth_status(request: Request):
70+ auth = get_auth(request)
71+ from ..auth import principal_from_request
72+ p = await principal_from_request(request, auth)
73+ return {"needs_setup": (await auth.user_count()) == 0, "authenticated": bool(p and p.has("admin")),
74+ "principal": {"kind": p.kind, "name": p.name} if p else None}
75+
76+
77+@router.post("/auth/setup")
78+async def auth_setup(body: SetupBody, request: Request, response: Response):
79+ auth = get_auth(request)
80+ if await auth.user_count() > 0:
81+ raise Conflict("Setup already completed.")
82+ uid = await auth.create_user(body.email, body.password)
83+ token = auth.make_session(uid, body.email.lower())
84+ _set_cookie(response, request, token)
85+ await request.app.state.db.audit("auth.setup", actor=body.email, ip=_ip(request))
86+ return {"ok": True, "email": body.email.lower()}
87+
88+
89+@router.post("/auth/login")
90+async def auth_login(body: LoginBody, request: Request, response: Response):
91+ auth = get_auth(request)
92+ if not login_limiter.check(_ip(request)):
93+ raise RateLimited("Too many login attempts. Try again in a minute.")
94+ user = await auth.authenticate(body.email, body.password)
95+ if not user:
96+ await request.app.state.db.audit("auth.login_failed", actor=body.email, ip=_ip(request))
97+ raise AuthError("Invalid email or password.", code="INVALID_CREDENTIALS")
98+ token = auth.make_session(user["id"], user["email"])
99+ _set_cookie(response, request, token)
100+ await request.app.state.db.audit("auth.login", actor=user["email"], ip=_ip(request))
101+ return {"ok": True, "email": user["email"]}
102+
103+
104+@router.post("/auth/logout")
105+async def auth_logout(request: Request, response: Response):
106+ response.delete_cookie(SESSION_COOKIE, path="/")
107+ return {"ok": True}
108+
109+
110+@router.get("/auth/me")
111+async def auth_me(request: Request, p: Principal = Depends(require_admin)):
112+ return {"kind": p.kind, "name": p.name, "scopes": sorted(p.scopes)}
113+
114+
115+@router.post("/auth/password")
116+async def auth_password(body: PasswordBody, request: Request, p: Principal = Depends(require_admin)):
117+ auth = get_auth(request)
118+ if p.kind != "session":
119+ raise APIError("Password changes require a dashboard session.")
120+ user = await auth.authenticate(p.name, body.current_password)
121+ if not user:
122+ raise AuthError("Current password is incorrect.", code="INVALID_CREDENTIALS")
123+ await auth.change_password(user["id"], body.new_password)
124+ await request.app.state.db.audit("auth.password_changed", actor=p.name, ip=_ip(request))
125+ return {"ok": True}
126+
127+
128+# ---------------------------------------------------------------------------
129+# Models
130+# ---------------------------------------------------------------------------
131+
132+
133+def _decorate(manager, m: dict) -> dict:
134+ st = manager.status_of(m["id"])
135+ m["status"] = st
136+ m["loaded"] = st == "ready"
137+ lm = manager.loaded.get(m["id"])
138+ if lm:
139+ m["worker"] = lm.to_dict()
140+ p = manager.progress.get(m["id"])
141+ if p:
142+ m["progress"] = p
143+ return m
144+
145+
146+@router.get("/models")
147+async def list_models(request: Request, p: Principal = Depends(require_admin), include_missing: bool = False):
148+ manager = request.app.state.manager
149+ models = await manager.registry.list_models(include_missing=include_missing)
150+ aliases = await manager.registry.aliases()
151+ by_model: dict[str, list[str]] = {}
152+ for a, mid in aliases.items():
153+ by_model.setdefault(mid, []).append(a)
154+ for m in models:
155+ _decorate(manager, m)
156+ m["aliases"] = by_model.get(m["id"], [])
157+ return {"models": models, "aliases": aliases, "count": len(models)}
158+
159+
160+@router.get("/models/{model_id}")
161+async def get_model(model_id: str, request: Request, p: Principal = Depends(require_admin)):
162+ manager = request.app.state.manager
163+ m = await manager.registry.resolve(model_id)
164+ if not m:
165+ raise ModelNotFound(f"Model '{model_id}' not found.")
166+ _decorate(manager, m)
167+ db = request.app.state.db
168+ m["benchmarks"] = await db.fetchall("SELECT * FROM model_benchmarks WHERE model_id=? ORDER BY created_at DESC LIMIT 30", (m["id"],))
169+ for b in m["benchmarks"]:
170+ for k in ("params", "notes"):
171+ if b.get(k):
172+ try:
173+ b[k] = json.loads(b[k])
174+ except Exception:
175+ pass
176+ m["events"] = await db.fetchall("SELECT * FROM model_events WHERE model_id=? ORDER BY created_at DESC LIMIT 50", (m["id"],))
177+ m["aliases"] = [a for a, mid in (await manager.registry.aliases()).items() if mid == m["id"]]
178+ m["recent_requests"] = await db.fetchall(
179+ "SELECT created_at, endpoint, prompt_tokens, completion_tokens, ttft_ms, total_ms, tps, status FROM inference_requests "
180+ "WHERE model_id=? ORDER BY created_at DESC LIMIT 20", (m["id"],))
181+ from ..models.estimator import CONTEXT_STEPS, estimate
182+ rt = m["runtime"]
183+ m["memory_curve"] = [estimate(m["weights_bytes"], rt, m["kv_bytes_per_token"] or 0, c, m["vision"]).to_dict()
184+ for c in CONTEXT_STEPS if not m["max_context"] or c <= max(m["max_context"], 2048)]
185+ return m
186+
187+
188+@router.get("/models/{model_id}/files")
189+async def model_files(model_id: str, request: Request, p: Principal = Depends(require_admin)):
190+ manager = request.app.state.manager
191+ m = await manager.registry.resolve(model_id)
192+ if not m:
193+ raise ModelNotFound(f"Model '{model_id}' not found.")
194+ return {"path": m["path"], "files": await manager.registry.files(m["id"])}
195+
196+
197+class LoadBody(BaseModel):
198+ context: int | None = None
199+ force: bool = False
200+
201+
202+@router.post("/models/{model_id}/load")
203+async def load_model(model_id: str, request: Request, body: LoadBody | None = None, p: Principal = Depends(require_admin)):
204+ manager = request.app.state.manager
205+ body = body or LoadBody()
206+ lm = await manager.load(model_id, context=body.context, force=body.force)
207+ await request.app.state.db.audit("model.load", actor=_actor(request), target=lm.model["id"], ip=_ip(request))
208+ return {"ok": True, "worker": lm.to_dict()}
209+
210+
211+@router.post("/models/{model_id}/unload")
212+async def unload_model(model_id: str, request: Request, p: Principal = Depends(require_admin)):
213+ manager = request.app.state.manager
214+ m = await manager.registry.resolve(model_id)
215+ if not m:
216+ raise ModelNotFound(f"Model '{model_id}' not found.")
217+ ok = await manager.unload(m["id"], reason="manual")
218+ await request.app.state.db.audit("model.unload", actor=_actor(request), target=m["id"], ip=_ip(request))
219+ return {"ok": ok, "status": manager.status_of(m["id"])}
220+
221+
222+class PatchModel(BaseModel):
223+ favorite: bool | None = None
224+ pinned: bool | None = None
225+ enabled: bool | None = None
226+ notes: str | None = None
227+ tags: list[str] | None = None
228+ name: str | None = None
229+ overrides: dict[str, Any] | None = None
230+
231+
232+@router.patch("/models/{model_id}")
233+async def patch_model(model_id: str, body: PatchModel, request: Request, p: Principal = Depends(require_admin)):
234+ manager = request.app.state.manager
235+ m = await manager.registry.resolve(model_id)
236+ if not m:
237+ raise ModelNotFound(f"Model '{model_id}' not found.")
238+ fields = {k: v for k, v in body.model_dump().items() if v is not None}
239+ if "overrides" in fields:
240+ ov = fields["overrides"]
241+ allowed = {"task", "vision", "embedding", "reranker", "family", "quantization", "max_context", "context", "kv_bits",
242+ "parameter_count", "pooling", "llama_args", "thinking", "tools", "name"}
243+ fields["overrides"] = {k: v for k, v in ov.items() if k in allowed}
244+ if fields:
245+ await manager.registry.update(m["id"], **fields)
246+ if "overrides" in fields:
247+ await manager.registry.rescan()
248+ await manager.refresh_model(m["id"])
249+ await request.app.state.db.audit("model.update", actor=_actor(request), target=m["id"], detail=fields, ip=_ip(request))
250+ return _decorate(manager, await manager.registry.get(m["id"])) # type: ignore[arg-type]
251+
252+
253+@router.post("/models/{model_id}/pin")
254+async def pin_model(model_id: str, request: Request, p: Principal = Depends(require_admin), pinned: bool = True):
255+ manager = request.app.state.manager
256+ m = await manager.registry.resolve(model_id)
257+ if not m:
258+ raise ModelNotFound(f"Model '{model_id}' not found.")
259+ await manager.registry.update(m["id"], pinned=pinned)
260+ await manager.refresh_model(m["id"])
261+ return {"ok": True, "pinned": pinned}
262+
263+
264+@router.post("/models/{model_id}/favorite")
265+async def favorite_model(model_id: str, request: Request, p: Principal = Depends(require_admin), favorite: bool = True):
266+ manager = request.app.state.manager
267+ m = await manager.registry.resolve(model_id)
268+ if not m:
269+ raise ModelNotFound(f"Model '{model_id}' not found.")
270+ await manager.registry.update(m["id"], favorite=favorite)
271+ return {"ok": True, "favorite": favorite}
272+
273+
274+class DeleteBody(BaseModel):
275+ confirm: str
276+ keep_benchmarks: bool = True
277+
278+
279+@router.delete("/models/{model_id}")
280+async def delete_model(model_id: str, body: DeleteBody, request: Request, p: Principal = Depends(require_admin)):
281+ manager = request.app.state.manager
282+ m = await manager.registry.resolve(model_id)
283+ if not m:
284+ raise ModelNotFound(f"Model '{model_id}' not found.")
285+ if body.confirm != m["id"]:
286+ raise APIError("Type the exact model id in 'confirm' to delete it.", code="CONFIRMATION_REQUIRED")
287+ if manager.loaded.get(m["id"]):
288+ raise Conflict("Unload the model before deleting it.")
289+ return await request.app.state.downloader.delete_model(m["id"], actor=_actor(request), keep_benchmarks=body.keep_benchmarks)
290+
291+
292+@router.post("/models/rescan")
293+async def rescan(request: Request, p: Principal = Depends(require_admin)):
294+ state = request.app.state
295+
296+ async def run(job):
297+ def prog(frac, name):
298+ state.jobs.update(job, progress=frac, current=name)
299+ summary = await state.manager.registry.rescan(progress=prog)
300+ bus.publish("models", {"event": "rescan", **summary})
301+ return summary
302+
303+ job = state.jobs.submit("scan", "Rescan model directory", {}, run)
304+ # Scans are quick: wait a bit so callers get the result directly when possible
305+ for _ in range(100):
306+ await asyncio.sleep(0.1)
307+ if job.status in ("completed", "failed"):
308+ break
309+ return {"job": job.to_dict()}
310+
311+
312+class BenchBody(BaseModel):
313+ max_tokens: int = 256
314+ runs: int = 2
315+ long_prompt: bool = True
316+
317+
318+@router.post("/models/{model_id}/benchmark")
319+async def benchmark(model_id: str, request: Request, body: BenchBody | None = None, p: Principal = Depends(require_admin)):
320+ state = request.app.state
321+ m = await state.manager.registry.resolve(model_id)
322+ if not m:
323+ raise ModelNotFound(f"Model '{model_id}' not found.")
324+ body = body or BenchBody()
325+ from ..bench import run_benchmark
326+
327+ async def run(job):
328+ return await run_benchmark(state, m["id"], job, body.model_dump())
329+
330+ job = state.jobs.submit("benchmark", f"Benchmark {m['name']}", {"model_id": m["id"], **body.model_dump()}, run)
331+ return {"job": job.to_dict()}
332+
333+
334+@router.get("/models/{model_id}/benchmarks")
335+async def benchmarks(model_id: str, request: Request, p: Principal = Depends(require_admin)):
336+ rows = await request.app.state.db.fetchall("SELECT * FROM model_benchmarks WHERE model_id=? ORDER BY created_at DESC LIMIT 100", (model_id,))
337+ for b in rows:
338+ for k in ("params", "notes"):
339+ if b.get(k):
340+ try:
341+ b[k] = json.loads(b[k])
342+ except Exception:
343+ pass
344+ return {"benchmarks": rows}
345+
346+
347+# ---------------------------------------------------------------------------
348+# Aliases
349+# ---------------------------------------------------------------------------
350+
351+
352+class AliasBody(BaseModel):
353+ alias: str = Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
354+ model_id: str
355+
356+
357+@router.get("/aliases")
358+async def list_aliases(request: Request, p: Principal = Depends(require_admin)):
359+ return {"aliases": await request.app.state.manager.registry.aliases()}
360+
361+
362+@router.put("/aliases")
363+async def put_alias(body: AliasBody, request: Request, p: Principal = Depends(require_admin)):
364+ reg = request.app.state.manager.registry
365+ m = await reg.get(body.model_id)
366+ if not m:
367+ raise ModelNotFound(f"Model '{body.model_id}' not found.")
368+ if await reg.get(body.alias):
369+ raise Conflict("An installed model already has this id.")
370+ if body.alias == "auto":
371+ raise APIError("'auto' is reserved.")
372+ await reg.set_alias(body.alias, m["id"])
373+ await request.app.state.db.audit("alias.set", actor=_actor(request), target=body.alias, detail=body.model_id)
374+ return {"aliases": await reg.aliases()}
375+
376+
377+@router.delete("/aliases/{alias}")
378+async def delete_alias(alias: str, request: Request, p: Principal = Depends(require_admin)):
379+ reg = request.app.state.manager.registry
380+ await reg.delete_alias(alias)
381+ return {"aliases": await reg.aliases()}
382+
383+
384+# ---------------------------------------------------------------------------
385+# System / runtime
386+# ---------------------------------------------------------------------------
387+
388+
389+@router.get("/system")
390+async def system(request: Request, p: Principal = Depends(require_admin)):
391+ state = request.app.state
392+ hw = detect_hardware(state.settings.models_dir).to_dict()
393+ tel = state.metrics.last or await state.metrics.sample()
394+ budget, absolute, max_models = await state.manager.budgets()
395+ return {"hardware": hw, "telemetry": tel, "manager": state.manager.snapshot(),
396+ "policy": {"max_model_memory_gb": budget, "absolute_max_memory_gb": absolute, "max_simultaneous_models": max_models,
397+ "macos_reserve_gb": state.settings.macos_reserve_gb,
398+ "min_free_disk_gb": float(await state.db.get_setting("min_free_disk_gb", state.settings.min_free_disk_gb))},
399+ "version": request.app.version, "started_at": state.metrics.started_at,
400+ "runtimes": await _runtime_versions()}
401+
402+
403+async def _runtime_versions() -> dict:
404+ out: dict[str, Any] = {}
405+ try:
406+ import mlx.core as mx
407+ import mlx_lm
408+ out["mlx"] = mx.__version__
409+ out["mlx_lm"] = mlx_lm.__version__
410+ except Exception:
411+ out["mlx"] = None
412+ try:
413+ import mlx_vlm
414+ out["mlx_vlm"] = mlx_vlm.__version__
415+ except Exception:
416+ out["mlx_vlm"] = None
417+ try:
418+ import shutil
419+ import subprocess
420+ b = shutil.which("llama-server") or "/opt/homebrew/bin/llama-server"
421+ r = subprocess.run([b, "--version"], capture_output=True, text=True, timeout=5)
422+ out["llama_cpp"] = (r.stdout + r.stderr).strip().splitlines()[0][:80] if (r.stdout or r.stderr) else None
423+ except Exception:
424+ out["llama_cpp"] = None
425+ return out
426+
427+
428+@router.get("/system/memory")
429+async def system_memory(request: Request, p: Principal = Depends(require_admin)):
430+ tel = await request.app.state.metrics.sample()
431+ return {k: tel[k] for k in tel if k.startswith(("mem_", "swap_", "worker_", "loaded_"))}
432+
433+
434+@router.get("/system/gpu")
435+async def system_gpu(request: Request, p: Principal = Depends(require_admin)):
436+ tel = await request.app.state.metrics.sample()
437+ hw = detect_hardware(request.app.state.settings.models_dir)
438+ return {"gpu_cores": hw.gpu_cores, "chip": hw.chip, "gpu_percent": tel["gpu_percent"],
439+ "gpu_renderer_percent": tel["gpu_renderer_percent"], "gpu_memory_gb": tel["gpu_memory_gb"],
440+ "thermal_state": tel["thermal_state"]}
441+
442+
443+@router.get("/system/storage")
444+async def system_storage(request: Request, p: Principal = Depends(require_admin)):
445+ state = request.app.state
446+ settings = state.settings
447+ import shutil
448+
449+ def du(path: Path) -> int:
450+ total = 0
451+ if path.exists():
452+ for f in path.rglob("*"):
453+ if f.is_file():
454+ try:
455+ total += f.stat().st_size
456+ except OSError:
457+ pass
458+ return total
459+
460+ usage = shutil.disk_usage(settings.models_dir)
461+ models_bytes, logs_bytes, db_bytes = await asyncio.gather(
462+ asyncio.to_thread(du, settings.models_dir), asyncio.to_thread(du, settings.logs_path),
463+ asyncio.to_thread(lambda: sum(du(p) for p in [settings.db_path] if p.exists()) + du(settings.data_path)))
464+ hf_cache = Path.home() / ".cache" / "huggingface"
465+ cache_bytes = await asyncio.to_thread(du, hf_cache)
466+ by_model = await state.db.fetchall("SELECT id, name, disk_size_bytes, runtime, last_used_at FROM models WHERE installed=1 ORDER BY disk_size_bytes DESC")
467+ return {"total_gb": round(usage.total / GB, 1), "used_gb": round(usage.used / GB, 1), "free_gb": round(usage.free / GB, 1),
468+ "models_gb": round(models_bytes / GB, 2), "logs_gb": round(logs_bytes / GB, 3), "database_gb": round(db_bytes / GB, 3),
469+ "cache_gb": round(cache_bytes / GB, 2), "other_gb": round(max(0, usage.used - models_bytes - logs_bytes - db_bytes - cache_bytes) / GB, 1),
470+ "model_root": str(settings.models_dir), "models": by_model,
471+ "min_free_gb": float(await state.db.get_setting("min_free_disk_gb", settings.min_free_disk_gb))}
472+
473+
474+@router.get("/system/processes")
475+async def system_processes(request: Request, p: Principal = Depends(require_admin)):
476+ import psutil
477+ manager = request.app.state.manager
478+ procs = []
479+ for lm in manager.loaded.values():
480+ try:
481+ pr = psutil.Process(lm.handle.pid)
482+ procs.append({"model_id": lm.model["id"], "pid": lm.handle.pid, "port": lm.handle.port, "runtime": lm.handle.runtime,
483+ "rss_gb": round(lm.handle.memory_bytes() / GB, 2), "cpu_percent": pr.cpu_percent(interval=None),
484+ "status": lm.status, "threads": pr.num_threads(), "created": pr.create_time()})
485+ except psutil.Error:
486+ pass
487+ me = psutil.Process()
488+ return {"workers": procs, "server": {"pid": me.pid, "rss_gb": round(me.memory_info().rss / GB, 3),
489+ "cpu_percent": me.cpu_percent(interval=None), "threads": me.num_threads()}}
490+
491+
492+@router.get("/system/health")
493+async def system_health_admin(request: Request):
494+ return await public_health(request)
495+
496+
497+async def public_health(request: Request) -> dict:
498+ state = request.app.state
499+ hw = detect_hardware(state.settings.models_dir)
500+ tel = state.metrics.last or await state.metrics.sample()
501+ cur = state.manager.current_model()
502+ return {"status": "ok", "hardware": {"chip": hw.chip, "memory_gb": hw.memory_gb, "gpu_cores": hw.gpu_cores},
503+ "model": {"loaded": cur is not None, "id": cur.model["id"] if cur else None,
504+ "loaded_models": [lm.model["id"] for lm in state.manager.loaded.values() if lm.status == "ready"]},
505+ "memory": {"used_gb": tel["mem_used_gb"], "available_gb": tel["mem_available_gb"], "pressure": tel["mem_pressure_level"]},
506+ "uptime_seconds": round(time.time() - state.metrics.started_at), "version": request.app.version}
507+
508+
509+@router.get("/system/metrics")
510+async def system_metrics(request: Request, p: Principal = Depends(require_admin), minutes: int = Query(60, ge=5, le=60 * 24 * 7),
511+ hours: int = Query(24, ge=1, le=24 * 30)):
512+ state = request.app.state
513+ return {"history": await state.metrics.history(minutes), "requests": await state.metrics.request_stats(hours),
514+ "live": state.metrics.last}
515+
516+
517+@router.get("/runtime/status")
518+async def runtime_status(request: Request, p: Principal = Depends(require_admin)):
519+ return request.app.state.manager.snapshot()
520+
521+
522+@router.get("/runtime/current-model")
523+async def runtime_current(request: Request, p: Principal = Depends(require_admin)):
524+ cur = request.app.state.manager.current_model()
525+ return {"model": cur.to_dict() if cur else None}
526+
527+
528+# ---------------------------------------------------------------------------
529+# Jobs / downloads / harvest
530+# ---------------------------------------------------------------------------
531+
532+
533+@router.get("/jobs")
534+async def jobs(request: Request, p: Principal = Depends(require_admin), kind: str | None = None, limit: int = 100):
535+ kinds = {kind} if kind else None
536+ return {"jobs": await request.app.state.jobs.history(kinds, limit)}
537+
538+
539+@router.post("/jobs/{job_id}/cancel")
540+async def cancel_job(job_id: str, request: Request, p: Principal = Depends(require_admin)):
541+ ok = await request.app.state.jobs.cancel(job_id)
542+ return {"ok": ok}
543+
544+
545+class InspectBody(BaseModel):
546+ repository: str
547+ quant: str | None = None
548+
549+
550+class DownloadBody(BaseModel):
551+ repository: str
552+ quant: str | None = None
553+ force: bool = False
554+
555+
556+@router.post("/models/inspect")
557+async def inspect_repo(body: InspectBody, request: Request, p: Principal = Depends(require_admin)):
558+ return await request.app.state.downloader.inspect(body.repository, body.quant)
559+
560+
561+@router.post("/models/download")
562+async def download(body: DownloadBody, request: Request, p: Principal = Depends(require_admin)):
563+ job = await request.app.state.downloader.start_download(body.repository, body.quant, force=body.force, actor=_actor(request))
564+ return {"job": job.to_dict()}
565+
566+
567+@router.get("/downloads")
568+async def downloads(request: Request, p: Principal = Depends(require_admin)):
569+ return {"downloads": await request.app.state.jobs.history({"download"}, 100)}
570+
571+
572+@router.post("/downloads/{job_id}/retry")
573+async def retry_download(job_id: str, request: Request, p: Principal = Depends(require_admin)):
574+ rows = await request.app.state.db.fetchone("SELECT payload FROM jobs WHERE id=?", (job_id,))
575+ if not rows:
576+ raise APIError("Job not found.", status_code=404, code="JOB_NOT_FOUND")
577+ payload = json.loads(rows["payload"] or "{}")
578+ job = await request.app.state.downloader.start_download(payload["repository"], payload.get("quant"), force=True, actor=_actor(request))
579+ return {"job": job.to_dict()}
580+
581+
582+class HarvestBody(BaseModel):
583+ runtimes: list[str] | None = None
584+ authors: list[str] | None = None
585+ limit_per_author: int = 150
586+ min_downloads: int = 500
587+ max_ram_gb: float | None = None
588+ families: list[str] | None = None
589+ tasks: list[str] | None = None
590+ search: str | None = None
591+
592+
593+@router.post("/harvest/scan")
594+async def harvest_scan(body: HarvestBody, request: Request, p: Principal = Depends(require_admin)):
595+ job = await request.app.state.harvester.start_scan(body.model_dump(), actor=_actor(request))
596+ return {"job": job.to_dict()}
597+
598+
599+@router.get("/harvest/candidates")
600+async def harvest_candidates(request: Request, p: Principal = Depends(require_admin), include_duplicates: bool = False,
601+ task: str | None = None, runtime: str | None = None, family: str | None = None,
602+ size_class: str | None = None, q: str | None = None, limit: int = 300):
603+ h = request.app.state.harvester
604+ rows = await h.candidates(include_duplicates=include_duplicates, task=task, runtime=runtime, family=family,
605+ size_class=size_class, q=q, limit=limit)
606+ last = await request.app.state.db.scalar("SELECT MAX(scanned_at) FROM harvest_candidates")
607+ return {"candidates": rows, "last_scan": last, "starter": await h.suggest_starter()}
608+
609+
610+class SelectBody(BaseModel):
611+ repo_id: str
612+ selected: bool = True
613+
614+
615+@router.post("/harvest/select")
616+async def harvest_select(body: SelectBody, request: Request, p: Principal = Depends(require_admin)):
617+ await request.app.state.harvester.select(body.repo_id, body.selected)
618+ return {"ok": True}
619+
620+
621+@router.post("/harvest/dismiss")
622+async def harvest_dismiss(body: SelectBody, request: Request, p: Principal = Depends(require_admin)):
623+ await request.app.state.harvester.dismiss(body.repo_id)
624+ return {"ok": True}
625+
626+
627+@router.post("/harvest/queue")
628+async def harvest_queue(request: Request, p: Principal = Depends(require_admin)):
629+ return {"queued": await request.app.state.harvester.queue_selected(actor=_actor(request))}
630+
631+
632+# ---------------------------------------------------------------------------
633+# API keys
634+# ---------------------------------------------------------------------------
635+
636+
637+class KeyBody(BaseModel):
638+ name: str = Field(min_length=1, max_length=64)
639+ scopes: list[str] = ["inference"]
640+
641+
642+@router.get("/keys")
643+async def list_keys(request: Request, p: Principal = Depends(require_admin)):
644+ rows = await request.app.state.db.fetchall(
645+ "SELECT id, name, prefix, scopes, created_at, last_used_at, request_count, revoked_at FROM api_keys ORDER BY created_at DESC")
646+ for r in rows:
647+ r["scopes"] = r["scopes"].split(",")
648+ return {"keys": rows}
649+
650+
651+@router.post("/keys")
652+async def create_key(body: KeyBody, request: Request, p: Principal = Depends(require_admin)):
653+ scopes = [s for s in body.scopes if s in ("inference", "admin")] or ["inference"]
654+ raw, row = await get_auth(request).create_key(body.name, scopes)
655+ await request.app.state.db.audit("key.create", actor=_actor(request), target=body.name, ip=_ip(request))
656+ row = dict(row)
657+ row["scopes"] = row["scopes"].split(",")
658+ return {"key": raw, "record": row}
659+
660+
661+class KeyPatch(BaseModel):
662+ name: str | None = None
663+
664+
665+@router.patch("/keys/{key_id}")
666+async def rename_key(key_id: int, body: KeyPatch, request: Request, p: Principal = Depends(require_admin)):
667+ if body.name:
668+ await request.app.state.db.execute("UPDATE api_keys SET name=? WHERE id=?", (body.name, key_id))
669+ return {"ok": True}
670+
671+
672+@router.delete("/keys/{key_id}")
673+async def revoke_key(key_id: int, request: Request, p: Principal = Depends(require_admin)):
674+ await request.app.state.db.execute("UPDATE api_keys SET revoked_at=? WHERE id=? AND revoked_at IS NULL", (time.time(), key_id))
675+ await request.app.state.db.audit("key.revoke", actor=_actor(request), target=str(key_id), ip=_ip(request))
676+ return {"ok": True}
677+
678+
679+# ---------------------------------------------------------------------------
680+# Settings / logs / events
681+# ---------------------------------------------------------------------------
682+
683+
684+@router.get("/settings")
685+async def get_settings_(request: Request, p: Principal = Depends(require_admin)):
686+ state = request.app.state
687+ s = state.settings
688+ defaults = {
689+ "max_model_memory_gb": s.max_model_memory_gb, "absolute_max_memory_gb": s.absolute_max_memory_gb,
690+ "min_free_disk_gb": s.min_free_disk_gb, "model_idle_timeout_minutes": s.model_idle_timeout_minutes,
691+ "max_simultaneous_models": s.max_simultaneous_models, "preferred_runtime": "mlx", "default_model": None,
692+ "preload_model": s.preload_model, "log_prompts": s.log_prompts, "allow_downloads": s.allow_downloads,
693+ "allow_gguf": s.enable_gguf, "allow_mlx": s.enable_mlx, "default_context": s.default_context,
694+ "default_max_tokens": s.default_max_tokens, "benchmark_max_tokens": 256, "benchmark_runs": 2,
695+ }
696+ stored = await state.db.all_settings()
697+ merged = {**defaults, **{k: v for k, v in stored.items() if k in defaults}}
698+ return {"settings": merged, "defaults": defaults, "paths": {"root": str(s.root), "models": str(s.models_dir),
699+ "data": str(s.data_path), "logs": str(s.logs_path), "db": str(s.db_path)},
700+ "hf_token_set": bool(s.hf_token), "public_url": s.public_url}
701+
702+
703+@router.patch("/settings")
704+async def patch_settings(request: Request, p: Principal = Depends(require_admin)):
705+ state = request.app.state
706+ body = await request.json()
707+ if not isinstance(body, dict):
708+ raise APIError("Body must be an object.")
709+ changed = {}
710+ hw = detect_hardware(state.settings.models_dir)
711+ for k, v in body.items():
712+ if k not in SETTINGS_KEYS:
713+ continue
714+ typ = SETTINGS_KEYS[k]
715+ try:
716+ if typ is bool:
717+ v = bool(v)
718+ elif v is None or v == "":
719+ v = None
720+ else:
721+ v = typ(v)
722+ except (TypeError, ValueError):
723+ raise APIError(f"Invalid value for {k}.", param=k)
724+ if k in ("max_model_memory_gb", "absolute_max_memory_gb") and v is not None:
725+ if v < 1 or v > hw.memory_gb - 4:
726+ raise APIError(f"{k} must be between 1 and {hw.memory_gb - 4:.0f} GB on this machine.", param=k)
727+ if k == "max_simultaneous_models" and v is not None and (v < 1 or v > 4):
728+ raise APIError("max_simultaneous_models must be 1-4.", param=k)
729+ changed[k] = v
730+ await state.db.set_setting(k, v)
731+ if changed:
732+ await state.db.audit("settings.update", actor=_actor(request), detail=changed, ip=_ip(request))
733+ if any(k in changed for k in ("max_model_memory_gb", "absolute_max_memory_gb")):
734+ await state.manager.registry.reevaluate_all()
735+ bus.publish("settings", changed)
736+ return {"changed": changed}
737+
738+
739+@router.get("/logs/audit")
740+async def audit_logs(request: Request, p: Principal = Depends(require_admin), limit: int = 200):
741+ rows = await request.app.state.db.fetchall("SELECT * FROM audit_logs ORDER BY created_at DESC LIMIT ?", (limit,))
742+ return {"logs": rows}
743+
744+
745+@router.get("/logs/events")
746+async def model_events(request: Request, p: Principal = Depends(require_admin), limit: int = 200):
747+ rows = await request.app.state.db.fetchall("SELECT * FROM model_events ORDER BY created_at DESC LIMIT ?", (limit,))
748+ return {"events": rows}
749+
750+
751+@router.get("/logs/requests")
752+async def request_logs(request: Request, p: Principal = Depends(require_admin), limit: int = 200):
753+ rows = await request.app.state.db.fetchall(
754+ "SELECT id, created_at, model_id, requested_model, endpoint, api_key_id, stream, prompt_tokens, completion_tokens, ttft_ms, "
755+ "total_ms, tps, load_wait_ms, status, error_code FROM inference_requests ORDER BY created_at DESC LIMIT ?", (limit,))
756+ return {"requests": rows}
757+
758+
759+@router.get("/logs/worker/{model_id}")
760+async def worker_log(model_id: str, request: Request, p: Principal = Depends(require_admin), lines: int = 200):
761+ path = request.app.state.settings.logs_path / "workers" / f"worker-{model_id}.log"
762+ if not path.exists():
763+ return {"lines": []}
764+ txt = path.read_text(errors="replace").splitlines()
765+ return {"lines": txt[-lines:]}
766+
767+
768+@router.get("/events")
769+async def events(request: Request, p: Principal = Depends(require_admin)):
770+ q = bus.subscribe()
771+ state = request.app.state
772+
773+ async def gen():
774+ try:
775+ # initial snapshot
776+ snap = {"seq": 0, "ts": time.time(), "type": "snapshot",
777+ "data": {"manager": state.manager.snapshot(), "metrics": state.metrics.last,
778+ "jobs": state.jobs.list(limit=30)}}
779+ yield bus.format_sse(snap)
780+ while True:
781+ try:
782+ ev = await asyncio.wait_for(q.get(), timeout=15)
783+ yield bus.format_sse(ev)
784+ except asyncio.TimeoutError:
785+ yield ": ping\n\n"
786+ if await request.is_disconnected():
787+ break
788+ finally:
789+ bus.unsubscribe(q)
790+
791+ return StreamingResponse(gen(), media_type="text/event-stream",
792+ headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})
793+
794+
795+@router.post("/models/{model_id}/tokenize")
796+async def tokenize(model_id: str, request: Request, p: Principal = Depends(require_admin)):
797+ manager = request.app.state.manager
798+ lm = manager.get_ready(model_id)
799+ if not lm:
800+ raise Conflict("Model is not loaded.")
801+ body = await request.json()
802+ if lm.handle.runtime == "mlx":
803+ r = await manager.client.post(f"{lm.handle.base_url}/tokenize", json=body, timeout=30)
804+ else:
805+ text = body.get("text") or " ".join(m.get("content", "") for m in body.get("messages", []) if isinstance(m.get("content"), str))
806+ r = await manager.client.post(f"{lm.handle.base_url}/tokenize", json={"content": text}, timeout=30)
807+ d = r.json()
808+ return {"tokens": len(d.get("tokens", [])), "max_context": lm.handle.context}
809+ return r.json()
added server/llm_api/api/openai_routes.py +369 −0
@@ -0,0 +1,369 @@
1+"""OpenAI-compatible endpoints. Requests are validated, the model is loaded on demand, then the
2+request is proxied to the local worker. Usage/timings are recorded from the worker's response."""
3+
4+from __future__ import annotations
5+
6+import asyncio
7+import json
8+import logging
9+import time
10+from typing import Any
11+
12+import httpx
13+from fastapi import APIRouter, Depends, Request
14+from fastapi.responses import JSONResponse, StreamingResponse
15+
16+from ..auth import Principal, require_inference
17+from ..errors import APIError, GenerationTimeout, WorkerCrashed
18+from ..events import bus
19+from ..manager import ModelManager
20+
21+log = logging.getLogger("llm_api.openai")
22+router = APIRouter()
23+
24+
25+def _manager(request: Request) -> ModelManager:
26+ return request.app.state.manager
27+
28+
29+async def _parse_body(request: Request) -> dict:
30+ settings = request.app.state.settings
31+ raw = await request.body()
32+ if len(raw) > settings.max_body_bytes:
33+ raise APIError("Request body too large.", status_code=413, code="BODY_TOO_LARGE")
34+ try:
35+ body = json.loads(raw or b"{}")
36+ except json.JSONDecodeError:
37+ raise APIError("Request body is not valid JSON.")
38+ if not isinstance(body, dict):
39+ raise APIError("Request body must be a JSON object.")
40+ return body
41+
42+
43+def _validate_sampling(body: dict) -> None:
44+ def num(k, lo, hi):
45+ v = body.get(k)
46+ if v is None:
47+ return
48+ if not isinstance(v, (int, float)) or v < lo or v > hi:
49+ raise APIError(f"{k} must be a number between {lo} and {hi}.", param=k)
50+ num("temperature", 0, 2)
51+ num("top_p", 0, 1)
52+ num("presence_penalty", -2, 2)
53+ num("frequency_penalty", -2, 2)
54+ mt = body.get("max_tokens", body.get("max_completion_tokens"))
55+ if mt is not None and (not isinstance(mt, int) or mt < 1 or mt > 200000):
56+ raise APIError("max_tokens must be a positive integer.", param="max_tokens")
57+ n = body.get("n")
58+ if n not in (None, 1):
59+ raise APIError("Only n=1 is supported.", param="n")
60+ stop = body.get("stop")
61+ if stop is not None and not isinstance(stop, (str, list)):
62+ raise APIError("stop must be a string or an array of strings.", param="stop")
63+ if isinstance(stop, list) and len(stop) > 8:
64+ raise APIError("At most 8 stop sequences are supported.", param="stop")
65+
66+
67+async def _pick_model(request: Request, body: dict, endpoint: str) -> tuple[str, dict]:
68+ """Resolve model name (alias/auto) -> registry row."""
69+ manager = _manager(request)
70+ name = body.get("model")
71+ if not name or not isinstance(name, str):
72+ # default model setting
73+ name = await request.app.state.db.get_setting("default_model")
74+ if not name:
75+ cur = manager.current_model()
76+ if cur:
77+ name = cur.model["id"]
78+ if not name:
79+ raise APIError("model is required.", param="model")
80+ if name == "auto":
81+ from ..routing import choose_auto
82+ name = await choose_auto(request.app.state, body, endpoint)
83+ model = await manager.registry.resolve(name)
84+ if not model:
85+ from ..errors import ModelNotFound
86+ raise ModelNotFound(f"The model '{name}' does not exist. Use GET /v1/models to list available models.")
87+ return name, model
88+
89+
90+async def _record(app_state, *, model_id: str | None, requested: str, endpoint: str, principal: Principal,
91+ stream: bool, usage: dict | None, timings: dict | None, status: int, error_code: str | None,
92+ started: float, load_wait_ms: float, prompt: Any = None, completion: str | None = None) -> None:
93+ db = app_state.db
94+ settings = app_state.settings
95+ log_prompts = bool(await db.get_setting("log_prompts", settings.log_prompts))
96+ usage = usage or {}
97+ timings = timings or {}
98+ tps = timings.get("generation_tps") or timings.get("predicted_per_second")
99+ ttft = timings.get("ttft_ms") or timings.get("prompt_ms")
100+ total_ms = (time.time() - started) * 1000
101+ ctoks = usage.get("completion_tokens") or timings.get("predicted_n")
102+ ptoks = usage.get("prompt_tokens") or timings.get("prompt_n")
103+ await db.execute(
104+ "INSERT INTO inference_requests(created_at, model_id, requested_model, endpoint, api_key_id, stream, prompt_tokens, "
105+ "completion_tokens, ttft_ms, total_ms, tps, load_wait_ms, status, error_code, prompt, completion) "
106+ "VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
107+ (started, model_id, requested, endpoint, principal.id if principal.kind == "api_key" else None, int(stream),
108+ ptoks, ctoks, ttft, round(total_ms, 1), tps, round(load_wait_ms, 1), status, error_code,
109+ json.dumps(prompt, ensure_ascii=False)[:20000] if (log_prompts and prompt is not None) else None,
110+ (completion or "")[:20000] if log_prompts else None))
111+ if model_id and status < 400:
112+ await app_state.manager.registry.record_usage(model_id, int(ctoks or 0), tps, ttft)
113+ app_state.manager.stats["tokens"] += int(ctoks or 0)
114+ bus.publish("request", {"model_id": model_id, "endpoint": endpoint, "status": status, "tps": tps, "ttft_ms": ttft,
115+ "completion_tokens": ctoks, "prompt_tokens": ptoks, "total_ms": round(total_ms), "stream": stream})
116+
117+
118+def _extract_timings(data: dict) -> dict:
119+ t = data.get("timings") or {}
120+ if "predicted_per_second" in t and "generation_tps" not in t: # llama.cpp
121+ t = {"generation_tps": t.get("predicted_per_second"), "prompt_tps": t.get("prompt_per_second"),
122+ "ttft_ms": t.get("prompt_ms"), "prompt_ms": t.get("prompt_ms"), "generation_ms": t.get("predicted_ms"),
123+ "predicted_n": t.get("predicted_n"), "prompt_n": t.get("prompt_n"), "cached_tokens": t.get("cache_n")}
124+ return t
125+
126+
127+async def _proxy(request: Request, endpoint: str, body: dict, principal: Principal, *, kind: str):
128+ """Common path for chat/completions/embeddings/rerank."""
129+ app_state = request.app.state
130+ manager: ModelManager = app_state.manager
131+ started = time.time()
132+ requested, model = await _pick_model(request, body, endpoint)
133+ if kind == "chat" or kind == "completion":
134+ if model.get("embedding") or model.get("reranker"):
135+ raise APIError(f"Model '{model['id']}' is an {'embedding' if model.get('embedding') else 'reranking'} model and cannot generate text.",
136+ code="WRONG_MODEL_TYPE")
137+ if kind == "embeddings" and not model.get("embedding"):
138+ # Allow causal LMs served by MLX to embed (last-token pooling) only if explicitly tagged; otherwise reject
139+ if model["runtime"] != "mlx":
140+ raise APIError(f"Model '{model['id']}' is not an embedding model.", code="WRONG_MODEL_TYPE")
141+ if kind == "rerank" and not model.get("reranker"):
142+ raise APIError(f"Model '{model['id']}' is not a reranking model.", code="WRONG_MODEL_TYPE")
143+ stream = bool(body.get("stream")) and kind in ("chat", "completion")
144+ t_load = time.time()
145+ try:
146+ lm = await manager.ensure_loaded(model["id"], reason=f"{endpoint}")
147+ except APIError as e:
148+ await _record(app_state, model_id=model["id"], requested=requested, endpoint=endpoint, principal=principal,
149+ stream=stream, usage=None, timings=None, status=e.status_code, error_code=e.code, started=started,
150+ load_wait_ms=(time.time() - t_load) * 1000)
151+ raise
152+ load_wait_ms = (time.time() - t_load) * 1000
153+ body = dict(body)
154+ body["model"] = model["id"]
155+ if stream and kind == "chat":
156+ so = dict(body.get("stream_options") or {})
157+ so["include_usage"] = True
158+ body["stream_options"] = so
159+ url = f"{lm.handle.base_url}{endpoint}"
160+ timeout = httpx.Timeout(connect=10.0, read=float(app_state.settings.generation_timeout_seconds), write=60.0, pool=10.0)
161+ prompt_for_log = body.get("messages") or body.get("prompt") or body.get("input")
162+
163+ async with manager.use(lm):
164+ if not stream:
165+ try:
166+ r = await manager.client.post(url, json=body, timeout=timeout)
167+ except httpx.ReadTimeout:
168+ raise GenerationTimeout("The model did not finish generating in time.")
169+ except httpx.HTTPError as e:
170+ if not lm.handle.alive():
171+ raise WorkerCrashed(f"The inference worker for '{model['id']}' crashed during the request.")
172+ raise APIError(f"Worker connection error: {e}", status_code=502, code="WORKER_UNREACHABLE", error_type="runtime_error")
173+ try:
174+ data = r.json()
175+ except Exception:
176+ data = {"error": {"message": r.text[:500], "type": "runtime_error", "code": "WORKER_BAD_RESPONSE"}}
177+ if r.status_code >= 400:
178+ err = data.get("error") if isinstance(data, dict) else None
179+ code = (err or {}).get("code") if isinstance(err, dict) else None
180+ msg = (err or {}).get("message") if isinstance(err, dict) else str(err)
181+ await _record(app_state, model_id=model["id"], requested=requested, endpoint=endpoint, principal=principal,
182+ stream=False, usage=None, timings=None, status=r.status_code, error_code=str(code or "WORKER_ERROR"),
183+ started=started, load_wait_ms=load_wait_ms, prompt=prompt_for_log)
184+ # normalise llama.cpp error shape
185+ return JSONResponse(status_code=r.status_code, content={"error": {"message": msg or "worker error",
186+ "type": (err or {}).get("type", "runtime_error") if isinstance(err, dict) else "runtime_error",
187+ "code": code or "WORKER_ERROR"}})
188+ data["model"] = requested if requested != "auto" else model["id"]
189+ timings = _extract_timings(data)
190+ if timings:
191+ data["timings"] = timings
192+ completion = None
193+ if kind == "chat":
194+ completion = ((data.get("choices") or [{}])[0].get("message") or {}).get("content")
195+ elif kind == "completion":
196+ completion = (data.get("choices") or [{}])[0].get("text")
197+ await _record(app_state, model_id=model["id"], requested=requested, endpoint=endpoint, principal=principal,
198+ stream=False, usage=data.get("usage"), timings=timings, status=200, error_code=None,
199+ started=started, load_wait_ms=load_wait_ms, prompt=prompt_for_log, completion=completion)
200+ return JSONResponse(content=data)
201+
202+ # ---- streaming ----------------------------------------------------
203+ req = manager.client.build_request("POST", url, json=body, timeout=timeout)
204+ try:
205+ resp = await manager.client.send(req, stream=True)
206+ except httpx.HTTPError as e:
207+ raise APIError(f"Worker connection error: {e}", status_code=502, code="WORKER_UNREACHABLE", error_type="runtime_error")
208+ if resp.status_code >= 400:
209+ raw = await resp.aread()
210+ await resp.aclose()
211+ try:
212+ data = json.loads(raw)
213+ except Exception:
214+ data = {"error": {"message": raw.decode(errors="replace")[:500], "type": "runtime_error", "code": "WORKER_ERROR"}}
215+ err = data.get("error") or {}
216+ await _record(app_state, model_id=model["id"], requested=requested, endpoint=endpoint, principal=principal,
217+ stream=True, usage=None, timings=None, status=resp.status_code, error_code=str(err.get("code") or "WORKER_ERROR"),
218+ started=started, load_wait_ms=load_wait_ms, prompt=prompt_for_log)
219+ return JSONResponse(status_code=resp.status_code, content={"error": {"message": err.get("message", "worker error"),
220+ "type": err.get("type", "runtime_error"),
221+ "code": err.get("code", "WORKER_ERROR")}})
222+
223+ lm.in_flight += 1 # held until the generator finishes
224+
225+ async def gen():
226+ usage = None
227+ timings: dict = {}
228+ completion_parts: list[str] = []
229+ status = 200
230+ error_code = None
231+ try:
232+ async for line in resp.aiter_lines():
233+ if not line:
234+ continue
235+ if line.startswith("data: "):
236+ payload = line[6:]
237+ if payload.strip() == "[DONE]":
238+ yield b"data: [DONE]\n\n"
239+ continue
240+ try:
241+ obj = json.loads(payload)
242+ except json.JSONDecodeError:
243+ yield (line + "\n\n").encode()
244+ continue
245+ if "error" in obj and "choices" not in obj:
246+ status = 500
247+ error_code = (obj["error"] or {}).get("code", "GENERATION_FAILED")
248+ yield (f"data: {json.dumps(obj)}\n\n").encode()
249+ continue
250+ obj["model"] = requested if requested != "auto" else model["id"]
251+ if obj.get("usage"):
252+ usage = obj["usage"]
253+ if obj.get("timings"):
254+ timings = _extract_timings(obj)
255+ obj["timings"] = timings
256+ for ch in obj.get("choices") or []:
257+ d = ch.get("delta") or {}
258+ if d.get("content"):
259+ completion_parts.append(d["content"])
260+ if ch.get("text"):
261+ completion_parts.append(ch["text"])
262+ yield (f"data: {json.dumps(obj, ensure_ascii=False)}\n\n").encode()
263+ else:
264+ yield (line + "\n").encode()
265+ except (httpx.ReadTimeout, asyncio.TimeoutError):
266+ status, error_code = 504, "GENERATION_TIMEOUT"
267+ yield (f"data: {json.dumps({'error': {'message': 'generation timed out', 'type': 'runtime_error', 'code': 'GENERATION_TIMEOUT'}})}\n\n").encode()
268+ except httpx.HTTPError:
269+ status, error_code = 502, "WORKER_CRASHED" if not lm.handle.alive() else "WORKER_STREAM_ERROR"
270+ yield (f"data: {json.dumps({'error': {'message': 'worker stream interrupted', 'type': 'runtime_error', 'code': error_code}})}\n\n").encode()
271+ finally:
272+ await resp.aclose()
273+ lm.in_flight = max(0, lm.in_flight - 1)
274+ lm.last_used = time.time()
275+ try:
276+ await _record(app_state, model_id=model["id"], requested=requested, endpoint=endpoint, principal=principal,
277+ stream=True, usage=usage, timings=timings, status=status, error_code=error_code,
278+ started=started, load_wait_ms=load_wait_ms, prompt=prompt_for_log,
279+ completion="".join(completion_parts))
280+ except Exception:
281+ log.exception("record failed")
282+
283+ return StreamingResponse(gen(), media_type="text/event-stream",
284+ headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "Connection": "keep-alive"})
285+
286+
287+# ---------------------------------------------------------------------------
288+
289+
290+@router.get("/v1/models")
291+async def list_models(request: Request, principal: Principal = Depends(require_inference)):
292+ manager = _manager(request)
293+ models = await manager.registry.list_models()
294+ aliases = await manager.registry.aliases()
295+ out = []
296+ for m in models:
297+ if not m.get("enabled"):
298+ continue
299+ st = manager.status_of(m["id"])
300+ out.append({
301+ "id": m["id"], "object": "model", "created": int(m["created_at"]), "owned_by": m.get("provider") or "local",
302+ "root": m["id"], "parent": None,
303+ # llm-api extensions (ignored by SDKs)
304+ "name": m["name"], "family": m["family"], "runtime": m["runtime"], "quantization": m["quantization"],
305+ "parameter_count": m["parameter_count"], "estimated_ram_gb": m["estimated_ram_gb"],
306+ "context": m["recommended_context"], "max_context": m["max_context"], "task": m["task"],
307+ "compatibility": m["compatibility_status"], "status": st, "loaded": st == "ready",
308+ "capabilities": {"vision": m["vision"], "embedding": m["embedding"], "reranking": m["reranker"],
309+ "tools": m["tools"], "thinking": m["thinking"]},
310+ })
311+ for alias, mid in aliases.items():
312+ out.append({"id": alias, "object": "model", "created": 0, "owned_by": "alias", "root": mid, "parent": mid,
313+ "alias_of": mid})
314+ return {"object": "list", "data": out}
315+
316+
317+@router.get("/v1/models/{model_id:path}")
318+async def get_model(model_id: str, request: Request, principal: Principal = Depends(require_inference)):
319+ manager = _manager(request)
320+ m = await manager.registry.resolve(model_id)
321+ if not m:
322+ from ..errors import ModelNotFound
323+ raise ModelNotFound(f"The model '{model_id}' does not exist.")
324+ return {"id": m["id"], "object": "model", "created": int(m["created_at"]), "owned_by": m.get("provider") or "local",
325+ "status": manager.status_of(m["id"])}
326+
327+
328+@router.post("/v1/chat/completions")
329+async def chat_completions(request: Request, principal: Principal = Depends(require_inference)):
330+ body = await _parse_body(request)
331+ msgs = body.get("messages")
332+ if not isinstance(msgs, list) or not msgs:
333+ raise APIError("messages must be a non-empty array.", param="messages")
334+ for m in msgs:
335+ if not isinstance(m, dict) or "role" not in m:
336+ raise APIError("Each message needs a role.", param="messages")
337+ _validate_sampling(body)
338+ return await _proxy(request, "/v1/chat/completions", body, principal, kind="chat")
339+
340+
341+@router.post("/v1/completions")
342+async def completions(request: Request, principal: Principal = Depends(require_inference)):
343+ body = await _parse_body(request)
344+ if "prompt" not in body:
345+ raise APIError("prompt is required.", param="prompt")
346+ _validate_sampling(body)
347+ return await _proxy(request, "/v1/completions", body, principal, kind="completion")
348+
349+
350+@router.post("/v1/embeddings")
351+async def embeddings(request: Request, principal: Principal = Depends(require_inference)):
352+ body = await _parse_body(request)
353+ if "input" not in body:
354+ raise APIError("input is required.", param="input")
355+ inp = body["input"]
356+ n = len(inp) if isinstance(inp, list) else 1
357+ if n > 256:
358+ raise APIError("At most 256 inputs per request.", param="input")
359+ return await _proxy(request, "/v1/embeddings", body, principal, kind="embeddings")
360+
361+
362+@router.post("/v1/rerank")
363+async def rerank(request: Request, principal: Principal = Depends(require_inference)):
364+ body = await _parse_body(request)
365+ if not body.get("query") or not isinstance(body.get("documents"), list):
366+ raise APIError("query and documents are required.")
367+ if len(body["documents"]) > 200:
368+ raise APIError("At most 200 documents per request.", param="documents")
369+ return await _proxy(request, "/v1/rerank", body, principal, kind="rerank")
added server/llm_api/auth.py +210 −0
Large diff (210 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/bench.py +158 −0
Large diff (158 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/cli.py +72 −0
Large diff (72 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/config.py +122 −0
Large diff (122 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/db.py +300 −0
Large diff (300 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/downloads.py +341 −0
Large diff (341 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/errors.py +122 −0
Large diff (122 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/events.py +48 −0
Large diff (48 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/hardware.py +254 −0
Large diff (254 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/harvester.py +397 −0
Large diff (397 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/jobs.py +162 −0
Large diff (162 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/main.py +166 −0
Large diff (166 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/manager.py +506 −0
Large diff (506 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/metrics.py +98 −0
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/proxy_ui.py +50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/routing.py +70 −0
Large diff (70 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/runtimes/__init__.py +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/runtimes/base.py +149 −0
Large diff (149 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/runtimes/llamacpp_adapter.py +69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/runtimes/mlx_adapter.py +58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/worker/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/worker/mlx_worker.py +1117 −0
Large diff (1,117 lines) is not loaded by default to keep the page fast.Load diff
added server/llm_api/worker/openai_types.py +239 −0
Large diff (239 lines) is not loaded by default to keep the page fast.Load diff
added server/pyproject.toml +48 −0
Large diff (48 lines) is not loaded by default to keep the page fast.Load diff
added server/tests/__init__.py +0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added server/tests/conftest.py +294 −0
Large diff (294 lines) is not loaded by default to keep the page fast.Load diff
added server/tests/test_api.py +298 −0
Large diff (298 lines) is not loaded by default to keep the page fast.Load diff
added server/tests/test_units.py +140 −0
Large diff (140 lines) is not loaded by default to keep the page fast.Load diff
added web/.gitignore +41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added web/README.md +36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added web/eslint.config.mjs +18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added web/next.config.ts +20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added web/package.json +27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added web/pnpm-lock.yaml +4184 −0
Large diff (4,184 lines) is not loaded by default to keep the page fast.Load diff
added web/pnpm-workspace.yaml +3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added web/postcss.config.mjs +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/docs/page.tsx +64 −0
Large diff (64 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/downloads/page.tsx +149 −0
Large diff (149 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/favicon.ico +0 −0

Binary file not shown.

added web/src/app/globals.css +101 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/harvester/page.tsx +159 −0
Large diff (159 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/keys/page.tsx +89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/layout.tsx +21 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/login/page.tsx +73 −0
Large diff (73 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/page.tsx +170 −0
Large diff (170 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/playground/page.tsx +199 −0
Large diff (199 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/settings/page.tsx +134 −0
Large diff (134 lines) is not loaded by default to keep the page fast.Load diff
added web/src/app/system/page.tsx +132 −0
Large diff (132 lines) is not loaded by default to keep the page fast.Load diff
added web/src/components/shell.tsx +157 −0
Large diff (157 lines) is not loaded by default to keep the page fast.Load diff
added web/src/components/ui.tsx +233 −0
Large diff (233 lines) is not loaded by default to keep the page fast.Load diff
added web/src/lib/api.ts +89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added web/src/lib/events.tsx +130 −0
Large diff (130 lines) is not loaded by default to keep the page fast.Load diff
added web/src/lib/format.ts +73 −0
Large diff (73 lines) is not loaded by default to keep the page fast.Load diff
added web/src/lib/types.ts +225 −0
Large diff (225 lines) is not loaded by default to keep the page fast.Load diff
added web/tsconfig.json +34 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere