1# LLM API — `https://www.llm-api.io`23A private, production-grade, **OpenAI-compatible local LLM API for Apple Silicon**. Store a large library of models on SSD, load only the one that is requested into unified memory, unload it when something else is asked for, and expose everything behind one clean, authenticated API plus a management console.45```6OpenAI API — but backed by my own Apple Silicon machine and my own locally stored models.7```89First deployment: Mac Studio M1 Max, 10-core CPU, 32-core GPU, 64 GB unified memory, 1.8 TB SSD (node `M1M64`).1011## What it does1213- **Load-on-demand**: `{"model": "qwen3.8-27b-4bit"}` loads that model from SSD if it is not resident, evicting the previous one (LRU + idle timeout), then serves the request. Switches are serialized; requests queue while a model loads.14- **Two runtimes**: MLX / MLX-LM (preferred, safetensors) and llama.cpp `llama-server` (GGUF, Metal). Each model runs in its **own worker process** on `127.0.0.1` — killing the process is how memory is guaranteed to come back.15- **Memory policy**: weights + KV cache + runtime overhead are estimated *before* loading; the default budget is 45 GB of 64 GB (never all of it). Too-large models are refused with a structured error. Swap is treated as a warning, never a feature.16- **Compatibility engine**: `compatible` / `compatible_with_restrictions` / `experimental` / `not_recommended` / `incompatible`, with a recommended context per model.17- **OpenAI endpoints**: `/v1/models`, `/v1/chat/completions` (streaming, tools, reasoning content), `/v1/completions`, `/v1/embeddings`, `/v1/rerank`. Works with the official OpenAI SDKs.18- **Management API + console**: model library, per-model pages, playground, downloads from Hugging Face (inspected first: size, RAM, disk reserve), **Model Harvester** (explores HF for models that truly fit, dedupes quantizations, proposes a download queue), benchmarks, API keys (hashed), settings, real-time telemetry (RAM, GPU, CPU, thermal, disk) via SSE.19- **Aliases and `auto`**: `fast`, `coder`, `reasoning`, `vision`, `embedding`, `default`… `model: "auto"` routes by prompt content (never when a model is named explicitly).2021## Layout2223```24server/ Python 3.13 · FastAPI · SQLite — API gateway, registry, model manager, workers25web/ Next.js 16 · React 19 · Tailwind 4 — console (proxied by the API in production)26scripts/ install.sh27docs/ architecture, api, models, security, deployment, troubleshooting28```2930## Quick start (clean Mac)3132```bash33git clone <repo> llm-api && cd llm-api34./scripts/install.sh --test # checks the Mac, installs uv/Node/llama.cpp, Python deps, builds the web app35$EDITOR ~/llm-api/.env # ADMIN_EMAIL, ADMIN_PASSWORD, HF_TOKEN, PUBLIC_URL36(cd ~/llm-api && ../path/to/server/.venv/bin/llm-api serve) # API on 127.0.0.1:830037(cd web && pnpm start -p 8301 -H 127.0.0.1) # console38```3940Then open `http://127.0.0.1:8300`, sign in, create an API key, and:4142```python43from openai import OpenAI44client = OpenAI(base_url="https://www.llm-api.io/v1", api_key="llm_live_xxxxx")45r = client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hello"}])46print(r.choices[0].message.content)47```4849```bash50curl https://www.llm-api.io/v1/chat/completions \51 -H "Authorization: Bearer llm_live_xxxxx" -H "Content-Type: application/json" \52 -d '{"model": "qwen3.8-27b-4bit", "messages": [{"role": "user", "content": "Explain monetary policy."}], "stream": true}'53```5455## Verified on the M1 Max (2026-09-10, all through the API, `temperature=0`)5657| Model | Runtime | Cold load | Generation | Memory measured |58|---|---|---|---|---|59| Qwen3-4B-Instruct-2507 4-bit | MLX | 1.7 s | ~100 tok/s | 2.4 GB |60| Llama-3.2-3B-Instruct 4-bit | MLX | 1.4 s | ~170 tok/s | 2.0 GB |61| gemma-3-1b-it Q4_K_M | llama.cpp | 1.1 s | ~150 tok/s | 0.7 GB |62| Qwen3.5-9B 4-bit (VLM) | MLX (mlx-vlm) | 3.5 s | 77 tok/s | 5.9 GB |63| gpt-oss-20b MXFP4 (reasoning → `reasoning_content`) | MLX | 4.9 s | 73 tok/s | 12.0 GB |64| Devstral-Small-2-24B 4-bit | MLX | 4.7 s | 22 tok/s | 14.0 GB |65| gemma-4-26B-A4B 4-bit (MoE) | MLX | 6.0 s | 80 tok/s | 15.0 GB |66| Qwen3.8-27B 4-bit | MLX | ~5 s | 20 tok/s | 16.3 GB |67| Qwen3.6-35B-A3B 4-bit (MoE) | MLX | ~5 s | 66 tok/s | 20.5 GB |68| Qwen3-Coder-30B-A3B 4-bit | MLX | ~5 s | 68 tok/s | 17.2 GB |69| **Llama-3.3-70B-Instruct 4-bit** (XL, ctx 8K) | MLX | 9.3 s | 8.6 tok/s | 37.7 GB worker · 47 GB system · **no swap** |70| Qwen3-Embedding-0.6B 8-bit · Qwen3-Reranker-0.6B 4-bit · embeddinggemma-300M (GGUF) | MLX / llama.cpp | < 2 s | — | < 1 GB, can stay resident |7172Also verified: A → B → A switching releases the previous worker's memory before the next load (e.g. 20.9 GB back after evicting Qwen3.6-35B); OpenAI Python SDK (models, chat, streaming, embeddings); memory rejection (`MODEL_TOO_LARGE`) when the budget is lowered; Harvester scan of 5 Hugging Face authors → 361 repos listed, 307 candidates, 205 unique base models in ~90 s.7374## Tests7576```bash77cd server && .venv/bin/python -m pytest -q78```7980The suite runs without MLX (fake workers) and covers: authentication, API keys (creation, rejection, revocation), registry scanning, missing-file detection, load-on-demand, streaming, model switching, concurrent requests + switch lock, memory rejection, worker crash, load timeout, deletion confirmation, low-disk refusal, benchmarks, restart cleanup, settings validation.8182## Docs8384- [docs/architecture.md](docs/architecture.md) — components, data flow, worker protocol, memory model85- [docs/api.md](docs/api.md) — OpenAI endpoints, extensions, management API, errors86- [docs/models.md](docs/models.md) — storage layout, discovery, compatibility, quantization policy, Harvester, starter library87- [docs/security.md](docs/security.md) — auth, keys, CSRF, sandboxing, what is never logged88- [docs/deployment.md](docs/deployment.md) — M1M64 deployment with `mld`, PM2, MacLustr Tunnel, DNS89- [docs/troubleshooting.md](docs/troubleshooting.md) — failure modes and what to look at9091## Non-negotiables (from CLAUDE.md)9293Never crash the host with an oversized model · never rely on swap · never expose secrets or raw workers · never mark a model ready before a successful warm-up · never delete model files automatically · never fake metrics · verify runtime APIs against the installed versions · always test load → inference → unload → memory recovery · keep the design ready for multiple Apple Silicon nodes.94