# LLM API — `https://www.llm-api.io`

A private, production-grade, **OpenAI-compatible local LLM API for Apple Silicon**. Store a large library of models on SSD, load only the one that is requested into unified memory, unload it when something else is asked for, and expose everything behind one clean, authenticated API plus a management console.

```
OpenAI API — but backed by my own Apple Silicon machine and my own locally stored models.
```

First deployment: Mac Studio M1 Max, 10-core CPU, 32-core GPU, 64 GB unified memory, 1.8 TB SSD (node `M1M64`).

## What it does

- **Load-on-demand**: `{"model": "qwen3.8-27b-4bit"}` loads that model from SSD if it is not resident, evicting the previous one (LRU + idle timeout), then serves the request. Switches are serialized; requests queue while a model loads.
- **Two runtimes**: MLX / MLX-LM (preferred, safetensors) and llama.cpp `llama-server` (GGUF, Metal). Each model runs in its **own worker process** on `127.0.0.1` — killing the process is how memory is guaranteed to come back.
- **Memory policy**: weights + KV cache + runtime overhead are estimated *before* loading; the default budget is 45 GB of 64 GB (never all of it). Too-large models are refused with a structured error. Swap is treated as a warning, never a feature.
- **Compatibility engine**: `compatible` / `compatible_with_restrictions` / `experimental` / `not_recommended` / `incompatible`, with a recommended context per model.
- **OpenAI endpoints**: `/v1/models`, `/v1/chat/completions` (streaming, tools, reasoning content), `/v1/completions`, `/v1/embeddings`, `/v1/rerank`. Works with the official OpenAI SDKs.
- **Management API + console**: model library, per-model pages, playground, downloads from Hugging Face (inspected first: size, RAM, disk reserve), **Model Harvester** (explores HF for models that truly fit, dedupes quantizations, proposes a download queue), benchmarks, API keys (hashed), settings, real-time telemetry (RAM, GPU, CPU, thermal, disk) via SSE.
- **Aliases and `auto`**: `fast`, `coder`, `reasoning`, `vision`, `embedding`, `default`… `model: "auto"` routes by prompt content (never when a model is named explicitly).

## Layout

```
server/   Python 3.13 · FastAPI · SQLite      — API gateway, registry, model manager, workers
web/      Next.js 16 · React 19 · Tailwind 4  — console (proxied by the API in production)
scripts/  install.sh
docs/     architecture, api, models, security, deployment, troubleshooting
```

## Quick start (clean Mac)

```bash
git clone <repo> llm-api && cd llm-api
./scripts/install.sh --test          # checks the Mac, installs uv/Node/llama.cpp, Python deps, builds the web app
$EDITOR ~/llm-api/.env               # ADMIN_EMAIL, ADMIN_PASSWORD, HF_TOKEN, PUBLIC_URL
(cd ~/llm-api && ../path/to/server/.venv/bin/llm-api serve)         # API on 127.0.0.1:8300
(cd web && pnpm start -p 8301 -H 127.0.0.1)                          # console
```

Then open `http://127.0.0.1:8300`, sign in, create an API key, and:

```python
from openai import OpenAI
client = OpenAI(base_url="https://www.llm-api.io/v1", api_key="llm_live_xxxxx")
r = client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hello"}])
print(r.choices[0].message.content)
```

```bash
curl https://www.llm-api.io/v1/chat/completions \
  -H "Authorization: Bearer llm_live_xxxxx" -H "Content-Type: application/json" \
  -d '{"model": "qwen3.8-27b-4bit", "messages": [{"role": "user", "content": "Explain monetary policy."}], "stream": true}'
```

## Verified on the M1 Max (2026-09-10, all through the API, `temperature=0`)

| Model | Runtime | Cold load | Generation | Memory measured |
|---|---|---|---|---|
| Qwen3-4B-Instruct-2507 4-bit | MLX | 1.7 s | ~100 tok/s | 2.4 GB |
| Llama-3.2-3B-Instruct 4-bit | MLX | 1.4 s | ~170 tok/s | 2.0 GB |
| gemma-3-1b-it Q4_K_M | llama.cpp | 1.1 s | ~150 tok/s | 0.7 GB |
| Qwen3.5-9B 4-bit (VLM) | MLX (mlx-vlm) | 3.5 s | 77 tok/s | 5.9 GB |
| gpt-oss-20b MXFP4 (reasoning → `reasoning_content`) | MLX | 4.9 s | 73 tok/s | 12.0 GB |
| Devstral-Small-2-24B 4-bit | MLX | 4.7 s | 22 tok/s | 14.0 GB |
| gemma-4-26B-A4B 4-bit (MoE) | MLX | 6.0 s | 80 tok/s | 15.0 GB |
| Qwen3.8-27B 4-bit | MLX | ~5 s | 20 tok/s | 16.3 GB |
| Qwen3.6-35B-A3B 4-bit (MoE) | MLX | ~5 s | 66 tok/s | 20.5 GB |
| Qwen3-Coder-30B-A3B 4-bit | MLX | ~5 s | 68 tok/s | 17.2 GB |
| **Llama-3.3-70B-Instruct 4-bit** (XL, ctx 8K) | MLX | 9.3 s | 8.6 tok/s | 37.7 GB worker · 47 GB system · **no swap** |
| Qwen3-Embedding-0.6B 8-bit · Qwen3-Reranker-0.6B 4-bit · embeddinggemma-300M (GGUF) | MLX / llama.cpp | < 2 s | — | < 1 GB, can stay resident |

Also verified: A → B → A switching releases the previous worker's memory before the next load (e.g. 20.9 GB back after evicting Qwen3.6-35B); OpenAI Python SDK (models, chat, streaming, embeddings); memory rejection (`MODEL_TOO_LARGE`) when the budget is lowered; Harvester scan of 5 Hugging Face authors → 361 repos listed, 307 candidates, 205 unique base models in ~90 s.

## Tests

```bash
cd server && .venv/bin/python -m pytest -q
```

The suite runs without MLX (fake workers) and covers: authentication, API keys (creation, rejection, revocation), registry scanning, missing-file detection, load-on-demand, streaming, model switching, concurrent requests + switch lock, memory rejection, worker crash, load timeout, deletion confirmation, low-disk refusal, benchmarks, restart cleanup, settings validation.

## Docs

- [docs/architecture.md](docs/architecture.md) — components, data flow, worker protocol, memory model
- [docs/api.md](docs/api.md) — OpenAI endpoints, extensions, management API, errors
- [docs/models.md](docs/models.md) — storage layout, discovery, compatibility, quantization policy, Harvester, starter library
- [docs/security.md](docs/security.md) — auth, keys, CSRF, sandboxing, what is never logged
- [docs/deployment.md](docs/deployment.md) — M1M64 deployment with `mld`, PM2, MacLustr Tunnel, DNS
- [docs/troubleshooting.md](docs/troubleshooting.md) — failure modes and what to look at

## Non-negotiables (from CLAUDE.md)

Never crash the host with an oversized model · never rely on swap · never expose secrets or raw workers · never mark a model ready before a successful warm-up · never delete model files automatically · never fake metrics · verify runtime APIs against the installed versions · always test load → inference → unload → memory recovery · keep the design ready for multiple Apple Silicon nodes.