# LLM API — https://www.llm-api.io

A private, production-grade, OpenAI-compatible local LLM API for Apple Silicon. Store a large library of models on SSD, load only the one that is requested into unified memory, unload it when something else is asked for, and expose everything behind one clean, authenticated API plus a management console.

OpenAI API — but backed by my own Apple Silicon machine and my own locally stored models.

First deployment: Mac Studio M1 Max, 10-core CPU, 32-core GPU, 64 GB unified memory, 1.8 TB SSD (node M1M64).

# What it does

• Load-on-demand: {"model": "qwen3.8-27b-4bit"} loads that model from SSD if it is not resident, evicting the previous one (LRU + idle timeout), then serves the request. Switches are serialized; requests queue while a model loads.
• Two runtimes: MLX / MLX-LM (preferred, safetensors) and llama.cpp llama-server (GGUF, Metal). Each model runs in its own worker process on 127.0.0.1 — killing the process is how memory is guaranteed to come back.
• Memory policy: weights + KV cache + runtime overhead are estimated before loading; the default budget is 45 GB of 64 GB (never all of it). Too-large models are refused with a structured error. Swap is treated as a warning, never a feature.
• Compatibility engine: compatible / compatible_with_restrictions / experimental / not_recommended / incompatible, with a recommended context per model.
• OpenAI endpoints: /v1/models, /v1/chat/completions (streaming, tools, reasoning content), /v1/completions, /v1/embeddings, /v1/rerank. Works with the official OpenAI SDKs.
• Management API + console: model library, per-model pages, playground, downloads from Hugging Face (inspected first: size, RAM, disk reserve), Model Harvester (explores HF for models that truly fit, dedupes quantizations, proposes a download queue), benchmarks, API keys (hashed), settings, real-time telemetry (RAM, GPU, CPU, thermal, disk) via SSE.
• Aliases and auto: fast, coder, reasoning, vision, embedding, default… model: "auto" routes by prompt content (never when a model is named explicitly).

# Layout

server/   Python 3.13 · FastAPI · SQLite      — API gateway, registry, model manager, workers
web/      Next.js 16 · React 19 · Tailwind 4  — console (proxied by the API in production)
scripts/  install.sh
docs/     architecture, api, models, security, deployment, troubleshooting

# Quick start (clean Mac)

git clone <repo> llm-api && cd llm-api
./scripts/install.sh --test          # checks the Mac, installs uv/Node/llama.cpp, Python deps, builds the web app
$EDITOR ~/llm-api/.env               # ADMIN_EMAIL, ADMIN_PASSWORD, HF_TOKEN, PUBLIC_URL
(cd ~/llm-api && ../path/to/server/.venv/bin/llm-api serve)         # API on 127.0.0.1:8300
(cd web && pnpm start -p 8301 -H 127.0.0.1)                          # console

Then open http://127.0.0.1:8300, sign in, create an API key, and:

from openai import OpenAI
client = OpenAI(base_url="https://www.llm-api.io/v1", api_key="llm_live_xxxxx")
r = client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hello"}])
print(r.choices[0].message.content)

curl https://www.llm-api.io/v1/chat/completions \
  -H "Authorization: Bearer llm_live_xxxxx" -H "Content-Type: application/json" \
  -d '{"model": "qwen3.8-27b-4bit", "messages": [{"role": "user", "content": "Explain monetary policy."}], "stream": true}'

# Verified on the M1 Max (2026-09-10)

# Tests

cd server && .venv/bin/python -m pytest -q

The suite runs without MLX (fake workers) and covers: authentication, API keys (creation, rejection, revocation), registry scanning, missing-file detection, load-on-demand, streaming, model switching, concurrent requests + switch lock, memory rejection, worker crash, load timeout, deletion confirmation, low-disk refusal, benchmarks, restart cleanup, settings validation.

# Docs

• docs/architecture.md — components, data flow, worker protocol, memory model
• docs/api.md — OpenAI endpoints, extensions, management API, errors
• docs/models.md — storage layout, discovery, compatibility, quantization policy, Harvester, starter library
• docs/security.md — auth, keys, CSRF, sandboxing, what is never logged
• docs/deployment.md — M1M64 deployment with mld, PM2, MacLustr Tunnel, DNS
• docs/troubleshooting.md — failure modes and what to look at

# Non-negotiables (from CLAUDE.md)

Never crash the host with an oversized model · never rely on swap · never expose secrets or raw workers · never mark a model ready before a successful warm-up · never delete model files automatically · never fake metrics · verify runtime APIs against the installed versions · always test load → inference → unload → memory recovery · keep the design ready for multiple Apple Silicon nodes.