SPB Git forge
spboucher.ai ↗

spb/llm-api

Public
0commits 0branches 0releases
0 Bsize
maindefault branch
—last push
6.4 KB
View source Raw Blame History

# LLM API — https://www.llm-api.io

A private, production-grade, OpenAI-compatible local LLM API for Apple Silicon. Store a large library of models on SSD, load only the one that is requested into unified memory, unload it when something else is asked for, and expose everything behind one clean, authenticated API plus a management console.

text
OpenAI API — but backed by my own Apple Silicon machine and my own locally stored models.

First deployment: Mac Studio M1 Max, 10-core CPU, 32-core GPU, 64 GB unified memory, 1.8 TB SSD (node M1M64).

# What it does

  • Load-on-demand: {"model": "qwen3.8-27b-4bit"} loads that model from SSD if it is not resident, evicting the previous one (LRU + idle timeout), then serves the request. Switches are serialized; requests queue while a model loads.
  • Two runtimes: MLX / MLX-LM (preferred, safetensors) and llama.cpp llama-server (GGUF, Metal). Each model runs in its own worker process on 127.0.0.1 — killing the process is how memory is guaranteed to come back.
  • Memory policy: weights + KV cache + runtime overhead are estimated before loading; the default budget is 45 GB of 64 GB (never all of it). Too-large models are refused with a structured error. Swap is treated as a warning, never a feature.
  • Compatibility engine: compatible / compatible_with_restrictions / experimental / not_recommended / incompatible, with a recommended context per model.
  • OpenAI endpoints: /v1/models, /v1/chat/completions (streaming, tools, reasoning content), /v1/completions, /v1/embeddings, /v1/rerank. Works with the official OpenAI SDKs.
  • Management API + console: model library, per-model pages, playground, downloads from Hugging Face (inspected first: size, RAM, disk reserve), Model Harvester (explores HF for models that truly fit, dedupes quantizations, proposes a download queue), benchmarks, API keys (hashed), settings, real-time telemetry (RAM, GPU, CPU, thermal, disk) via SSE.
  • Aliases and auto: fast, coder, reasoning, vision, embedding, default… model: "auto" routes by prompt content (never when a model is named explicitly).

# Layout

text
server/   Python 3.13 · FastAPI · SQLite      — API gateway, registry, model manager, workers
web/      Next.js 16 · React 19 · Tailwind 4  — console (proxied by the API in production)
scripts/  install.sh
docs/     architecture, api, models, security, deployment, troubleshooting

# Quick start (clean Mac)

bash
git clone <repo> llm-api && cd llm-api
./scripts/install.sh --test          # checks the Mac, installs uv/Node/llama.cpp, Python deps, builds the web app
$EDITOR ~/llm-api/.env               # ADMIN_EMAIL, ADMIN_PASSWORD, HF_TOKEN, PUBLIC_URL
(cd ~/llm-api && ../path/to/server/.venv/bin/llm-api serve)         # API on 127.0.0.1:8300
(cd web && pnpm start -p 8301 -H 127.0.0.1)                          # console

Then open http://127.0.0.1:8300, sign in, create an API key, and:

python
from openai import OpenAI
client = OpenAI(base_url="https://www.llm-api.io/v1", api_key="llm_live_xxxxx")
r = client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hello"}])
print(r.choices[0].message.content)
bash
curl https://www.llm-api.io/v1/chat/completions \
  -H "Authorization: Bearer llm_live_xxxxx" -H "Content-Type: application/json" \
  -d '{"model": "qwen3.8-27b-4bit", "messages": [{"role": "user", "content": "Explain monetary policy."}], "stream": true}'

# Verified on the M1 Max (2026-09-10, all through the API, temperature=0)

Model Runtime Cold load Generation Memory measured
Qwen3-4B-Instruct-2507 4-bit MLX 1.7 s ~100 tok/s 2.4 GB
Llama-3.2-3B-Instruct 4-bit MLX 1.4 s ~170 tok/s 2.0 GB
gemma-3-1b-it Q4_K_M llama.cpp 1.1 s ~150 tok/s 0.7 GB
Qwen3.5-9B 4-bit (VLM) MLX (mlx-vlm) 3.5 s 77 tok/s 5.9 GB
gpt-oss-20b MXFP4 (reasoning → reasoning_content) MLX 4.9 s 73 tok/s 12.0 GB
Devstral-Small-2-24B 4-bit MLX 4.7 s 22 tok/s 14.0 GB
gemma-4-26B-A4B 4-bit (MoE) MLX 6.0 s 80 tok/s 15.0 GB
Qwen3.8-27B 4-bit MLX ~5 s 20 tok/s 16.3 GB
Qwen3.6-35B-A3B 4-bit (MoE) MLX ~5 s 66 tok/s 20.5 GB
Qwen3-Coder-30B-A3B 4-bit MLX ~5 s 68 tok/s 17.2 GB
Llama-3.3-70B-Instruct 4-bit (XL, ctx 8K) MLX 9.3 s 8.6 tok/s 37.7 GB worker · 47 GB system · no swap
Qwen3-Embedding-0.6B 8-bit · Qwen3-Reranker-0.6B 4-bit · embeddinggemma-300M (GGUF) MLX / llama.cpp < 2 s — < 1 GB, can stay resident

Also verified: A → B → A switching releases the previous worker's memory before the next load (e.g. 20.9 GB back after evicting Qwen3.6-35B); OpenAI Python SDK (models, chat, streaming, embeddings); memory rejection (MODEL_TOO_LARGE) when the budget is lowered; Harvester scan of 5 Hugging Face authors → 361 repos listed, 307 candidates, 205 unique base models in ~90 s.

# Tests

bash
cd server && .venv/bin/python -m pytest -q

The suite runs without MLX (fake workers) and covers: authentication, API keys (creation, rejection, revocation), registry scanning, missing-file detection, load-on-demand, streaming, model switching, concurrent requests + switch lock, memory rejection, worker crash, load timeout, deletion confirmation, low-disk refusal, benchmarks, restart cleanup, settings validation.

# Docs

# Non-negotiables (from CLAUDE.md)

Never crash the host with an oversized model · never rely on swap · never expose secrets or raw workers · never mark a model ready before a successful warm-up · never delete model files automatically · never fake metrics · verify runtime APIs against the installed versions · always test load → inference → unload → memory recovery · keep the design ready for multiple Apple Silicon nodes.