MacLustr LLM API
A complete local AI inference platform on Apple Silicon: an OpenAI-compatible API served by Apple MLX on a Mac mini M4 Pro, with API keys, quotas, a priority queue, a developer portal, a playground and a passkey-protected admin dashboard.
Author: Simon-Pierre Boucher
Table of contents
- Overview · Features · Measured performance · Architecture
- Stack · Repository layout · Getting started · Configuration
- API · Data model · Security & privacy · Testing
- Deployment · Troubleshooting · Known limitations · Author
🧭 Overview
https://llm.spboucher.ai is the single entry point for LLM requests from the author's web, macOS and iOS apps
and agents. Inference is entirely local: no external LLM service is involved, and no prompt or response is
stored.
from openai import OpenAI
client = OpenAI(base_url="https://llm.spboucher.ai/v1", api_key="YOUR_MACLUSTR_API_KEY")
r = client.chat.completions.create(model="maclustr-qwen3",
messages=[{"role": "user", "content": "Explique les modèles Transformer."}])
print(r.choices[0].message.content)| URL | What |
|---|---|
/ · /docs · /status |
public developer portal (15 navigation entries, generated parameter tables, search) |
/playground |
real-API playground (streaming, tools, JSON mode, live metrics) |
/v1/* |
OpenAI-compatible API (Bearer key) |
/admin |
private dashboard, passkey only |
/setup/<token> |
one-time administrator activation link |
Model served: maclustr-qwen3 = mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (Apache-2.0, revision pinned,
SHA-256 verified), chosen on measurements against three other candidates — see docs/BENCHMARKS.md.
✨ Features
Inference engine (engine_main.py, hardened mlx_lm server): model kept loaded, continuous batching, batched
prefill, KV prefix cache bounded in bytes, token streaming, cancellation, tool-call parsing, model locked (a request
can never load another model), loopback-only, /internal/stats with real MLX memory.
API gateway: OpenAI formats for chat/completions, completions, models; SSE streaming with keep-alives;
tool calling; validated JSON mode (json_object / json_schema, one automatic repair, 422 instead of bad JSON);
usage with cached tokens; request ids; standard error envelope; per-request timeouts; cancellation on disconnect.
Capacity control: per-key requests/min, tokens/min, daily and monthly token quotas, concurrency; four priority classes with aging (no starvation); bounded queue with immediate 503 back-pressure; in-flight token budget that bounds KV-cache memory; guard on real macOS memory pressure.
Administration: passkeys (WebAuthn/FIDO2, Touch ID / Face ID, no password), one-time 15-minute setup link printed on the Mac mini, local recovery, hashed sessions, CSRF, recent-auth for critical actions, audit log; dashboard with live overview, Mac monitoring, analytics, API keys, models, security.
Portal: static site without build step or external resource (strict CSP), light and dark themes, reduced-motion
support, parameter and error tables generated at runtime from /openapi.json.
📈 Measured performance
Mac mini M4 Pro, 48 GB, through the production gateway (full data and method in
docs/BENCHMARKS.md, raw reports in docs/bench-reports/).
| Measure | Value |
|---|---|
| Single stream | 80.6 tokens/s, first token in 111 ms |
| 8 concurrent generations | 155.8 tokens/s aggregate, 20.4 per request, first token p95 695 ms |
| Prefix cache (2 304-token prefix) | first token 3 564 ms cold → 147 ms warm |
| Idle MLX memory (model loaded) | 17.2 GB |
| Peak MLX memory, 82 000 tokens in flight | 33.9 GB, no swap, normal pressure |
| Quality suite (41 auto-graded items) | 40/41 |
🏗 Architecture
flowchart LR
C["Clients<br/>OpenAI SDKs, apps, agents"] -- HTTPS --> P["Caddy (TLS)<br/>cluster gateway"]
P -- WireGuard --> G
subgraph M["Mac mini M4 Pro · 48 GB"]
G["API gateway :8810<br/>auth · limits · priority queue · site"]
E["MLX engine 127.0.0.1:8811<br/>continuous batching · KV prefix cache"]
D[("SQLite<br/>hashed keys · sessions · usage · audit")]
G --> E --> GPU["Apple GPU (Metal)"]
G --- D
end
sequenceDiagram
participant T as Terminal (Mac mini)
participant B as Browser
participant S as Gateway
T->>S: maclustr-llm setup-link
S-->>T: https://…/setup/<token> (15 min, single use, stored hashed)
B->>S: POST /api/setup/options {token}
S-->>B: WebAuthn creation options + challenge
B->>B: Touch ID / Face ID (system sheet)
B->>S: POST /api/setup/complete {credential}
S->>S: one transaction: verify, create admin, store passkey, burn link
S-->>B: session cookie (HttpOnly, Secure, SameSite=Strict)
More: docs/ARCHITECTURE.md.
🧰 Stack
| Layer | Technology | Role |
|---|---|---|
| Inference | Apple MLX 0.32.4, MLX-LM 0.32.2 (Metal) | model execution, batching, KV cache |
| Gateway | Python 3.12, FastAPI, uvicorn, httpx, Pydantic 2 | API, scheduling, admin |
| Auth | webauthn (py_webauthn) |
passkey registration and assertion |
| Storage | SQLite (WAL) | state; no prompt or response |
| Monitoring | psutil, sysctl, ioreg, pmset |
real CPU / GPU / memory / pressure |
| Frontend | HTML, CSS, ES modules | portal, playground, dashboard (no build, no dependency) |
| Packaging | uv + uv.lock |
reproducible installs |
| Services | launchd | engine, gateway, daily backup |
🗂 Repository layout
src/maclustr_llm/ engine_main.py (MLX engine) · app.py · routes_{openai,public,admin}.py · inference.py
scheduler.py · limits.py · keys.py · auth.py · metrics.py · sysmon.py · openapi.py · cli.py …
web/ index, docs, playground, status, admin, setup + assets/ (tokens.css, site/, admin/, data/)
config/models.json model catalog (public alias, repo, pinned revision) and engines (capacity, token budget)
deploy/ install.sh · verify.sh · run-engine.sh · run-gateway.sh · *.env · launchd/*.plist
bench/ bench.py (speed) · quality.py (41 graded items) · memtest.py (memory) · compare.sh
scripts/ model.py (pinned download + SHA-256) · backup.sh · ci.sh · e2e_live.py · publish_bench.py
tests/ test_api.py · test_limits.py · test_auth.py · fake_engine.py (test double) · conftest.py
docs/ AUDIT-INITIAL · ARCHITECTURE · SECURITY · OPERATIONS · BENCHMARKS · RAPPORT-FINAL · bench-reports/🚀 Getting started
uv sync --frozen --extra dev # development machine
scripts/ci.sh # ruff, mypy --strict, pytest, pip-audit
.venv/bin/python scripts/dev_server.py # real gateway + echo test engine on http://localhost:8910
deploy/install.sh # on the serving Mac: locked deps, migrations, model, launchd services
maclustr-llm setup-link # print the one-time administrator activation link⚙️ Configuration
Non-secret settings live in deploy/gateway.env and deploy/engine.env; optional local overrides in
~/.config/maclustr-llm/{gateway,engine}.env.
| Variable | Purpose |
|---|---|
MLLM_PUBLIC_ORIGIN, MLLM_RP_ID |
public origin and WebAuthn relying-party id |
MLLM_HOST, MLLM_PORT, MLLM_TRUST_PROXY |
listening address, proxy headers |
MLLM_DATA_DIR, MLLM_CATALOG, MLLM_WEB_DIR |
state directory, model catalog, static site |
MLLM_MAX_QUEUE, MLLM_QUEUE_TIMEOUT_S, MLLM_AGING_S |
queue size, wait limit, priority aging |
MLLM_REQUEST_TIMEOUT_S, MLLM_NONSTREAM_TIMEOUT_S, MLLM_FIRST_TOKEN_TIMEOUT_S |
generation limits |
MLLM_MAX_BODY_BYTES, MLLM_MAX_MESSAGES |
request size limits |
MLLM_SESSION_TTL_S, MLLM_SESSION_IDLE_S, MLLM_REAUTH_WINDOW_S, MLLM_SETUP_TTL_S |
admin session policy |
ENGINE_MODEL_DIR, ENGINE_PORT, ENGINE_DECODE_CONCURRENCY, ENGINE_PROMPT_CONCURRENCY |
engine model and batch sizes |
ENGINE_PROMPT_CACHE_SIZE, ENGINE_PROMPT_CACHE_BYTES, ENGINE_MEMORY_LIMIT, ENGINE_CACHE_LIMIT |
engine memory bounds |
🔌 API
| Method | Path | Auth | Notes |
|---|---|---|---|
| GET | /v1/models, /v1/models/{id} |
API key | models visible to the key |
| POST | /v1/chat/completions |
API key | streaming, tools (tool_choice: auto/none), validated JSON mode |
| POST | /v1/completions |
API key | raw text completion |
| GET | /health, /ready, /api/status, /openapi.json |
public | liveness, engine readiness, public status, contract |
| POST | /api/setup/{status,options,complete} |
one-time token | administrator activation / recovery |
| * | /admin/api/* (session, login, reauth, overview, system, analytics, requests, keys, models, engine, security, passkeys, sessions, audit) |
passkey session + CSRF | dashboard |
Not offered because the engine does not support them: forced tool_choice, grammar-constrained decoding,
embeddings, image input, /v1/responses.
🗃 Data model
admin_users · passkeys · setup_tokens · challenges · sessions · api_keys · usage_minute ·
request_log · audit_log · sys_samples · settings. Secrets are stored only as HMAC-SHA-256 digests keyed by
a server secret kept outside the database.
🔒 Security & privacy
No prompt or response is stored or logged. API keys, session ids, CSRF tokens and setup tokens are stored hashed.
The engine listens on loopback only. Strict CSP, HSTS, __Host- session cookie, CSRF token and origin check,
recent passkey authentication for critical actions, IP throttling on authentication routes, audit log.
Details and known limits: docs/SECURITY.md.
🧪 Testing
| Suite | Count | Covers |
|---|---|---|
tests/test_api.py |
23 | OpenAI formats, streaming, tools, JSON mode, validation, permissions, engine failure, official OpenAI SDK |
tests/test_limits.py |
15 | rate limits, quotas, queue, back-pressure, priorities and aging, token budget, cancellation |
tests/test_auth.py |
15 | setup link, passkeys with real signatures, sessions, CSRF, recent auth, recovery, no secret at rest |
scripts/e2e_live.py |
14 checks | the deployed instance over HTTPS, with the real model |
Tests run the real gateway and a protocol-faithful test engine on real sockets. scripts/ci.sh adds ruff,
mypy --strict and pip-audit.
🚢 Deployment
Deployed with the cluster orchestrator mld (app maclustr-llm, pinned to node m4mm): three launchd services
(ai.spboucher.maclustr-llm.engine, .gateway, .backup), public route https://llm.spboucher.ai through the
MacLustr Tunnel (Caddy with automatic TLS, WireGuard to the node). Update, rollback, backup, recovery and model
change procedures: docs/OPERATIONS.md. Initial machine audit:
docs/AUDIT-INITIAL.md. Final report: docs/RAPPORT-FINAL.md.
🩺 Troubleshooting
| Symptom | Action |
|---|---|
503 engine_unavailable |
engine reloading: launchctl print gui/$(id -u)/ai.spboucher.maclustr-llm.engine, logs/engine.log |
503 engine_overloaded / queue_timeout |
demand above capacity; check Analytics, quotas, max_active |
| 504 on non-streaming calls | generation longer than 280 s: use stream: true |
| model download stalls | the Xet client hangs on this network; scripts/model.py fetch forces plain HTTP |
| passkeys lost | maclustr-llm recover on the Mac mini |
🚧 Known limitations
- Prefill is the bottleneck (about 450–650 tokens/s): a never-seen 12 000-token prompt waits about 27 s for its first token; context window served is 32 768 tokens.
- JSON mode is validated, not grammar-constrained; validity is guaranteed for non-streaming responses only.
- API error messages are in French.
- Passkey flows were verified with a virtual authenticator in Chromium, not yet on a physical Touch ID / Face ID device.
- No independent security audit.
👤 Author
Simon-Pierre Boucher — design, development, operations
www.spboucher.ai · git.spboucher.ai · contact@spboucher.ai
📄 License
All rights reserved © 2026 Simon-Pierre Boucher. The model weights are distributed by their authors under Apache-2.0.