Repository Public
1<div align="center">23<img src="docs/brand/logo.svg" alt="MacLustr LLM API logo" width="560">45<br><br>67<img src="docs/brand/icon.png" alt="MacLustr LLM API icon" width="112" height="112">89# MacLustr LLM API1011**A complete local AI inference platform on Apple Silicon: an OpenAI-compatible API served by Apple MLX on a Mac mini M4 Pro, with API keys, quotas, a priority queue, a developer portal, a playground and a passkey-protected admin dashboard.**1213**Author: [Simon-Pierre Boucher](https://www.spboucher.ai)**1415161718192021222324252627282930[](https://git.spboucher.ai/maclustr-llm/commits)31[](https://git.spboucher.ai/maclustr-llm/commits)32[](https://git.spboucher.ai/maclustr-llm/insights)33[](https://git.spboucher.ai/maclustr-llm)34[](https://git.spboucher.ai/maclustr-llm/insights)35[](https://git.spboucher.ai/maclustr-llm)36[](https://git.spboucher.ai/maclustr-llm/branches)37[](https://git.spboucher.ai/maclustr-llm/tags)38[](https://git.spboucher.ai/maclustr-llm)3940414243444546-0ea5e9?style=flat-square)47-0ea5e9?style=flat-square)484950-0ea5e9?style=flat-square)5152</div>5354---5556## Table of contents5758- [Overview](#-overview) · [Features](#-features) · [Measured performance](#-measured-performance) · [Architecture](#-architecture)59- [Stack](#-stack) · [Repository layout](#-repository-layout) · [Getting started](#-getting-started) · [Configuration](#-configuration)60- [API](#-api) · [Data model](#-data-model) · [Security & privacy](#-security--privacy) · [Testing](#-testing)61- [Deployment](#-deployment) · [Troubleshooting](#-troubleshooting) · [Known limitations](#-known-limitations) · [Author](#-author)6263## 🧭 Overview6465`https://llm.spboucher.ai` is the single entry point for LLM requests from the author's web, macOS and iOS apps66and agents. Inference is **entirely local**: no external LLM service is involved, and no prompt or response is67stored.6869```python70from openai import OpenAI7172client = OpenAI(base_url="https://llm.spboucher.ai/v1", api_key="YOUR_MACLUSTR_API_KEY")73r = client.chat.completions.create(model="maclustr-qwen3",74 messages=[{"role": "user", "content": "Explique les modèles Transformer."}])75print(r.choices[0].message.content)76```7778| URL | What |79|---|---|80| `/` · `/docs` · `/status` | public developer portal (15 navigation entries, generated parameter tables, search) |81| `/playground` | real-API playground (streaming, tools, JSON mode, live metrics) |82| `/v1/*` | OpenAI-compatible API (Bearer key) |83| `/admin` | private dashboard, passkey only |84| `/setup/<token>` | one-time administrator activation link |8586Model served: **`maclustr-qwen3`** = `mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit` (Apache-2.0, revision pinned,87SHA-256 verified), chosen on measurements against three other candidates — see [docs/BENCHMARKS.md](docs/BENCHMARKS.md).8889## ✨ Features9091**Inference engine** (`engine_main.py`, hardened `mlx_lm` server): model kept loaded, continuous batching, batched92prefill, KV prefix cache bounded in bytes, token streaming, cancellation, tool-call parsing, model locked (a request93can never load another model), loopback-only, `/internal/stats` with real MLX memory.9495**API gateway**: OpenAI formats for `chat/completions`, `completions`, `models`; SSE streaming with keep-alives;96tool calling; validated JSON mode (`json_object` / `json_schema`, one automatic repair, 422 instead of bad JSON);97usage with cached tokens; request ids; standard error envelope; per-request timeouts; cancellation on disconnect.9899**Capacity control**: per-key requests/min, tokens/min, daily and monthly token quotas, concurrency; four priority100classes with aging (no starvation); bounded queue with immediate 503 back-pressure; **in-flight token budget** that101bounds KV-cache memory; guard on real macOS memory pressure.102103**Administration**: passkeys (WebAuthn/FIDO2, Touch ID / Face ID, no password), one-time 15-minute setup link printed104on the Mac mini, local recovery, hashed sessions, CSRF, recent-auth for critical actions, audit log; dashboard with105live overview, Mac monitoring, analytics, API keys, models, security.106107**Portal**: static site without build step or external resource (strict CSP), light and dark themes, reduced-motion108support, parameter and error tables generated at runtime from `/openapi.json`.109110## 📈 Measured performance111112Mac mini M4 Pro, 48 GB, through the production gateway (full data and method in113[docs/BENCHMARKS.md](docs/BENCHMARKS.md), raw reports in `docs/bench-reports/`).114115| Measure | Value |116|---|---|117| Single stream | 80.6 tokens/s, first token in 111 ms |118| 8 concurrent generations | 155.8 tokens/s aggregate, 20.4 per request, first token p95 695 ms |119| Prefix cache (2 304-token prefix) | first token 3 564 ms cold → 147 ms warm |120| Idle MLX memory (model loaded) | 17.2 GB |121| Peak MLX memory, 82 000 tokens in flight | 33.9 GB, no swap, normal pressure |122| Quality suite (41 auto-graded items) | 40/41 |123124## 🏗 Architecture125126```mermaid127flowchart LR128 C["Clients<br/>OpenAI SDKs, apps, agents"] -- HTTPS --> P["Caddy (TLS)<br/>cluster gateway"]129 P -- WireGuard --> G130 subgraph M["Mac mini M4 Pro · 48 GB"]131 G["API gateway :8810<br/>auth · limits · priority queue · site"]132 E["MLX engine 127.0.0.1:8811<br/>continuous batching · KV prefix cache"]133 D[("SQLite<br/>hashed keys · sessions · usage · audit")]134 G --> E --> GPU["Apple GPU (Metal)"]135 G --- D136 end137```138139```mermaid140sequenceDiagram141 participant T as Terminal (Mac mini)142 participant B as Browser143 participant S as Gateway144 T->>S: maclustr-llm setup-link145 S-->>T: https://…/setup/<token> (15 min, single use, stored hashed)146 B->>S: POST /api/setup/options {token}147 S-->>B: WebAuthn creation options + challenge148 B->>B: Touch ID / Face ID (system sheet)149 B->>S: POST /api/setup/complete {credential}150 S->>S: one transaction: verify, create admin, store passkey, burn link151 S-->>B: session cookie (HttpOnly, Secure, SameSite=Strict)152```153154More: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).155156## 🧰 Stack157158| Layer | Technology | Role |159|---|---|---|160| Inference | Apple MLX 0.32.4, MLX-LM 0.32.2 (Metal) | model execution, batching, KV cache |161| Gateway | Python 3.12, FastAPI, uvicorn, httpx, Pydantic 2 | API, scheduling, admin |162| Auth | `webauthn` (py_webauthn) | passkey registration and assertion |163| Storage | SQLite (WAL) | state; no prompt or response |164| Monitoring | psutil, `sysctl`, `ioreg`, `pmset` | real CPU / GPU / memory / pressure |165| Frontend | HTML, CSS, ES modules | portal, playground, dashboard (no build, no dependency) |166| Packaging | uv + `uv.lock` | reproducible installs |167| Services | launchd | engine, gateway, daily backup |168169## 🗂 Repository layout170171```text172src/maclustr_llm/ engine_main.py (MLX engine) · app.py · routes_{openai,public,admin}.py · inference.py173 scheduler.py · limits.py · keys.py · auth.py · metrics.py · sysmon.py · openapi.py · cli.py …174web/ index, docs, playground, status, admin, setup + assets/ (tokens.css, site/, admin/, data/)175config/models.json model catalog (public alias, repo, pinned revision) and engines (capacity, token budget)176deploy/ install.sh · verify.sh · run-engine.sh · run-gateway.sh · *.env · launchd/*.plist177bench/ bench.py (speed) · quality.py (41 graded items) · memtest.py (memory) · compare.sh178scripts/ model.py (pinned download + SHA-256) · backup.sh · ci.sh · e2e_live.py · publish_bench.py179tests/ test_api.py · test_limits.py · test_auth.py · fake_engine.py (test double) · conftest.py180docs/ AUDIT-INITIAL · ARCHITECTURE · SECURITY · OPERATIONS · BENCHMARKS · RAPPORT-FINAL · bench-reports/181```182183## 🚀 Getting started184185```bash186uv sync --frozen --extra dev # development machine187scripts/ci.sh # ruff, mypy --strict, pytest, pip-audit188.venv/bin/python scripts/dev_server.py # real gateway + echo test engine on http://localhost:8910189190deploy/install.sh # on the serving Mac: locked deps, migrations, model, launchd services191maclustr-llm setup-link # print the one-time administrator activation link192```193194## ⚙️ Configuration195196Non-secret settings live in `deploy/gateway.env` and `deploy/engine.env`; optional local overrides in197`~/.config/maclustr-llm/{gateway,engine}.env`.198199| Variable | Purpose |200|---|---|201| `MLLM_PUBLIC_ORIGIN`, `MLLM_RP_ID` | public origin and WebAuthn relying-party id |202| `MLLM_HOST`, `MLLM_PORT`, `MLLM_TRUST_PROXY` | listening address, proxy headers |203| `MLLM_DATA_DIR`, `MLLM_CATALOG`, `MLLM_WEB_DIR` | state directory, model catalog, static site |204| `MLLM_MAX_QUEUE`, `MLLM_QUEUE_TIMEOUT_S`, `MLLM_AGING_S` | queue size, wait limit, priority aging |205| `MLLM_REQUEST_TIMEOUT_S`, `MLLM_NONSTREAM_TIMEOUT_S`, `MLLM_FIRST_TOKEN_TIMEOUT_S` | generation limits |206| `MLLM_MAX_BODY_BYTES`, `MLLM_MAX_MESSAGES` | request size limits |207| `MLLM_SESSION_TTL_S`, `MLLM_SESSION_IDLE_S`, `MLLM_REAUTH_WINDOW_S`, `MLLM_SETUP_TTL_S` | admin session policy |208| `ENGINE_MODEL_DIR`, `ENGINE_PORT`, `ENGINE_DECODE_CONCURRENCY`, `ENGINE_PROMPT_CONCURRENCY` | engine model and batch sizes |209| `ENGINE_PROMPT_CACHE_SIZE`, `ENGINE_PROMPT_CACHE_BYTES`, `ENGINE_MEMORY_LIMIT`, `ENGINE_CACHE_LIMIT` | engine memory bounds |210211## 🔌 API212213| Method | Path | Auth | Notes |214|---|---|---|---|215| GET | `/v1/models`, `/v1/models/{id}` | API key | models visible to the key |216| POST | `/v1/chat/completions` | API key | streaming, tools (`tool_choice`: `auto`/`none`), validated JSON mode |217| POST | `/v1/completions` | API key | raw text completion |218| GET | `/health`, `/ready`, `/api/status`, `/openapi.json` | public | liveness, engine readiness, public status, contract |219| POST | `/api/setup/{status,options,complete}` | one-time token | administrator activation / recovery |220| * | `/admin/api/*` (session, login, reauth, overview, system, analytics, requests, keys, models, engine, security, passkeys, sessions, audit) | passkey session + CSRF | dashboard |221222Not offered because the engine does not support them: forced `tool_choice`, grammar-constrained decoding,223embeddings, image input, `/v1/responses`.224225## 🗃 Data model226227`admin_users` · `passkeys` · `setup_tokens` · `challenges` · `sessions` · `api_keys` · `usage_minute` ·228`request_log` · `audit_log` · `sys_samples` · `settings`. Secrets are stored only as HMAC-SHA-256 digests keyed by229a server secret kept outside the database.230231## 🔒 Security & privacy232233No prompt or response is stored or logged. API keys, session ids, CSRF tokens and setup tokens are stored hashed.234The engine listens on loopback only. Strict CSP, HSTS, `__Host-` session cookie, CSRF token and origin check,235recent passkey authentication for critical actions, IP throttling on authentication routes, audit log.236Details and known limits: [docs/SECURITY.md](docs/SECURITY.md).237238## 🧪 Testing239240| Suite | Count | Covers |241|---|---|---|242| `tests/test_api.py` | 23 | OpenAI formats, streaming, tools, JSON mode, validation, permissions, engine failure, **official OpenAI SDK** |243| `tests/test_limits.py` | 15 | rate limits, quotas, queue, back-pressure, priorities and aging, token budget, cancellation |244| `tests/test_auth.py` | 15 | setup link, passkeys with real signatures, sessions, CSRF, recent auth, recovery, no secret at rest |245| `scripts/e2e_live.py` | 14 checks | the deployed instance over HTTPS, with the real model |246247Tests run the real gateway and a protocol-faithful test engine on real sockets. `scripts/ci.sh` adds ruff,248`mypy --strict` and `pip-audit`.249250## 🚢 Deployment251252Deployed with the cluster orchestrator `mld` (app `maclustr-llm`, pinned to node `m4mm`): three launchd services253(`ai.spboucher.maclustr-llm.engine`, `.gateway`, `.backup`), public route `https://llm.spboucher.ai` through the254MacLustr Tunnel (Caddy with automatic TLS, WireGuard to the node). Update, rollback, backup, recovery and model255change procedures: [docs/OPERATIONS.md](docs/OPERATIONS.md). Initial machine audit:256[docs/AUDIT-INITIAL.md](docs/AUDIT-INITIAL.md). Final report: [docs/RAPPORT-FINAL.md](docs/RAPPORT-FINAL.md).257258## 🩺 Troubleshooting259260| Symptom | Action |261|---|---|262| 503 `engine_unavailable` | engine reloading: `launchctl print gui/$(id -u)/ai.spboucher.maclustr-llm.engine`, `logs/engine.log` |263| 503 `engine_overloaded` / `queue_timeout` | demand above capacity; check Analytics, quotas, `max_active` |264| 504 on non-streaming calls | generation longer than 280 s: use `stream: true` |265| model download stalls | the Xet client hangs on this network; `scripts/model.py fetch` forces plain HTTP |266| passkeys lost | `maclustr-llm recover` on the Mac mini |267268## 🚧 Known limitations269270- Prefill is the bottleneck (about 450–650 tokens/s): a never-seen 12 000-token prompt waits about 27 s for its271 first token; context window served is 32 768 tokens.272- JSON mode is validated, not grammar-constrained; validity is guaranteed for non-streaming responses only.273- API error messages are in French.274- Passkey flows were verified with a virtual authenticator in Chromium, not yet on a physical Touch ID / Face ID device.275- No independent security audit.276277## 👤 Author278279<div align="center"><img src="docs/brand/icon.png" width="64" height="64" alt=""><br>280281**Simon-Pierre Boucher** — design, development, operations<br>282[www.spboucher.ai](https://www.spboucher.ai) · [git.spboucher.ai](https://git.spboucher.ai) · contact@spboucher.ai</div>283284## 📄 License285286All rights reserved © 2026 Simon-Pierre Boucher. The model weights are distributed by their authors under Apache-2.0.287