SPB Git

Repository Public

README.md 16.2 KB 287 lines markdown

1<div align="center">23<img src="docs/brand/logo.svg" alt="MacLustr LLM API logo" width="560">45<br><br>67<img src="docs/brand/icon.png" alt="MacLustr LLM API icon" width="112" height="112">89# MacLustr LLM API1011**A complete local AI inference platform on Apple Silicon: an OpenAI-compatible API served by Apple MLX on a Mac mini M4 Pro, with API keys, quotas, a priority queue, a developer portal, a playground and a passkey-protected admin dashboard.**1213**Author: [Simon-Pierre Boucher](https://www.spboucher.ai)**1415![website](https://img.shields.io/website?url=https%3A%2F%2Fllm.spboucher.ai%2Fhealth&style=for-the-badge&label=llm.spboucher.ai)16![version](https://img.shields.io/badge/version-1.0.0-2563eb?style=for-the-badge)17![platform](https://img.shields.io/badge/platform-macOS%20·%20Apple%20Silicon-000000?style=for-the-badge&logo=apple)18![license](https://img.shields.io/badge/license-all%20rights%20reserved-64748b?style=for-the-badge)19![author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-0A66C2?style=for-the-badge)2021![Python](https://img.shields.io/badge/Python-3.12-3776AB?style=flat-square&logo=python&logoColor=white)22![MLX](https://img.shields.io/badge/MLX-0.32.4-000000?style=flat-square&logo=apple)23![MLX-LM](https://img.shields.io/badge/MLX--LM-0.32.2-000000?style=flat-square)24![FastAPI](https://img.shields.io/badge/FastAPI-ASGI-009688?style=flat-square&logo=fastapi&logoColor=white)25![SQLite](https://img.shields.io/badge/SQLite-WAL-003B57?style=flat-square&logo=sqlite&logoColor=white)26![WebAuthn](https://img.shields.io/badge/WebAuthn-passkeys-3423A6?style=flat-square&logo=webauthn&logoColor=white)27![uv](https://img.shields.io/badge/uv-locked%20deps-DE5FE9?style=flat-square)28![frontend](https://img.shields.io/badge/frontend-static%20·%20no%20build-5ef2c4?style=flat-square)2930[![commits](https://git.spboucher.ai/badge/maclustr-llm/commits.svg?style=flat-square)](https://git.spboucher.ai/maclustr-llm/commits)31[![last commit](https://git.spboucher.ai/badge/maclustr-llm/last-commit.svg?style=flat-square)](https://git.spboucher.ai/maclustr-llm/commits)32[![activity](https://git.spboucher.ai/badge/maclustr-llm/activity.svg?style=flat-square)](https://git.spboucher.ai/maclustr-llm/insights)33[![language](https://git.spboucher.ai/badge/maclustr-llm/language.svg?style=flat-square)](https://git.spboucher.ai/maclustr-llm)34[![languages](https://git.spboucher.ai/badge/maclustr-llm/languages.svg?style=flat-square)](https://git.spboucher.ai/maclustr-llm/insights)35[![size](https://git.spboucher.ai/badge/maclustr-llm/size.svg?style=flat-square)](https://git.spboucher.ai/maclustr-llm)36[![branches](https://git.spboucher.ai/badge/maclustr-llm/branches.svg?style=flat-square)](https://git.spboucher.ai/maclustr-llm/branches)37[![tags](https://git.spboucher.ai/badge/maclustr-llm/tags.svg?style=flat-square)](https://git.spboucher.ai/maclustr-llm/tags)38[![clones](https://git.spboucher.ai/badge/maclustr-llm/clones.svg?style=flat-square)](https://git.spboucher.ai/maclustr-llm)3940![python](https://img.shields.io/badge/backend-21%20modules%20·%204%20691%20lines-7c3aed?style=flat-square)41![tests](https://img.shields.io/badge/tests-53%20passing-16a34a?style=flat-square)42![typing](https://img.shields.io/badge/mypy-strict-7c3aed?style=flat-square)43![tables](https://img.shields.io/badge/SQLite%20tables-11-7c3aed?style=flat-square)44![docs](https://img.shields.io/badge/doc%20sections-13-7c3aed?style=flat-square)4546![node](https://img.shields.io/badge/node-m4mm%20(Mac%20mini%20M4%20Pro%2C%2048%20GB)-0ea5e9?style=flat-square)47![ports](https://img.shields.io/badge/ports-8810%20gateway%20·%208811%20engine%20(loopback)-0ea5e9?style=flat-square)48![launchd](https://img.shields.io/badge/launchd-3%20services-0ea5e9?style=flat-square)49![deploy](https://img.shields.io/badge/deploy-mld-0ea5e9?style=flat-square)50![tunnel](https://img.shields.io/badge/tunnel-MacLustr%20Tunnel%20(Caddy%20%2B%20WireGuard)-0ea5e9?style=flat-square)5152</div>5354---5556## Table of contents5758- [Overview](#-overview) · [Features](#-features) · [Measured performance](#-measured-performance) · [Architecture](#-architecture)59- [Stack](#-stack) · [Repository layout](#-repository-layout) · [Getting started](#-getting-started) · [Configuration](#-configuration)60- [API](#-api) · [Data model](#-data-model) · [Security & privacy](#-security--privacy) · [Testing](#-testing)61- [Deployment](#-deployment) · [Troubleshooting](#-troubleshooting) · [Known limitations](#-known-limitations) · [Author](#-author)6263## 🧭 Overview6465`https://llm.spboucher.ai` is the single entry point for LLM requests from the author's web, macOS and iOS apps66and agents. Inference is **entirely local**: no external LLM service is involved, and no prompt or response is67stored.6869```python70from openai import OpenAI7172client = OpenAI(base_url="https://llm.spboucher.ai/v1", api_key="YOUR_MACLUSTR_API_KEY")73r = client.chat.completions.create(model="maclustr-qwen3",74                                   messages=[{"role": "user", "content": "Explique les modèles Transformer."}])75print(r.choices[0].message.content)76```7778| URL | What |79|---|---|80| `/` · `/docs` · `/status` | public developer portal (15 navigation entries, generated parameter tables, search) |81| `/playground` | real-API playground (streaming, tools, JSON mode, live metrics) |82| `/v1/*` | OpenAI-compatible API (Bearer key) |83| `/admin` | private dashboard, passkey only |84| `/setup/<token>` | one-time administrator activation link |8586Model served: **`maclustr-qwen3`** = `mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit` (Apache-2.0, revision pinned,87SHA-256 verified), chosen on measurements against three other candidates — see [docs/BENCHMARKS.md](docs/BENCHMARKS.md).8889## ✨ Features9091**Inference engine** (`engine_main.py`, hardened `mlx_lm` server): model kept loaded, continuous batching, batched92prefill, KV prefix cache bounded in bytes, token streaming, cancellation, tool-call parsing, model locked (a request93can never load another model), loopback-only, `/internal/stats` with real MLX memory.9495**API gateway**: OpenAI formats for `chat/completions`, `completions`, `models`; SSE streaming with keep-alives;96tool calling; validated JSON mode (`json_object` / `json_schema`, one automatic repair, 422 instead of bad JSON);97usage with cached tokens; request ids; standard error envelope; per-request timeouts; cancellation on disconnect.9899**Capacity control**: per-key requests/min, tokens/min, daily and monthly token quotas, concurrency; four priority100classes with aging (no starvation); bounded queue with immediate 503 back-pressure; **in-flight token budget** that101bounds KV-cache memory; guard on real macOS memory pressure.102103**Administration**: passkeys (WebAuthn/FIDO2, Touch ID / Face ID, no password), one-time 15-minute setup link printed104on the Mac mini, local recovery, hashed sessions, CSRF, recent-auth for critical actions, audit log; dashboard with105live overview, Mac monitoring, analytics, API keys, models, security.106107**Portal**: static site without build step or external resource (strict CSP), light and dark themes, reduced-motion108support, parameter and error tables generated at runtime from `/openapi.json`.109110## 📈 Measured performance111112Mac mini M4 Pro, 48 GB, through the production gateway (full data and method in113[docs/BENCHMARKS.md](docs/BENCHMARKS.md), raw reports in `docs/bench-reports/`).114115| Measure | Value |116|---|---|117| Single stream | 80.6 tokens/s, first token in 111 ms |118| 8 concurrent generations | 155.8 tokens/s aggregate, 20.4 per request, first token p95 695 ms |119| Prefix cache (2 304-token prefix) | first token 3 564 ms cold → 147 ms warm |120| Idle MLX memory (model loaded) | 17.2 GB |121| Peak MLX memory, 82 000 tokens in flight | 33.9 GB, no swap, normal pressure |122| Quality suite (41 auto-graded items) | 40/41 |123124## 🏗 Architecture125126```mermaid127flowchart LR128    C["Clients<br/>OpenAI SDKs, apps, agents"] -- HTTPS --> P["Caddy (TLS)<br/>cluster gateway"]129    P -- WireGuard --> G130    subgraph M["Mac mini M4 Pro · 48 GB"]131        G["API gateway :8810<br/>auth · limits · priority queue · site"]132        E["MLX engine 127.0.0.1:8811<br/>continuous batching · KV prefix cache"]133        D[("SQLite<br/>hashed keys · sessions · usage · audit")]134        G --> E --> GPU["Apple GPU (Metal)"]135        G --- D136    end137```138139```mermaid140sequenceDiagram141    participant T as Terminal (Mac mini)142    participant B as Browser143    participant S as Gateway144    T->>S: maclustr-llm setup-link145    S-->>T: https://…/setup/<token> (15 min, single use, stored hashed)146    B->>S: POST /api/setup/options {token}147    S-->>B: WebAuthn creation options + challenge148    B->>B: Touch ID / Face ID (system sheet)149    B->>S: POST /api/setup/complete {credential}150    S->>S: one transaction: verify, create admin, store passkey, burn link151    S-->>B: session cookie (HttpOnly, Secure, SameSite=Strict)152```153154More: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).155156## 🧰 Stack157158| Layer | Technology | Role |159|---|---|---|160| Inference | Apple MLX 0.32.4, MLX-LM 0.32.2 (Metal) | model execution, batching, KV cache |161| Gateway | Python 3.12, FastAPI, uvicorn, httpx, Pydantic 2 | API, scheduling, admin |162| Auth | `webauthn` (py_webauthn) | passkey registration and assertion |163| Storage | SQLite (WAL) | state; no prompt or response |164| Monitoring | psutil, `sysctl`, `ioreg`, `pmset` | real CPU / GPU / memory / pressure |165| Frontend | HTML, CSS, ES modules | portal, playground, dashboard (no build, no dependency) |166| Packaging | uv + `uv.lock` | reproducible installs |167| Services | launchd | engine, gateway, daily backup |168169## 🗂 Repository layout170171```text172src/maclustr_llm/   engine_main.py (MLX engine) · app.py · routes_{openai,public,admin}.py · inference.py173                    scheduler.py · limits.py · keys.py · auth.py · metrics.py · sysmon.py · openapi.py · cli.py …174web/                index, docs, playground, status, admin, setup + assets/ (tokens.css, site/, admin/, data/)175config/models.json  model catalog (public alias, repo, pinned revision) and engines (capacity, token budget)176deploy/             install.sh · verify.sh · run-engine.sh · run-gateway.sh · *.env · launchd/*.plist177bench/              bench.py (speed) · quality.py (41 graded items) · memtest.py (memory) · compare.sh178scripts/            model.py (pinned download + SHA-256) · backup.sh · ci.sh · e2e_live.py · publish_bench.py179tests/              test_api.py · test_limits.py · test_auth.py · fake_engine.py (test double) · conftest.py180docs/               AUDIT-INITIAL · ARCHITECTURE · SECURITY · OPERATIONS · BENCHMARKS · RAPPORT-FINAL · bench-reports/181```182183## 🚀 Getting started184185```bash186uv sync --frozen --extra dev            # development machine187scripts/ci.sh                           # ruff, mypy --strict, pytest, pip-audit188.venv/bin/python scripts/dev_server.py  # real gateway + echo test engine on http://localhost:8910189190deploy/install.sh                       # on the serving Mac: locked deps, migrations, model, launchd services191maclustr-llm setup-link                 # print the one-time administrator activation link192```193194## ⚙️ Configuration195196Non-secret settings live in `deploy/gateway.env` and `deploy/engine.env`; optional local overrides in197`~/.config/maclustr-llm/{gateway,engine}.env`.198199| Variable | Purpose |200|---|---|201| `MLLM_PUBLIC_ORIGIN`, `MLLM_RP_ID` | public origin and WebAuthn relying-party id |202| `MLLM_HOST`, `MLLM_PORT`, `MLLM_TRUST_PROXY` | listening address, proxy headers |203| `MLLM_DATA_DIR`, `MLLM_CATALOG`, `MLLM_WEB_DIR` | state directory, model catalog, static site |204| `MLLM_MAX_QUEUE`, `MLLM_QUEUE_TIMEOUT_S`, `MLLM_AGING_S` | queue size, wait limit, priority aging |205| `MLLM_REQUEST_TIMEOUT_S`, `MLLM_NONSTREAM_TIMEOUT_S`, `MLLM_FIRST_TOKEN_TIMEOUT_S` | generation limits |206| `MLLM_MAX_BODY_BYTES`, `MLLM_MAX_MESSAGES` | request size limits |207| `MLLM_SESSION_TTL_S`, `MLLM_SESSION_IDLE_S`, `MLLM_REAUTH_WINDOW_S`, `MLLM_SETUP_TTL_S` | admin session policy |208| `ENGINE_MODEL_DIR`, `ENGINE_PORT`, `ENGINE_DECODE_CONCURRENCY`, `ENGINE_PROMPT_CONCURRENCY` | engine model and batch sizes |209| `ENGINE_PROMPT_CACHE_SIZE`, `ENGINE_PROMPT_CACHE_BYTES`, `ENGINE_MEMORY_LIMIT`, `ENGINE_CACHE_LIMIT` | engine memory bounds |210211## 🔌 API212213| Method | Path | Auth | Notes |214|---|---|---|---|215| GET | `/v1/models`, `/v1/models/{id}` | API key | models visible to the key |216| POST | `/v1/chat/completions` | API key | streaming, tools (`tool_choice`: `auto`/`none`), validated JSON mode |217| POST | `/v1/completions` | API key | raw text completion |218| GET | `/health`, `/ready`, `/api/status`, `/openapi.json` | public | liveness, engine readiness, public status, contract |219| POST | `/api/setup/{status,options,complete}` | one-time token | administrator activation / recovery |220| * | `/admin/api/*` (session, login, reauth, overview, system, analytics, requests, keys, models, engine, security, passkeys, sessions, audit) | passkey session + CSRF | dashboard |221222Not offered because the engine does not support them: forced `tool_choice`, grammar-constrained decoding,223embeddings, image input, `/v1/responses`.224225## 🗃 Data model226227`admin_users` · `passkeys` · `setup_tokens` · `challenges` · `sessions` · `api_keys` · `usage_minute` ·228`request_log` · `audit_log` · `sys_samples` · `settings`. Secrets are stored only as HMAC-SHA-256 digests keyed by229a server secret kept outside the database.230231## 🔒 Security & privacy232233No prompt or response is stored or logged. API keys, session ids, CSRF tokens and setup tokens are stored hashed.234The engine listens on loopback only. Strict CSP, HSTS, `__Host-` session cookie, CSRF token and origin check,235recent passkey authentication for critical actions, IP throttling on authentication routes, audit log.236Details and known limits: [docs/SECURITY.md](docs/SECURITY.md).237238## 🧪 Testing239240| Suite | Count | Covers |241|---|---|---|242| `tests/test_api.py` | 23 | OpenAI formats, streaming, tools, JSON mode, validation, permissions, engine failure, **official OpenAI SDK** |243| `tests/test_limits.py` | 15 | rate limits, quotas, queue, back-pressure, priorities and aging, token budget, cancellation |244| `tests/test_auth.py` | 15 | setup link, passkeys with real signatures, sessions, CSRF, recent auth, recovery, no secret at rest |245| `scripts/e2e_live.py` | 14 checks | the deployed instance over HTTPS, with the real model |246247Tests run the real gateway and a protocol-faithful test engine on real sockets. `scripts/ci.sh` adds ruff,248`mypy --strict` and `pip-audit`.249250## 🚢 Deployment251252Deployed with the cluster orchestrator `mld` (app `maclustr-llm`, pinned to node `m4mm`): three launchd services253(`ai.spboucher.maclustr-llm.engine`, `.gateway`, `.backup`), public route `https://llm.spboucher.ai` through the254MacLustr Tunnel (Caddy with automatic TLS, WireGuard to the node). Update, rollback, backup, recovery and model255change procedures: [docs/OPERATIONS.md](docs/OPERATIONS.md). Initial machine audit:256[docs/AUDIT-INITIAL.md](docs/AUDIT-INITIAL.md). Final report: [docs/RAPPORT-FINAL.md](docs/RAPPORT-FINAL.md).257258## 🩺 Troubleshooting259260| Symptom | Action |261|---|---|262| 503 `engine_unavailable` | engine reloading: `launchctl print gui/$(id -u)/ai.spboucher.maclustr-llm.engine`, `logs/engine.log` |263| 503 `engine_overloaded` / `queue_timeout` | demand above capacity; check Analytics, quotas, `max_active` |264| 504 on non-streaming calls | generation longer than 280 s: use `stream: true` |265| model download stalls | the Xet client hangs on this network; `scripts/model.py fetch` forces plain HTTP |266| passkeys lost | `maclustr-llm recover` on the Mac mini |267268## 🚧 Known limitations269270- Prefill is the bottleneck (about 450–650 tokens/s): a never-seen 12 000-token prompt waits about 27 s for its271  first token; context window served is 32 768 tokens.272- JSON mode is validated, not grammar-constrained; validity is guaranteed for non-streaming responses only.273- API error messages are in French.274- Passkey flows were verified with a virtual authenticator in Chromium, not yet on a physical Touch ID / Face ID device.275- No independent security audit.276277## 👤 Author278279<div align="center"><img src="docs/brand/icon.png" width="64" height="64" alt=""><br>280281**Simon-Pierre Boucher** — design, development, operations<br>282[www.spboucher.ai](https://www.spboucher.ai) · [git.spboucher.ai](https://git.spboucher.ai) · contact@spboucher.ai</div>283284## 📄 License285286All rights reserved © 2026 Simon-Pierre Boucher. The model weights are distributed by their authors under Apache-2.0.287
Files main
Menu

Theme