SPB Git

Repository Public

12 entries at the root
History

Latest commit docs: README, final deployment report, published benchmark data fe036fc

NameLast commitUpdated
bench Tune engine from measurements; add memory test, web portal work in progress
config Tune engine from measurements; add memory test, web portal work in progress
deploy Tune engine from measurements; add memory test, web portal work in progress
docs docs: README, final deployment report, published benchmark data
scripts Web portal, admin dashboard, published benchmarks; guaranteed lease release for unread streams
src Web portal, admin dashboard, published benchmarks; guaranteed lease release for unread streams
tests Web portal, admin dashboard, published benchmarks; guaranteed lease release for unread streams
web Web portal, admin dashboard, published benchmarks; guaranteed lease release for unread streams
.gitignore MacLustr LLM API: gateway, MLX engine launcher, auth, benchmarks, deployment
pyproject.toml MacLustr LLM API: gateway, MLX engine launcher, auth, benchmarks, deployment
README.md docs: README, final deployment report, published benchmark data
uv.lock MacLustr LLM API: gateway, MLX engine launcher, auth, benchmarks, deployment
README.md View source
MacLustr LLM API logo



MacLustr LLM API icon

# MacLustr LLM API

A complete local AI inference platform on Apple Silicon: an OpenAI-compatible API served by Apple MLX on a Mac mini M4 Pro, with API keys, quotas, a priority queue, a developer portal, a playground and a passkey-protected admin dashboard.

Author: Simon-Pierre Boucher

website version platform license author

Python MLX MLX-LM FastAPI SQLite WebAuthn uv frontend

commits last commit activity language languages size branches tags clones

python tests typing tables docs

node ports launchd deploy tunnel


# Table of contents

# 🧭 Overview

https://llm.spboucher.ai is the single entry point for LLM requests from the author's web, macOS and iOS apps and agents. Inference is entirely local: no external LLM service is involved, and no prompt or response is stored.

python
from openai import OpenAI

client = OpenAI(base_url="https://llm.spboucher.ai/v1", api_key="YOUR_MACLUSTR_API_KEY")
r = client.chat.completions.create(model="maclustr-qwen3",
                                   messages=[{"role": "user", "content": "Explique les modèles Transformer."}])
print(r.choices[0].message.content)
URL What
/ · /docs · /status public developer portal (15 navigation entries, generated parameter tables, search)
/playground real-API playground (streaming, tools, JSON mode, live metrics)
/v1/* OpenAI-compatible API (Bearer key)
/admin private dashboard, passkey only
/setup/<token> one-time administrator activation link

Model served: maclustr-qwen3 = mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit (Apache-2.0, revision pinned, SHA-256 verified), chosen on measurements against three other candidates — see docs/BENCHMARKS.md.

# ✨ Features

Inference engine (engine_main.py, hardened mlx_lm server): model kept loaded, continuous batching, batched prefill, KV prefix cache bounded in bytes, token streaming, cancellation, tool-call parsing, model locked (a request can never load another model), loopback-only, /internal/stats with real MLX memory.

API gateway: OpenAI formats for chat/completions, completions, models; SSE streaming with keep-alives; tool calling; validated JSON mode (json_object / json_schema, one automatic repair, 422 instead of bad JSON); usage with cached tokens; request ids; standard error envelope; per-request timeouts; cancellation on disconnect.

Capacity control: per-key requests/min, tokens/min, daily and monthly token quotas, concurrency; four priority classes with aging (no starvation); bounded queue with immediate 503 back-pressure; in-flight token budget that bounds KV-cache memory; guard on real macOS memory pressure.

Administration: passkeys (WebAuthn/FIDO2, Touch ID / Face ID, no password), one-time 15-minute setup link printed on the Mac mini, local recovery, hashed sessions, CSRF, recent-auth for critical actions, audit log; dashboard with live overview, Mac monitoring, analytics, API keys, models, security.

Portal: static site without build step or external resource (strict CSP), light and dark themes, reduced-motion support, parameter and error tables generated at runtime from /openapi.json.

# 📈 Measured performance

Mac mini M4 Pro, 48 GB, through the production gateway (full data and method in docs/BENCHMARKS.md, raw reports in docs/bench-reports/).

Measure Value
Single stream 80.6 tokens/s, first token in 111 ms
8 concurrent generations 155.8 tokens/s aggregate, 20.4 per request, first token p95 695 ms
Prefix cache (2 304-token prefix) first token 3 564 ms cold → 147 ms warm
Idle MLX memory (model loaded) 17.2 GB
Peak MLX memory, 82 000 tokens in flight 33.9 GB, no swap, normal pressure
Quality suite (41 auto-graded items) 40/41

# 🏗 Architecture

flowchart LR
    C["Clients<br/>OpenAI SDKs, apps, agents"] -- HTTPS --> P["Caddy (TLS)<br/>cluster gateway"]
    P -- WireGuard --> G
    subgraph M["Mac mini M4 Pro · 48 GB"]
        G["API gateway :8810<br/>auth · limits · priority queue · site"]
        E["MLX engine 127.0.0.1:8811<br/>continuous batching · KV prefix cache"]
        D[("SQLite<br/>hashed keys · sessions · usage · audit")]
        G --> E --> GPU["Apple GPU (Metal)"]
        G --- D
    end
sequenceDiagram
    participant T as Terminal (Mac mini)
    participant B as Browser
    participant S as Gateway
    T->>S: maclustr-llm setup-link
    S-->>T: https://…/setup/<token> (15 min, single use, stored hashed)
    B->>S: POST /api/setup/options {token}
    S-->>B: WebAuthn creation options + challenge
    B->>B: Touch ID / Face ID (system sheet)
    B->>S: POST /api/setup/complete {credential}
    S->>S: one transaction: verify, create admin, store passkey, burn link
    S-->>B: session cookie (HttpOnly, Secure, SameSite=Strict)

More: docs/ARCHITECTURE.md.

# 🧰 Stack

Layer Technology Role
Inference Apple MLX 0.32.4, MLX-LM 0.32.2 (Metal) model execution, batching, KV cache
Gateway Python 3.12, FastAPI, uvicorn, httpx, Pydantic 2 API, scheduling, admin
Auth webauthn (py_webauthn) passkey registration and assertion
Storage SQLite (WAL) state; no prompt or response
Monitoring psutil, sysctl, ioreg, pmset real CPU / GPU / memory / pressure
Frontend HTML, CSS, ES modules portal, playground, dashboard (no build, no dependency)
Packaging uv + uv.lock reproducible installs
Services launchd engine, gateway, daily backup

# 🗂 Repository layout

text
src/maclustr_llm/   engine_main.py (MLX engine) · app.py · routes_{openai,public,admin}.py · inference.py
                    scheduler.py · limits.py · keys.py · auth.py · metrics.py · sysmon.py · openapi.py · cli.py …
web/                index, docs, playground, status, admin, setup + assets/ (tokens.css, site/, admin/, data/)
config/models.json  model catalog (public alias, repo, pinned revision) and engines (capacity, token budget)
deploy/             install.sh · verify.sh · run-engine.sh · run-gateway.sh · *.env · launchd/*.plist
bench/              bench.py (speed) · quality.py (41 graded items) · memtest.py (memory) · compare.sh
scripts/            model.py (pinned download + SHA-256) · backup.sh · ci.sh · e2e_live.py · publish_bench.py
tests/              test_api.py · test_limits.py · test_auth.py · fake_engine.py (test double) · conftest.py
docs/               AUDIT-INITIAL · ARCHITECTURE · SECURITY · OPERATIONS · BENCHMARKS · RAPPORT-FINAL · bench-reports/

# 🚀 Getting started

bash
uv sync --frozen --extra dev            # development machine
scripts/ci.sh                           # ruff, mypy --strict, pytest, pip-audit
.venv/bin/python scripts/dev_server.py  # real gateway + echo test engine on http://localhost:8910

deploy/install.sh                       # on the serving Mac: locked deps, migrations, model, launchd services
maclustr-llm setup-link                 # print the one-time administrator activation link

# ⚙️ Configuration

Non-secret settings live in deploy/gateway.env and deploy/engine.env; optional local overrides in ~/.config/maclustr-llm/{gateway,engine}.env.

Variable Purpose
MLLM_PUBLIC_ORIGIN, MLLM_RP_ID public origin and WebAuthn relying-party id
MLLM_HOST, MLLM_PORT, MLLM_TRUST_PROXY listening address, proxy headers
MLLM_DATA_DIR, MLLM_CATALOG, MLLM_WEB_DIR state directory, model catalog, static site
MLLM_MAX_QUEUE, MLLM_QUEUE_TIMEOUT_S, MLLM_AGING_S queue size, wait limit, priority aging
MLLM_REQUEST_TIMEOUT_S, MLLM_NONSTREAM_TIMEOUT_S, MLLM_FIRST_TOKEN_TIMEOUT_S generation limits
MLLM_MAX_BODY_BYTES, MLLM_MAX_MESSAGES request size limits
MLLM_SESSION_TTL_S, MLLM_SESSION_IDLE_S, MLLM_REAUTH_WINDOW_S, MLLM_SETUP_TTL_S admin session policy
ENGINE_MODEL_DIR, ENGINE_PORT, ENGINE_DECODE_CONCURRENCY, ENGINE_PROMPT_CONCURRENCY engine model and batch sizes
ENGINE_PROMPT_CACHE_SIZE, ENGINE_PROMPT_CACHE_BYTES, ENGINE_MEMORY_LIMIT, ENGINE_CACHE_LIMIT engine memory bounds

# 🔌 API

Method Path Auth Notes
GET /v1/models, /v1/models/{id} API key models visible to the key
POST /v1/chat/completions API key streaming, tools (tool_choice: auto/none), validated JSON mode
POST /v1/completions API key raw text completion
GET /health, /ready, /api/status, /openapi.json public liveness, engine readiness, public status, contract
POST /api/setup/{status,options,complete} one-time token administrator activation / recovery
* /admin/api/* (session, login, reauth, overview, system, analytics, requests, keys, models, engine, security, passkeys, sessions, audit) passkey session + CSRF dashboard

Not offered because the engine does not support them: forced tool_choice, grammar-constrained decoding, embeddings, image input, /v1/responses.

# 🗃 Data model

admin_users · passkeys · setup_tokens · challenges · sessions · api_keys · usage_minute · request_log · audit_log · sys_samples · settings. Secrets are stored only as HMAC-SHA-256 digests keyed by a server secret kept outside the database.

# 🔒 Security & privacy

No prompt or response is stored or logged. API keys, session ids, CSRF tokens and setup tokens are stored hashed. The engine listens on loopback only. Strict CSP, HSTS, __Host- session cookie, CSRF token and origin check, recent passkey authentication for critical actions, IP throttling on authentication routes, audit log. Details and known limits: docs/SECURITY.md.

# 🧪 Testing

Suite Count Covers
tests/test_api.py 23 OpenAI formats, streaming, tools, JSON mode, validation, permissions, engine failure, official OpenAI SDK
tests/test_limits.py 15 rate limits, quotas, queue, back-pressure, priorities and aging, token budget, cancellation
tests/test_auth.py 15 setup link, passkeys with real signatures, sessions, CSRF, recent auth, recovery, no secret at rest
scripts/e2e_live.py 14 checks the deployed instance over HTTPS, with the real model

Tests run the real gateway and a protocol-faithful test engine on real sockets. scripts/ci.sh adds ruff, mypy --strict and pip-audit.

# 🚢 Deployment

Deployed with the cluster orchestrator mld (app maclustr-llm, pinned to node m4mm): three launchd services (ai.spboucher.maclustr-llm.engine, .gateway, .backup), public route https://llm.spboucher.ai through the MacLustr Tunnel (Caddy with automatic TLS, WireGuard to the node). Update, rollback, backup, recovery and model change procedures: docs/OPERATIONS.md. Initial machine audit: docs/AUDIT-INITIAL.md. Final report: docs/RAPPORT-FINAL.md.

# 🩺 Troubleshooting

Symptom Action
503 engine_unavailable engine reloading: launchctl print gui/$(id -u)/ai.spboucher.maclustr-llm.engine, logs/engine.log
503 engine_overloaded / queue_timeout demand above capacity; check Analytics, quotas, max_active
504 on non-streaming calls generation longer than 280 s: use stream: true
model download stalls the Xet client hangs on this network; scripts/model.py fetch forces plain HTTP
passkeys lost maclustr-llm recover on the Mac mini

# 🚧 Known limitations

  • Prefill is the bottleneck (about 450–650 tokens/s): a never-seen 12 000-token prompt waits about 27 s for its first token; context window served is 32 768 tokens.
  • JSON mode is validated, not grammar-constrained; validity is guaranteed for non-streaming responses only.
  • API error messages are in French.
  • Passkey flows were verified with a virtual authenticator in Chromium, not yet on a physical Touch ID / Face ID device.
  • No independent security audit.

# 👤 Author


Simon-Pierre Boucher — design, development, operations
www.spboucher.ai · git.spboucher.ai · contact@spboucher.ai

# 📄 License

All rights reserved © 2026 Simon-Pierre Boucher. The model weights are distributed by their authors under Apache-2.0.

Clone maclustr-llm

Anonymous clone and fetch. Pushing requires the owner's personal access token.

Snapshot of main

Files main
Menu

Theme