# CLAUDE.md — LLM API (www.llm-api.io)

Private OpenAI-compatible local model server for Apple Silicon. Full product spec: see docs/ and README. Runs on M1M64 (Mac Studio M1 Max 64 GB, rented, user simon), deployed with mld deploy llm-api --node M1M64 (manifest M1M32:~/dispatch/apps/llm-api.json), public via MacLustr Tunnel (BHS64) at https://www.llm-api.io.

# Layout

server/ Python 3.13 FastAPI (llm_api): gateway, registry, manager, workers (llm_api/worker/mlx_worker.py, llama-server), tests (pytest, fake workers, no MLX needed).
web/ Next.js 16 console (proxied by the API in production; pnpm dev proxies /api,/v1 to API_URL).
Data root on the node: ~/llm-api/ (models/ library, data/ SQLite + .secret + .env, logs/). Code: ~/apps/llm-api/.

# Rules (non-negotiable)

Never load a model whose estimate exceeds the budget; never rely on swap; never fake metrics.
Workers only on 127.0.0.1; memory release = kill the worker process.
Never mark ready before warm-up; never delete model files automatically.
Verify MLX / mlx-lm / mlx-vlm / llama.cpp / huggingface-hub APIs against the installed versions before changing runtime code.
Always test load → inference → unload → memory recovery after touching the manager or workers.

# Dev loop

bash

cd server && python -m compileall -q llm_api && rsync -az --delete --exclude __pycache__ --exclude .venv ./ M1M64:~/apps/llm-api/server/
ssh M1M64 'source ~/llm-api/.venv/bin/activate; cd ~/apps/llm-api/server && python -m pytest -q --basetemp=/tmp/llmapi-tests; rm -rf /tmp/llmapi-tests'
cd web && pnpm exec tsc --noEmit && pnpm build

Gotchas: pytest temp dirs must stay sparse (--basetemp + sparse fixtures — a real 400-layer fixture once filled the 1.8 TB SSD); socket.getfqdn hangs on the rented Mac (never use stock HTTPServer.server_bind); MLX work must stay on one thread per worker.