# CLAUDE.md — LLM API (www.llm-api.io)

Private OpenAI-compatible local model server for Apple Silicon. Full product spec: see `docs/` and README.
Runs on **M1M64** (Mac Studio M1 Max 64 GB, rented, user `simon`), deployed with `mld deploy llm-api --node M1M64`
(manifest `M1M32:~/dispatch/apps/llm-api.json`), public via MacLustr Tunnel (BHS64) at https://www.llm-api.io.

## Layout
- `server/` Python 3.13 FastAPI (`llm_api`): gateway, registry, manager, workers (`llm_api/worker/mlx_worker.py`, llama-server), tests (`pytest`, fake workers, no MLX needed).
- `web/` Next.js 16 console (proxied by the API in production; `pnpm dev` proxies /api,/v1 to `API_URL`).
- Data root on the node: `~/llm-api/` (`models/` library, `data/` SQLite + `.secret` + `.env`, `logs/`). Code: `~/apps/llm-api/`.

## Rules (non-negotiable)
1. Never load a model whose estimate exceeds the budget; never rely on swap; never fake metrics.
2. Workers only on 127.0.0.1; memory release = kill the worker process.
3. Never mark ready before warm-up; never delete model files automatically.
4. Verify MLX / mlx-lm / mlx-vlm / llama.cpp / huggingface-hub APIs against the installed versions before changing runtime code.
5. Always test load → inference → unload → memory recovery after touching the manager or workers.

## Dev loop
```bash
cd server && python -m compileall -q llm_api && rsync -az --delete --exclude __pycache__ --exclude .venv ./ M1M64:~/apps/llm-api/server/
ssh M1M64 'source ~/llm-api/.venv/bin/activate; cd ~/apps/llm-api/server && python -m pytest -q --basetemp=/tmp/llmapi-tests; rm -rf /tmp/llmapi-tests'
cd web && pnpm exec tsc --noEmit && pnpm build
```
Gotchas: pytest temp dirs must stay sparse (`--basetemp` + sparse fixtures — a real 400-layer fixture once filled the 1.8 TB SSD); `socket.getfqdn` hangs on the rented Mac (never use stock `HTTPServer.server_bind`); MLX work must stay on one thread per worker.