# PolyLLM — cluster deployment (MacLustr)

Production runs on the owner's private Apple Silicon cluster and is exposed through the MacLustr Tunnel (WireGuard + Caddy on the OVH gateway BHS64, since 2026-09-10 — ngrok is gone) at https://www.polyllm.io. Everything is orchestrated by mld (maclustr-dispatch) from the gateway node M1M32.

text

Internet → www.polyllm.io (A → 51.161.112.61, BHS64) → Caddy TLS → WireGuard wg1 → M3U96a:8240 Next.js `next start` → PostgreSQL 17 (localhost)

Item	Value
Node	M3U96a (pinned: local PostgreSQL 17, same host as fetcha / spinza / rareindex)
Directory	`~/apps/polyllm` (rsync'd by mld from the laptop staging area)
Processes	PM2 `polyllm-web` (Next 16, port 8240, binds 0.0.0.0). Public route = Caddy on BHS64 (`mld tunnel route polyllm`), repointed automatically on deploy/move
Database	`postgres://localhost:5432/polyllm` (created by the post-sync hook if missing)
Manifest	`M1M32:~/dispatch/apps/polyllm.json` — the only place secrets live (gitignored copy `deploy/polyllm.mld.json`)
Health	`GET /api/health` (liveness), `GET /api/health/ready` (DB ping; used by mld)
Registry	`mld status` / `mld status --live`

# Environment variables (production)

See .env.example. Required: NODE_ENV=production, PORT=8240, PUBLIC_APP_URL=https://www.polyllm.io, DATABASE_URL, AUTH_SECRET (32+ bytes), API_KEY_ENCRYPTION_SECRET (32+ bytes — rotating it invalidates every stored user key), RESEND_API_KEY, RESEND_FROM_EMAIL, ADMIN_EMAILS. Optional owner keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_GEMINI_API_KEY, XAI_API_KEY) are used only for the scheduled model-registry refresh and the admin diagnostics; user requests always use the user's own encrypted key.

# Build & deploy

bash

# laptop
cd ~/Desktop/Projets/apps-web/polyllm
pnpm typecheck && pnpm lint && pnpm test          # gates
scp deploy/polyllm.mld.json M1M32:~/dispatch/apps/polyllm.json   # first time / when secrets change
~/Desktop/cluster-skill/mld stage . polyllm       # laptop → M1M32 staging
~/Desktop/cluster-skill/mld deploy polyllm --node M3U96a

mld deploy runs the manifest hooks on the node: pnpm install --frozen-lockfile → createdb polyllm (if needed) + pnpm db:migrate (Drizzle migrations in drizzle/, version-controlled) → pnpm build → PM2 start/reload of polyllm-web → local health check (/api/health/ready) → tunnel route https://www.polyllm.io → M3U96a:8240 → public health check → registry update.

Restart / logs on the node:

bash

ssh M3U96a 'pm2 restart polyllm-web; pm2 logs polyllm-web --lines 100 --nostream'
~/Desktop/cluster-skill/mld tunnel status | grep polyllm

Migrations are applied by the hook before the new build starts; they must stay backward compatible with the running version (add columns, never drop in the same release).

# Domain / TLS (MacLustr Tunnel)

DNS (GoDaddy): www.polyllm.io A → 51.161.112.61 (BHS64). The apex polyllm.io redirects to https://www.polyllm.io.
TLS terminates on Caddy (Let's Encrypt, automatic). The app sees X-Forwarded-For / X-Forwarded-Proto; Better Auth is configured with baseURL = PUBLIC_APP_URL and secure cookies in production.
Streaming: SSE responses send a : ping heartbeat every 15 s so idle long reasoning turns are never closed by proxies.
mld heal (every 5 min on M1M32) re-creates the wg1 peer / Caddy route if they disappear after an outage.

# Process management

PM2 with pm2 startup (LaunchAgent) on the node → processes survive terminal closure and reboots; autorestart: true, max_memory_restart: 2G. The model-registry scheduler runs inside the web process (src/instrumentation.ts, every MODEL_SYNC_INTERVAL_MINUTES, default 360) plus a startup sync.

# Backups

Daily pg_dump at 03:45 via launchd dev.polyllm.backup on M3U96a → ~/backups/polyllm/polyllm-YYYYMMDD.sql.gz, 14-day retention (find -mtime +14 -delete). Install with deploy/backup/install.sh (see that file).
Restore: gunzip -c ~/backups/polyllm/polyllm-YYYYMMDD.sql.gz | psql polyllm_restore after createdb polyllm_restore; swap with ALTER DATABASE … RENAME during a maintenance window. Test restore on the 1st of each month into a scratch DB and run select count(*) from users; to validate.
Encrypted user keys are only decryptable with the same API_KEY_ENCRYPTION_SECRET: back up the manifest too (M1M32:~/dispatch/apps/polyllm.json is included in the spbgit gateway backup).

# Rollback

bash

ssh M3U96a 'cd ~/apps/polyllm && git log --oneline -5'          # deployed commit
# redeploy the previous tag from the laptop:
git checkout <previous-tag> && ~/Desktop/cluster-skill/mld stage . polyllm && ~/Desktop/cluster-skill/mld deploy polyllm --node M3U96a

If a migration must be reverted, restore last night's dump (see Backups) — Drizzle migrations are forward-only.

# Moving to another node

mld move polyllm --to <node> handles rsync, PM2 and the tunnel route, but the database is local: dump on M3U96a, restore on the target (PostgreSQL 17 via Homebrew), then update DATABASE_URL in the manifest before the move.