spb/polyllm

1# PolyLLM — cluster deployment (MacLustr)23Production runs on the owner's private Apple Silicon cluster and is exposed through the **MacLustr Tunnel**4(WireGuard + Caddy on the OVH gateway BHS64, since 2026-09-10 — ngrok is gone) at **https://www.polyllm.io**.5Everything is orchestrated by `mld` (maclustr-dispatch) from the gateway node M1M32.67```8Internet → www.polyllm.io (A → 51.161.112.61, BHS64) → Caddy TLS → WireGuard wg1 → M3U96a:8240 Next.js `next start` → PostgreSQL 17 (localhost)9```1011| Item | Value |12| --- | --- |13| Node | M3U96a (pinned: local PostgreSQL 17, same host as fetcha / spinza / rareindex) |14| Directory | `~/apps/polyllm` (rsync'd by mld from the laptop staging area) |15| Processes | PM2 `polyllm-web` (Next 16, port 8240, binds 0.0.0.0). Public route = Caddy on BHS64 (`mld tunnel route polyllm`), repointed automatically on deploy/move |16| Database | `postgres://localhost:5432/polyllm` (created by the post-sync hook if missing) |17| Manifest | `M1M32:~/dispatch/apps/polyllm.json` — **the only place secrets live** (gitignored copy `deploy/polyllm.mld.json`) |18| Health | `GET /api/health` (liveness), `GET /api/health/ready` (DB ping; used by mld) |19| Registry | `mld status` / `mld status --live` |2021## Environment variables (production)2223See `.env.example`. Required: `NODE_ENV=production`, `PORT=8240`, `PUBLIC_APP_URL=https://www.polyllm.io`,24`DATABASE_URL`, `AUTH_SECRET` (32+ bytes), `API_KEY_ENCRYPTION_SECRET` (32+ bytes — **rotating it invalidates every25stored user key**), `RESEND_API_KEY`, `RESEND_FROM_EMAIL`, `ADMIN_EMAILS`. Optional owner keys26(`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_GEMINI_API_KEY`, `XAI_API_KEY`) are used **only** for the scheduled27model-registry refresh and the admin diagnostics; user requests always use the user's own encrypted key.2829## Build & deploy3031```bash32# laptop33cd ~/Desktop/Projets/apps-web/polyllm34pnpm typecheck && pnpm lint && pnpm test          # gates35scp deploy/polyllm.mld.json M1M32:~/dispatch/apps/polyllm.json   # first time / when secrets change36~/Desktop/cluster-skill/mld stage . polyllm       # laptop → M1M32 staging37~/Desktop/cluster-skill/mld deploy polyllm --node M3U96a38```3940`mld deploy` runs the manifest hooks on the node: `pnpm install --frozen-lockfile` → `createdb polyllm` (if needed) +41`pnpm db:migrate` (Drizzle migrations in `drizzle/`, version-controlled) → `pnpm build` → PM2 start/reload of42`polyllm-web` → local health check (`/api/health/ready`) → tunnel route `https://www.polyllm.io → M3U96a:8240` → public health check → registry update.4344Restart / logs on the node:4546```bash47ssh M3U96a 'pm2 restart polyllm-web; pm2 logs polyllm-web --lines 100 --nostream'48~/Desktop/cluster-skill/mld tunnel status | grep polyllm49```5051Migrations are applied by the hook before the new build starts; they must stay backward compatible with the running52version (add columns, never drop in the same release).5354## Domain / TLS (MacLustr Tunnel)5556- DNS (GoDaddy): `www.polyllm.io` A → `51.161.112.61` (BHS64). The apex `polyllm.io` redirects to `https://www.polyllm.io`.57- TLS terminates on Caddy (Let's Encrypt, automatic). The app sees `X-Forwarded-For` / `X-Forwarded-Proto`; Better Auth is58  configured with `baseURL = PUBLIC_APP_URL` and secure cookies in production.59- Streaming: SSE responses send a `: ping` heartbeat every 15 s so idle long reasoning turns are never closed by proxies.60- `mld heal` (every 5 min on M1M32) re-creates the wg1 peer / Caddy route if they disappear after an outage.6162## Process management6364PM2 with `pm2 startup` (LaunchAgent) on the node → processes survive terminal closure and reboots; `autorestart: true`,65`max_memory_restart: 2G`. The model-registry scheduler runs inside the web process (`src/instrumentation.ts`, every66`MODEL_SYNC_INTERVAL_MINUTES`, default 360) plus a startup sync.6768## Backups6970- Daily `pg_dump` at 03:45 via launchd `dev.polyllm.backup` on M3U96a → `~/backups/polyllm/polyllm-YYYYMMDD.sql.gz`,71  14-day retention (`find -mtime +14 -delete`). Install with `deploy/backup/install.sh` (see that file).72- Restore: `gunzip -c ~/backups/polyllm/polyllm-YYYYMMDD.sql.gz | psql polyllm_restore` after `createdb polyllm_restore`;73  swap with `ALTER DATABASE … RENAME` during a maintenance window. Test restore on the 1st of each month into a scratch DB74  and run `select count(*) from users;` to validate.75- Encrypted user keys are only decryptable with the same `API_KEY_ENCRYPTION_SECRET`: back up the manifest too76  (`M1M32:~/dispatch/apps/polyllm.json` is included in the spbgit gateway backup).7778## Rollback7980```bash81ssh M3U96a 'cd ~/apps/polyllm && git log --oneline -5'          # deployed commit82# redeploy the previous tag from the laptop:83git checkout <previous-tag> && ~/Desktop/cluster-skill/mld stage . polyllm && ~/Desktop/cluster-skill/mld deploy polyllm --node M3U96a84```8586If a migration must be reverted, restore last night's dump (see Backups) — Drizzle migrations are forward-only.8788## Moving to another node8990`mld move polyllm --to <node>` handles rsync, PM2 and the tunnel route, but the database is local: dump on M3U96a, restore on the91target (PostgreSQL 17 via Homebrew), then update `DATABASE_URL` in the manifest before the move.92