# PolyLLM — cluster deployment (MacLustr)

Production runs on the owner's private Apple Silicon cluster and is exposed through the **MacLustr Tunnel**
(WireGuard + Caddy on the OVH gateway BHS64, since 2026-09-10 — ngrok is gone) at **https://www.polyllm.io**.
Everything is orchestrated by `mld` (maclustr-dispatch) from the gateway node M1M32.

```
Internet → www.polyllm.io (A → 51.161.112.61, BHS64) → Caddy TLS → WireGuard wg1 → M3U96a:8240 Next.js `next start` → PostgreSQL 17 (localhost)
```

| Item | Value |
| --- | --- |
| Node | M3U96a (pinned: local PostgreSQL 17, same host as fetcha / spinza / rareindex) |
| Directory | `~/apps/polyllm` (rsync'd by mld from the laptop staging area) |
| Processes | PM2 `polyllm-web` (Next 16, port 8240, binds 0.0.0.0). Public route = Caddy on BHS64 (`mld tunnel route polyllm`), repointed automatically on deploy/move |
| Database | `postgres://localhost:5432/polyllm` (created by the post-sync hook if missing) |
| Manifest | `M1M32:~/dispatch/apps/polyllm.json` — **the only place secrets live** (gitignored copy `deploy/polyllm.mld.json`) |
| Health | `GET /api/health` (liveness), `GET /api/health/ready` (DB ping; used by mld) |
| Registry | `mld status` / `mld status --live` |

## Environment variables (production)

See `.env.example`. Required: `NODE_ENV=production`, `PORT=8240`, `PUBLIC_APP_URL=https://www.polyllm.io`,
`DATABASE_URL`, `AUTH_SECRET` (32+ bytes), `API_KEY_ENCRYPTION_SECRET` (32+ bytes — **rotating it invalidates every
stored user key**), `RESEND_API_KEY`, `RESEND_FROM_EMAIL`, `ADMIN_EMAILS`. Optional owner keys
(`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_GEMINI_API_KEY`, `XAI_API_KEY`) are used **only** for the scheduled
model-registry refresh and the admin diagnostics; user requests always use the user's own encrypted key.

## Build & deploy

```bash
# laptop
cd ~/Desktop/Projets/apps-web/polyllm
pnpm typecheck && pnpm lint && pnpm test          # gates
scp deploy/polyllm.mld.json M1M32:~/dispatch/apps/polyllm.json   # first time / when secrets change
~/Desktop/cluster-skill/mld stage . polyllm       # laptop → M1M32 staging
~/Desktop/cluster-skill/mld deploy polyllm --node M3U96a
```

`mld deploy` runs the manifest hooks on the node: `pnpm install --frozen-lockfile` → `createdb polyllm` (if needed) +
`pnpm db:migrate` (Drizzle migrations in `drizzle/`, version-controlled) → `pnpm build` → PM2 start/reload of
`polyllm-web` → local health check (`/api/health/ready`) → tunnel route `https://www.polyllm.io → M3U96a:8240` → public health check → registry update.

Restart / logs on the node:

```bash
ssh M3U96a 'pm2 restart polyllm-web; pm2 logs polyllm-web --lines 100 --nostream'
~/Desktop/cluster-skill/mld tunnel status | grep polyllm
```

Migrations are applied by the hook before the new build starts; they must stay backward compatible with the running
version (add columns, never drop in the same release).

## Domain / TLS (MacLustr Tunnel)

- DNS (GoDaddy): `www.polyllm.io` A → `51.161.112.61` (BHS64). The apex `polyllm.io` redirects to `https://www.polyllm.io`.
- TLS terminates on Caddy (Let's Encrypt, automatic). The app sees `X-Forwarded-For` / `X-Forwarded-Proto`; Better Auth is
  configured with `baseURL = PUBLIC_APP_URL` and secure cookies in production.
- Streaming: SSE responses send a `: ping` heartbeat every 15 s so idle long reasoning turns are never closed by proxies.
- `mld heal` (every 5 min on M1M32) re-creates the wg1 peer / Caddy route if they disappear after an outage.

## Process management

PM2 with `pm2 startup` (LaunchAgent) on the node → processes survive terminal closure and reboots; `autorestart: true`,
`max_memory_restart: 2G`. The model-registry scheduler runs inside the web process (`src/instrumentation.ts`, every
`MODEL_SYNC_INTERVAL_MINUTES`, default 360) plus a startup sync.

## Backups

- Daily `pg_dump` at 03:45 via launchd `dev.polyllm.backup` on M3U96a → `~/backups/polyllm/polyllm-YYYYMMDD.sql.gz`,
  14-day retention (`find -mtime +14 -delete`). Install with `deploy/backup/install.sh` (see that file).
- Restore: `gunzip -c ~/backups/polyllm/polyllm-YYYYMMDD.sql.gz | psql polyllm_restore` after `createdb polyllm_restore`;
  swap with `ALTER DATABASE … RENAME` during a maintenance window. Test restore on the 1st of each month into a scratch DB
  and run `select count(*) from users;` to validate.
- Encrypted user keys are only decryptable with the same `API_KEY_ENCRYPTION_SECRET`: back up the manifest too
  (`M1M32:~/dispatch/apps/polyllm.json` is included in the spbgit gateway backup).

## Rollback

```bash
ssh M3U96a 'cd ~/apps/polyllm && git log --oneline -5'          # deployed commit
# redeploy the previous tag from the laptop:
git checkout <previous-tag> && ~/Desktop/cluster-skill/mld stage . polyllm && ~/Desktop/cluster-skill/mld deploy polyllm --node M3U96a
```

If a migration must be reverted, restore last night's dump (see Backups) — Drizzle migrations are forward-only.

## Moving to another node

`mld move polyllm --to <node>` handles rsync, PM2 and the tunnel route, but the database is local: dump on M3U96a, restore on the
target (PostgreSQL 17 via Homebrew), then update `DATABASE_URL` in the manifest before the move.