# Deploying CancerIndex on the MacLustr cluster

> **Since 2026-09-14 production runs on the OVH server BHS128b** (`ssh BHS128b`, ubuntu@51.161.112.85, Ubuntu 24.04, 16 threads / 128 GB),
> outside `mld`: code in `~/apps/cancerindex`, Postgres 17 native (db/role `cancerindex`/`cancerindex`, pgvector + pg_trgm + unaccent, en_US.UTF-8),
> raw lake + backups under `~/apps/cancerindex/{data,backups}`, PM2 under systemd (`pm2-ubuntu`) with `~/apps/pm2.cancerindex.config.cjs` generated
> from the mld manifest by `~/apps/.manifests/gen-pm2.py` (secrets, 0600). Tunnel route: `mlt add www.cancerindex.io BHS128b:8250`
> (BHS64 Caddy → wg1 10.67.0.62); firewall `ufw allow in on wg1 from 10.67.0.1 to any port 8250 proto tcp`.
> **Release on BHS128b**: `rsync -az --exclude node_modules --exclude .git --exclude 'apps/web/.next' --exclude data --exclude backups --exclude logs . BHS128b:apps/cancerindex/`
> then `ssh BHS128b 'cd apps/cancerindex && pnpm install --frozen-lockfile && DATABASE_URL=postgres://cancerindex:cancerindex@127.0.0.1:5432/cancerindex pnpm db:migrate && NODE_ENV=production NEXT_PUBLIC_SITE_URL=https://www.cancerindex.io CI_API_URL=http://127.0.0.1:8251 DATABASE_URL=postgres://cancerindex:cancerindex@127.0.0.1:5432/cancerindex pnpm --filter @cancerindex/web build && pm2 restart cancerindex-api cancerindex-web cancerindex-worker && pm2 save'`.
> pnpm ≥ 10 needs `onlyBuiltDependencies` (esbuild, sharp, @tailwindcss/oxide, unrs-resolver) in `pnpm-workspace.yaml` on a fresh install.
> The M4M64b copy (code, Postgres `cancerindex`) is kept as a cold copy, retired from `mld` (`--keep-dir`); the `mld` procedure below still describes it.

Everything goes through the gateway **M1M32** and the `mld` orchestrator (`~/Desktop/cluster-skill/mld`
on the laptop relays to `M1M32:~/dispatch`). Nothing in this folder runs on the cluster by itself;
these are the files and the procedure.

## What gets deployed

| Piece | Where | Port |
|---|---|---|
| `cancerindex-web` (PM2) | `~/apps/cancerindex/apps/web`, `next start -p 8250 -H 0.0.0.0` | 8250 (public via ngrok) |
| `cancerindex-api` (PM2) | `~/apps/cancerindex`, `tsx apps/api/src/server.ts` | 8251 (127.0.0.1) |
| `cancerindex-worker` (PM2) | `~/apps/cancerindex`, `tsx workers/main.ts` | — |
| `cancerindex-backup` (PM2, cron `20 5 * * *`, `autorestart: false`) | `~/apps/cancerindex`, `/bin/bash deploy/backup.sh` | — |
| `cancerindex-ngrok` | `www.cancerindex.io` → 8250 (reserved domain) | — |
| PostgreSQL 17 | database `cancerindex` on the node (extensions `pg_trgm`, `unaccent`, `vector`) | 5432 |

Preferred node: **M4M64b** (Postgres 17 + pgvector, pnpm, redis, 772 GB disk). Avoid M3U96b
(hfmarketdata) and M1M32 (gateway). Requires ≈ 6 GB RAM.

## Procedure

1. **Manifest** — copy `deploy/mld-manifest.cancerindex.json` to `M1M32:~/dispatch/apps/cancerindex.json`
   and replace the `{{ADMIN_TOKEN}}` placeholder with a real secret (or set it through
   `env_overrides`). `{{HOME}}` is expanded by `mld`.

   ```bash
   scp deploy/mld-manifest.cancerindex.json M1M32:~/dispatch/apps/cancerindex.json
   ssh M1M32 "sed -i '' 's/{{ADMIN_TOKEN}}/'\"$(openssl rand -hex 24)\"'/g' ~/dispatch/apps/cancerindex.json"
   ```

2. **Stage** — from a clean export of the repository (no `node_modules`, no `.env`, no data lake):

   ```bash
   cd /path/to/cancerindex
   rm -rf /tmp/cancerindex-stage && mkdir -p /tmp/cancerindex-stage
   git archive HEAD | tar -x -C /tmp/cancerindex-stage
   ~/Desktop/cluster-skill/mld stage /tmp/cancerindex-stage cancerindex
   ```

   Source of truth for the code is spbgit `cancerindex.git` (`ssh gitsrv`); push there first.

3. **Deploy** — `mld` scores the nodes, syncs to `~/apps/cancerindex`, runs the `post_sync` hooks
   (`pnpm install --frozen-lockfile`, `createdb cancerindex` if missing, extensions, `db:migrate`,
   `db:seed`, `cix sources:sync`, `mkdir data/raw data/cache logs`, `next build`), starts the PM2
   processes and the ngrok tunnel, and checks `/healthz`:

   ```bash
   ~/Desktop/cluster-skill/mld plan cancerindex
   ~/Desktop/cluster-skill/mld deploy cancerindex            # or --node M4M64b
   ~/Desktop/cluster-skill/mld status --live
   ```

4. **First ingestion** — on the node, in order (terminology → genes → evidence/genomics/variants →
   trials → literature → epidemiology), then counters and rankings:

   ```bash
   ssh M4M64b 'cd ~/apps/cancerindex && bash deploy/first-run.sh'
   ```

   `first-run.sh` is idempotent: connectors are restartable (cursors), records are hashed, and the
   script re-runs a connector until its last run is `succeeded` or the retry budget is spent.
   Expect ClinicalTrials.gov and PubMed to need several 45-minute windows; the worker's cron picks
   up where the script stops.

5. **Verify**

   ```bash
   curl -fsS http://127.0.0.1:8251/healthz
   curl -fsS http://127.0.0.1:8251/v1/stats | jq .data
   curl -fsS "http://127.0.0.1:8251/v1/sources" | jq '.data[] | {slug, status, health: .connector.health}'
   pm2 ls && pm2 logs cancerindex-worker --lines 50
   curl -fsS https://www.cancerindex.io/healthz
   ```

## Day-2 operations

- **Redeploy code**: push to spbgit, `git archive` → `mld stage` → `mld deploy cancerindex` (hooks
  rebuild the web app; PM2 restarts). Migrations run in `post_sync` (`pnpm db:migrate`).
- **Move node**: `mld move cancerindex --to <node>` — the data lake (`data/raw`) is excluded from sync;
  copy it separately (`rsync -a M4M64b:~/apps/cancerindex/data/raw/ <node>:~/apps/cancerindex/data/raw/`)
  and dump/restore the database (`pg_dump -Fc cancerindex`).
- **Backups** (§172): the PM2 process `cancerindex-backup` runs `deploy/backup.sh` every day at
  05:20 UTC (`cron_restart`, `autorestart: false` — PM2 restarts it on the cron only). It writes
  `pg_dump -Fc` to `~/apps/cancerindex/backups/cancerindex-YYYYMMDD-HHMM.dump`, verifies the archive
  with `pg_restore --list` (core tables must be present), keeps the last **14 daily + 8 weekly**
  dumps, logs to `logs/backup.log` and exits non-zero on any failure (`pm2 ls` shows it `errored`;
  `pm2 logs cancerindex-backup`). Run it by hand with `bash deploy/backup.sh`.
  The raw data lake (`data/raw`, append-only gzip JSON Lines) is **not** in the dump — mirror it with
  `rsync -a ~/apps/cancerindex/data/raw/ <nas-or-node>:/…/cancerindex-raw/`. Both are needed for TRACE.
- **Restore test / disaster recovery**: `bash deploy/restore.sh backups/<file>.dump` creates
  `cancerindex_restore_<timestamp>` on the same server (never overwrites `cancerindex`), restores in
  parallel (`--jobs 4`), prints row counts. To promote: stop the PM2 processes, `ALTER DATABASE
  cancerindex RENAME TO cancerindex_old; ALTER DATABASE cancerindex_restore_… RENAME TO cancerindex;`
  (or point `DATABASE_URL` at the restored database), start the processes, run `pnpm cix doctor`.
  Test a restore after every schema migration.
- **Readiness / health**: `pnpm cix doctor` (env, database + extensions + pending migrations, table
  sizes, every connector's health / last success / last run / anomaly / drift / cursor, stale
  connectors, unresolved labels, `data/raw` disk usage, ranking freshness, open alerts; exit 1 on a
  hard failure) and `pnpm cix alerts` (open alerts; `ack <id>` / `resolve <id>`). The API exposes
  `GET /v1/admin/alerts` (admin token).
- **Stopping a run cleanly**: `pm2 stop cancerindex-worker` sends SIGTERM — the active connector
  saves its cursor and marks the run `aborted`; the next scheduled run resumes at the last completed
  page (docs/connectors/README.md).
- **Trigger a run**: `pnpm cix run <id>` on the node, or `POST /v1/admin/connectors/<id>/run` with
  `x-admin-token`, or `tsx workers/cli.ts run <id>`.
- **Pause a connector**: `POST /v1/admin/connectors/<id>/pause` (worker skips scheduled runs).
- **Logs**: pino JSON via PM2 (`pm2 logs cancerindex-api`), `CI_SERVICE` labels api/worker.
- **Schedules** (UTC, from manifests): civic 02:00 daily, clinicaltrials 02:30 daily, pubmed 03:00
  daily, ncit-evs 03:00 on the 1st, oncotree 04:00 Mondays, hgnc/clinvar 05:00 Tuesdays, gdc 06:00
  on the 1st; counters 06:00, rank 06:30, health probe hourly at :15. Check with
  `tsx workers/cli.ts schedules`.

## Gotchas

- `pg-boss` creates its own schema `pgboss` in the same database on first worker start; the API only
  enqueues (no supervision). Queue policy is `stately` (one job per connector queued/active).
- `mld` syncs with the `sync_excludes` above: `.env*`, `data/raw`, `.next` are never transferred;
  production env comes from the manifest `env` blocks.
- The web build needs `NEXT_PUBLIC_SITE_URL` and `CI_API_URL` at build time (baked into the bundle);
  they are set in the `post_sync` build command.
- M2U64 has no graphical session — not relevant here (PM2 processes), but launchd services there
  would need LaunchDaemons.
- Never commit real values into the manifest; `{{ADMIN_TOKEN}}` is a placeholder on purpose.