# Deploying CancerIndex on the MacLustr cluster

Since 2026-09-14 production runs on the OVH server BHS128b (ssh BHS128b, ubuntu@51.161.112.85, Ubuntu 24.04, 16 threads / 128 GB), outside mld: code in ~/apps/cancerindex, Postgres 17 native (db/role cancerindex/cancerindex, pgvector + pg_trgm + unaccent, en_US.UTF-8), raw lake + backups under ~/apps/cancerindex/{data,backups}, PM2 under systemd (pm2-ubuntu) with ~/apps/pm2.cancerindex.config.cjs generated from the mld manifest by ~/apps/.manifests/gen-pm2.py (secrets, 0600). Tunnel route: mlt add www.cancerindex.io BHS128b:8250 (BHS64 Caddy → wg1 10.67.0.62); firewall ufw allow in on wg1 from 10.67.0.1 to any port 8250 proto tcp. Release on BHS128b: rsync -az --exclude node_modules --exclude .git --exclude 'apps/web/.next' --exclude data --exclude backups --exclude logs . BHS128b:apps/cancerindex/ then ssh BHS128b 'cd apps/cancerindex && pnpm install --frozen-lockfile && DATABASE_URL=postgres://cancerindex:cancerindex@127.0.0.1:5432/cancerindex pnpm db:migrate && NODE_ENV=production NEXT_PUBLIC_SITE_URL=https://www.cancerindex.io CI_API_URL=http://127.0.0.1:8251 DATABASE_URL=postgres://cancerindex:cancerindex@127.0.0.1:5432/cancerindex pnpm --filter @cancerindex/web build && pm2 restart cancerindex-api cancerindex-web cancerindex-worker && pm2 save'. pnpm ≥ 10 needs onlyBuiltDependencies (esbuild, sharp, @tailwindcss/oxide, unrs-resolver) in pnpm-workspace.yaml on a fresh install. The M4M64b copy (code, Postgres cancerindex) is kept as a cold copy, retired from mld (--keep-dir); the mld procedure below still describes it.

Everything goes through the gateway M1M32 and the mld orchestrator (~/Desktop/cluster-skill/mld on the laptop relays to M1M32:~/dispatch). Nothing in this folder runs on the cluster by itself; these are the files and the procedure.

# What gets deployed

Preferred node: M4M64b (Postgres 17 + pgvector, pnpm, redis, 772 GB disk). Avoid M3U96b (hfmarketdata) and M1M32 (gateway). Requires ≈ 6 GB RAM.

# Procedure

1. Manifest — copy deploy/mld-manifest.cancerindex.json to M1M32:~/dispatch/apps/cancerindex.json and replace the {{ADMIN_TOKEN}} placeholder with a real secret (or set it through env_overrides). {{HOME}} is expanded by mld.

   bashCopy

   scp deploy/mld-manifest.cancerindex.json M1M32:~/dispatch/apps/cancerindex.json
   ssh M1M32 "sed -i '' 's/{{ADMIN_TOKEN}}/'\"$(openssl rand -hex 24)\"'/g' ~/dispatch/apps/cancerindex.json"

2. Stage — from a clean export of the repository (no node_modules, no .env, no data lake):

   bashCopy

   cd /path/to/cancerindex
   rm -rf /tmp/cancerindex-stage && mkdir -p /tmp/cancerindex-stage
   git archive HEAD | tar -x -C /tmp/cancerindex-stage
   ~/Desktop/cluster-skill/mld stage /tmp/cancerindex-stage cancerindex

   Source of truth for the code is spbgit cancerindex.git (ssh gitsrv); push there first.

3. Deploy — mld scores the nodes, syncs to ~/apps/cancerindex, runs the post_sync hooks (pnpm install --frozen-lockfile, createdb cancerindex if missing, extensions, db:migrate, db:seed, cix sources:sync, mkdir data/raw data/cache logs, next build), starts the PM2 processes and the ngrok tunnel, and checks /healthz:

   bashCopy

   ~/Desktop/cluster-skill/mld plan cancerindex
   ~/Desktop/cluster-skill/mld deploy cancerindex            # or --node M4M64b
   ~/Desktop/cluster-skill/mld status --live

4. First ingestion — on the node, in order (terminology → genes → evidence/genomics/variants → trials → literature → epidemiology), then counters and rankings:

   bashCopy

   ssh M4M64b 'cd ~/apps/cancerindex && bash deploy/first-run.sh'

   first-run.sh is idempotent: connectors are restartable (cursors), records are hashed, and the script re-runs a connector until its last run is succeeded or the retry budget is spent. Expect ClinicalTrials.gov and PubMed to need several 45-minute windows; the worker's cron picks up where the script stops.

5. Verify

   bashCopy

   curl -fsS http://127.0.0.1:8251/healthz
   curl -fsS http://127.0.0.1:8251/v1/stats | jq .data
   curl -fsS "http://127.0.0.1:8251/v1/sources" | jq '.data[] | {slug, status, health: .connector.health}'
   pm2 ls && pm2 logs cancerindex-worker --lines 50
   curl -fsS https://www.cancerindex.io/healthz

# Day-2 operations

• Redeploy code: push to spbgit, git archive → mld stage → mld deploy cancerindex (hooks rebuild the web app; PM2 restarts). Migrations run in post_sync (pnpm db:migrate).
• Move node: mld move cancerindex --to <node> — the data lake (data/raw) is excluded from sync; copy it separately (rsync -a M4M64b:~/apps/cancerindex/data/raw/ <node>:~/apps/cancerindex/data/raw/) and dump/restore the database (pg_dump -Fc cancerindex).
• Backups (§172): the PM2 process cancerindex-backup runs deploy/backup.sh every day at 05:20 UTC (cron_restart, autorestart: false — PM2 restarts it on the cron only). It writes pg_dump -Fc to ~/apps/cancerindex/backups/cancerindex-YYYYMMDD-HHMM.dump, verifies the archive with pg_restore --list (core tables must be present), keeps the last 14 daily + 8 weekly dumps, logs to logs/backup.log and exits non-zero on any failure (pm2 ls shows it errored; pm2 logs cancerindex-backup). Run it by hand with bash deploy/backup.sh. The raw data lake (data/raw, append-only gzip JSON Lines) is not in the dump — mirror it with rsync -a ~/apps/cancerindex/data/raw/ <nas-or-node>:/…/cancerindex-raw/. Both are needed for TRACE.
• Restore test / disaster recovery: bash deploy/restore.sh backups/<file>.dump creates cancerindex_restore_<timestamp> on the same server (never overwrites cancerindex), restores in parallel (--jobs 4), prints row counts. To promote: stop the PM2 processes, ALTER DATABASE cancerindex RENAME TO cancerindex_old; ALTER DATABASE cancerindex_restore_… RENAME TO cancerindex; (or point DATABASE_URL at the restored database), start the processes, run pnpm cix doctor. Test a restore after every schema migration.
• Readiness / health: pnpm cix doctor (env, database + extensions + pending migrations, table sizes, every connector's health / last success / last run / anomaly / drift / cursor, stale connectors, unresolved labels, data/raw disk usage, ranking freshness, open alerts; exit 1 on a hard failure) and pnpm cix alerts (open alerts; ack <id> / resolve <id>). The API exposes GET /v1/admin/alerts (admin token).
• Stopping a run cleanly: pm2 stop cancerindex-worker sends SIGTERM — the active connector saves its cursor and marks the run aborted; the next scheduled run resumes at the last completed page (docs/connectors/README.md).
• Trigger a run: pnpm cix run <id> on the node, or POST /v1/admin/connectors/<id>/run with x-admin-token, or tsx workers/cli.ts run <id>.
• Pause a connector: POST /v1/admin/connectors/<id>/pause (worker skips scheduled runs).
• Logs: pino JSON via PM2 (pm2 logs cancerindex-api), CI_SERVICE labels api/worker.
• Schedules (UTC, from manifests): civic 02:00 daily, clinicaltrials 02:30 daily, pubmed 03:00 daily, ncit-evs 03:00 on the 1st, oncotree 04:00 Mondays, hgnc/clinvar 05:00 Tuesdays, gdc 06:00 on the 1st; counters 06:00, rank 06:30, health probe hourly at :15. Check with tsx workers/cli.ts schedules.

# Gotchas

• pg-boss creates its own schema pgboss in the same database on first worker start; the API only enqueues (no supervision). Queue policy is stately (one job per connector queued/active).
• mld syncs with the sync_excludes above: .env*, data/raw, .next are never transferred; production env comes from the manifest env blocks.
• The web build needs NEXT_PUBLIC_SITE_URL and CI_API_URL at build time (baked into the bundle); they are set in the post_sync build command.
• M2U64 has no graphical session — not relevant here (PM2 processes), but launchd services there would need LaunchDaemons.
• Never commit real values into the manifest; {{ADMIN_TOKEN}} is a placeholder on purpose.