spb/cancerindex

1# Deploying CancerIndex on the MacLustr cluster23> **Since 2026-09-14 production runs on the OVH server BHS128b** (`ssh BHS128b`, ubuntu@51.161.112.85, Ubuntu 24.04, 16 threads / 128 GB),4> outside `mld`: code in `~/apps/cancerindex`, Postgres 17 native (db/role `cancerindex`/`cancerindex`, pgvector + pg_trgm + unaccent, en_US.UTF-8),5> raw lake + backups under `~/apps/cancerindex/{data,backups}`, PM2 under systemd (`pm2-ubuntu`) with `~/apps/pm2.cancerindex.config.cjs` generated6> from the mld manifest by `~/apps/.manifests/gen-pm2.py` (secrets, 0600). Tunnel route: `mlt add www.cancerindex.io BHS128b:8250`7> (BHS64 Caddy → wg1 10.67.0.62); firewall `ufw allow in on wg1 from 10.67.0.1 to any port 8250 proto tcp`.8> **Release on BHS128b**: `rsync -az --exclude node_modules --exclude .git --exclude 'apps/web/.next' --exclude data --exclude backups --exclude logs . BHS128b:apps/cancerindex/`9> then `ssh BHS128b 'cd apps/cancerindex && pnpm install --frozen-lockfile && DATABASE_URL=postgres://cancerindex:cancerindex@127.0.0.1:5432/cancerindex pnpm db:migrate && NODE_ENV=production NEXT_PUBLIC_SITE_URL=https://www.cancerindex.io CI_API_URL=http://127.0.0.1:8251 DATABASE_URL=postgres://cancerindex:cancerindex@127.0.0.1:5432/cancerindex pnpm --filter @cancerindex/web build && pm2 restart cancerindex-api cancerindex-web cancerindex-worker && pm2 save'`.10> pnpm ≥ 10 needs `onlyBuiltDependencies` (esbuild, sharp, @tailwindcss/oxide, unrs-resolver) in `pnpm-workspace.yaml` on a fresh install.11> The M4M64b copy (code, Postgres `cancerindex`) is kept as a cold copy, retired from `mld` (`--keep-dir`); the `mld` procedure below still describes it.1213Everything goes through the gateway **M1M32** and the `mld` orchestrator (`~/Desktop/cluster-skill/mld`14on the laptop relays to `M1M32:~/dispatch`). Nothing in this folder runs on the cluster by itself;15these are the files and the procedure.1617## What gets deployed1819| Piece | Where | Port |20|---|---|---|21| `cancerindex-web` (PM2) | `~/apps/cancerindex/apps/web`, `next start -p 8250 -H 0.0.0.0` | 8250 (public via ngrok) |22| `cancerindex-api` (PM2) | `~/apps/cancerindex`, `tsx apps/api/src/server.ts` | 8251 (127.0.0.1) |23| `cancerindex-worker` (PM2) | `~/apps/cancerindex`, `tsx workers/main.ts` | — |24| `cancerindex-backup` (PM2, cron `20 5 * * *`, `autorestart: false`) | `~/apps/cancerindex`, `/bin/bash deploy/backup.sh` | — |25| `cancerindex-ngrok` | `www.cancerindex.io` → 8250 (reserved domain) | — |26| PostgreSQL 17 | database `cancerindex` on the node (extensions `pg_trgm`, `unaccent`, `vector`) | 5432 |2728Preferred node: **M4M64b** (Postgres 17 + pgvector, pnpm, redis, 772 GB disk). Avoid M3U96b29(hfmarketdata) and M1M32 (gateway). Requires ≈ 6 GB RAM.3031## Procedure32331. **Manifest** — copy `deploy/mld-manifest.cancerindex.json` to `M1M32:~/dispatch/apps/cancerindex.json`34   and replace the `{{ADMIN_TOKEN}}` placeholder with a real secret (or set it through35   `env_overrides`). `{{HOME}}` is expanded by `mld`.3637   ```bash38   scp deploy/mld-manifest.cancerindex.json M1M32:~/dispatch/apps/cancerindex.json39   ssh M1M32 "sed -i '' 's/{{ADMIN_TOKEN}}/'\"$(openssl rand -hex 24)\"'/g' ~/dispatch/apps/cancerindex.json"40   ```41422. **Stage** — from a clean export of the repository (no `node_modules`, no `.env`, no data lake):4344   ```bash45   cd /path/to/cancerindex46   rm -rf /tmp/cancerindex-stage && mkdir -p /tmp/cancerindex-stage47   git archive HEAD | tar -x -C /tmp/cancerindex-stage48   ~/Desktop/cluster-skill/mld stage /tmp/cancerindex-stage cancerindex49   ```5051   Source of truth for the code is spbgit `cancerindex.git` (`ssh gitsrv`); push there first.52533. **Deploy** — `mld` scores the nodes, syncs to `~/apps/cancerindex`, runs the `post_sync` hooks54   (`pnpm install --frozen-lockfile`, `createdb cancerindex` if missing, extensions, `db:migrate`,55   `db:seed`, `cix sources:sync`, `mkdir data/raw data/cache logs`, `next build`), starts the PM256   processes and the ngrok tunnel, and checks `/healthz`:5758   ```bash59   ~/Desktop/cluster-skill/mld plan cancerindex60   ~/Desktop/cluster-skill/mld deploy cancerindex            # or --node M4M64b61   ~/Desktop/cluster-skill/mld status --live62   ```63644. **First ingestion** — on the node, in order (terminology → genes → evidence/genomics/variants →65   trials → literature → epidemiology), then counters and rankings:6667   ```bash68   ssh M4M64b 'cd ~/apps/cancerindex && bash deploy/first-run.sh'69   ```7071   `first-run.sh` is idempotent: connectors are restartable (cursors), records are hashed, and the72   script re-runs a connector until its last run is `succeeded` or the retry budget is spent.73   Expect ClinicalTrials.gov and PubMed to need several 45-minute windows; the worker's cron picks74   up where the script stops.75765. **Verify**7778   ```bash79   curl -fsS http://127.0.0.1:8251/healthz80   curl -fsS http://127.0.0.1:8251/v1/stats | jq .data81   curl -fsS "http://127.0.0.1:8251/v1/sources" | jq '.data[] | {slug, status, health: .connector.health}'82   pm2 ls && pm2 logs cancerindex-worker --lines 5083   curl -fsS https://www.cancerindex.io/healthz84   ```8586## Day-2 operations8788- **Redeploy code**: push to spbgit, `git archive` → `mld stage` → `mld deploy cancerindex` (hooks89  rebuild the web app; PM2 restarts). Migrations run in `post_sync` (`pnpm db:migrate`).90- **Move node**: `mld move cancerindex --to <node>` — the data lake (`data/raw`) is excluded from sync;91  copy it separately (`rsync -a M4M64b:~/apps/cancerindex/data/raw/ <node>:~/apps/cancerindex/data/raw/`)92  and dump/restore the database (`pg_dump -Fc cancerindex`).93- **Backups** (§172): the PM2 process `cancerindex-backup` runs `deploy/backup.sh` every day at94  05:20 UTC (`cron_restart`, `autorestart: false` — PM2 restarts it on the cron only). It writes95  `pg_dump -Fc` to `~/apps/cancerindex/backups/cancerindex-YYYYMMDD-HHMM.dump`, verifies the archive96  with `pg_restore --list` (core tables must be present), keeps the last **14 daily + 8 weekly**97  dumps, logs to `logs/backup.log` and exits non-zero on any failure (`pm2 ls` shows it `errored`;98  `pm2 logs cancerindex-backup`). Run it by hand with `bash deploy/backup.sh`.99  The raw data lake (`data/raw`, append-only gzip JSON Lines) is **not** in the dump — mirror it with100  `rsync -a ~/apps/cancerindex/data/raw/ <nas-or-node>:/…/cancerindex-raw/`. Both are needed for TRACE.101- **Restore test / disaster recovery**: `bash deploy/restore.sh backups/<file>.dump` creates102  `cancerindex_restore_<timestamp>` on the same server (never overwrites `cancerindex`), restores in103  parallel (`--jobs 4`), prints row counts. To promote: stop the PM2 processes, `ALTER DATABASE104  cancerindex RENAME TO cancerindex_old; ALTER DATABASE cancerindex_restore_… RENAME TO cancerindex;`105  (or point `DATABASE_URL` at the restored database), start the processes, run `pnpm cix doctor`.106  Test a restore after every schema migration.107- **Readiness / health**: `pnpm cix doctor` (env, database + extensions + pending migrations, table108  sizes, every connector's health / last success / last run / anomaly / drift / cursor, stale109  connectors, unresolved labels, `data/raw` disk usage, ranking freshness, open alerts; exit 1 on a110  hard failure) and `pnpm cix alerts` (open alerts; `ack <id>` / `resolve <id>`). The API exposes111  `GET /v1/admin/alerts` (admin token).112- **Stopping a run cleanly**: `pm2 stop cancerindex-worker` sends SIGTERM — the active connector113  saves its cursor and marks the run `aborted`; the next scheduled run resumes at the last completed114  page (docs/connectors/README.md).115- **Trigger a run**: `pnpm cix run <id>` on the node, or `POST /v1/admin/connectors/<id>/run` with116  `x-admin-token`, or `tsx workers/cli.ts run <id>`.117- **Pause a connector**: `POST /v1/admin/connectors/<id>/pause` (worker skips scheduled runs).118- **Logs**: pino JSON via PM2 (`pm2 logs cancerindex-api`), `CI_SERVICE` labels api/worker.119- **Schedules** (UTC, from manifests): civic 02:00 daily, clinicaltrials 02:30 daily, pubmed 03:00120  daily, ncit-evs 03:00 on the 1st, oncotree 04:00 Mondays, hgnc/clinvar 05:00 Tuesdays, gdc 06:00121  on the 1st; counters 06:00, rank 06:30, health probe hourly at :15. Check with122  `tsx workers/cli.ts schedules`.123124## Gotchas125126- `pg-boss` creates its own schema `pgboss` in the same database on first worker start; the API only127  enqueues (no supervision). Queue policy is `stately` (one job per connector queued/active).128- `mld` syncs with the `sync_excludes` above: `.env*`, `data/raw`, `.next` are never transferred;129  production env comes from the manifest `env` blocks.130- The web build needs `NEXT_PUBLIC_SITE_URL` and `CI_API_URL` at build time (baked into the bundle);131  they are set in the `post_sync` build command.132- M2U64 has no graphical session — not relevant here (PM2 processes), but launchd services there133  would need LaunchDaemons.134- Never commit real values into the manifest; `{{ADMIN_TOKEN}}` is a placeholder on purpose.135