spb/cancerindex

1# Deploying CancerIndex on the MacLustr cluster23Everything goes through the gateway **M1M32** and the `mld` orchestrator (`~/Desktop/cluster-skill/mld`4on the laptop relays to `M1M32:~/dispatch`). Nothing in this folder runs on the cluster by itself;5these are the files and the procedure.67## What gets deployed89| Piece | Where | Port |10|---|---|---|11| `cancerindex-web` (PM2) | `~/apps/cancerindex/apps/web`, `next start -p 8250 -H 0.0.0.0` | 8250 (public via ngrok) |12| `cancerindex-api` (PM2) | `~/apps/cancerindex`, `tsx apps/api/src/server.ts` | 8251 (127.0.0.1) |13| `cancerindex-worker` (PM2) | `~/apps/cancerindex`, `tsx workers/main.ts` | — |14| `cancerindex-backup` (PM2, cron `20 5 * * *`, `autorestart: false`) | `~/apps/cancerindex`, `/bin/bash deploy/backup.sh` | — |15| `cancerindex-ngrok` | `www.cancerindex.io` → 8250 (reserved domain) | — |16| PostgreSQL 17 | database `cancerindex` on the node (extensions `pg_trgm`, `unaccent`, `vector`) | 5432 |1718Preferred node: **M4M64b** (Postgres 17 + pgvector, pnpm, redis, 772 GB disk). Avoid M3U96b19(hfmarketdata) and M1M32 (gateway). Requires ≈ 6 GB RAM.2021## Procedure22231. **Manifest** — copy `deploy/mld-manifest.cancerindex.json` to `M1M32:~/dispatch/apps/cancerindex.json`24   and replace the `{{ADMIN_TOKEN}}` placeholder with a real secret (or set it through25   `env_overrides`). `{{HOME}}` is expanded by `mld`.2627   ```bash28   scp deploy/mld-manifest.cancerindex.json M1M32:~/dispatch/apps/cancerindex.json29   ssh M1M32 "sed -i '' 's/{{ADMIN_TOKEN}}/'\"$(openssl rand -hex 24)\"'/g' ~/dispatch/apps/cancerindex.json"30   ```31322. **Stage** — from a clean export of the repository (no `node_modules`, no `.env`, no data lake):3334   ```bash35   cd /path/to/cancerindex36   rm -rf /tmp/cancerindex-stage && mkdir -p /tmp/cancerindex-stage37   git archive HEAD | tar -x -C /tmp/cancerindex-stage38   ~/Desktop/cluster-skill/mld stage /tmp/cancerindex-stage cancerindex39   ```4041   Source of truth for the code is spbgit `cancerindex.git` (`ssh gitsrv`); push there first.42433. **Deploy** — `mld` scores the nodes, syncs to `~/apps/cancerindex`, runs the `post_sync` hooks44   (`pnpm install --frozen-lockfile`, `createdb cancerindex` if missing, extensions, `db:migrate`,45   `db:seed`, `cix sources:sync`, `mkdir data/raw data/cache logs`, `next build`), starts the PM246   processes and the ngrok tunnel, and checks `/healthz`:4748   ```bash49   ~/Desktop/cluster-skill/mld plan cancerindex50   ~/Desktop/cluster-skill/mld deploy cancerindex            # or --node M4M64b51   ~/Desktop/cluster-skill/mld status --live52   ```53544. **First ingestion** — on the node, in order (terminology → genes → evidence/genomics/variants →55   trials → literature → epidemiology), then counters and rankings:5657   ```bash58   ssh M4M64b 'cd ~/apps/cancerindex && bash deploy/first-run.sh'59   ```6061   `first-run.sh` is idempotent: connectors are restartable (cursors), records are hashed, and the62   script re-runs a connector until its last run is `succeeded` or the retry budget is spent.63   Expect ClinicalTrials.gov and PubMed to need several 45-minute windows; the worker's cron picks64   up where the script stops.65665. **Verify**6768   ```bash69   curl -fsS http://127.0.0.1:8251/healthz70   curl -fsS http://127.0.0.1:8251/v1/stats | jq .data71   curl -fsS "http://127.0.0.1:8251/v1/sources" | jq '.data[] | {slug, status, health: .connector.health}'72   pm2 ls && pm2 logs cancerindex-worker --lines 5073   curl -fsS https://www.cancerindex.io/healthz74   ```7576## Day-2 operations7778- **Redeploy code**: push to spbgit, `git archive` → `mld stage` → `mld deploy cancerindex` (hooks79  rebuild the web app; PM2 restarts). Migrations run in `post_sync` (`pnpm db:migrate`).80- **Move node**: `mld move cancerindex --to <node>` — the data lake (`data/raw`) is excluded from sync;81  copy it separately (`rsync -a M4M64b:~/apps/cancerindex/data/raw/ <node>:~/apps/cancerindex/data/raw/`)82  and dump/restore the database (`pg_dump -Fc cancerindex`).83- **Backups** (§172): the PM2 process `cancerindex-backup` runs `deploy/backup.sh` every day at84  05:20 UTC (`cron_restart`, `autorestart: false` — PM2 restarts it on the cron only). It writes85  `pg_dump -Fc` to `~/apps/cancerindex/backups/cancerindex-YYYYMMDD-HHMM.dump`, verifies the archive86  with `pg_restore --list` (core tables must be present), keeps the last **14 daily + 8 weekly**87  dumps, logs to `logs/backup.log` and exits non-zero on any failure (`pm2 ls` shows it `errored`;88  `pm2 logs cancerindex-backup`). Run it by hand with `bash deploy/backup.sh`.89  The raw data lake (`data/raw`, append-only gzip JSON Lines) is **not** in the dump — mirror it with90  `rsync -a ~/apps/cancerindex/data/raw/ <nas-or-node>:/…/cancerindex-raw/`. Both are needed for TRACE.91- **Restore test / disaster recovery**: `bash deploy/restore.sh backups/<file>.dump` creates92  `cancerindex_restore_<timestamp>` on the same server (never overwrites `cancerindex`), restores in93  parallel (`--jobs 4`), prints row counts. To promote: stop the PM2 processes, `ALTER DATABASE94  cancerindex RENAME TO cancerindex_old; ALTER DATABASE cancerindex_restore_… RENAME TO cancerindex;`95  (or point `DATABASE_URL` at the restored database), start the processes, run `pnpm cix doctor`.96  Test a restore after every schema migration.97- **Readiness / health**: `pnpm cix doctor` (env, database + extensions + pending migrations, table98  sizes, every connector's health / last success / last run / anomaly / drift / cursor, stale99  connectors, unresolved labels, `data/raw` disk usage, ranking freshness, open alerts; exit 1 on a100  hard failure) and `pnpm cix alerts` (open alerts; `ack <id>` / `resolve <id>`). The API exposes101  `GET /v1/admin/alerts` (admin token).102- **Stopping a run cleanly**: `pm2 stop cancerindex-worker` sends SIGTERM — the active connector103  saves its cursor and marks the run `aborted`; the next scheduled run resumes at the last completed104  page (docs/connectors/README.md).105- **Trigger a run**: `pnpm cix run <id>` on the node, or `POST /v1/admin/connectors/<id>/run` with106  `x-admin-token`, or `tsx workers/cli.ts run <id>`.107- **Pause a connector**: `POST /v1/admin/connectors/<id>/pause` (worker skips scheduled runs).108- **Logs**: pino JSON via PM2 (`pm2 logs cancerindex-api`), `CI_SERVICE` labels api/worker.109- **Schedules** (UTC, from manifests): civic 02:00 daily, clinicaltrials 02:30 daily, pubmed 03:00110  daily, ncit-evs 03:00 on the 1st, oncotree 04:00 Mondays, hgnc/clinvar 05:00 Tuesdays, gdc 06:00111  on the 1st; counters 06:00, rank 06:30, health probe hourly at :15. Check with112  `tsx workers/cli.ts schedules`.113114## Gotchas115116- `pg-boss` creates its own schema `pgboss` in the same database on first worker start; the API only117  enqueues (no supervision). Queue policy is `stately` (one job per connector queued/active).118- `mld` syncs with the `sync_excludes` above: `.env*`, `data/raw`, `.next` are never transferred;119  production env comes from the manifest `env` blocks.120- The web build needs `NEXT_PUBLIC_SITE_URL` and `CI_API_URL` at build time (baked into the bundle);121  they are set in the `post_sync` build command.122- M2U64 has no graphical session — not relevant here (PM2 processes), but launchd services there123  would need LaunchDaemons.124- Never commit real values into the manifest; `{{ADMIN_TOKEN}}` is a placeholder on purpose.125