spb/cancerindex

1# Deploying CancerIndex on the MacLustr cluster23Everything goes through the gateway **M1M32** and the `mld` orchestrator (`~/Desktop/cluster-skill/mld`4on the laptop relays to `M1M32:~/dispatch`). Nothing in this folder runs on the cluster by itself;5these are the files and the procedure.67## What gets deployed89| Piece | Where | Port |10|---|---|---|11| `cancerindex-web` (PM2) | `~/apps/cancerindex/apps/web`, `next start -p 8250 -H 0.0.0.0` | 8250 (public via ngrok) |12| `cancerindex-api` (PM2) | `~/apps/cancerindex`, `tsx apps/api/src/server.ts` | 8251 (127.0.0.1) |13| `cancerindex-worker` (PM2) | `~/apps/cancerindex`, `tsx workers/main.ts` | — |14| `cancerindex-ngrok` | `www.cancerindex.io` → 8250 (reserved domain) | — |15| PostgreSQL 17 | database `cancerindex` on the node (extensions `pg_trgm`, `unaccent`, `vector`) | 5432 |1617Preferred node: **M4M64b** (Postgres 17 + pgvector, pnpm, redis, 772 GB disk). Avoid M3U96b18(hfmarketdata) and M1M32 (gateway). Requires ≈ 6 GB RAM.1920## Procedure21221. **Manifest** — copy `deploy/mld-manifest.cancerindex.json` to `M1M32:~/dispatch/apps/cancerindex.json`23   and replace the `{{ADMIN_TOKEN}}` placeholder with a real secret (or set it through24   `env_overrides`). `{{HOME}}` is expanded by `mld`.2526   ```bash27   scp deploy/mld-manifest.cancerindex.json M1M32:~/dispatch/apps/cancerindex.json28   ssh M1M32 "sed -i '' 's/{{ADMIN_TOKEN}}/'\"$(openssl rand -hex 24)\"'/g' ~/dispatch/apps/cancerindex.json"29   ```30312. **Stage** — from a clean export of the repository (no `node_modules`, no `.env`, no data lake):3233   ```bash34   cd /path/to/cancerindex35   rm -rf /tmp/cancerindex-stage && mkdir -p /tmp/cancerindex-stage36   git archive HEAD | tar -x -C /tmp/cancerindex-stage37   ~/Desktop/cluster-skill/mld stage /tmp/cancerindex-stage cancerindex38   ```3940   Source of truth for the code is spbgit `cancerindex.git` (`ssh gitsrv`); push there first.41423. **Deploy** — `mld` scores the nodes, syncs to `~/apps/cancerindex`, runs the `post_sync` hooks43   (`pnpm install --frozen-lockfile`, `createdb cancerindex` if missing, extensions, `db:migrate`,44   `db:seed`, `cix sources:sync`, `mkdir data/raw data/cache logs`, `next build`), starts the PM245   processes and the ngrok tunnel, and checks `/healthz`:4647   ```bash48   ~/Desktop/cluster-skill/mld plan cancerindex49   ~/Desktop/cluster-skill/mld deploy cancerindex            # or --node M4M64b50   ~/Desktop/cluster-skill/mld status --live51   ```52534. **First ingestion** — on the node, in order (terminology → genes → evidence/genomics/variants →54   trials → literature → epidemiology), then counters and rankings:5556   ```bash57   ssh M4M64b 'cd ~/apps/cancerindex && bash deploy/first-run.sh'58   ```5960   `first-run.sh` is idempotent: connectors are restartable (cursors), records are hashed, and the61   script re-runs a connector until its last run is `succeeded` or the retry budget is spent.62   Expect ClinicalTrials.gov and PubMed to need several 45-minute windows; the worker's cron picks63   up where the script stops.64655. **Verify**6667   ```bash68   curl -fsS http://127.0.0.1:8251/healthz69   curl -fsS http://127.0.0.1:8251/v1/stats | jq .data70   curl -fsS "http://127.0.0.1:8251/v1/sources" | jq '.data[] | {slug, status, health: .connector.health}'71   pm2 ls && pm2 logs cancerindex-worker --lines 5072   curl -fsS https://www.cancerindex.io/healthz73   ```7475## Day-2 operations7677- **Redeploy code**: push to spbgit, `git archive` → `mld stage` → `mld deploy cancerindex` (hooks78  rebuild the web app; PM2 restarts). Migrations run in `post_sync` (`pnpm db:migrate`).79- **Move node**: `mld move cancerindex --to <node>` — the data lake (`data/raw`) is excluded from sync;80  copy it separately (`rsync -a M4M64b:~/apps/cancerindex/data/raw/ <node>:~/apps/cancerindex/data/raw/`)81  and dump/restore the database (`pg_dump -Fc cancerindex`).82- **Backups**: `pg_dump -Fc cancerindex` + `data/raw` (append-only). Both are needed for TRACE.83- **Trigger a run**: `pnpm cix run <id>` on the node, or `POST /v1/admin/connectors/<id>/run` with84  `x-admin-token`, or `tsx workers/cli.ts run <id>`.85- **Pause a connector**: `POST /v1/admin/connectors/<id>/pause` (worker skips scheduled runs).86- **Logs**: pino JSON via PM2 (`pm2 logs cancerindex-api`), `CI_SERVICE` labels api/worker.87- **Schedules** (UTC, from manifests): civic 02:00 daily, clinicaltrials 02:30 daily, pubmed 03:0088  daily, ncit-evs 03:00 on the 1st, oncotree 04:00 Mondays, hgnc/clinvar 05:00 Tuesdays, gdc 06:0089  on the 1st; counters 06:00, rank 06:30, health probe hourly at :15. Check with90  `tsx workers/cli.ts schedules`.9192## Gotchas9394- `pg-boss` creates its own schema `pgboss` in the same database on first worker start; the API only95  enqueues (no supervision). Queue policy is `stately` (one job per connector queued/active).96- `mld` syncs with the `sync_excludes` above: `.env*`, `data/raw`, `.next` are never transferred;97  production env comes from the manifest `env` blocks.98- The web build needs `NEXT_PUBLIC_SITE_URL` and `CI_API_URL` at build time (baked into the bundle);99  they are set in the `post_sync` build command.100- M2U64 has no graphical session — not relevant here (PM2 processes), but launchd services there101  would need LaunchDaemons.102- Never commit real values into the manifest; `{{ADMIN_TOKEN}}` is a placeholder on purpose.103