# Deploying CancerIndex on the MacLustr cluster

Everything goes through the gateway **M1M32** and the `mld` orchestrator (`~/Desktop/cluster-skill/mld`
on the laptop relays to `M1M32:~/dispatch`). Nothing in this folder runs on the cluster by itself;
these are the files and the procedure.

## What gets deployed

| Piece | Where | Port |
|---|---|---|
| `cancerindex-web` (PM2) | `~/apps/cancerindex/apps/web`, `next start -p 8250 -H 0.0.0.0` | 8250 (public via ngrok) |
| `cancerindex-api` (PM2) | `~/apps/cancerindex`, `tsx apps/api/src/server.ts` | 8251 (127.0.0.1) |
| `cancerindex-worker` (PM2) | `~/apps/cancerindex`, `tsx workers/main.ts` | — |
| `cancerindex-ngrok` | `www.cancerindex.io` → 8250 (reserved domain) | — |
| PostgreSQL 17 | database `cancerindex` on the node (extensions `pg_trgm`, `unaccent`, `vector`) | 5432 |

Preferred node: **M4M64b** (Postgres 17 + pgvector, pnpm, redis, 772 GB disk). Avoid M3U96b
(hfmarketdata) and M1M32 (gateway). Requires ≈ 6 GB RAM.

## Procedure

1. **Manifest** — copy `deploy/mld-manifest.cancerindex.json` to `M1M32:~/dispatch/apps/cancerindex.json`
   and replace the `{{ADMIN_TOKEN}}` placeholder with a real secret (or set it through
   `env_overrides`). `{{HOME}}` is expanded by `mld`.

   ```bash
   scp deploy/mld-manifest.cancerindex.json M1M32:~/dispatch/apps/cancerindex.json
   ssh M1M32 "sed -i '' 's/{{ADMIN_TOKEN}}/'\"$(openssl rand -hex 24)\"'/g' ~/dispatch/apps/cancerindex.json"
   ```

2. **Stage** — from a clean export of the repository (no `node_modules`, no `.env`, no data lake):

   ```bash
   cd /path/to/cancerindex
   rm -rf /tmp/cancerindex-stage && mkdir -p /tmp/cancerindex-stage
   git archive HEAD | tar -x -C /tmp/cancerindex-stage
   ~/Desktop/cluster-skill/mld stage /tmp/cancerindex-stage cancerindex
   ```

   Source of truth for the code is spbgit `cancerindex.git` (`ssh gitsrv`); push there first.

3. **Deploy** — `mld` scores the nodes, syncs to `~/apps/cancerindex`, runs the `post_sync` hooks
   (`pnpm install --frozen-lockfile`, `createdb cancerindex` if missing, extensions, `db:migrate`,
   `db:seed`, `cix sources:sync`, `mkdir data/raw data/cache logs`, `next build`), starts the PM2
   processes and the ngrok tunnel, and checks `/healthz`:

   ```bash
   ~/Desktop/cluster-skill/mld plan cancerindex
   ~/Desktop/cluster-skill/mld deploy cancerindex            # or --node M4M64b
   ~/Desktop/cluster-skill/mld status --live
   ```

4. **First ingestion** — on the node, in order (terminology → genes → evidence/genomics/variants →
   trials → literature → epidemiology), then counters and rankings:

   ```bash
   ssh M4M64b 'cd ~/apps/cancerindex && bash deploy/first-run.sh'
   ```

   `first-run.sh` is idempotent: connectors are restartable (cursors), records are hashed, and the
   script re-runs a connector until its last run is `succeeded` or the retry budget is spent.
   Expect ClinicalTrials.gov and PubMed to need several 45-minute windows; the worker's cron picks
   up where the script stops.

5. **Verify**

   ```bash
   curl -fsS http://127.0.0.1:8251/healthz
   curl -fsS http://127.0.0.1:8251/v1/stats | jq .data
   curl -fsS "http://127.0.0.1:8251/v1/sources" | jq '.data[] | {slug, status, health: .connector.health}'
   pm2 ls && pm2 logs cancerindex-worker --lines 50
   curl -fsS https://www.cancerindex.io/healthz
   ```

## Day-2 operations

- **Redeploy code**: push to spbgit, `git archive` → `mld stage` → `mld deploy cancerindex` (hooks
  rebuild the web app; PM2 restarts). Migrations run in `post_sync` (`pnpm db:migrate`).
- **Move node**: `mld move cancerindex --to <node>` — the data lake (`data/raw`) is excluded from sync;
  copy it separately (`rsync -a M4M64b:~/apps/cancerindex/data/raw/ <node>:~/apps/cancerindex/data/raw/`)
  and dump/restore the database (`pg_dump -Fc cancerindex`).
- **Backups**: `pg_dump -Fc cancerindex` + `data/raw` (append-only). Both are needed for TRACE.
- **Trigger a run**: `pnpm cix run <id>` on the node, or `POST /v1/admin/connectors/<id>/run` with
  `x-admin-token`, or `tsx workers/cli.ts run <id>`.
- **Pause a connector**: `POST /v1/admin/connectors/<id>/pause` (worker skips scheduled runs).
- **Logs**: pino JSON via PM2 (`pm2 logs cancerindex-api`), `CI_SERVICE` labels api/worker.
- **Schedules** (UTC, from manifests): civic 02:00 daily, clinicaltrials 02:30 daily, pubmed 03:00
  daily, ncit-evs 03:00 on the 1st, oncotree 04:00 Mondays, hgnc/clinvar 05:00 Tuesdays, gdc 06:00
  on the 1st; counters 06:00, rank 06:30, health probe hourly at :15. Check with
  `tsx workers/cli.ts schedules`.

## Gotchas

- `pg-boss` creates its own schema `pgboss` in the same database on first worker start; the API only
  enqueues (no supervision). Queue policy is `stately` (one job per connector queued/active).
- `mld` syncs with the `sync_excludes` above: `.env*`, `data/raw`, `.next` are never transferred;
  production env comes from the manifest `env` blocks.
- The web build needs `NEXT_PUBLIC_SITE_URL` and `CI_API_URL` at build time (baked into the bundle);
  they are set in the `post_sync` build command.
- M2U64 has no graphical session — not relevant here (PM2 processes), but launchd services there
  would need LaunchDaemons.
- Never commit real values into the manifest; `{{ADMIN_TOKEN}}` is a placeholder on purpose.