# CancerIndex — Architecture

CancerIndex.io is a provenance-first oncology knowledge platform: a canonical cancer ontology
connected to epidemiology, genomics, biomarkers, therapies, clinical trials, regulatory status and
literature, with transparent, reproducible rankings. This document describes how the system is
layered, which packages implement each layer, how data flows, and how the platform is deployed.
The condensed rules every contributor follows are in `CLAUDE.md`; the full specification is
`docs/SPEC-original.md`.

## 1. Layers

The platform is organised as six separable layers (CLAUDE.md §2). Each layer only reads from the
layer below it and writes to its own tables; nothing is ever overwritten across layers.

| Layer | What lives here | Tables (examples) | Written by |
|---|---|---|---|
| **RAW** | Exact upstream payloads, gzip JSON Lines in the data lake, plus a hash per record | `source_records`, files under `data/raw/{source}/{date}/{entity}/{runId}-{part}.jsonl.gz` | connectors (`ctx.upsertSourceRecord`) |
| **NORMALIZED** | Source-native structures kept intact but typed and validated (CIViC evidence items, ClinVar interpretations, trial conditions before mapping) | `civic_evidence_items`, `variant_clinical_significance`, `trial_conditions`, `trial_interventions`, `connector_field_stats` | connectors |
| **CANONICAL** | One row per real-world concept with stable `CI-*` identifiers, aliases, cross-reference codes, multi-dimensional hierarchy | `cancers`, `cancer_aliases`, `cancer_codes`, `cancer_hierarchy`, `genes`, `variants`, `drugs`, `clinical_trials`, `publications`, `geographies`, `anatomical_sites` | connectors through `CancerResolver` reconciliation |
| **DERIVED** | Numbers computed from canonical data, always with provenance and a formula version | `epidemiology_observations`, `survival_observations`, `cancer_gene_frequencies`, `literature_counts`, `entity_counters`, `knowledge_edges`, `trial_pulse` | connectors (observations) and `@cancerindex/ranking` (`refreshCounters`) |
| **RANKED** | Ranking snapshots per metric × scope × formula version with per-row lineage | `metric_definitions`, `ranking_snapshots`, `rankings` | `@cancerindex/ranking` (`computeAllRankings`) |
| **AI** | Cached, database-grounded syntheses with model and prompt version (not implemented in Phase 1, see `docs/AI.md`) | `ai_answers`, `entity_embeddings` | — |

Cross-cutting tables: `sources` (registry + license status), `ingest_runs`, `connector_cursors`,
`provenance`, `unresolved_labels`, `change_events`, `audit_log`, `entity_merges`, `api_keys`.

Two invariants hold across every layer:

1. **Provenance first.** No scientific number exists without a `provenance` row (source, dataset,
   version, retrieval time, URL, evidence type, license). Derived values also carry
   `formula_version` and their inputs. `traceValue()` in `@cancerindex/ranking` walks a ranked value
   back to the observation, the provenance row and the raw record.
2. **Never fake data.** Missing data is absent, never zero. The API returns `counters: null` when
   counters have not been computed, and a ranking without a snapshot returns `status: "not_available"`.

## 2. Repository and packages

```
cancerindex/
├── apps/
│   ├── web/                 Next.js 16 (webpack), Tailwind v4, server components; proxies /api/v1/* → API
│   └── api/                 Fastify 5 public API (/v1), OpenAPI from zod route schemas, rate limiting, admin
├── workers/                 pg-boss scheduler (connector cron, counters, rankings, health probes)
├── packages/
│   ├── shared/              ids (CI-*), provenance types, normalization, pino logger, env helpers
│   ├── database/            Drizzle schema (snake_case), migrations, seed (metrics + geographies)
│   ├── ontology/            qualifier rules, CancerResolver, TOP_LEVEL_CANCERS, NCIt roots
│   ├── connectors/          SDK (manifest, HttpClient, RawLake, RunContext, validators) + connectors/<id>/
│   └── ranking/             counters, ranking engine (snapshots + lineage), trace
├── scripts/ci.ts            operator CLI (`pnpm cix …`)
├── deploy/                  mld manifest, first-run bootstrap, deployment notes
├── docs/                    this folder
└── data/raw                 data lake (gitignored)
```

Dependency direction (no cycles): `shared` ← `database` ← `ontology` ← `connectors` ← `ranking`;
`apps/api` depends on `database`, `shared`, `ranking`, `ontology`; `workers` additionally depends
on `connectors`; `apps/web` never imports `connectors` (CLAUDE.md conventions).

### Connector SDK

A connector is a class extending `Connector` with a validated `manifest` (zod, see
`packages/connectors/src/sdk/manifest.ts`), a `healthCheck(ctx)` and a `sync(ctx)`. The `RunContext`
provides a rate-limited `HttpClient` (token bucket, bounded concurrency, exponential backoff,
`Retry-After`), a `RawLake` writer, idempotent `upsertSourceRecord` (sha256 payload hash),
`addProvenance`, `recordUnresolved` (curation queue), `recordChange`, a restartable `cursor`, and a
time budget checked with `shouldStop()`. Schema drift is detected by recording observed field
names/types per entity (`connector_field_stats`). `runConnector()` handles bookkeeping in
`ingest_runs` / `connector_cursors`, the credentials gate (`awaiting_credentials`) and the license
gate (`licenseStatus: blocked` never ingests).

### Reconciliation

Labels are mapped to canonical cancers by `CancerResolver`: identifiers first (NCIt, DOID, OncoTree,
UMLS, MeSH…), then curated aliases, then normalized strings. Every mapping stores a `match_type`
(`EXACT_IDENTIFIER`, `CURATED_EXACT`, `ONTOLOGY_EXACT`, `CURATED_BROADER`, `CURATED_NARROWER`,
`ALIAS`, `PROBABILISTIC`, `UNRESOLVED`). Unknown labels are stored in `unresolved_labels` with an
optional trigram suggestion and are resolved by a curator through the admin API, which adds a curated
alias so the next connector run picks it up.

## 3. Data flow

```
 upstream source ──HTTP──▶ connector.sync(ctx)
        │                     │  upsertSourceRecord → source_records + data/raw/*.jsonl.gz   (RAW)
        │                     │  validate / typed rows                                         (NORMALIZED)
        │                     │  CancerResolver → canonical ids, match_type; misses → unresolved_labels
        │                     │  addProvenance → provenance; observations / evidence / edges   (CANONICAL, DERIVED)
        │                     └  recordChange → change_events ; ingest_runs finalised
        ▼
 workers: maintenance.counters → refreshCounters() → entity_counters                            (DERIVED)
          maintenance.rank     → computeAllRankings() → ranking_snapshots + rankings            (RANKED)
        ▼
 apps/api (/v1) ── envelope { data, sources, dataRelease, generatedAt } ──▶ apps/web, API consumers
```

Runs are scheduled by the worker from each manifest's `schedule` (cron, UTC) and can be triggered
by an operator (`pnpm cix run <id>` or `POST /v1/admin/connectors/:id/run`). A `connector.run` job is
`stately` per connector id (never queued twice, never concurrent). When a run creates or updates
records, the worker enqueues `maintenance.counters` with `thenRank: true`, so derived and ranked
layers follow canonical changes. A daily counters (06:00 UTC) and rank (06:30 UTC) pass covers
everything else; an hourly `health.probe` refreshes `connector_cursors.health`.

## 4. Public API

`apps/api` is a read-only Fastify 5 service. Route schemas are written in zod (via
`fastify-type-provider-zod`) and the OpenAPI 3.1 document is generated from them (`/v1/openapi.json`,
Swagger UI at `/v1/docs`) — documentation is never maintained by hand (CLAUDE.md §357).

Every `/v1` response uses the envelope `{ data, sources, dataRelease, generatedAt, …pagination }`.
`sources` lists the distinct upstream sources behind the payload with license and attribution;
`dataRelease` is `CancerIndex <YYYY-MM>` derived from the latest successful ingest.

Other properties: request correlation id (`x-request-id`, honoured when supplied), pino structured
logs, `@fastify/rate-limit` (anonymous 60/min per IP; API keys looked up by sha256 hash with
per-key limits, `X-RateLimit-*` headers), ETag + gzip/brotli, CORS. Admin endpoints under
`/v1/admin/*` require `x-admin-token` (constant-time compare) and write `audit_log` rows for every
mutation; they never execute ingestion in-process, they enqueue pg-boss jobs.

See `docs/API.md` for endpoint examples.

## 5. Web application

`apps/web` (Next.js 16, server components) renders the public site. It talks to the API through a
`/api/v1/*` proxy to `http://127.0.0.1:${API_PORT}/v1/*`, so the browser only ever sees one origin.
The UI follows the scientific-editorial aesthetic and shows, for every number: source badge, unit,
population, period and freshness; empty states read "Data not yet available".

## 6. Ports, processes and environment

| Process | Command | Port | Notes |
|---|---|---|---|
| `cancerindex-web` | `next start -p 8250 -H 0.0.0.0` | 8250 | public entry (ngrok → www.cancerindex.io) |
| `cancerindex-api` | `tsx apps/api/src/server.ts` | 8251 (127.0.0.1) | reached only via the web proxy / localhost |
| `cancerindex-worker` | `tsx workers/main.ts` | — | pg-boss schema `pgboss` in the same database |
| PostgreSQL 17 | database `cancerindex`, extensions `pg_trgm`, `unaccent`, `vector` | 5432 | single node |

Key environment variables (see `.env.example`): `DATABASE_URL`, `API_HOST`/`API_PORT`, `WEB_PORT`,
`CI_API_URL`, `CI_DATA_DIR` (data lake root), `ADMIN_TOKEN`, `NCBI_TOOL`/`NCBI_EMAIL`/`NCBI_API_KEY`,
`SEER_API_KEY`, `WORKER_CONCURRENCY`, `CI_MAX_RUN_MINUTES`, `LOG_LEVEL`, `CI_SERVICE` (log label).

## 7. Deployment

Production runs on the MacLustr cluster through the `mld` orchestrator (gateway M1M32). The
manifest is `deploy/mld-manifest.cancerindex.json` (copied to `M1M32:~/dispatch/apps/cancerindex.json`);
`mld stage <dir> cancerindex` uploads a `git archive` of the repository and `mld deploy cancerindex`
places it on the preferred node (M4M64b: Postgres 17 + pgvector, pnpm, large disk), runs the
`post_sync` hooks (`pnpm install`, `createdb` if missing, migrate, seed, `sources:sync`, web build)
and starts three PM2 processes plus the ngrok tunnel. `deploy/first-run.sh` bootstraps the data in
the recommended connector order. Details in `deploy/README.md`.

## 7b. Operations: checkpoints, anomaly guard, alerts, backups, doctor

Reference: `docs/connectors/README.md` (runtime guarantees), `deploy/README.md` (day-2 procedures),
`docs/schema-changes-ops.md` (ops tables and indexes added outside the generated migrations).

- **Checkpoints (§90).** `RunContext` persists `connector_cursors.cursor` (and
  `ingest_runs.cursor_after`) automatically every `manifest.checkpointEvery` upserted records
  (default 2 000) or 60 s when the cursor changed, on `ctx.saveCursor()`, and on SIGTERM/SIGINT
  (`runConnector` installs handlers for the duration of the run: first signal → cursor saved, run
  `aborted`, connector stops at its next `shouldStop()`; second signal → immediate exit). A killed
  run therefore resumes from the last completed page; payload hashes make the overlap idempotent.
  `RawLake` serialises raw writes through one promise chain (single shared drain, no listener
  pile-up) and can `flush()` gzip blocks before a kill.
- **Anomaly guard (§171).** `ctx.guardCount(entity, total)` compares a declared total with the
  previous successful run's `records_fetched` (`anomalyGuard.minRatioOfPrevious`, default 0.5) and
  refuses to continue: run `failed` + `ingest_runs.anomaly`, critical alert, previous data untouched.
  Wired into OncoTree, NCIt and the ClinicalTrials.gov full crawl; connectors keep their fixed floors
  (HGNC, GDC, USCS, CIViC…). Nothing is ever mass-deleted on a shrunken response.
- **Alerts (§170).** Table `system_alerts` (`kind`, `severity info|warn|critical`, `connector_id`,
  `message`, `detail`, `first_seen_at`, `last_seen_at`, `count`, `status open|acknowledged|resolved`),
  deduplicated on kind + connector + message. Raised by `runConnector` (failure, anomaly, aborted,
  schema drift) and by the worker's hourly health probe (`source_failing`, `source_stale` = no
  success within 2× the cron interval of an active connector); resolved on the next success.
  Surfaces: `pnpm cix alerts`, `pnpm cix doctor`, `GET /v1/admin/alerts`.
- **Backups (§172).** PM2 process `cancerindex-backup` (`cron_restart "20 5 * * *"`,
  `autorestart: false`) runs `deploy/backup.sh`: `pg_dump -Fc` to
  `~/apps/cancerindex/backups/cancerindex-YYYYMMDD-HHMM.dump`, verified with `pg_restore --list`,
  14 daily + 8 weekly retained, `logs/backup.log`, non-zero exit on failure. `deploy/restore.sh
  <dump>` restores into a fresh `cancerindex_restore_<ts>` database (never over production). The raw
  lake (`data/raw`) is mirrored separately with rsync; both are needed for TRACE.
- **Doctor.** `pnpm cix doctor` prints the readiness report (environment, database + extensions +
  pending migrations vs `drizzle.__drizzle_migrations`, table sizes, every connector's status /
  license / health / last success age / last run status-anomaly-drift / cursor summary / stale flag,
  unresolved-label backlog, `data/raw` disk usage and free space, ranking-snapshot freshness, open
  alerts) and exits 1 on a hard failure (unreachable database, missing required extension, pending
  migration, unwritable data dir, missing ops schema, critical alert).
- **Indexes.** Indexes that drizzle cannot express or that were added by the performance review are
  created idempotently in `packages/database/src/migrate.ts` after the SQL migrations
  (`createPerformanceIndexes`): GIN on `civic_evidence_items.gene_ids/variant_ids/therapy_ids`
  (the counters use the containment form `@> ARRAY[id]::text[]`), `trial_conditions(cancer_id,
  trial_id)`, `clinical_trials(overall_status, study_type)` and `(overall_status,
  last_update_posted_date)`, `cancer_aliases(normalized text_pattern_ops)`,
  `ingest_runs(connector_id, status, started_at)`.

## 8. Phase 1 boundaries

- Rankings are count-based (trials, literature, curated evidence, genes, cohorts) for all entities
  and burden/lethality/gap metrics only where epidemiology observations exist for the scope.
- No composite score (ADR-006). No AI synthesis (docs/AI.md). No HGVS normalisation / liftover
  (ADR-002 plans Python/DuckDB workers for that).
- GLOBOCAN stays in license review and SEER in `awaiting_credentials`; the platform is honest about
  the resulting gaps ("awaiting license review").