# CancerIndex — Architecture

CancerIndex.io is a provenance-first oncology knowledge platform: a canonical cancer ontology connected to epidemiology, genomics, biomarkers, therapies, clinical trials, regulatory status and literature, with transparent, reproducible rankings. This document describes how the system is layered, which packages implement each layer, how data flows, and how the platform is deployed. The condensed rules every contributor follows are in CLAUDE.md; the full specification is docs/SPEC-original.md.

# 1. Layers

The platform is organised as six separable layers (CLAUDE.md §2). Each layer only reads from the layer below it and writes to its own tables; nothing is ever overwritten across layers.

Cross-cutting tables: sources (registry + license status), ingest_runs, connector_cursors, provenance, unresolved_labels, change_events, audit_log, entity_merges, api_keys.

Two invariants hold across every layer:

1. Provenance first. No scientific number exists without a provenance row (source, dataset, version, retrieval time, URL, evidence type, license). Derived values also carry formula_version and their inputs. traceValue() in @cancerindex/ranking walks a ranked value back to the observation, the provenance row and the raw record.
2. Never fake data. Missing data is absent, never zero. The API returns counters: null when counters have not been computed, and a ranking without a snapshot returns status: "not_available".

# 2. Repository and packages

cancerindex/
├── apps/
│   ├── web/                 Next.js 16 (webpack), Tailwind v4, server components; proxies /api/v1/* → API
│   └── api/                 Fastify 5 public API (/v1), OpenAPI from zod route schemas, rate limiting, admin
├── workers/                 pg-boss scheduler (connector cron, counters, rankings, health probes)
├── packages/
│   ├── shared/              ids (CI-*), provenance types, normalization, pino logger, env helpers
│   ├── database/            Drizzle schema (snake_case), migrations, seed (metrics + geographies)
│   ├── ontology/            qualifier rules, CancerResolver, TOP_LEVEL_CANCERS, NCIt roots
│   ├── connectors/          SDK (manifest, HttpClient, RawLake, RunContext, validators) + connectors/<id>/
│   └── ranking/             counters, ranking engine (snapshots + lineage), trace
├── scripts/ci.ts            operator CLI (`pnpm cix …`)
├── deploy/                  mld manifest, first-run bootstrap, deployment notes
├── docs/                    this folder
└── data/raw                 data lake (gitignored)

Dependency direction (no cycles): shared ← database ← ontology ← connectors ← ranking; apps/api depends on database, shared, ranking, ontology; workers additionally depends on connectors; apps/web never imports connectors (CLAUDE.md conventions).

# Connector SDK

A connector is a class extending Connector with a validated manifest (zod, see packages/connectors/src/sdk/manifest.ts), a healthCheck(ctx) and a sync(ctx). The RunContext provides a rate-limited HttpClient (token bucket, bounded concurrency, exponential backoff, Retry-After), a RawLake writer, idempotent upsertSourceRecord (sha256 payload hash), addProvenance, recordUnresolved (curation queue), recordChange, a restartable cursor, and a time budget checked with shouldStop(). Schema drift is detected by recording observed field names/types per entity (connector_field_stats). runConnector() handles bookkeeping in ingest_runs / connector_cursors, the credentials gate (awaiting_credentials) and the license gate (licenseStatus: blocked never ingests).

# Reconciliation

Labels are mapped to canonical cancers by CancerResolver: identifiers first (NCIt, DOID, OncoTree, UMLS, MeSH…), then curated aliases, then normalized strings. Every mapping stores a match_type (EXACT_IDENTIFIER, CURATED_EXACT, ONTOLOGY_EXACT, CURATED_BROADER, CURATED_NARROWER, ALIAS, PROBABILISTIC, UNRESOLVED). Unknown labels are stored in unresolved_labels with an optional trigram suggestion and are resolved by a curator through the admin API, which adds a curated alias so the next connector run picks it up.

# 3. Data flow

 upstream source ──HTTP──▶ connector.sync(ctx)
        │                     │  upsertSourceRecord → source_records + data/raw/*.jsonl.gz   (RAW)
        │                     │  validate / typed rows                                         (NORMALIZED)
        │                     │  CancerResolver → canonical ids, match_type; misses → unresolved_labels
        │                     │  addProvenance → provenance; observations / evidence / edges   (CANONICAL, DERIVED)
        │                     └  recordChange → change_events ; ingest_runs finalised
        ▼
 workers: maintenance.counters → refreshCounters() → entity_counters                            (DERIVED)
          maintenance.rank     → computeAllRankings() → ranking_snapshots + rankings            (RANKED)
        ▼
 apps/api (/v1) ── envelope { data, sources, dataRelease, generatedAt } ──▶ apps/web, API consumers

Runs are scheduled by the worker from each manifest's schedule (cron, UTC) and can be triggered by an operator (pnpm cix run <id> or POST /v1/admin/connectors/:id/run). A connector.run job is stately per connector id (never queued twice, never concurrent). When a run creates or updates records, the worker enqueues maintenance.counters with thenRank: true, so derived and ranked layers follow canonical changes. A daily counters (06:00 UTC) and rank (06:30 UTC) pass covers everything else; an hourly health.probe refreshes connector_cursors.health.

# 4. Public API

apps/api is a read-only Fastify 5 service. Route schemas are written in zod (via fastify-type-provider-zod) and the OpenAPI 3.1 document is generated from them (/v1/openapi.json, Swagger UI at /v1/docs) — documentation is never maintained by hand (CLAUDE.md §357).

Every /v1 response uses the envelope { data, sources, dataRelease, generatedAt, …pagination }. sources lists the distinct upstream sources behind the payload with license and attribution; dataRelease is CancerIndex <YYYY-MM> derived from the latest successful ingest.

Other properties: request correlation id (x-request-id, honoured when supplied), pino structured logs, @fastify/rate-limit (anonymous 60/min per IP; API keys looked up by sha256 hash with per-key limits, X-RateLimit-* headers), ETag + gzip/brotli, CORS. Admin endpoints under /v1/admin/* require x-admin-token (constant-time compare) and write audit_log rows for every mutation; they never execute ingestion in-process, they enqueue pg-boss jobs.

See docs/API.md for endpoint examples.

# 5. Web application

apps/web (Next.js 16, server components) renders the public site. It talks to the API through a /api/v1/* proxy to http://127.0.0.1:${API_PORT}/v1/*, so the browser only ever sees one origin. The UI follows the scientific-editorial aesthetic and shows, for every number: source badge, unit, population, period and freshness; empty states read "Data not yet available".

# 6. Ports, processes and environment

Key environment variables (see .env.example): DATABASE_URL, API_HOST/API_PORT, WEB_PORT, CI_API_URL, CI_DATA_DIR (data lake root), ADMIN_TOKEN, NCBI_TOOL/NCBI_EMAIL/NCBI_API_KEY, SEER_API_KEY, WORKER_CONCURRENCY, CI_MAX_RUN_MINUTES, LOG_LEVEL, CI_SERVICE (log label).

# 7. Deployment

Production runs on the MacLustr cluster through the mld orchestrator (gateway M1M32). The manifest is deploy/mld-manifest.cancerindex.json (copied to M1M32:~/dispatch/apps/cancerindex.json); mld stage <dir> cancerindex uploads a git archive of the repository and mld deploy cancerindex places it on the preferred node (M4M64b: Postgres 17 + pgvector, pnpm, large disk), runs the post_sync hooks (pnpm install, createdb if missing, migrate, seed, sources:sync, web build) and starts three PM2 processes plus the ngrok tunnel. deploy/first-run.sh bootstraps the data in the recommended connector order. Details in deploy/README.md.

# 7b. Operations: checkpoints, anomaly guard, alerts, backups, doctor

Reference: docs/connectors/README.md (runtime guarantees), deploy/README.md (day-2 procedures), docs/schema-changes-ops.md (ops tables and indexes added outside the generated migrations).

• Checkpoints (§90). RunContext persists connector_cursors.cursor (and ingest_runs.cursor_after) automatically every manifest.checkpointEvery upserted records (default 2 000) or 60 s when the cursor changed, on ctx.saveCursor(), and on SIGTERM/SIGINT (runConnector installs handlers for the duration of the run: first signal → cursor saved, run aborted, connector stops at its next shouldStop(); second signal → immediate exit). A killed run therefore resumes from the last completed page; payload hashes make the overlap idempotent. RawLake serialises raw writes through one promise chain (single shared drain, no listener pile-up) and can flush() gzip blocks before a kill.
• Anomaly guard (§171). ctx.guardCount(entity, total) compares a declared total with the previous successful run's records_fetched (anomalyGuard.minRatioOfPrevious, default 0.5) and refuses to continue: run failed + ingest_runs.anomaly, critical alert, previous data untouched. Wired into OncoTree, NCIt and the ClinicalTrials.gov full crawl; connectors keep their fixed floors (HGNC, GDC, USCS, CIViC…). Nothing is ever mass-deleted on a shrunken response.
• Alerts (§170). Table system_alerts (kind, severity info|warn|critical, connector_id, message, detail, first_seen_at, last_seen_at, count, status open|acknowledged|resolved), deduplicated on kind + connector + message. Raised by runConnector (failure, anomaly, aborted, schema drift) and by the worker's hourly health probe (source_failing, source_stale = no success within 2× the cron interval of an active connector); resolved on the next success. Surfaces: pnpm cix alerts, pnpm cix doctor, GET /v1/admin/alerts.
• Backups (§172). PM2 process cancerindex-backup (cron_restart "20 5 * * *", autorestart: false) runs deploy/backup.sh: pg_dump -Fc to ~/apps/cancerindex/backups/cancerindex-YYYYMMDD-HHMM.dump, verified with pg_restore --list, 14 daily + 8 weekly retained, logs/backup.log, non-zero exit on failure. deploy/restore.sh <dump> restores into a fresh cancerindex_restore_<ts> database (never over production). The raw lake (data/raw) is mirrored separately with rsync; both are needed for TRACE.
• Doctor. pnpm cix doctor prints the readiness report (environment, database + extensions + pending migrations vs drizzle.__drizzle_migrations, table sizes, every connector's status / license / health / last success age / last run status-anomaly-drift / cursor summary / stale flag, unresolved-label backlog, data/raw disk usage and free space, ranking-snapshot freshness, open alerts) and exits 1 on a hard failure (unreachable database, missing required extension, pending migration, unwritable data dir, missing ops schema, critical alert).
• Indexes. Indexes that drizzle cannot express or that were added by the performance review are created idempotently in packages/database/src/migrate.ts after the SQL migrations (createPerformanceIndexes): GIN on civic_evidence_items.gene_ids/variant_ids/therapy_ids (the counters use the containment form @> ARRAY[id]::text[]), trial_conditions(cancer_id, trial_id), clinical_trials(overall_status, study_type) and (overall_status, last_update_posted_date), cancer_aliases(normalized text_pattern_ops), ingest_runs(connector_id, status, started_at).

# 8. Phase 1 boundaries

• Rankings are count-based (trials, literature, curated evidence, genes, cohorts) for all entities and burden/lethality/gap metrics only where epidemiology observations exist for the scope.
• No composite score (ADR-006). No AI synthesis (docs/AI.md). No HGVS normalisation / liftover (ADR-002 plans Python/DuckDB workers for that).
• GLOBOCAN stays in license review and SEER in awaiting_credentials; the platform is honest about the resulting gaps ("awaiting license review").