# CancerIndex — Architecture

CancerIndex.io is a provenance-first oncology knowledge platform: a canonical cancer ontology connected to epidemiology, genomics, biomarkers, therapies, clinical trials, regulatory status and literature, with transparent, reproducible rankings. This document describes how the system is layered, which packages implement each layer, how data flows, and how the platform is deployed. The condensed rules every contributor follows are in CLAUDE.md; the full specification is docs/SPEC-original.md.

# 1. Layers

The platform is organised as six separable layers (CLAUDE.md §2). Each layer only reads from the layer below it and writes to its own tables; nothing is ever overwritten across layers.

Cross-cutting tables: sources (registry + license status), ingest_runs, connector_cursors, provenance, unresolved_labels, change_events, audit_log, entity_merges, api_keys.

Two invariants hold across every layer:

1. Provenance first. No scientific number exists without a provenance row (source, dataset, version, retrieval time, URL, evidence type, license). Derived values also carry formula_version and their inputs. traceValue() in @cancerindex/ranking walks a ranked value back to the observation, the provenance row and the raw record.
2. Never fake data. Missing data is absent, never zero. The API returns counters: null when counters have not been computed, and a ranking without a snapshot returns status: "not_available".

# 2. Repository and packages

cancerindex/
├── apps/
│   ├── web/                 Next.js 16 (webpack), Tailwind v4, server components; proxies /api/v1/* → API
│   └── api/                 Fastify 5 public API (/v1), OpenAPI from zod route schemas, rate limiting, admin
├── workers/                 pg-boss scheduler (connector cron, counters, rankings, health probes)
├── packages/
│   ├── shared/              ids (CI-*), provenance types, normalization, pino logger, env helpers
│   ├── database/            Drizzle schema (snake_case), migrations, seed (metrics + geographies)
│   ├── ontology/            qualifier rules, CancerResolver, TOP_LEVEL_CANCERS, NCIt roots
│   ├── connectors/          SDK (manifest, HttpClient, RawLake, RunContext, validators) + connectors/<id>/
│   └── ranking/             counters, ranking engine (snapshots + lineage), trace
├── scripts/ci.ts            operator CLI (`pnpm cix …`)
├── deploy/                  mld manifest, first-run bootstrap, deployment notes
├── docs/                    this folder
└── data/raw                 data lake (gitignored)

Dependency direction (no cycles): shared ← database ← ontology ← connectors ← ranking; apps/api depends on database, shared, ranking, ontology; workers additionally depends on connectors; apps/web never imports connectors (CLAUDE.md conventions).

# Connector SDK

A connector is a class extending Connector with a validated manifest (zod, see packages/connectors/src/sdk/manifest.ts), a healthCheck(ctx) and a sync(ctx). The RunContext provides a rate-limited HttpClient (token bucket, bounded concurrency, exponential backoff, Retry-After), a RawLake writer, idempotent upsertSourceRecord (sha256 payload hash), addProvenance, recordUnresolved (curation queue), recordChange, a restartable cursor, and a time budget checked with shouldStop(). Schema drift is detected by recording observed field names/types per entity (connector_field_stats). runConnector() handles bookkeeping in ingest_runs / connector_cursors, the credentials gate (awaiting_credentials) and the license gate (licenseStatus: blocked never ingests).

# Reconciliation

Labels are mapped to canonical cancers by CancerResolver: identifiers first (NCIt, DOID, OncoTree, UMLS, MeSH…), then curated aliases, then normalized strings. Every mapping stores a match_type (EXACT_IDENTIFIER, CURATED_EXACT, ONTOLOGY_EXACT, CURATED_BROADER, CURATED_NARROWER, ALIAS, PROBABILISTIC, UNRESOLVED). Unknown labels are stored in unresolved_labels with an optional trigram suggestion and are resolved by a curator through the admin API, which adds a curated alias so the next connector run picks it up.

# 3. Data flow

 upstream source ──HTTP──▶ connector.sync(ctx)
        │                     │  upsertSourceRecord → source_records + data/raw/*.jsonl.gz   (RAW)
        │                     │  validate / typed rows                                         (NORMALIZED)
        │                     │  CancerResolver → canonical ids, match_type; misses → unresolved_labels
        │                     │  addProvenance → provenance; observations / evidence / edges   (CANONICAL, DERIVED)
        │                     └  recordChange → change_events ; ingest_runs finalised
        ▼
 workers: maintenance.counters → refreshCounters() → entity_counters                            (DERIVED)
          maintenance.rank     → computeAllRankings() → ranking_snapshots + rankings            (RANKED)
        ▼
 apps/api (/v1) ── envelope { data, sources, dataRelease, generatedAt } ──▶ apps/web, API consumers

Runs are scheduled by the worker from each manifest's schedule (cron, UTC) and can be triggered by an operator (pnpm cix run <id> or POST /v1/admin/connectors/:id/run). A connector.run job is stately per connector id (never queued twice, never concurrent). When a run creates or updates records, the worker enqueues maintenance.counters with thenRank: true, so derived and ranked layers follow canonical changes. A daily counters (06:00 UTC) and rank (06:30 UTC) pass covers everything else; an hourly health.probe refreshes connector_cursors.health.

# 4. Public API

apps/api is a read-only Fastify 5 service. Route schemas are written in zod (via fastify-type-provider-zod) and the OpenAPI 3.1 document is generated from them (/v1/openapi.json, Swagger UI at /v1/docs) — documentation is never maintained by hand (CLAUDE.md §357).

Every /v1 response uses the envelope { data, sources, dataRelease, generatedAt, …pagination }. sources lists the distinct upstream sources behind the payload with license and attribution; dataRelease is CancerIndex <YYYY-MM> derived from the latest successful ingest.

Other properties: request correlation id (x-request-id, honoured when supplied), pino structured logs, @fastify/rate-limit (anonymous 60/min per IP; API keys looked up by sha256 hash with per-key limits, X-RateLimit-* headers), ETag + gzip/brotli, CORS. Admin endpoints under /v1/admin/* require x-admin-token (constant-time compare) and write audit_log rows for every mutation; they never execute ingestion in-process, they enqueue pg-boss jobs.

See docs/API.md for endpoint examples.

# 5. Web application

apps/web (Next.js 16, server components) renders the public site. It talks to the API through a /api/v1/* proxy to http://127.0.0.1:${API_PORT}/v1/*, so the browser only ever sees one origin. The UI follows the scientific-editorial aesthetic and shows, for every number: source badge, unit, population, period and freshness; empty states read "Data not yet available".

# 6. Ports, processes and environment

Key environment variables (see .env.example): DATABASE_URL, API_HOST/API_PORT, WEB_PORT, CI_API_URL, CI_DATA_DIR (data lake root), ADMIN_TOKEN, NCBI_TOOL/NCBI_EMAIL/NCBI_API_KEY, SEER_API_KEY, WORKER_CONCURRENCY, CI_MAX_RUN_MINUTES, LOG_LEVEL, CI_SERVICE (log label).

# 7. Deployment

Production runs on the MacLustr cluster through the mld orchestrator (gateway M1M32). The manifest is deploy/mld-manifest.cancerindex.json (copied to M1M32:~/dispatch/apps/cancerindex.json); mld stage <dir> cancerindex uploads a git archive of the repository and mld deploy cancerindex places it on the preferred node (M4M64b: Postgres 17 + pgvector, pnpm, large disk), runs the post_sync hooks (pnpm install, createdb if missing, migrate, seed, sources:sync, web build) and starts three PM2 processes plus the ngrok tunnel. deploy/first-run.sh bootstraps the data in the recommended connector order. Details in deploy/README.md.

# 8. Phase 1 boundaries

• Rankings are count-based (trials, literature, curated evidence, genes, cohorts) for all entities and burden/lethality/gap metrics only where epidemiology observations exist for the scope.
• No composite score (ADR-006). No AI synthesis (docs/AI.md). No HGVS normalisation / liftover (ADR-002 plans Python/DuckDB workers for that).
• GLOBOCAN stays in license review and SEER in awaiting_credentials; the platform is honest about the resulting gaps ("awaiting license review").