CancerIndex.io

Understand cancer through data.
A provenance-first, continuously updated, transparently sourced index of every recognized cancer entity — epidemiology, genomics, biomarkers, drugs, regulatory approvals, clinical trials, research and derived intelligence, cross-linked in one coherent dataset.

www.cancerindex.io · API docs · Methodology · Sources · Pulse

# Table of contents

1. What CancerIndex is
2. What it refuses to do
3. The index in numbers
4. Product map
5. Architecture
6. Repository layout
7. Data model
8. Connectors
9. Derived intelligence layer
10. Rankings and metrics
11. Public API
12. Operator CLI
13. Local development
14. Configuration
15. Quality: tests, QA, accessibility, performance
16. Deployment and operations
17. Brand
18. Design language
19. Known gotchas
20. Roadmap
21. Licensing, attribution and contact

# What CancerIndex is

CancerIndex organizes fragmented oncology data into one structured, navigable system. A reader moves from a cancer to its incidence and mortality, to the genes altered in it, to a variant, to the drugs that target it, to the regulatory approvals of those drugs in each jurisdiction, to the clinical trials that investigate them and to the papers that describe them — every step a link, every number a sourced record.

It is built for researchers, students, journalists, policymakers, biotech professionals and informed members of the public. It feels like a research terminal, not a health-content website: dense tables, sparse charts, one accent colour, a source badge on every figure.

Five things make it different from a cancer information site:

• A canonical taxonomy first. 9 510 active disease entities anchored on the NCI Thesaurus, with OncoTree, ICD-10, ICD-O, DOID, UMLS, MeSH and MONDO cross-references; 36 mutually exclusive top-level site groups (GLOBOCAN / ICD-10 ranges) for burden rankings without double counting.
• Provenance on every value. Each observation, edge, approval and count points to a provenance row (source, dataset, version, URL, retrieval date, population, methodology); raw payloads are kept in a gzip JSON lake and can be replayed.
• Layers that stay separable. RAW → NORMALIZED → CANONICAL → DERIVED → RANKED. Derived values carry a formula_version and the inputs that produced them ("Why this rank?").
• Claim categories that never merge. Observed · published · curated · regulatory · guideline · computed · AI-generated — labelled everywhere, mixed nowhere.
• No invented numbers. Missing data is shown as Data not yet available. No estimates, no placeholder statistics, no composite "worst cancer" score.

# What it refuses to do

• It is not a physician: no diagnosis, no individual prognosis, no dosing, no treatment recommendation. Population statistics never predict an individual outcome and the UI says so.
• It does not rank cancers by an opaque composite; each ranking is one metric, one scope, one formula.
• It does not infer facts: no edge is created by a language model, no reason for a terminated trial is guessed, no indication is derived from an ATC class, no cancer is attached to an approval unless the official text names exactly one.
• It does not redistribute data it may not redistribute: every connector manifest is licence-reviewed before it goes live (IARC / GLOBOCAN stays under review, SEER awaits credentials).

# The index in numbers

Production database on 2026-09-11 (SELECT count(*) — live counts, never estimates):

# Product map

Every page has a light and a dark theme (toggle in the header, prefers-color-scheme by default), a canonical URL, structured metadata and a server-rendered Open Graph / Twitter image.

# Architecture

                   ┌──────────────────────── sources (19 connectors) ────────────────────────┐
                   │ NCIt EVS · OncoTree · HGNC · CIViC · ClinVar · GDC · cBioPortal · MeSH   │
                   │ ClinicalTrials.gov · PubMed · CDC WONDER · CDC USCS · ChEMBL · openFDA   │
                   │ Health Canada DPD · EMA · (IARC GLOBOCAN: review) · (SEER: credentials)  │
                   └───────────────┬───────────────────────────────────────────────────────────┘
                                   │ HTTP / bulk files, rate-limited, restartable cursors
                                   ▼
   workers/main.ts  ──►  packages/connectors (SDK: manifest · HttpClient · RawLake · RunContext)
   pg-boss scheduler          │ RAW  → data/raw/{source}/{date}/{entity}/*.jsonl.gz + source_records
   cron per manifest          │ NORMALIZED → validators, schema-drift stats, unresolved_labels
   counters 06:00 UTC         │ CANONICAL → cancers · genes · variants · drugs · trials · publications
   intel    06:15 UTC         │             observations · approvals · knowledge_edges · provenance
   rank     06:30 UTC         ▼
                        PostgreSQL 17 (+ pg_trgm, unaccent, pgvector) — Drizzle schema, snake_case
                                   │
                    packages/ranking: counters → intelligence → rankings (DERIVED / RANKED)
                                   │
              ┌────────────────────┴────────────────────┐
              ▼                                         ▼
   apps/api  Fastify 5 · /v1 · zod · OpenAPI      apps/web  Next.js 16 (webpack) · server components
   envelope { data, sources, dataRelease }         Tailwind v4 tokens · pure SVG charts · dark mode
   rate limits · API keys · x-request-id           /api/v1/* proxied to the API · ISR caching
              └────────────────────┬────────────────────┘
                                   ▼
             MacLustr node M4M64b · PM2 (web :8250, api :8251, worker, backup)
             MacLustr Tunnel (WireGuard + Caddy on BHS64) → https://www.cancerindex.io

Principles baked into the code

• Reconciliation before ingestion. Labels map to entities by shared identifier, then curated alias, then normalized string; every mapping stores a match_type (EXACT_IDENTIFIER, CURATED_EXACT, ONTOLOGY_EXACT, CURATED_BROADER, ALIAS, PROBABILISTIC, UNRESOLVED). Unknown labels go to unresolved_labels, never to /dev/null.
• Time-aware observations. A value for year X never overwrites year Y; multi-year aggregates keep both bounds; rates keep their standard population.
• Idempotent, restartable connectors. Payload hashes, cursor checkpoints every 2 000 records or 60 s, SIGTERM-safe aborts, anomaly guard refusing destructive updates when a source shrinks by more than half.
• Deterministic derived layers. Counters, intelligence tables and rankings are full rebuilds in one transaction; snapshots are immutable and keep previous_rank for change explanation.

# Repository layout

apps/
  web/                 Next.js 16 app (React 19, server components, Tailwind v4, --webpack)
    src/app/           routes (see Product map), opengraph-image.tsx per entity, icon.svg, manifest.ts
    src/components/    ui · charts (SVG) · cancer tabs · graph · explorer · country · home modules · layout
    src/lib/           db · format · queries/* (server-only SQL) · graph-model · explorer-* · og.tsx · site.ts
    src/assets/fonts/  WOFF copies of Newsreader / Inter / IBM Plex Mono for server-side images
    public/brand/      logo.svg · logo-mark.svg · logo-mark-dark.svg · PNG icons
    qa/smoke.mjs       read-only smoke suite (44 routes, weight caps, mobile overflow, console errors)
  api/                 Fastify 5 public API (/v1), zod schemas, OpenAPI at /v1/docs, admin routes
packages/
  shared/              CI-XXX-00000001 ids, provenance types, normalization, logger, env
  database/            Drizzle schema (12 files, ~50 tables), migrations 0000–0002, seeds (metrics,
                       geographies, biomarkers), alerts, ids
  ontology/            qualifier rules, CancerResolver (alias/code reconciliation), TOP_LEVEL_CANCERS
  connectors/          SDK (manifest · HttpClient · RawLake · RunContext · validators · doctor) and
                       connectors/<id>/{manifest.ts,index.ts,normalize.ts,fixtures/,*.test.ts}
  ranking/             counters · intelligence (trial-intelligence, trial-sites, drug-pipeline,
                       drug-duplicates, research-gap) · engine (snapshots) · trace · country-codes
workers/               pg-boss scheduler and job handlers
scripts/ci.ts          operator CLI (`pnpm cix …`)
deploy/                mld manifest, first-run bootstrap, backup / restore scripts
docs/                  ARCHITECTURE · DATA-MODEL · METHODOLOGY · API · SECURITY · AI · source-policy ·
                       schema-changes-ops · adr/ (6 ADRs) · connectors/ (18 pages) · methodology/ (7 pages)
CLAUDE.md              operational rules every contributor (human or agent) follows

# Data model

Source of truth: packages/database/src/schema/*.ts (documented in docs/DATA-MODEL.md). Public identifiers are CI-<NS>-00000001 minted per namespace (CAN, GENE, VAR, DRUG, TRIAL, PUB, BIO, STUDY, METRIC, SOURCE, GEO, PROV…) — never database integers.

Schema changes are migrations generated with pnpm db:generate (one per integration) and applied by pnpm db:migrate, which also creates the extensions and the trigram / GIN performance indexes.

# Connectors

Each connector is a class with a manifest (licence, terms review date, documentation verification date, rate limits, schedule, anomaly guard), a health check and a sync(); it is tested against sanitized fixtures (normal, empty, pagination, rate-limit, server error, malformed) and documented in docs/connectors/<id>.md. pnpm cix sources:sync seeds manifests into sources; /sources shows them.

Run order for a fresh database: deploy/first-run.sh (terminology → genes → evidence/genomics/variants → trials → literature → epidemiology), then the regulatory connectors, pnpm cix reconcile-drugs, pnpm cix counters, pnpm cix intel, pnpm cix rank.

# Derived intelligence layer

Recomputed daily by the worker (maintenance.intel, between counters and rankings) or with pnpm cix intel (≈ 50 s on the production database). Every row stores formula_version and inputs. Methods: docs/methodology/*.md and /methodology.

# Rankings and metrics

25 metric definitions live in metric_definitions (seeded from packages/database/src/seed-data/metrics.ts) and are rendered live on /methodology#metrics. A ranking snapshot is one metric × one scope (geo=USA|sex=all|age=all|year=2024|level=top) × one formula version; rows keep rank, percentile, confidence, previous rank and the exact inputs.

Burden, lethality and gap metrics exist only for scopes with licensed observations (currently the United States, per year and sex, top level). No composite score is published (ADR-006).

# Public API

Base URL https://www.cancerindex.io/api/v1 (proxied to Fastify). Read-only JSON. Every response is an envelope { data, sources, dataRelease, generatedAt, total?, limit?, offset?, hasMore? } where sources lists the upstream sources, licences and attributions behind the returned data. Swagger UI at /api/v1/docs, OpenAPI 3.1 at /api/v1/openapi.json. Rate limits per IP; an optional bearer API key raises them.

Details and examples: docs/API.md.

# Operator CLI

pnpm cix connectors                          list connectors, licence status, health, last success
pnpm cix run <id> [--mode full|incremental|backfill|dry_run] [--max-records N] [--max-minutes M] [--reset-cursor]
pnpm cix run-all [--max-minutes M]           every active connector in registry order
pnpm cix health <id>                         source liveness probe
pnpm cix sources:sync                        manifests → sources table
pnpm cix reconcile-drugs [--remap]           trial interventions → canonical drugs
pnpm cix counters                            rebuild entity_counters
pnpm cix intel                               trial intelligence · trial sites · drug pipeline · research gap
pnpm cix rank                                recompute every ranking snapshot
pnpm cix stats | trace <type> <id> | doctor [--no-disk] | alerts [ack|resolve <id>]

--mode backfill replays a connector's raw lake without HTTP (used after a mapping-rule change); dry_run never writes and never moves a cursor.

# Local development

Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with pg_trgm, unaccent and vector; rsvg-convert and Python Pillow only if you regenerate brand rasters.

createdb cancerindex && cp .env.example .env          # set NCBI_EMAIL, ADMIN_TOKEN
pnpm install
pnpm db:migrate && pnpm db:seed && pnpm cix sources:sync
pnpm cix run oncotree --mode dry_run                  # smoke against the live API, no writes
bash deploy/first-run.sh                              # ordered first ingestion (hours; restartable)
pnpm cix run health-canada-dpd && pnpm cix run ema && pnpm cix run openfda
pnpm cix reconcile-drugs && pnpm cix counters && pnpm cix intel && pnpm cix rank
pnpm dev:api &                                        # http://127.0.0.1:8251/v1/docs
pnpm worker &                                         # schedules from connector manifests
pnpm dev:web                                          # http://localhost:8250

A faster path for UI work is to restore a production dump (bash deploy/restore.sh <dump> --target cancerindex_prodcopy, then rename databases) and run pnpm db:migrate && pnpm db:seed. Note that Next.js allows a single next dev per app directory (.next/dev/lock).

# Configuration

.env at the repository root (read by the API, the worker and the web app).

# Quality: tests, QA, accessibility, performance

• pnpm typecheck — strict TypeScript across the 8 workspaces.
• pnpm test — 528 vitest tests: connector fixtures (every connector), SDK (lake, run, checkpoints), formulas (shares, log ratios, HHI, growth, termination share, stop-reason rules, pipeline stages, country codes, map scales, projections, explorer comparability, CSV escaping, graph layout), API smoke tests against the local database (skipped when unreachable).
• pnpm --filter @cancerindex/web qa / qa:prod — read-only smoke suite over 44 routes: HTTP 200, expected text, per-route weight caps, no "Data not yet available" where data must exist, no horizontal overflow at 390 px, console-error scan (Playwright when available).
• Accessibility: keyboard navigation, focus rings, colour never the only carrier (badges carry text, charts have legends and data tables, maps have equivalent tables), aria labels on SVG figures.
• Performance: ISR caching (revalidate per page, Cache-Control on public routes), materialized derived tables, GIN / trigram indexes, sql.param for array parameters, request budgets on heavy pages (graph groups, city layers).

# Deployment and operations

Production runs on the MacLustr cluster (node M4M64b: Postgres 17 + pgvector, PM2) behind the MacLustr Tunnel (WireGuard + Caddy on the BHS64 gateway) at https://www.cancerindex.io.

# from the laptop — gateway M1M32 orchestrates (mld)
rsync -a --exclude '/data/' --exclude node_modules --exclude .next --exclude .git --exclude '.env*' ./ /tmp/cancerindex-deploy/
mld stage /tmp/cancerindex-deploy cancerindex
mld deploy cancerindex --node M4M64b          # install, migrate, seed, sources:sync, build, PM2 restart, health checks

Operational visibility: /data-updates (connector state, ingestion log), /admin (runs, unresolved labels, alerts), pnpm cix doctor, system_alerts. Deploy manifest: deploy/mld-manifest.cancerindex.json (note the excludes are anchored — /coverage/, /data/ — so route folders with those names are kept).

# Brand

The mark reads CI: an open ring — a cell with its nucleus, the "C" — and a graduated index scale in the accent teal, the "I". It is drawn once in SVG and used everywhere: the header and footer (BrandMark, theme-aware through currentColor and the accent token), the favicon (app/icon.svg, favicon.ico, apple-icon.png), the web manifest and the share images.

Colours: paper #fafaf7, ink #1c1c1a, accent teal #0f5f63 (dark theme #6cc3c6). Type: Newsreader (display), Inter (UI), IBM Plex Mono (identifiers).

# Design language

Scientific, editorial, institutional. Off-white paper, charcoal ink, one restrained teal accent, thin rules, tabular numerals, dense tables, sparklines and small multiples rather than decorative charts. All charts are server-rendered SVG with a legend, units, the dataset year and a source line; every value shows a claim badge (observed · published · curated · regulatory · computed) and a source badge whose popover gives dataset, version, retrieval date and URL. Mobile is first-class (tables scroll inside their wrap; no horizontal overflow at 390 px). Light and dark themes share one token set (--color-*, --color-series-*).

# Known gotchas

• A JS array inside a Drizzle template — sql`x = ANY(${arr}::text[])` — becomes a ($1,$2,…) tuple and fails; use sql.param(arr) or sql.raw for constant lists.
• Columns declared with the updatedAt() helper are physically updated_at even when named computedAt.
• db.execute<T> needs a type alias, not an interface, for T.
• A prop named ref cannot be passed to a server component (React reserves it) — the page fails in production with minified error #441; use another name (entityRef).
• SVG <title> must be a single text node or hydration fails; array_agg over empty arrays errors — use unnest / string_agg.
• opengraph-image.tsx cannot live inside an optional catch-all segment ([[...tab]]); place it one level up.
• ranking_snapshots.scope_key has no source dimension: two sources for the same geography/year/sex overwrite each other's current snapshot, so a preferred source is chosen per scope.
• A shared alias between a concept and its descendant ("breast cancer") must resolve to the broadest concept when building a dictionary; the narrowest-of-lineage rule is only for sentence-level mentions.
• OncoTree's WAF rejects any User-Agent containing a URL; NCBI throttles at 3 req/s without a key; openFDA allows 1 000 requests/day without a key; the EMA xlsx has 39 columns and header row 9.
• mld sync_excludes patterns must be anchored (/data/, /coverage/), otherwise route folders with the same name are dropped from the deploy.

# Roadmap

• Global burden once IARC / GLOBOCAN terms are settled (human decision) and SEER credentials are set; survival observations and the five_year_survival metric follow.
• MHRA and TGA regulatory connectors; populating drug_approvals.biomarker_ids from label text.
• Risk factors and attributable burden, screening and guideline registries (metadata and links only, with change detection), hereditary syndromes, pediatric and rare-cancer views.
• Researcher, institution and funding entities (OpenAlex, NIH RePORTER, CIHR) for the funding-gap view.
• Watchlists, alerts and a grounded, cited /ask layer that summarizes indexed sources only.
• Lighter default views for the heaviest pages (/graph, biomarker pages with hundreds of approvals).

# Licensing, attribution and contact

• Code: private repository (spbgit cancerindex.git).
• Derived data (rankings, intelligence tables, CSV exports): © CancerIndex, CC BY 4.0, with attribution rows naming the underlying providers.
• Source data remains under the licence of each provider — see /sources and the sources array of every API response (NCIt CC BY 4.0, OncoTree CC BY 4.0, HGNC CC0, CIViC CC0, openFDA CC0, Health Canada OGL, EMA with acknowledgement, ClinVar / PubMed / ClinicalTrials.gov / CDC public domain, GDC open tier, ChEMBL CC BY-SA).
• Not medical advice. CancerIndex provides research and educational information only.

Built and maintained by Simon-Pierre Boucher — contact contact@spboucher.ai — hosted on MacLustr (https://www.maclustr.io).