SPB Git
spboucher.ai

spb/cancerindex

Public

README: detailed project guide with brand mark, product map, architecture, data model, connectors, derived layer, metrics, API, CLI, setup, operations, gotchas, roadmap

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 11, 2026) parent d64e723 Browse files

1 changed file +463 −68

Unified Split
modified README.md +463 −68
@@ -1,97 +1,492 @@
1−# CancerIndex.io — the global index of cancer 1+<p align="center">
2+ <img src="apps/web/public/brand/logo.svg" alt="CancerIndex — the global index of cancer" width="360">
3+</p>
2 4
3−CancerIndex is a provenance-first, continuously updated, transparently sourced oncology knowledge 5+<h1 align="center">CancerIndex.io</h1>
4−platform. It connects a canonical cancer ontology (NCIt-anchored, OncoTree and registry dimensions)
5−to epidemiology, genomics, biomarkers, therapies, clinical trials, regulatory evidence and literature
6−— for every recognised malignant disease entity — and publishes reproducible rankings with their
7−scope, formula version and lineage down to the raw record.
8 6
9−It is not a physician, not a diagnostic tool, and gives no treatment recommendations. 7+<p align="center"><strong>Understand cancer through data.</strong><br>
8+A provenance-first, continuously updated, transparently sourced index of every recognized cancer entity —
9+epidemiology, genomics, biomarkers, drugs, regulatory approvals, clinical trials, research and derived
10+intelligence, cross-linked in one coherent dataset.</p>
10 11
11−## Status 12+<p align="center">
13+ <a href="https://www.cancerindex.io">www.cancerindex.io</a> ·
14+ <a href="https://www.cancerindex.io/api/v1/docs">API docs</a> ·
15+ <a href="https://www.cancerindex.io/methodology">Methodology</a> ·
16+ <a href="https://www.cancerindex.io/sources">Sources</a> ·
17+ <a href="https://www.cancerindex.io/pulse">Pulse</a>
18+</p>
12 19
13−Phase 1 (2026-09). Foundation, connectors, public API, worker and web app are being built in 20+---
14−parallel. Data currently ingested locally: OncoTree (865 cancer entities), with NCIt EVS, HGNC,
15−CIViC, GDC, ClinVar, ClinicalTrials.gov, PubMed and CDC WONDER connectors in progress. GLOBOCAN
16−remains under license review and SEER awaits credentials — the platform says so rather than showing
17−numbers it cannot source.
18 21
19−## Repository 22+## Table of contents
23+
24+1. [What CancerIndex is](#what-cancerindex-is)
25+2. [What it refuses to do](#what-it-refuses-to-do)
26+3. [The index in numbers](#the-index-in-numbers)
27+4. [Product map](#product-map)
28+5. [Architecture](#architecture)
29+6. [Repository layout](#repository-layout)
30+7. [Data model](#data-model)
31+8. [Connectors](#connectors)
32+9. [Derived intelligence layer](#derived-intelligence-layer)
33+10. [Rankings and metrics](#rankings-and-metrics)
34+11. [Public API](#public-api)
35+12. [Operator CLI](#operator-cli)
36+13. [Local development](#local-development)
37+14. [Configuration](#configuration)
38+15. [Quality: tests, QA, accessibility, performance](#quality-tests-qa-accessibility-performance)
39+16. [Deployment and operations](#deployment-and-operations)
40+17. [Brand](#brand)
41+18. [Design language](#design-language)
42+19. [Known gotchas](#known-gotchas)
43+20. [Roadmap](#roadmap)
44+21. [Licensing, attribution and contact](#licensing-attribution-and-contact)
45+
46+---
47+
48+## What CancerIndex is
49+
50+CancerIndex organizes fragmented oncology data into one structured, navigable system. A reader moves
51+from a **cancer** to its **incidence and mortality**, to the **genes** altered in it, to a **variant**, to the
52+**drugs** that target it, to the **regulatory approvals** of those drugs in each jurisdiction, to the
53+**clinical trials** that investigate them and to the **papers** that describe them — every step a link, every
54+number a sourced record.
55+
56+It is built for researchers, students, journalists, policymakers, biotech professionals and informed
57+members of the public. It feels like a research terminal, not a health-content website: dense tables,
58+sparse charts, one accent colour, a source badge on every figure.
59+
60+Five things make it different from a cancer information site:
61+
62+- **A canonical taxonomy first.** 9 510 active disease entities anchored on the NCI Thesaurus, with
63+ OncoTree, ICD-10, ICD-O, DOID, UMLS, MeSH and MONDO cross-references; 36 mutually exclusive
64+ top-level site groups (GLOBOCAN / ICD-10 ranges) for burden rankings without double counting.
65+- **Provenance on every value.** Each observation, edge, approval and count points to a `provenance`
66+ row (source, dataset, version, URL, retrieval date, population, methodology); raw payloads are kept
67+ in a gzip JSON lake and can be replayed.
68+- **Layers that stay separable.** RAW → NORMALIZED → CANONICAL → DERIVED → RANKED. Derived values
69+ carry a `formula_version` and the inputs that produced them ("Why this rank?").
70+- **Claim categories that never merge.** Observed · published · curated · regulatory · guideline ·
71+ computed · AI-generated — labelled everywhere, mixed nowhere.
72+- **No invented numbers.** Missing data is shown as *Data not yet available*. No estimates, no
73+ placeholder statistics, no composite "worst cancer" score.
74+
75+## What it refuses to do
76+
77+- It is **not a physician**: no diagnosis, no individual prognosis, no dosing, no treatment
78+ recommendation. Population statistics never predict an individual outcome and the UI says so.
79+- It does not rank cancers by an opaque composite; each ranking is one metric, one scope, one formula.
80+- It does not infer facts: no edge is created by a language model, no reason for a terminated trial is
81+ guessed, no indication is derived from an ATC class, no cancer is attached to an approval unless the
82+ official text names exactly one.
83+- It does not redistribute data it may not redistribute: every connector manifest is licence-reviewed
84+ before it goes live (IARC / GLOBOCAN stays under review, SEER awaits credentials).
85+
86+## The index in numbers
87+
88+Production database on 2026-09-11 (`SELECT count(*)` — live counts, never estimates):
89+
90+| Layer | Entity | Count |
91+|---|---|---:|
92+| Taxonomy | active malignant cancer entities (of 9 510 active entities) | 5 595 |
93+| Taxonomy | top-level site groups (ranking scope) | 36 |
94+| Genomics | genes (HGNC) | 45 170 |
95+| Genomics | variants (CIViC, ClinVar) | 417 000 |
96+| Genomics | curated CIViC evidence items | 11 984 |
97+| Biomarkers | canonical biomarkers (NCIt-verified) | 54 |
98+| Drugs | canonical drugs (CIViC, ChEMBL, openFDA, Health Canada, EMA) | 814 |
99+| Regulatory | approval records — US 1 570 · CA 1 534 · EU 459 | 3 563 |
100+| Trials | ClinicalTrials.gov oncology studies | 126 238 |
101+| Trials | registrant-entered study locations (1.19 M geocoded) | 1 212 144 |
102+| Literature | PubMed records linked to entities (plus 19 080 per-cancer count windows) | 4 287 |
103+| Epidemiology | US observations (CDC WONDER, U.S. Cancer Statistics), 1999–2024 | 10 674 |
104+| Graph | active knowledge edges with cancer context | 18 822 |
105+| Derived | trial-intelligence rows · drug-pipeline rows · research-gap components | 2 240 · 5 498 · 2 966 |
106+| Derived | country × cancer × phase trial-site aggregates | 11 805 |
107+| Rankings | current snapshots over 25 metric definitions | 875 |
108+| Provenance | source records · provenance rows | 629 889 · 106 263 |
109+| Sources | connectors registered (17 active, 1 under licence review, 1 awaiting credentials) | 19 |
110+| Storage | PostgreSQL 17 | 2.5 GB |
111+
112+## Product map
113+
114+| Area | Route(s) | What it shows |
115+|---|---|---|
116+| Home | `/` | Hero, live ticker (cancers, active trials, recruiting Phase III, approved drugs), US burden, data-explorer chart, trial intelligence, new approvals, rankings preview, fastest-rising incidence, gap ratios, curated evidence; aside: where trials recruit, graph teaser, taxonomy, rare spotlight, sources |
117+| Cancer profile | `/cancer/[slug]` + tabs `statistics` `survival` `genomics` `evidence` `drugs` `trials` `research` `rankings` `sources` | Header with codes and badges; observations by geography/year/sex with charts; cohort alteration frequencies with denominators; CIViC evidence with native levels; jurisdiction-aware approvals; trials with an intelligence strip; literature; "Why this rank?"; every source behind the page |
118+| Cancers, taxonomy | `/cancers`, `/taxonomy` | Filterable entity table (level, type, malignant, hematologic, pediatric, rare); tree browser across hierarchy types |
119+| Data explorer | `/explore`, `/explore/coverage`, `/api/export/epidemiology.csv` | Metric × cancers × geography × sex × age × years → comparable chart groups (metric, unit, geography, source, standard population, age never mixed), observations table, permalink, CSV with attribution, API call; coverage matrix |
120+| Countries | `/countries`, `/country/[slug]` | Latest-year summary, top cancers per metric, trends, clinical trial activity (sites by cancer and phase), the jurisdiction's approval records, sources |
121+| Compare | `/compare?ids=` | 2–4 cancers side by side (registry figures, counters, ranks) |
122+| Trials | `/trials`, `/trial/[nct]`, `/trials/intelligence`, `/trials/terminated`, `/trials/map`, `/api/export/trial-intelligence.csv` | Search and filters; study record with mappings, locations, publications; per-cancer trial metrics (growth, enrollment, sponsor and country concentration, termination share, trials per 1 000 deaths); failure tracking with registrant-stated reasons classified by explicit rules; Equal Earth choropleth of sites |
123+| Drugs | `/drugs`, `/drug/[slug]`, `/pipeline` | Canonical drugs with brands as aliases; identifiers (ATC, DIN, UNII, ChEMBL…), approvals by jurisdiction, development stage per cancer, evidence, trials; pipeline funnel per cancer |
124+| Approvals | `/approvals` | Dated feed of FDA, Health Canada and EMA records grouped by month, filters by authority, jurisdiction, cancer, status, year |
125+| Genes, variants | `/genes`, `/gene/[symbol]`, `/variant/[slug]` | HGNC genes; evidence by cancer; variants with ClinVar interpretations; cohort frequencies; publications |
126+| Biomarkers | `/biomarkers`, `/biomarker/[slug]` | 54 curated biomarkers with NCIt codes verified against EVS; derived cancers, drugs, approvals (tumour-agnostic from real rows), trials, publications |
127+| Rankings | `/rankings`, `/rankings/[metric]` | One metric, one scope, one formula version per table; lineage on every row; CSV export |
128+| Research gap | `/research-gap`, `/api/export/research-gap.csv` | Death share vs trial and publication shares, log₂ gap ratios, per-1 000-deaths intensities, log-log scatter |
129+| Knowledge graph | `/graph?focus=type:ref` | Contextual neighbourhood (source-native edges vs derived registry links), cancer → gene → variant → drug → approval → trials paths, accessible edge table |
130+| Pulse, history | `/pulse`, `/year/[year]`, `/data-updates` | What changed (approvals, new recruiting Phase III, registration momentum, ranking moves, publications, dataset refreshes); a year in cancer 1999–now; connector state and ingestion log |
131+| Sources, method | `/sources`, `/source/[slug]`, `/methodology`, `/methodology/trial-map`, `/trust`, `/about`, `/developers`, `/data` | Licence registry and connector health; every formula and threshold; policies; team and hosting; API guide; redistributable downloads |
132+| Search | ⌘K / Ctrl+K anywhere, `/search` | Cancer, gene, variant, drug, trial, publication, source — exact › alias › prefix › fuzzy |
133+| Admin | `/admin/*` (token) | Connectors, runs, unresolved labels, rankings, trace |
134+
135+Every page has a light and a dark theme (toggle in the header, `prefers-color-scheme` by default),
136+a canonical URL, structured metadata and a server-rendered Open Graph / Twitter image.
137+
138+## Architecture
139+
140+```
141+ ┌──────────────────────── sources (19 connectors) ────────────────────────┐
142+ │ NCIt EVS · OncoTree · HGNC · CIViC · ClinVar · GDC · cBioPortal · MeSH │
143+ │ ClinicalTrials.gov · PubMed · CDC WONDER · CDC USCS · ChEMBL · openFDA │
144+ │ Health Canada DPD · EMA · (IARC GLOBOCAN: review) · (SEER: credentials) │
145+ └───────────────┬───────────────────────────────────────────────────────────┘
146+ │ HTTP / bulk files, rate-limited, restartable cursors
147+ ▼
148+ workers/main.ts ──► packages/connectors (SDK: manifest · HttpClient · RawLake · RunContext)
149+ pg-boss scheduler │ RAW → data/raw/{source}/{date}/{entity}/*.jsonl.gz + source_records
150+ cron per manifest │ NORMALIZED → validators, schema-drift stats, unresolved_labels
151+ counters 06:00 UTC │ CANONICAL → cancers · genes · variants · drugs · trials · publications
152+ intel 06:15 UTC │ observations · approvals · knowledge_edges · provenance
153+ rank 06:30 UTC ▼
154+ PostgreSQL 17 (+ pg_trgm, unaccent, pgvector) — Drizzle schema, snake_case
155+ │
156+ packages/ranking: counters → intelligence → rankings (DERIVED / RANKED)
157+ │
158+ ┌────────────────────┴────────────────────┐
159+ ▼ ▼
160+ apps/api Fastify 5 · /v1 · zod · OpenAPI apps/web Next.js 16 (webpack) · server components
161+ envelope { data, sources, dataRelease } Tailwind v4 tokens · pure SVG charts · dark mode
162+ rate limits · API keys · x-request-id /api/v1/* proxied to the API · ISR caching
163+ └────────────────────┬────────────────────┘
164+ ▼
165+ MacLustr node M4M64b · PM2 (web :8250, api :8251, worker, backup)
166+ MacLustr Tunnel (WireGuard + Caddy on BHS64) → https://www.cancerindex.io
167+```
168+
169+**Principles baked into the code**
170+
171+- *Reconciliation before ingestion.* Labels map to entities by shared identifier, then curated alias,
172+ then normalized string; every mapping stores a `match_type` (`EXACT_IDENTIFIER`, `CURATED_EXACT`,
173+ `ONTOLOGY_EXACT`, `CURATED_BROADER`, `ALIAS`, `PROBABILISTIC`, `UNRESOLVED`). Unknown labels go to
174+ `unresolved_labels`, never to `/dev/null`.
175+- *Time-aware observations.* A value for year X never overwrites year Y; multi-year aggregates keep
176+ both bounds; rates keep their standard population.
177+- *Idempotent, restartable connectors.* Payload hashes, cursor checkpoints every 2 000 records or 60 s,
178+ SIGTERM-safe aborts, anomaly guard refusing destructive updates when a source shrinks by more than half.
179+- *Deterministic derived layers.* Counters, intelligence tables and rankings are full rebuilds in one
180+ transaction; snapshots are immutable and keep `previous_rank` for change explanation.
181+
182+## Repository layout
183+
184+```
185+apps/
186+ web/ Next.js 16 app (React 19, server components, Tailwind v4, --webpack)
187+ src/app/ routes (see Product map), opengraph-image.tsx per entity, icon.svg, manifest.ts
188+ src/components/ ui · charts (SVG) · cancer tabs · graph · explorer · country · home modules · layout
189+ src/lib/ db · format · queries/* (server-only SQL) · graph-model · explorer-* · og.tsx · site.ts
190+ src/assets/fonts/ WOFF copies of Newsreader / Inter / IBM Plex Mono for server-side images
191+ public/brand/ logo.svg · logo-mark.svg · logo-mark-dark.svg · PNG icons
192+ qa/smoke.mjs read-only smoke suite (44 routes, weight caps, mobile overflow, console errors)
193+ api/ Fastify 5 public API (/v1), zod schemas, OpenAPI at /v1/docs, admin routes
194+packages/
195+ shared/ CI-XXX-00000001 ids, provenance types, normalization, logger, env
196+ database/ Drizzle schema (12 files, ~50 tables), migrations 0000–0002, seeds (metrics,
197+ geographies, biomarkers), alerts, ids
198+ ontology/ qualifier rules, CancerResolver (alias/code reconciliation), TOP_LEVEL_CANCERS
199+ connectors/ SDK (manifest · HttpClient · RawLake · RunContext · validators · doctor) and
200+ connectors/<id>/{manifest.ts,index.ts,normalize.ts,fixtures/,*.test.ts}
201+ ranking/ counters · intelligence (trial-intelligence, trial-sites, drug-pipeline,
202+ drug-duplicates, research-gap) · engine (snapshots) · trace · country-codes
203+workers/ pg-boss scheduler and job handlers
204+scripts/ci.ts operator CLI (`pnpm cix …`)
205+deploy/ mld manifest, first-run bootstrap, backup / restore scripts
206+docs/ ARCHITECTURE · DATA-MODEL · METHODOLOGY · API · SECURITY · AI · source-policy ·
207+ schema-changes-ops · adr/ (6 ADRs) · connectors/ (18 pages) · methodology/ (7 pages)
208+CLAUDE.md operational rules every contributor (human or agent) follows
209+```
210+
211+## Data model
212+
213+Source of truth: `packages/database/src/schema/*.ts` (documented in `docs/DATA-MODEL.md`). Public
214+identifiers are `CI-<NS>-00000001` minted per namespace (`CAN`, `GENE`, `VAR`, `DRUG`, `TRIAL`, `PUB`,
215+`BIO`, `STUDY`, `METRIC`, `SOURCE`, `GEO`, `PROV`…) — never database integers.
216+
217+| Group | Tables | Notes |
218+|---|---|---|
219+| Registry & operations | `sources`, `ingest_runs`, `connector_cursors`, `connector_field_stats`, `source_records`, `provenance`, `unresolved_labels`, `change_events`, `audit_log`, `entity_merges`, `system_alerts`, `api_keys`, `id_sequences` | licence status per source; one row per run with counters and log; raw-lake index with payload hashes; merge queue (proposed, never automatic) |
220+| Cancer ontology | `cancers`, `cancer_aliases`, `cancer_hierarchy` (ncit · oncotree · anatomical · …), `cancer_codes`, `anatomical_sites`, `cancer_anatomy`, `geographies`, `cohort_definitions` | one row per disease concept, aliases and codes searchable, several trees coexist |
221+| Genomics | `genes`, `gene_aliases`, `variants`, `variant_aliases`, `variant_clinical_significance`, `biomarkers`, `genomic_cohorts`, `cancer_gene_frequencies`, `entity_embeddings` | coordinates carry their assembly; frequencies carry their denominator |
222+| Drugs & regulatory | `drugs`, `drug_aliases`, `drug_codes`, `treatment_regimens`, `drug_approvals` | brands are aliases; approvals are per jurisdiction, authority, application/DIN/EMA number, dated, with status and verbatim source status |
223+| Trials | `clinical_trials`, `trial_conditions`, `trial_interventions`, `trial_locations`, `trial_pulse` | conditions and interventions reconciled with `match_type`; 1.2 M locations |
224+| Literature | `publications`, `publication_entity_edges`, `literature_counts` | the exact PubMed query is stored with every count |
225+| Evidence & graph | `knowledge_edges`, `civic_evidence_items`, `risk_factors` | edges carry cancer context, direction, native evidence level, category, provenance ids, status (`active` · `superseded`) |
226+| Epidemiology | `epidemiology_observations`, `survival_observations` | metric, unit, sex, age group, year(s), CI, standard population, estimate type, site definition |
227+| Derived | `entity_counters`, `trial_intelligence`, `trial_site_country_counts`, `drug_pipeline`, `research_gap_components`, `metric_definitions`, `ranking_snapshots`, `rankings`, `ai_answers` | every row has a `formula_version` and `inputs` |
228+
229+Schema changes are migrations generated with `pnpm db:generate` (one per integration) and applied by
230+`pnpm db:migrate`, which also creates the extensions and the trigram / GIN performance indexes.
231+
232+## Connectors
233+
234+Each connector is a class with a manifest (licence, terms review date, documentation verification
235+date, rate limits, schedule, anomaly guard), a health check and a `sync()`; it is tested against
236+sanitized fixtures (normal, empty, pagination, rate-limit, server error, malformed) and documented in
237+`docs/connectors/<id>.md`. `pnpm cix sources:sync` seeds manifests into `sources`; `/sources` shows them.
238+
239+| Connector | Organization | Category | Licence status | What it feeds |
240+|---|---|---|---|---|
241+| `ncit-evs` | NCI EVS | terminology | approved (CC BY 4.0) | cancers, aliases, codes, hierarchy — the ontology backbone |
242+| `oncotree` | MSKCC | terminology | approved (CC BY 4.0) | OncoTree codes and tree mapped onto NCIt concepts |
243+| `mesh` | NLM | terminology | approved | MeSH headings as aliases (PubMed queries, EMA therapeutic areas) |
244+| `hgnc` | HGNC | genes | approved (CC0) | 45 k gene records and aliases |
245+| `civic` | CIViC | evidence | approved (CC0) | evidence items, variants, therapies (mints drugs), edges |
246+| `clinvar` | NCBI | variants | approved (public domain) | 417 k variants with interpretations |
247+| `gdc` | NCI GDC | genomics | approved (open tier) | TCGA cohorts and gene alteration frequencies |
248+| `cbioportal` | cBioPortal | genomics | approved | additional cohorts and frequencies |
249+| `clinicaltrials` | ClinicalTrials.gov | trials | approved (public domain) | 126 k studies, conditions, interventions, locations, `trial_pulse`; intervention → drug reconciliation |
250+| `pubmed` | NLM | literature | approved | per-cancer literature counts (query stored) and linked records |
251+| `cdc-wonder` | CDC | epidemiology | approved | US mortality 1999–2024 |
252+| `cdc-uscs` | CDC | epidemiology | approved | US incidence and mortality (USCS), sex-specific handling |
253+| `chembl` | EMBL-EBI | drugs | approved (CC BY-SA) | drug enrichment (ids, mechanism, targets) |
254+| `openfda` | US FDA | regulatory | approved (CC0) | Drugs@FDA applications and labels → US approval rows, indication-text cancer mapping, edges |
255+| `health-canada-dpd` | Health Canada | regulatory | approved (OGL Canada) | Drug Product Database (ATC L01/L02/L03/V10) → DIN-level Canadian records, drug minting, codes |
256+| `ema` | European Medicines Agency | regulatory | approved (attribution) | medicines data xlsx → EU authorisations with dates and statuses, indication mapping, edges |
257+| `seer-explorer` | NCI SEER | epidemiology | approved | explorer datasets (no schedule) |
258+| `seer` | NCI SEER API | epidemiology | awaiting credentials | survival and rates once `SEER_API_KEY` is set |
259+| `iarc-globocan` | IARC | epidemiology | **review** | global burden — gated by `IARC_TERMS_ACCEPTED_BY`; nothing is ingested without a human decision |
260+
261+Run order for a fresh database: `deploy/first-run.sh` (terminology → genes → evidence/genomics/variants
262+→ trials → literature → epidemiology), then the regulatory connectors, `pnpm cix reconcile-drugs`,
263+`pnpm cix counters`, `pnpm cix intel`, `pnpm cix rank`.
264+
265+## Derived intelligence layer
266+
267+Recomputed daily by the worker (`maintenance.intel`, between counters and rankings) or with
268+`pnpm cix intel` (≈ 50 s on the production database). Every row stores `formula_version` and `inputs`.
269+Methods: `docs/methodology/*.md` and `/methodology`.
270+
271+| Module | Table | Formula version | Highlights |
272+|---|---|---|---|
273+| Clinical trial intelligence | `trial_intelligence` | `ci-trial-intel-v1`, stop reasons `ci-stop-reasons-v1` | per cancer (top and all levels, descendants included): total / active / recruiting / Phase I–IV counts, growth of first-posted studies (12 m vs prior 12 m, ≥ 20), enrollment mean/median, sponsor and country HHI, industry and US shares, termination share (≥ 2010, ≥ 30 terminal), stop-reason breakdown by explicit keyword rules, trials per 1 000 deaths / per 100 k cases (US, latest year, deaths ≥ 100) |
274+| Trial map | `trial_site_country_counts` | `ci-trial-sites-v1` | sites and studies per country × (all / each top-level cancer) × (any / each phase) × (all / recruiting only); ISO 3166-1 alpha-3 mapping with explicit unmapped historical names |
275+| Drug pipeline | `drug_pipeline`, `entity_merges` | `ci-drug-pipeline-v1` | stage per drug and per drug × top-level cancer (approved › withdrawn › highest registry phase › phase not stated); salt-form duplicates proposed to the merge queue |
276+| Research Gap Index | `research_gap_components` + 4 ranking metrics | `ci-research-gap-components-v1` | for each burden scope: death share vs active-trial and publication shares over the eligible set (deaths ≥ 100), `log₂` gap ratios, per-1 000-deaths intensities |
277+| Trial → drug reconciliation | `trial_interventions.drug_id` | (in the ClinicalTrials connector) | exact alias › salt/dose/label-stripped alias › probabilistic head; shared aliases resolved by rule (own generic name › brand › base molecule) or left unresolved — 78 956 rows linked |
278+
279+## Rankings and metrics
280+
281+25 metric definitions live in `metric_definitions` (seeded from `packages/database/src/seed-data/metrics.ts`)
282+and are rendered live on `/methodology#metrics`. A ranking snapshot is one metric × one scope
283+(`geo=USA|sex=all|age=all|year=2024|level=top`) × one formula version; rows keep rank, percentile,
284+confidence, previous rank and the exact inputs.
285+
286+| Category | Metrics |
287+|---|---|
288+| Burden | `incidence_count`, `mortality_count`, `as_incidence_rate`, `as_mortality_rate` |
289+| Lethality | `mortality_incidence_ratio`, `five_year_survival` (awaits survival observations) |
290+| Clinical research | `active_trials`, `recruiting_trials`, `phase3_trials`, `phase3_recruiting_trials`, `trial_termination_share`, `sponsor_concentration` |
291+| Research activity & trends | `publications_5y`, `publications_12m`, `publication_growth`, `trial_growth_yoy` |
292+| Molecular knowledge | `curated_evidence_items`, `associated_genes`, `genomic_cohorts` |
293+| Unmet need | `trial_gap`, `research_gap` (percentile-based), `trial_gap_ratio`, `research_gap_ratio` (share-based, log₂), `trials_per_1000_deaths`, `publications_per_1000_deaths` |
294+
295+Burden, lethality and gap metrics exist only for scopes with licensed observations (currently the
296+United States, per year and sex, top level). No composite score is published (ADR-006).
297+
298+## Public API
299+
300+Base URL `https://www.cancerindex.io/api/v1` (proxied to Fastify). Read-only JSON. Every response is an
301+envelope `{ data, sources, dataRelease, generatedAt, total?, limit?, offset?, hasMore? }` where `sources`
302+lists the upstream sources, licences and attributions behind the returned data. Swagger UI at `/api/v1/docs`,
303+OpenAPI 3.1 at `/api/v1/openapi.json`. Rate limits per IP; an optional bearer API key raises them.
304+
305+| Endpoints | Purpose |
306+|---|---|
307+| `/cancers`, `/cancers/:id`, `/cancers/:id/{statistics,survival,genes,variants,drugs,trials,publications}` | taxonomy, observations, evidence, approvals, trials per cancer (descendants included) |
308+| `/genes`, `/genes/:symbol`, `/variants/:id`, `/drugs`, `/drugs/:id`, `/biomarkers`, `/biomarkers/:slug` | entity details with derived links |
309+| `/trials`, `/trials/:nct`, `/trials/intelligence[/:cancer]`, `/trials/terminated`, `/trials/sites` | search, study record, per-cancer intelligence, failures with classified reasons, country/city site aggregates |
310+| `/epidemiology`, `/epidemiology/coverage`, `/epidemiology/metrics` | time-aware observations with provenance (Data explorer) |
311+| `/approvals`, `/approvals/recent`, `/pipeline`, `/pipeline/summary` | jurisdiction-aware regulatory records and development stages |
312+| `/research-gap`, `/research-gap/scopes` | gap components per burden scope |
313+| `/graph/:type/:id`, `/graph/cancer/:id/paths` | knowledge-graph neighbourhoods and chains |
314+| `/rankings/metrics`, `/rankings`, `/rankings/:metric/:cancerId/explain` | catalogue, snapshots, "Why this rank?" |
315+| `/search`, `/sources`, `/sources/:slug`, `/stats`, `/changes`, `/healthz` | search, registry and health, live counts, change events |
316+| `/admin/*` (`x-admin-token`) | connectors, runs, unresolved labels, jobs, trace, audit |
317+
318+Details and examples: `docs/API.md`.
319+
320+## Operator CLI
20 321
21 ``` 322 ```
22−apps/web Next.js 16 public site (port 8250), proxies /api/v1/* to the API 323+pnpm cix connectors list connectors, licence status, health, last success
23−apps/api Fastify 5 public API /v1 (port 8251), OpenAPI at /v1/docs 324+pnpm cix run <id> [--mode full|incremental|backfill|dry_run] [--max-records N] [--max-minutes M] [--reset-cursor]
24−workers/ pg-boss scheduler: connector cron, counters, rankings, health probes 325+pnpm cix run-all [--max-minutes M] every active connector in registry order
25−packages/ shared · database (Drizzle) · ontology · connectors (SDK + connectors) · ranking 326+pnpm cix health <id> source liveness probe
26−scripts/ci.ts operator CLI (pnpm cix …) 327+pnpm cix sources:sync manifests → sources table
27−deploy/ MacLustr mld manifest, first-run bootstrap 328+pnpm cix reconcile-drugs [--remap] trial interventions → canonical drugs
28−docs/ architecture, data model, methodology, API, security, AI policy, ADRs, connector docs 329+pnpm cix counters rebuild entity_counters
330+pnpm cix intel trial intelligence · trial sites · drug pipeline · research gap
331+pnpm cix rank recompute every ranking snapshot
332+pnpm cix stats | trace <type> <id> | doctor [--no-disk] | alerts [ack|resolve <id>]
29 ``` 333 ```
30 334
31−Start with `CLAUDE.md` (operational rules), then `docs/ARCHITECTURE.md`, `docs/DATA-MODEL.md` and 335+`--mode backfill` replays a connector's raw lake without HTTP (used after a mapping-rule change); `dry_run`
32−`docs/METHODOLOGY.md`. 336+never writes and never moves a cursor.
33 337
34−## Quickstart 338+## Local development
35 339
36−Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with `pg_trgm`, `unaccent` and `vector`. 340+Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with `pg_trgm`, `unaccent` and `vector`; `rsvg-convert`
341+and Python Pillow only if you regenerate brand rasters.
37 342
38 ```bash 343 ```bash
39−createdb cancerindex && cp .env.example .env # set NCBI_EMAIL, ADMIN_TOKEN 344+createdb cancerindex && cp .env.example .env # set NCBI_EMAIL, ADMIN_TOKEN
40 pnpm install 345 pnpm install
41 pnpm db:migrate && pnpm db:seed && pnpm cix sources:sync 346 pnpm db:migrate && pnpm db:seed && pnpm cix sources:sync
42−pnpm cix run oncotree --mode dry_run # smoke test against the live API (no writes) 347+pnpm cix run oncotree --mode dry_run # smoke against the live API, no writes
43−pnpm cix run oncotree # first real ingest 348+bash deploy/first-run.sh # ordered first ingestion (hours; restartable)
44−pnpm cix counters && pnpm cix rank 349+pnpm cix run health-canada-dpd && pnpm cix run ema && pnpm cix run openfda
45−pnpm dev:api & # http://127.0.0.1:8251/v1/docs 350+pnpm cix reconcile-drugs && pnpm cix counters && pnpm cix intel && pnpm cix rank
46−pnpm worker & # schedules from connector manifests 351+pnpm dev:api & # http://127.0.0.1:8251/v1/docs
47−pnpm dev:web # http://localhost:8250 352+pnpm worker & # schedules from connector manifests
353+pnpm dev:web # http://localhost:8250
48 ``` 354 ```
49 355
50−Verify: `curl -s localhost:8251/healthz`, `curl -s "localhost:8251/v1/search?q=glio"`. 356+A faster path for UI work is to restore a production dump (`bash deploy/restore.sh <dump> --target
357+cancerindex_prodcopy`, then rename databases) and run `pnpm db:migrate && pnpm db:seed`.
358+Note that Next.js allows a single `next dev` per app directory (`.next/dev/lock`).
51 359
52−## CLI 360+## Configuration
53 361
54−``` 362+`.env` at the repository root (read by the API, the worker and the web app).
55−pnpm cix connectors list connectors, license status, health 363+
56−pnpm cix run <id> [--mode full|incremental|dry_run] [--max-records N] [--max-minutes M] [--reset-cursor] 364+| Variable | Purpose |
57−pnpm cix run-all [--max-minutes M] every active connector in registry order 365+|---|---|
58−pnpm cix health <id> source liveness probe 366+| `DATABASE_URL`, `DB_POOL_MAX` | PostgreSQL connection |
59−pnpm cix sources:sync manifests → sources table 367+| `WEB_PORT` (8250), `API_PORT` (8251), `API_HOST`, `CI_API_URL` | ports and the API URL the web app proxies to |
60−pnpm cix counters rebuild entity_counters 368+| `NEXT_PUBLIC_SITE_URL` | canonical site URL (metadata, sitemaps, share images) |
61−pnpm cix rank recompute ranking snapshots 369+| `CI_DATA_DIR` | raw data lake root (`data/raw`, `data/cache`) |
62−pnpm cix stats table counts 370+| `ADMIN_TOKEN` | `/admin` console and `/v1/admin/*` |
63−pnpm cix trace <table> <id> lineage of a value 371+| `NCBI_TOOL`, `NCBI_EMAIL`, `NCBI_API_KEY` | E-utilities (PubMed, ClinVar); the key raises the limit to 10 req/s |
64− 372+| `SEER_API_KEY` | unlocks the SEER connector |
65−tsx workers/cli.ts run <id> | counters [--then-rank] | rank | health | schedules | queues 373+| `OPENFDA_API_KEY` | raises openFDA from 1 000 to 120 000 requests/day |
66−pnpm --filter @cancerindex/api create-key -- --label "Lab" --tier research --rpm 600 374+| `IARC_TERMS_ACCEPTED_BY` | human gate for GLOBOCAN (stays unset until terms are settled) |
67−``` 375+| `WORKER_CONCURRENCY`, `CI_MAX_RUN_MINUTES`, `LOG_LEVEL` | worker tuning |
376+| `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `OPENAI_BASE_URL` | reserved for the future `/ask` layer (not enabled) |
377+
378+## Quality: tests, QA, accessibility, performance
68 379
69−## Quality gates 380+- `pnpm typecheck` — strict TypeScript across the 8 workspaces.
381+- `pnpm test` — 528 vitest tests: connector fixtures (every connector), SDK (lake, run, checkpoints),
382+ formulas (shares, log ratios, HHI, growth, termination share, stop-reason rules, pipeline stages,
383+ country codes, map scales, projections, explorer comparability, CSV escaping, graph layout), API
384+ smoke tests against the local database (skipped when unreachable).
385+- `pnpm --filter @cancerindex/web qa` / `qa:prod` — read-only smoke suite over 44 routes: HTTP 200,
386+ expected text, per-route weight caps, no "Data not yet available" where data must exist, no horizontal
387+ overflow at 390 px, console-error scan (Playwright when available).
388+- Accessibility: keyboard navigation, focus rings, colour never the only carrier (badges carry text,
389+ charts have legends and data tables, maps have equivalent tables), `aria` labels on SVG figures.
390+- Performance: ISR caching (`revalidate` per page, `Cache-Control` on public routes), materialized
391+ derived tables, GIN / trigram indexes, `sql.param` for array parameters, request budgets on heavy
392+ pages (graph groups, city layers).
393+
394+## Deployment and operations
395+
396+Production runs on the MacLustr cluster (node **M4M64b**: Postgres 17 + pgvector, PM2) behind the
397+MacLustr Tunnel (WireGuard + Caddy on the BHS64 gateway) at https://www.cancerindex.io.
70 398
71 ```bash 399 ```bash
72−pnpm typecheck # every workspace 400+# from the laptop — gateway M1M32 orchestrates (mld)
73−pnpm -r test # vitest; connector tests use fixtures, API smoke tests skip without a DB 401+rsync -a --exclude '/data/' --exclude node_modules --exclude .next --exclude .git --exclude '.env*' ./ /tmp/cancerindex-deploy/
402+mld stage /tmp/cancerindex-deploy cancerindex
403+mld deploy cancerindex --node M4M64b # install, migrate, seed, sources:sync, build, PM2 restart, health checks
74 ``` 404 ```
75 405
76−## Deployment 406+| Process (PM2) | Role |
407+|---|---|
408+| `cancerindex-web` (:8250) | `next start` |
409+| `cancerindex-api` (:8251) | Fastify API |
410+| `cancerindex-worker` | pg-boss: connector schedules, `maintenance.counters` 06:00 UTC → `maintenance.intel` 06:15 → `maintenance.rank` 06:30, hourly health probes and alerts |
411+| `cancerindex-backup` | nightly `pg_dump` 05:20, 14 daily + 8 weekly retained (`deploy/backup.sh`, `restore.sh`) |
412+
413+Operational visibility: `/data-updates` (connector state, ingestion log), `/admin` (runs, unresolved
414+labels, alerts), `pnpm cix doctor`, `system_alerts`. Deploy manifest: `deploy/mld-manifest.cancerindex.json`
415+(note the excludes are anchored — `/coverage/`, `/data/` — so route folders with those names are kept).
416+
417+## Brand
418+
419+<p>
420+ <img src="apps/web/public/brand/logo-mark.svg" alt="CancerIndex mark" width="72" align="left" hspace="12">
421+ The mark reads <strong>CI</strong>: an open ring — a cell with its nucleus, the "C" — and a graduated
422+ index scale in the accent teal, the "I". It is drawn once in SVG and used everywhere: the header and
423+ footer (<code>BrandMark</code>, theme-aware through <code>currentColor</code> and the accent token),
424+ the favicon (<code>app/icon.svg</code>, <code>favicon.ico</code>, <code>apple-icon.png</code>), the web
425+ manifest and the share images.
426+</p>
427+<br clear="all">
428+
429+| Asset | Path |
430+|---|---|
431+| Logo with wordmark | `apps/web/public/brand/logo.svg` |
432+| Mark (light / dark backgrounds) | `apps/web/public/brand/logo-mark.svg`, `logo-mark-dark.svg` (+ PNG 512) |
433+| Favicon and app icons | `apps/web/src/app/icon.svg`, `favicon.ico`, `apple-icon.png`, `public/brand/icon-{192,512}.png` |
434+| Share images | `apps/web/src/lib/og.tsx` (next/og, 1200 × 630, site fonts); `app/opengraph-image.tsx` (live counts) and `opengraph-image.tsx` under `cancer/[slug]`, `gene/[symbol]`, `drug/[slug]`, `biomarker/[slug]`, `trial/[nct]` |
435+
436+Colours: paper `#fafaf7`, ink `#1c1c1a`, accent teal `#0f5f63` (dark theme `#6cc3c6`). Type: Newsreader
437+(display), Inter (UI), IBM Plex Mono (identifiers).
438+
439+## Design language
440+
441+Scientific, editorial, institutional. Off-white paper, charcoal ink, one restrained teal accent, thin
442+rules, tabular numerals, dense tables, sparklines and small multiples rather than decorative charts. All
443+charts are server-rendered SVG with a legend, units, the dataset year and a source line; every value shows
444+a claim badge (observed · published · curated · regulatory · computed) and a source badge whose popover
445+gives dataset, version, retrieval date and URL. Mobile is first-class (tables scroll inside their wrap; no
446+horizontal overflow at 390 px). Light and dark themes share one token set (`--color-*`, `--color-series-*`).
447+
448+## Known gotchas
449+
450+- A JS array inside a Drizzle template — `` sql`x = ANY(${arr}::text[])` `` — becomes a `($1,$2,…)` tuple
451+ and fails; use `sql.param(arr)` or `sql.raw` for constant lists.
452+- Columns declared with the `updatedAt()` helper are physically `updated_at` even when named `computedAt`.
453+- `db.execute<T>` needs a `type` alias, not an `interface`, for `T`.
454+- A prop named `ref` cannot be passed to a server component (React reserves it) — the page fails in
455+ production with minified error #441; use another name (`entityRef`).
456+- SVG `<title>` must be a single text node or hydration fails; `array_agg` over empty arrays errors — use
457+ `unnest` / `string_agg`.
458+- `opengraph-image.tsx` cannot live inside an optional catch-all segment (`[[...tab]]`); place it one
459+ level up.
460+- `ranking_snapshots.scope_key` has no source dimension: two sources for the same geography/year/sex
461+ overwrite each other's current snapshot, so a preferred source is chosen per scope.
462+- A shared alias between a concept and its descendant ("breast cancer") must resolve to the **broadest**
463+ concept when building a dictionary; the narrowest-of-lineage rule is only for sentence-level mentions.
464+- OncoTree's WAF rejects any User-Agent containing a URL; NCBI throttles at 3 req/s without a key;
465+ openFDA allows 1 000 requests/day without a key; the EMA xlsx has 39 columns and header row 9.
466+- mld `sync_excludes` patterns must be anchored (`/data/`, `/coverage/`), otherwise route folders with
467+ the same name are dropped from the deploy.
77 468
78−Production runs on the MacLustr cluster through `mld` (gateway M1M32): manifest 469+## Roadmap
79−`deploy/mld-manifest.cancerindex.json`, `mld stage <dir> cancerindex`, `mld deploy cancerindex`,
80−then `deploy/first-run.sh` for the ordered first ingestion. PM2 processes `cancerindex-web`,
81−`cancerindex-api`, `cancerindex-worker`; ngrok `www.cancerindex.io` → 8250. See `deploy/README.md`.
82 470
83−## Principles (short form) 471+- Global burden once IARC / GLOBOCAN terms are settled (human decision) and SEER credentials are set;
472+ survival observations and the `five_year_survival` metric follow.
473+- MHRA and TGA regulatory connectors; populating `drug_approvals.biomarker_ids` from label text.
474+- Risk factors and attributable burden, screening and guideline registries (metadata and links only,
475+ with change detection), hereditary syndromes, pediatric and rare-cancer views.
476+- Researcher, institution and funding entities (OpenAlex, NIH RePORTER, CIHR) for the funding-gap view.
477+- Watchlists, alerts and a grounded, cited `/ask` layer that summarizes indexed sources only.
478+- Lighter default views for the heaviest pages (`/graph`, biomarker pages with hundreds of approvals).
84 479
85−1. No scientific number without provenance; derived values carry a formula version and inputs. 480+## Licensing, attribution and contact
86−2. Layers stay separable: RAW → NORMALIZED → CANONICAL → DERIVED → RANKED → AI.
87−3. Never fake data: missing is missing ("Data not yet available"), never zero.
88−4. Identifiers are first-class: every upstream id is kept; public ids are `CI-<NS>-00000001`.
89−5. Reconciliation before ingestion; unknown labels go to a curation queue, never dropped.
90−6. Time-aware observations: a new year is a new row.
91−7. Every edge has context: cancer, direction, evidence level, provenance.
92−8. Licensing gate on every source; scientific safety labels are never merged.
93 481
94−## License and attribution 482+- **Code**: private repository (spbgit `cancerindex.git`).
483+- **Derived data** (rankings, intelligence tables, CSV exports): © CancerIndex, CC BY 4.0, with
484+ attribution rows naming the underlying providers.
485+- **Source data** remains under the licence of each provider — see `/sources` and the `sources` array of
486+ every API response (NCIt CC BY 4.0, OncoTree CC BY 4.0, HGNC CC0, CIViC CC0, openFDA CC0, Health Canada
487+ OGL, EMA with acknowledgement, ClinVar / PubMed / ClinicalTrials.gov / CDC public domain, GDC open tier,
488+ ChEMBL CC BY-SA).
489+- **Not medical advice.** CancerIndex provides research and educational information only.
95 490
96−Code: private (Groupe/SPB). Data: each source keeps its own license and attribution, surfaced in 491+Built and maintained by **Simon-Pierre Boucher** — contact **contact@spboucher.ai** — hosted on
97−every API response (`sources`) and on `/sources`. See `docs/source-policy.md`. 492+**MacLustr** (<https://www.maclustr.io>).
98 493
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere