spb/cancerindex

1# CancerIndex — Methodology23This is the public methodology of CancerIndex.io. Every number shown on the site or returned by the4API is either an **observation** imported from a named source with its provenance, or a **computed**5value whose formula and version are listed here. CancerIndex is not a physician, not a diagnostic6tool and gives no treatment recommendations. Population statistics describe groups of people7diagnosed in the past; they never predict an individual outcome.89## 1. Scientific safety labels1011Every value carries one of these categories and the categories are never merged:1213| Label | Meaning | Where it comes from |14|---|---|---|15| `observed_data` | Registry or database observation (counts, rates, cohort frequencies) | `epidemiology_observations`, `survival_observations`, `cancer_gene_frequencies`, trial records |16| `published_evidence` | A publication or a fact extracted from one | `publications`, `publication_entity_edges` |17| `curated_evidence` | Expert-curated assertion (CIViC, ClinVar) with source-native level | `civic_evidence_items`, `variant_clinical_significance` |18| `regulatory_status` | Jurisdiction-specific approval status | `drug_approvals` |19| `clinical_guideline` | Guideline statements (not ingested in Phase 1) | — |20| `computed_metric` | Derived by CancerIndex with a versioned formula | `entity_counters`, `literature_counts`, `rankings` |21| `ai_generated_synthesis` | Model-written text grounded in the above (not shipped in Phase 1) | `ai_answers` |2223## 2. Metric catalog2425Metric definitions are seeded from `packages/database/src/seed-data/metrics.ts` into26`metric_definitions` and served at `GET /v1/rankings/metrics`. The formula text below is the exact27text stored with each definition; `formula_version` changes whenever the computation changes, and28snapshots computed with different versions are never compared.2930### Burden (observations)3132| Slug | Name | Formula | Version | Unit | Eligibility |33|---|---|---|---|---|---|34| `incidence_count` | Annual new cases | `epidemiology_observations.value WHERE metric = incidence_count` | `ci-incidence-count-v1` | count | requires incidence_count; top level |35| `mortality_count` | Annual deaths | `epidemiology_observations.value WHERE metric = mortality_count` | `ci-mortality-count-v1` | count | requires mortality_count; top level |36| `as_incidence_rate` | Age-standardized incidence rate | `epidemiology_observations.value WHERE metric = as_incidence_rate (standard_population fixed per scope)` | `ci-asir-v1` | per 100 000 | requires as_incidence_rate; top level |37| `as_mortality_rate` | Age-standardized mortality rate | `epidemiology_observations.value WHERE metric = as_mortality_rate (standard_population fixed per scope)` | `ci-asmr-v1` | per 100 000 | requires as_mortality_rate; top level |3839Sources: CDC WONDER (US), SEER (US, awaiting credentials), IARC GLOBOCAN (global, license review).40Registry counts are `observed`; national/global figures may be `estimated` and are labeled as such.41Rates standardized to different standard populations are never compared in one ranking.4243### Lethality4445| Slug | Name | Formula | Version | Unit | Eligibility |46|---|---|---|---|---|---|47| `mortality_incidence_ratio` | Mortality-to-incidence ratio | `mortality_count / incidence_count (same geography, year, sex, source family)` | `ci-mir-v1` | ratio | both counts, incidence ≥ 100; top level |48| `five_year_survival` | 5-year relative survival | `survival_observations.probability WHERE survival_type = relative AND duration_months = 60 AND stage IS NULL` | `ci-5ys-v1` | probability | cohort ≥ 50; top level; source SEER |4950The mortality-to-incidence ratio is a crude proxy of lethality: it is not a survival probability and51is affected by incidence trends and registration completeness. Confidence is `HIGH` when incidence52≥ 1000, otherwise `MEDIUM`.5354### Clinical research (ClinicalTrials.gov)5556| Slug | Name | Formula | Version | Unit |57|---|---|---|---|---|58| `active_trials` | Active clinical trials | `COUNT(DISTINCT trial) FROM trial_conditions JOIN clinical_trials WHERE cancer_id IN descendants(cancer) AND study_type = INTERVENTIONAL AND overall_status IN (RECRUITING, NOT_YET_RECRUITING, ENROLLING_BY_INVITATION, ACTIVE_NOT_RECRUITING)` | `ci-active-trials-v1` | count |59| `recruiting_trials` | Recruiting clinical trials | `COUNT(DISTINCT trial) … overall_status = RECRUITING AND study_type = INTERVENTIONAL` | `ci-recruiting-trials-v1` | count |60| `phase3_trials` | Active Phase III trials | `COUNT(DISTINCT trial) … active statuses AND PHASE3 = ANY(phases)` | `ci-phase3-trials-v1` | count |6162Trials are attached to a cancer through its free-text conditions, mapped with a recorded63`match_type`; unmapped conditions sit in the curation queue and do not count. Counts aggregate over64the cancer's descendants in every hierarchy dimension (depth ≤ 12).6566### Research activity (PubMed)6768| Slug | Name | Formula | Version | Unit | Eligibility |69|---|---|---|---|---|---|70| `publications_5y` | Publications, last 5 years | `literature_counts.count WHERE window_key = 5y (query stored per row)` | `ci-pubs-5y-v1` | count | a stored PubMed query |71| `publications_12m` | Publications, last 12 months | `literature_counts.count WHERE window_key = 12m` | `ci-pubs-12m-v1` | count | a stored PubMed query |72| `publication_growth` | Publication growth | `publications_12m / (publications_5y_prior / 5)` | `ci-pub-growth-v1` | ratio | ≥ 50 publications in the prior 5-year window |7374The exact PubMed query string is stored next to every count (`literature_counts.query`), so any75count can be re-run against PubMed.7677### Molecular knowledge7879| Slug | Name | Formula | Version | Unit |80|---|---|---|---|---|81| `curated_evidence_items` | Curated clinical evidence items | `COUNT(civic_evidence_items) WHERE status = ACCEPTED AND cancer_id IN descendants(cancer)` | `ci-civic-evidence-v1` | count |82| `associated_genes` | Genes with curated or cohort evidence | `COUNT(DISTINCT gene) FROM (civic accepted evidence ∪ gdc frequency ≥ 0.05 AND cases_affected ≥ 20)` | `ci-genes-v1` | count |83| `genomic_cohorts` | Public genomic cohorts | `COUNT(genomic_cohorts) WHERE cancer_id IN descendants(cancer)` | `ci-cohorts-v1` | count |8485Only accepted CIViC evidence counts; CIViC levels (A–E) are shown as-is and never re-scaled.86Cohort frequencies always come with their denominator (`cases_affected / cases_profiled`).8788### Gap indexes (derived; require burden)8990| Slug | Name | Formula | Version | Unit | Eligibility |91|---|---|---|---|---|---|92| `trial_gap` | Trial Gap Index | `percentile(mortality_count) − percentile(active_trials)` | `ci-trial-gap-v1` | percentile points | mortality_count + active_trials in the same scope; ≥ 5 eligible entities |93| `research_gap` | Research Gap Index | `percentile(mortality_count) − percentile(publications_5y)` | `ci-research-gap-v1` | percentile points | mortality_count + publications_5y; ≥ 5 eligible entities |9495Positive values flag cancers with a high mortality burden but comparatively little activity. This is96a quantitative signal, not an accusation, and it depends on the epidemiology source of the scope.9798## 3. Scope and eligibility99100A **ranking snapshot** is one metric × one scope × one formula version at one time. The scope key is101`geo=<WORLD|ISO3>|sex=<all|male|female>|age=<all|…>|year=<YYYY|latest>|level=<top|all>` and is stored102on the snapshot with the `source_ids` that fed it. Snapshots are immutable; recomputation creates a103new snapshot and marks the previous one `is_current = false`, so history is preserved and104`previous_rank` can be explained.105106- **Entity levels.** `top` is the mutually exclusive set of 36 site groups (see §5); `all` is every107  active malignant canonical entity. Count metrics are computed for both; burden, lethality and gap108  metrics only for `top` (double-counting rule).109- **Geography-year scopes** are created only where a source provides ≥ 10 top-level observations for110  the same geography, year, sex and source; each scope uses a single source (no mixing of registries).111- **Eligibility** rules per metric are in the `eligibility` column (`requires`, `minIncidence`,112  `minCohort`, `minPublications5y`, `entityLevel`). Entities that do not meet them are not ranked113  rather than ranked with a zero; publication metrics require a stored PubMed query.114- **Ties** share a rank (competition ranking: 1, 2, 2, 4). `percentile` is the share of eligible115  entities ranked at or below the entity in the ranking direction (100 = most extreme).116117## 4. Uncertainty and confidence labels118119| Label | Rule |120|---|---|121| `HIGH` | Observed registry value (not an estimate); for MIR, incidence ≥ 1000 |122| `MEDIUM` | Default for computed counts and estimates |123| `LOW` | Reserved for small denominators / projected values |124| `INSUFFICIENT_DATA` | Entity excluded from the ranking; shown as "Data not yet available" |125126Estimates are always labeled `estimated` or `projected` on the observation; confidence intervals are127shown when the source provides them.128129## 5. The top-level ranking set and double counting130131Global burden rankings only make sense over **mutually exclusive** entities. Registry site groups132overlap with the disease taxonomy (a lung adenocarcinoma is also a lung cancer), so CancerIndex133ranks burden over a curated set of 36 site groups aligned with GLOBOCAN / ICD-10 chapters, each134anchored to an NCIt concept (`packages/ontology/src/top-level.ts`, `cancers.top_level = true`).135Subtypes, histologies and molecular entities are ranked separately at `level=all` for count metrics,136where aggregation over descendants is explicit and expected. A trial mapped to a subtype counts for137the subtype and for its ancestors; it is never counted twice within one snapshot because each138snapshot ranks one entity level and `COUNT(DISTINCT trial)` is applied per entity.139140## 6. What is not computed yet141142- **Composite scores** (impact/priority indexes): deliberately absent in Phase 1 (ADR-006). Any future143  composite will publish its weights and components in `rankings.breakdown`.144- **Global incidence/mortality rankings**: GLOBOCAN remains under license review, so worldwide burden145  metrics are shown as "awaiting license review"; US registry data (CDC WONDER) is the first burden146  scope.147- **Survival**: requires SEER (awaiting credentials); until then `five_year_survival` has no snapshot.148- **Prevalence, DALYs, trends over time, age-specific rankings**: not implemented.149- **Guideline status and biomarker-defined subgroups**: entities exist in the model but no connector150  ingests them in Phase 1.151152## 7. Reproducing a ranking153154Every snapshot stores an `inputs_hash`: the sha256 (first 24 hex chars) of the sorted155`<cancerId>=<value>;` pairs that entered the ranking. To reproduce:156157```bash158pnpm cix counters            # rebuild entity_counters deterministically from canonical tables159pnpm cix rank                # recompute every snapshot; prints metric, scope and eligible count160psql cancerindex -c "SELECT metric_slug, scope_key, formula_version, inputs_hash, eligible_entities FROM ranking_snapshots WHERE is_current"161```162163If the underlying observations have not changed, the new snapshot carries the same `inputs_hash`164as the previous one. The API returns the hash in every ranking response (`snapshot.inputsHash`) and165in `GET /v1/rankings/:metric/:cancerId/explain`.166167## 8. TRACE — from a rank to a raw record168169"Why this rank?" is answered by `GET /v1/rankings/:metric/:cancerId/explain`, which returns the row's170`inputs` (observation ids, counter names, formula inputs), `previous_rank`, the snapshot metadata and171a `trace`: for each referenced observation, the observation row, its `provenance` row (source,172dataset, version, retrieval time, URL, license) and the `source_records` entry (payload hash and raw173lake path). The same lineage is available to operators with `pnpm cix trace <table> <id>` and174`GET /v1/admin/trace/:table/:id`.175176## 9. Change tracking177178`change_events` records creations, updates, new trials/approvals, ranking changes, merges and179deprecations with before/after snapshots and the ingest run that caused them180(`GET /v1/changes`). Ranking rows keep `previous_rank`, and every response states its181`dataRelease` (`CancerIndex YYYY-MM`) and `generatedAt`.182183184## 6. Intelligence layer (wave 3, 2026-09-11)185186Derived tables recomputed daily by the worker (`maintenance.intel`, 06:15 UTC, between counters and187rankings) or by hand with `pnpm cix intel`. Each has its own method page with formulas, thresholds188and caveats; every row stores `formula_version` and an `inputs` JSON.189190| Module | Table(s) | Formula version | Method |191|---|---|---|---|192| Clinical trial intelligence (counts, growth, enrollment, sponsor & country concentration, termination share and stop reasons, burden-normalized intensity) | `trial_intelligence` | `ci-trial-intel-v1`, stop-reason rules `ci-stop-reasons-v1` | [docs/methodology/trial-intelligence.md](methodology/trial-intelligence.md) |193| Clinical trial map (country aggregates of registrant-entered sites) | `trial_site_country_counts` | `ci-trial-sites-v1` | [docs/methodology/trial-map.md](methodology/trial-map.md) |194| Research Gap Index (death share vs trial / publication share, log₂ ratios, per-1,000-deaths intensities) | `research_gap_components` + snapshots `trial_gap_ratio`, `research_gap_ratio`, `trials_per_1000_deaths`, `publications_per_1000_deaths` | `ci-research-gap-components-v1` | [docs/methodology/research-gap.md](methodology/research-gap.md) |195| Drug development pipeline (stage per drug and per drug × top-level cancer; duplicate-drug proposals) | `drug_pipeline`, `entity_merges` | `ci-drug-pipeline-v1` | [docs/methodology/pipeline.md](methodology/pipeline.md) |196| Knowledge graph (contextual neighbourhoods; source-native edges vs derived registry links) | query-time over `knowledge_edges` + relations | — (no derived numbers stored) | [docs/methodology/knowledge-graph.md](methodology/knowledge-graph.md) |197| Data explorer (comparability groups, computed defaults, permalinks, CSV) | query-time over `epidemiology_observations` | — | [docs/methodology/data-explorer.md](methodology/data-explorer.md) |198199Trial interventions are reconciled to canonical drugs by `reconcileInterventionDrugs`200(`packages/connectors/src/connectors/clinicaltrials/drugs.ts`): exact normalized alias → salt / dose /201label-stripped alias → probabilistic head match, each recorded as its own `match_type`; aliases shared202by several drugs are resolved only by a deterministic preference (own generic name > brand > base203molecule) and otherwise left unresolved. Salt-form duplicate drugs are *proposed* to `entity_merges`,204never merged automatically.205