# CancerIndex — Methodology

This is the public methodology of CancerIndex.io. Every number shown on the site or returned by the
API is either an **observation** imported from a named source with its provenance, or a **computed**
value whose formula and version are listed here. CancerIndex is not a physician, not a diagnostic
tool and gives no treatment recommendations. Population statistics describe groups of people
diagnosed in the past; they never predict an individual outcome.

## 1. Scientific safety labels

Every value carries one of these categories and the categories are never merged:

| Label | Meaning | Where it comes from |
|---|---|---|
| `observed_data` | Registry or database observation (counts, rates, cohort frequencies) | `epidemiology_observations`, `survival_observations`, `cancer_gene_frequencies`, trial records |
| `published_evidence` | A publication or a fact extracted from one | `publications`, `publication_entity_edges` |
| `curated_evidence` | Expert-curated assertion (CIViC, ClinVar) with source-native level | `civic_evidence_items`, `variant_clinical_significance` |
| `regulatory_status` | Jurisdiction-specific approval status | `drug_approvals` |
| `clinical_guideline` | Guideline statements (not ingested in Phase 1) | — |
| `computed_metric` | Derived by CancerIndex with a versioned formula | `entity_counters`, `literature_counts`, `rankings` |
| `ai_generated_synthesis` | Model-written text grounded in the above (not shipped in Phase 1) | `ai_answers` |

## 2. Metric catalog

Metric definitions are seeded from `packages/database/src/seed-data/metrics.ts` into
`metric_definitions` and served at `GET /v1/rankings/metrics`. The formula text below is the exact
text stored with each definition; `formula_version` changes whenever the computation changes, and
snapshots computed with different versions are never compared.

### Burden (observations)

| Slug | Name | Formula | Version | Unit | Eligibility |
|---|---|---|---|---|---|
| `incidence_count` | Annual new cases | `epidemiology_observations.value WHERE metric = incidence_count` | `ci-incidence-count-v1` | count | requires incidence_count; top level |
| `mortality_count` | Annual deaths | `epidemiology_observations.value WHERE metric = mortality_count` | `ci-mortality-count-v1` | count | requires mortality_count; top level |
| `as_incidence_rate` | Age-standardized incidence rate | `epidemiology_observations.value WHERE metric = as_incidence_rate (standard_population fixed per scope)` | `ci-asir-v1` | per 100 000 | requires as_incidence_rate; top level |
| `as_mortality_rate` | Age-standardized mortality rate | `epidemiology_observations.value WHERE metric = as_mortality_rate (standard_population fixed per scope)` | `ci-asmr-v1` | per 100 000 | requires as_mortality_rate; top level |

Sources: CDC WONDER (US), SEER (US, awaiting credentials), IARC GLOBOCAN (global, license review).
Registry counts are `observed`; national/global figures may be `estimated` and are labeled as such.
Rates standardized to different standard populations are never compared in one ranking.

### Lethality

| Slug | Name | Formula | Version | Unit | Eligibility |
|---|---|---|---|---|---|
| `mortality_incidence_ratio` | Mortality-to-incidence ratio | `mortality_count / incidence_count (same geography, year, sex, source family)` | `ci-mir-v1` | ratio | both counts, incidence ≥ 100; top level |
| `five_year_survival` | 5-year relative survival | `survival_observations.probability WHERE survival_type = relative AND duration_months = 60 AND stage IS NULL` | `ci-5ys-v1` | probability | cohort ≥ 50; top level; source SEER |

The mortality-to-incidence ratio is a crude proxy of lethality: it is not a survival probability and
is affected by incidence trends and registration completeness. Confidence is `HIGH` when incidence
≥ 1000, otherwise `MEDIUM`.

### Clinical research (ClinicalTrials.gov)

| Slug | Name | Formula | Version | Unit |
|---|---|---|---|---|
| `active_trials` | Active clinical trials | `COUNT(DISTINCT trial) FROM trial_conditions JOIN clinical_trials WHERE cancer_id IN descendants(cancer) AND study_type = INTERVENTIONAL AND overall_status IN (RECRUITING, NOT_YET_RECRUITING, ENROLLING_BY_INVITATION, ACTIVE_NOT_RECRUITING)` | `ci-active-trials-v1` | count |
| `recruiting_trials` | Recruiting clinical trials | `COUNT(DISTINCT trial) … overall_status = RECRUITING AND study_type = INTERVENTIONAL` | `ci-recruiting-trials-v1` | count |
| `phase3_trials` | Active Phase III trials | `COUNT(DISTINCT trial) … active statuses AND PHASE3 = ANY(phases)` | `ci-phase3-trials-v1` | count |

Trials are attached to a cancer through its free-text conditions, mapped with a recorded
`match_type`; unmapped conditions sit in the curation queue and do not count. Counts aggregate over
the cancer's descendants in every hierarchy dimension (depth ≤ 12).

### Research activity (PubMed)

| Slug | Name | Formula | Version | Unit | Eligibility |
|---|---|---|---|---|---|
| `publications_5y` | Publications, last 5 years | `literature_counts.count WHERE window_key = 5y (query stored per row)` | `ci-pubs-5y-v1` | count | a stored PubMed query |
| `publications_12m` | Publications, last 12 months | `literature_counts.count WHERE window_key = 12m` | `ci-pubs-12m-v1` | count | a stored PubMed query |
| `publication_growth` | Publication growth | `publications_12m / (publications_5y_prior / 5)` | `ci-pub-growth-v1` | ratio | ≥ 50 publications in the prior 5-year window |

The exact PubMed query string is stored next to every count (`literature_counts.query`), so any
count can be re-run against PubMed.

### Molecular knowledge

| Slug | Name | Formula | Version | Unit |
|---|---|---|---|---|
| `curated_evidence_items` | Curated clinical evidence items | `COUNT(civic_evidence_items) WHERE status = ACCEPTED AND cancer_id IN descendants(cancer)` | `ci-civic-evidence-v1` | count |
| `associated_genes` | Genes with curated or cohort evidence | `COUNT(DISTINCT gene) FROM (civic accepted evidence ∪ gdc frequency ≥ 0.05 AND cases_affected ≥ 20)` | `ci-genes-v1` | count |
| `genomic_cohorts` | Public genomic cohorts | `COUNT(genomic_cohorts) WHERE cancer_id IN descendants(cancer)` | `ci-cohorts-v1` | count |

Only accepted CIViC evidence counts; CIViC levels (A–E) are shown as-is and never re-scaled.
Cohort frequencies always come with their denominator (`cases_affected / cases_profiled`).

### Gap indexes (derived; require burden)

| Slug | Name | Formula | Version | Unit | Eligibility |
|---|---|---|---|---|---|
| `trial_gap` | Trial Gap Index | `percentile(mortality_count) − percentile(active_trials)` | `ci-trial-gap-v1` | percentile points | mortality_count + active_trials in the same scope; ≥ 5 eligible entities |
| `research_gap` | Research Gap Index | `percentile(mortality_count) − percentile(publications_5y)` | `ci-research-gap-v1` | percentile points | mortality_count + publications_5y; ≥ 5 eligible entities |

Positive values flag cancers with a high mortality burden but comparatively little activity. This is
a quantitative signal, not an accusation, and it depends on the epidemiology source of the scope.

## 3. Scope and eligibility

A **ranking snapshot** is one metric × one scope × one formula version at one time. The scope key is
`geo=<WORLD|ISO3>|sex=<all|male|female>|age=<all|…>|year=<YYYY|latest>|level=<top|all>` and is stored
on the snapshot with the `source_ids` that fed it. Snapshots are immutable; recomputation creates a
new snapshot and marks the previous one `is_current = false`, so history is preserved and
`previous_rank` can be explained.

- **Entity levels.** `top` is the mutually exclusive set of 36 site groups (see §5); `all` is every
  active malignant canonical entity. Count metrics are computed for both; burden, lethality and gap
  metrics only for `top` (double-counting rule).
- **Geography-year scopes** are created only where a source provides ≥ 10 top-level observations for
  the same geography, year, sex and source; each scope uses a single source (no mixing of registries).
- **Eligibility** rules per metric are in the `eligibility` column (`requires`, `minIncidence`,
  `minCohort`, `minPublications5y`, `entityLevel`). Entities that do not meet them are not ranked
  rather than ranked with a zero; publication metrics require a stored PubMed query.
- **Ties** share a rank (competition ranking: 1, 2, 2, 4). `percentile` is the share of eligible
  entities ranked at or below the entity in the ranking direction (100 = most extreme).

## 4. Uncertainty and confidence labels

| Label | Rule |
|---|---|
| `HIGH` | Observed registry value (not an estimate); for MIR, incidence ≥ 1000 |
| `MEDIUM` | Default for computed counts and estimates |
| `LOW` | Reserved for small denominators / projected values |
| `INSUFFICIENT_DATA` | Entity excluded from the ranking; shown as "Data not yet available" |

Estimates are always labeled `estimated` or `projected` on the observation; confidence intervals are
shown when the source provides them.

## 5. The top-level ranking set and double counting

Global burden rankings only make sense over **mutually exclusive** entities. Registry site groups
overlap with the disease taxonomy (a lung adenocarcinoma is also a lung cancer), so CancerIndex
ranks burden over a curated set of 36 site groups aligned with GLOBOCAN / ICD-10 chapters, each
anchored to an NCIt concept (`packages/ontology/src/top-level.ts`, `cancers.top_level = true`).
Subtypes, histologies and molecular entities are ranked separately at `level=all` for count metrics,
where aggregation over descendants is explicit and expected. A trial mapped to a subtype counts for
the subtype and for its ancestors; it is never counted twice within one snapshot because each
snapshot ranks one entity level and `COUNT(DISTINCT trial)` is applied per entity.

## 6. What is not computed yet

- **Composite scores** (impact/priority indexes): deliberately absent in Phase 1 (ADR-006). Any future
  composite will publish its weights and components in `rankings.breakdown`.
- **Global incidence/mortality rankings**: GLOBOCAN remains under license review, so worldwide burden
  metrics are shown as "awaiting license review"; US registry data (CDC WONDER) is the first burden
  scope.
- **Survival**: requires SEER (awaiting credentials); until then `five_year_survival` has no snapshot.
- **Prevalence, DALYs, trends over time, age-specific rankings**: not implemented.
- **Guideline status and biomarker-defined subgroups**: entities exist in the model but no connector
  ingests them in Phase 1.

## 7. Reproducing a ranking

Every snapshot stores an `inputs_hash`: the sha256 (first 24 hex chars) of the sorted
`<cancerId>=<value>;` pairs that entered the ranking. To reproduce:

```bash
pnpm cix counters            # rebuild entity_counters deterministically from canonical tables
pnpm cix rank                # recompute every snapshot; prints metric, scope and eligible count
psql cancerindex -c "SELECT metric_slug, scope_key, formula_version, inputs_hash, eligible_entities FROM ranking_snapshots WHERE is_current"
```

If the underlying observations have not changed, the new snapshot carries the same `inputs_hash`
as the previous one. The API returns the hash in every ranking response (`snapshot.inputsHash`) and
in `GET /v1/rankings/:metric/:cancerId/explain`.

## 8. TRACE — from a rank to a raw record

"Why this rank?" is answered by `GET /v1/rankings/:metric/:cancerId/explain`, which returns the row's
`inputs` (observation ids, counter names, formula inputs), `previous_rank`, the snapshot metadata and
a `trace`: for each referenced observation, the observation row, its `provenance` row (source,
dataset, version, retrieval time, URL, license) and the `source_records` entry (payload hash and raw
lake path). The same lineage is available to operators with `pnpm cix trace <table> <id>` and
`GET /v1/admin/trace/:table/:id`.

## 9. Change tracking

`change_events` records creations, updates, new trials/approvals, ranking changes, merges and
deprecations with before/after snapshots and the ingest run that caused them
(`GET /v1/changes`). Ranking rows keep `previous_rank`, and every response states its
`dataRelease` (`CancerIndex YYYY-MM`) and `generatedAt`.


## 6. Intelligence layer (wave 3, 2026-09-11)

Derived tables recomputed daily by the worker (`maintenance.intel`, 06:15 UTC, between counters and
rankings) or by hand with `pnpm cix intel`. Each has its own method page with formulas, thresholds
and caveats; every row stores `formula_version` and an `inputs` JSON.

| Module | Table(s) | Formula version | Method |
|---|---|---|---|
| Clinical trial intelligence (counts, growth, enrollment, sponsor & country concentration, termination share and stop reasons, burden-normalized intensity) | `trial_intelligence` | `ci-trial-intel-v1`, stop-reason rules `ci-stop-reasons-v1` | [docs/methodology/trial-intelligence.md](methodology/trial-intelligence.md) |
| Clinical trial map (country aggregates of registrant-entered sites) | `trial_site_country_counts` | `ci-trial-sites-v1` | [docs/methodology/trial-map.md](methodology/trial-map.md) |
| Research Gap Index (death share vs trial / publication share, log₂ ratios, per-1,000-deaths intensities) | `research_gap_components` + snapshots `trial_gap_ratio`, `research_gap_ratio`, `trials_per_1000_deaths`, `publications_per_1000_deaths` | `ci-research-gap-components-v1` | [docs/methodology/research-gap.md](methodology/research-gap.md) |
| Drug development pipeline (stage per drug and per drug × top-level cancer; duplicate-drug proposals) | `drug_pipeline`, `entity_merges` | `ci-drug-pipeline-v1` | [docs/methodology/pipeline.md](methodology/pipeline.md) |
| Knowledge graph (contextual neighbourhoods; source-native edges vs derived registry links) | query-time over `knowledge_edges` + relations | — (no derived numbers stored) | [docs/methodology/knowledge-graph.md](methodology/knowledge-graph.md) |
| Data explorer (comparability groups, computed defaults, permalinks, CSV) | query-time over `epidemiology_observations` | — | [docs/methodology/data-explorer.md](methodology/data-explorer.md) |

Trial interventions are reconciled to canonical drugs by `reconcileInterventionDrugs`
(`packages/connectors/src/connectors/clinicaltrials/drugs.ts`): exact normalized alias → salt / dose /
label-stripped alias → probabilistic head match, each recorded as its own `match_type`; aliases shared
by several drugs are resolved only by a deterministic preference (own generic name > brand > base
molecule) and otherwise left unresolved. Salt-form duplicate drugs are *proposed* to `entity_merges`,
never merged automatically.