# Data explorer — methodology

`/explore` charts and exports registry observations stored in `epidemiology_observations` (see
`docs/DATA-MODEL.md` and `docs/METHODOLOGY.md` §1–2). It shows values **exactly as published by the
source**; CancerIndex harmonizes units (`count`, `per_100k`) and labels, never the numbers. Nothing
is estimated, interpolated, summed across site groups or extrapolated beyond the years a source
publishes. Every chart and table states unit, geography, sex, age group, years, standard population,
source and retrieval date, and each row keeps its `provenance_id`.

Population statistics describe groups defined by geography, period, sex and age; they never predict
an individual outcome (METHODOLOGY §325).

## 1. Comparability rules

Two observations are drawn on **one axis** only when they share all of:

| Dimension | Why it matters |
|---|---|
| `metric` | deaths, new cases, crude rates and age-standardized rates are different quantities |
| `unit` | `count` vs `per_100k` |
| `geography` | a country and one of its subdivisions are not comparable series |
| `source` | registries differ in coverage, site definitions (e.g. USCS "Pancreas" = ICD-10 C25 vs another grouping), vintage and revisions; the same year from CDC WONDER and USCS is two observations, not one |
| `standard_population` | rates standardized to the "2000 U.S. standard population" and to the "World (Segi)" standard are not on the same scale; crude rates and counts have no standard population |
| `age_group` | all ages vs 65+ etc. |

Observations that differ on any of these dimensions are rendered as **separate charts**, and a caption
names the dimension(s) that differ, e.g. *"Shown as 2 separate charts: the observations differ by
source (cdc-uscs vs cdc-wonder) and standard population (2000 U.S. standard population (19 age groups)
vs 2000 U.S. Std. Population)."* The groups are never overlaid, indexed or rescaled to appear
comparable. Inside one group the series are **cancer × sex** (× estimate type × site definition when a
source publishes two definitions for one cancer). Series whose `estimate_type` is not `observed`
(estimated, projected) are dashed and labelled.

Implementation: `apps/web/src/lib/explorer-series.ts` (`groupComparable`, `explainSplit`), unit-tested
in `apps/web/test/explorer-series.test.ts`.

### Small multiples

When more than 4 series would be overlaid, or when the user chooses `view=multiples`, a group is
split into one panel per cancer. Panels **share one y-axis** (0 to the group maximum, stated above the
panels) so heights remain comparable. Colour follows the entity (cancer × sex) across panels and
groups and is never cycled; beyond eight identities, colour follows the sex inside each panel and the
panel title carries the cancer. Colour is never the only carrier: every series has a text legend
entry, a tooltip with the series name, and the observations table lists every value.

### Sex

`sex=all` shows the source's "both sexes" observation (as published — not the sum of male and
female). `sex=any` shows every sex the source publishes as separate series inside the same group.

## 2. Defaults (computed, not curated)

When the URL carries no parameter:

- `metric` = `mortality_count` when present in the database, else the first metric present;
- `geography` = `united-states` when it carries observations, else the first geography that does;
- `sex` = `all`, `age` = `all`;
- `cancers` = the **5 top-level cancers with the highest value in the latest year** of the selected
  metric, geography, sex and age group (`topCancersByLatest`, one row per cancer — when two sources
  publish the same year, the larger value decides membership only; both sources are then charted
  separately). The results header says so and names the year;
- `from`/`to` = the full year span of the metric for the geography.

The list of metrics, geographies, sexes, age groups, years and sources offered by the form is read from
the observations (`explorerOptions`), so the explorer never advertises data it does not hold. Sources
registered for epidemiology but not yet ingested (IARC/GLOBOCAN under license review, SEER awaiting
credentials) are listed as such.

## 3. Permalink parameters

The URL is the state. `serializeExplorerParams` writes every dimension explicitly so a link stays
stable when the computed defaults change.

| Parameter | Values | Notes |
|---|---|---|
| `metric` | `mortality_count`, `as_mortality_rate`, `mortality_rate`, `incidence_count`, `as_incidence_rate`, … | one metric per view |
| `cancers` | comma-separated slugs or `CI-CAN-…` ids, max 6 | repeated `cancers=` also accepted; unknown refs are ignored and reported |
| `geography` | slug or ISO3 (`united-states`, `USA`) | |
| `sex` | `all` (default), `male`, `female`, `any` | `any` = every sex, one series each |
| `age` | age group label as stored (`all` default) | |
| `from`, `to` | years (inclusive); swapped when inverted | multi-year observations match when the span overlaps |
| `view` | `lines` (default), `multiples` | |
| `normalize` | `none` (only value in Phase 1) | reserved for indexed views |
| `page` | observations table page (50 rows/page) | omitted from downloads |

Examples:

- `/explore` — defaults;
- `/explore?metric=as_mortality_rate&cancers=malignant-lung-neoplasm,malignant-breast-neoplasm&geography=united-states&sex=any&age=all&from=1999&to=2024`;
- `/explore?metric=mortality_count&cancers=malignant-pancreatic-neoplasm&geography=united-states&sex=all&age=all&from=2018&to=2024&view=multiples`.

## 4. Downloads and API

- **CSV** — `/api/export/epidemiology.csv?<same parameters>` (metric, cancers, geography required).
  Header rows prefixed `#`: CancerIndex attribution and licence of the harmonization (CC BY 4.0),
  each underlying source with its licence text, dataset, version and retrieval timestamp, source
  attribution sentences, methodology URL and comparability reminder, generation time and row count.
  Columns: `cancer_id, cancer_slug, cancer_name, geography_id, geography_slug, geography_name, iso3,
  year, year_end, sex, age_group, metric, value, unit, lower_ci, upper_ci, standard_population,
  estimate_type, site_definition, source_slug, source_name, provenance_id, dataset, dataset_version,
  source_url, retrieved_at`. RFC 4180 quoting. Capped at 50 000 rows (stated in the header).
- **JSON** — `GET /api/v1/epidemiology?metric=&cancer=<repeatable, max 8>&geography=&sex=&age=&from=&to=&source=&estimateType=&limit&offset`
  (paginated, max 200 per page, envelope with `sources`), `GET /api/v1/epidemiology/coverage?cancer=&geography=&metric=`
  and `GET /api/v1/epidemiology/metrics`. Each observation carries `standardPopulation`,
  `estimateType`, `siteDefinition`, `source` and `provenance {id, dataset, datasetVersion, sourceUrl, retrievedAt}`
  so consumers can apply the same comparability rules.
- **Coverage matrix** — `/explore/coverage`: metric × geography × sex × age × source × standard
  population with year span, distinct years, observations and cancers. The empty state of `/explore`
  shows the same matrix restricted to the selected cancers/geography so a user sees what exists.

## 5. Citation

The "Cite" line combines CancerIndex (URL, access date) with every underlying source, its dataset and
retrieval date. Underlying observations remain under their providers' licences (CDC WONDER and USCS:
US Government work, statistical reporting only); CancerIndex's harmonization is CC BY 4.0.