# Source policy — licensing, precedence, attribution (CLAUDE.md §142-143, §275-277, §299)

This document governs every connector in `packages/connectors`. It is enforced by the manifest schema (`sdk/manifest.ts`), `sources:sync` (which seeds the `sources` table = the public license registry shown on `/sources`) and `runConnector` (which refuses blocked sources). Last revised 2026-09-08.

## 1. License registry fields (per manifest → `sources`)

| field | meaning | operational effect |
|---|---|---|
| `license` | free-text license or terms summary, with the governing instrument named | displayed on `/sources` and copied to every `provenance.license` |
| `licenseStatus` | `approved` · `review` · `restricted` · `blocked` | see §3 |
| `commercialUse` | `allowed` · `restricted` · `prohibited` · `unknown` | `prohibited`/`restricted` must be justified in `termsNotes`; drives the "commercial use" badge |
| `redistribution` | `allowed` · `attribution` · `restricted` · `prohibited` · `unknown` | `attribution` requires a non-empty `attribution` string |
| `attribution` | the exact citation the source asks for | rendered next to every number from that source (§5) |
| `termsUrl`, `termsReviewedAt`, `termsNotes` | where the terms were read, when, and the verbatim/paraphrased findings (with what could not be verified) | audit trail; `termsReviewedAt` becomes `sources.license_reviewed_at` |
| `documentationVerifiedAt` | date the *technical* docs were checked with a smoke test (CLAUDE.md §9) | a connector cannot be `active` without it |
| `accessLevel` (provenance) | `open` · `registration_required` · `controlled` · `licensed` | shown as the access badge on values |

The connector's `docs/connectors/<id>.md` must quote the terms verbatim with URLs (see cdc-wonder.md, cdc-uscs.md, seer.md, iarc-globocan.md) so a human can re-check them without re-running the crawl.

## 2. Review workflow

1. **Discover** — locate official documentation and the terms page; record URLs.
2. **Read the terms** — capture verbatim the clauses on reproduction, redistribution, commercial use, systematic retrieval/bulk download, attribution, suppression/confidentiality. If the page is JavaScript-rendered, read the underlying bundle/JSON (as done for GCO) and say so.
3. **Classify** — propose `licenseStatus`, `commercialUse`, `redistribution` with a one-paragraph rationale in `termsNotes`.
4. **Smoke test** — verify the technical docs against the live source (curl/Node), save sanitized fixtures, write tests (normal / empty / pagination / rate-limit / server error / malformed).
5. **Human confirmation** — a maintainer confirms the classification in code review; `approved` for anything other than a clear public-domain / permissive license (CC BY, CC0, US Government work) requires that confirmation to be named in `termsNotes`. IARC/GLOBOCAN additionally requires IARC's written permission on file and `IARC_TERMS_ACCEPTED_BY` (see iarc-globocan.md).
6. **Register** — `pnpm cix sources:sync`; `/sources` shows the status; `approved_for_production` is true only for `approved`.
7. **Re-review** — at least yearly, and whenever `documentationVerifiedAt` is refreshed or the source changes its terms URL. A change from `approved` to anything else stops scheduled runs (worker skips non-active statuses) and must be followed by a decision on already-stored data.

## 3. What the statuses mean operationally

| status | ingestion | display | example |
|---|---|---|---|
| `approved` | connector may run and persist | values shown with source badge + attribution | oncotree (CC BY 4.0), cdc-wonder, cdc-uscs, seer, seer-explorer (US Government works) |
| `review` | connector may exist and run **only in dry_run/probe modes**; nothing is persisted; the manifest `status` is `review` so the scheduler skips it | `/sources` shows "awaiting license review"; every page that would use it shows "Data not yet available (awaiting license review)" — never a placeholder number | (none currently; former state of iarc-globocan) |
| `restricted` | like `review`, plus a named condition that must be met before approval (permission letter, non-commercial deployment, fee). `sync()` must fail loudly (`iarc-globocan` throws `license review pending — ingestion disabled`) | `/sources` shows "restricted — permission required" | iarc-globocan |
| `blocked` | **never ingests**: `runConnector` aborts before `sync()` ("license status blocked — connector will not ingest") | `/sources` lists the source and the reason; no values anywhere | a source whose terms forbid our use and where no permission will be sought |

Rules: a source in `review`/`restricted`/`blocked` never contributes to rankings, counters or AI syntheses; already-persisted values from a source later downgraded are hidden (not deleted) until the decision is recorded in `docs/adr/`.

## 4. Precedence by field (which source wins when two disagree)

Precedence applies to *canonical* fields; all observations are kept side by side with their own provenance (CLAUDE.md §71), never averaged or merged.

| field | authority (in order) |
|---|---|
| cancer entity identity, preferred name, hierarchy | NCIt (ncit-evs) → OncoTree (second hierarchy dimension) → curated aliases |
| gene symbol, gene identity | **HGNC** → NCBI Gene / Ensembl cross-references (never a paper's spelling) |
| variant nomenclature / clinical significance | ClinVar (regulatory-grade curation) → CIViC (curated evidence) → cohort frequencies (GDC) |
| trial registration, status, phase | **ClinicalTrials.gov** (registry of record) → EU CTR / WHO ICTRP when added |
| US regulatory status of a therapy | **FDA** (labels, approvals) → NCCN/guidelines for practice context |
| US registry statistics (counts, age-adjusted rates) | **SEER / CDC**: cdc-uscs (NPCR+SEER, 100% population coverage) for incidence; cdc-wonder (NVSS, final mortality) and cdc-uscs for mortality; seer-explorer for relative survival and SEER-area rates. Within one ranking scope a single `source_id` is used — sources are never mixed in one ranking |
| global / country burden | **IARC GLOBOCAN** — once `approved`; until then no global burden ranking exists and the UI says so |
| literature counts | PubMed (query stored with the count) |
| age standardization | the source's declared standard population is stored on every rate; rates standardized to different populations (US 2000 vs World Segi) are never compared in one ranking |

Where two approved US sources cover the same statistic (e.g. USCS mortality count vs WONDER mortality count, which use different site groupings for lung and lymphoma), both are stored with their `site_definition`, rankings are computed per source (the source is part of the scope), and the UI should prefer by default the source whose `site_definition` equals the entity's ICD-10 range exactly, listing the other under "also reported as" (UI rule to implement; not enforced in the data layer).

## 5. Attribution requirements per current source

| source | required text (verbatim where the source prescribes it) |
|---|---|
| ncit-evs | "NCI Thesaurus (NCIt), National Cancer Institute, version <release>" — public domain |
| oncotree | "OncoTree (Memorial Sloan Kettering Cancer Center), CC BY 4.0, https://oncotree.mskcc.org" |
| cdc-wonder | "Centers for Disease Control and Prevention, National Center for Health Statistics. Underlying Cause of Death 2018-2024 on CDC WONDER Online Database. Data are from the Multiple Cause of Death Files, compiled from data provided by the 57 vital statistics jurisdictions through the Vital Statistics Cooperative Program. Accessed at http://wonder.cdc.gov/ucd-icd10-expanded.html." + the query footnotes/caveats; never publish counts ≤ 9 (connector skips suppressed cells) |
| cdc-uscs | "U.S. Cancer Statistics Working Group. U.S. Cancer Statistics Data Visualizations Tool. U.S. Department of Health and Human Services, Centers for Disease Control and Prevention and National Cancer Institute; https://www.cdc.gov/cancer/dataviz, released in June 2026." |
| seer (API) | "appropriate acknowledgement of SEER's contribution": "Surveillance, Epidemiology, and End Results (SEER) Program (www.seer.cancer.gov), National Cancer Institute, DCCPS, Surveillance Research Program." |
| seer-explorer | "SEER*Explorer: An interactive website for SEER cancer statistics [Internet]. Surveillance Research Program, National Cancer Institute; <release date>. [cited <date>]. Available from: https://seer.cancer.gov/statistics-network/explorer/." + the export's Data Source footnote (submission, registries) — labelled as SEER registry areas, not the whole US |
| iarc-globocan (when approved) | "Ferlay J, Ervik M, Lam F, Laversanne M, Colombet M, Mery L, Piñeros M, Znaor A, Soerjomataram I, Bray F (2024). Global Cancer Observatory: Cancer Today (version 1.1). Lyon, France: International Agency for Research on Cancer. Available from: https://gco.iarc.who.int/today, accessed [DD Month YYYY]." |
| hgnc, clinicaltrials, pubmed, civic, clinvar, gdc | see their connector docs; all require the source name + identifier on each value |

Every displayed number carries: source badge, unit, population, period, standard population (rates), `site_definition`, freshness (`retrieved_at`) and a link to the provenance record (CLAUDE.md UI conventions).

## 6. Compliance checklist (before a connector goes `active`)

- [ ] Official documentation URL(s) recorded; `documentationVerifiedAt` set after a live smoke test with the real client (Node fetch — note Akamai rejects curl for CDC hosts).
- [ ] Terms read and quoted verbatim in `docs/connectors/<id>.md`; `termsUrl`, `termsReviewedAt`, `termsNotes` filled.
- [ ] `licenseStatus`, `commercialUse`, `redistribution`, `attribution` set and justified; human confirmation named for non-trivial licenses; permission letters filed for `restricted → approved` transitions.
- [ ] Source-imposed constraints implemented in code, not in prose: rate/frequency limits (WONDER: 1 query / 2 min, single robot), suppression rules (WONDER ≤ 9, USCS < 16 never re-derived), national-only restrictions, key handling (`credentialsMissing()`), no scraping of undocumented endpoints.
- [ ] Entities located through identifiers (`cancer_codes`, HGNC ids, NCT ids), never by name; unknown labels go to `unresolved_labels`.
- [ ] Every row has a provenance row (`dataset`, `dataset_version`, `evidence_type`, `access_level`, `geography`, `population`, `methodology`) and raw payloads are in the lake when retention is allowed.
- [ ] Fixtures are sanitized/trimmed real responses (or clearly labelled synthetic shapes) and tests cover normal, empty, malformed, rate-limit and server-error cases.
- [ ] Idempotent upserts on the natural key; anomaly guard against shrunken responses; restartable cursor.
- [ ] Time-aware: new years are added, never overwrite older ones; estimates vs observed values labelled (`estimate_type`).
- [ ] Never present a national/registry statistic as global (SEER/CDC = United States; SEER areas noted in `population`).
- [ ] `pnpm cix sources:sync` run; `/sources` shows the intended status; `pnpm cix run <id> --mode dry_run` passes before the first full run.