SPB Git forge
spboucher.ai ↗

spb/cancerindex

Public
37commits 1branches 0releases
2.9 MBsize
maindefault branch
10 days agolast push
TypeScript 97.2% SQL 1.5% CSS 0.6% JavaScript 0.5%
11.0 KB
View source Raw Blame History

# Source policy — licensing, precedence, attribution (CLAUDE.md §142-143, §275-277, §299)

This document governs every connector in packages/connectors. It is enforced by the manifest schema (sdk/manifest.ts), sources:sync (which seeds the sources table = the public license registry shown on /sources) and runConnector (which refuses blocked sources). Last revised 2026-09-08.

# 1. License registry fields (per manifest → sources)

field meaning operational effect
license free-text license or terms summary, with the governing instrument named displayed on /sources and copied to every provenance.license
licenseStatus approved · review · restricted · blocked see §3
commercialUse allowed · restricted · prohibited · unknown prohibited/restricted must be justified in termsNotes; drives the "commercial use" badge
redistribution allowed · attribution · restricted · prohibited · unknown attribution requires a non-empty attribution string
attribution the exact citation the source asks for rendered next to every number from that source (§5)
termsUrl, termsReviewedAt, termsNotes where the terms were read, when, and the verbatim/paraphrased findings (with what could not be verified) audit trail; termsReviewedAt becomes sources.license_reviewed_at
documentationVerifiedAt date the technical docs were checked with a smoke test (CLAUDE.md §9) a connector cannot be active without it
accessLevel (provenance) open · registration_required · controlled · licensed shown as the access badge on values

The connector's docs/connectors/<id>.md must quote the terms verbatim with URLs (see cdc-wonder.md, cdc-uscs.md, seer.md, iarc-globocan.md) so a human can re-check them without re-running the crawl.

# 2. Review workflow

  1. Discover — locate official documentation and the terms page; record URLs.
  2. Read the terms — capture verbatim the clauses on reproduction, redistribution, commercial use, systematic retrieval/bulk download, attribution, suppression/confidentiality. If the page is JavaScript-rendered, read the underlying bundle/JSON (as done for GCO) and say so.
  3. Classify — propose licenseStatus, commercialUse, redistribution with a one-paragraph rationale in termsNotes.
  4. Smoke test — verify the technical docs against the live source (curl/Node), save sanitized fixtures, write tests (normal / empty / pagination / rate-limit / server error / malformed).
  5. Human confirmation — a maintainer confirms the classification in code review; approved for anything other than a clear public-domain / permissive license (CC BY, CC0, US Government work) requires that confirmation to be named in termsNotes. IARC/GLOBOCAN additionally requires IARC's written permission on file and IARC_TERMS_ACCEPTED_BY (see iarc-globocan.md).
  6. Register — pnpm cix sources:sync; /sources shows the status; approved_for_production is true only for approved.
  7. Re-review — at least yearly, and whenever documentationVerifiedAt is refreshed or the source changes its terms URL. A change from approved to anything else stops scheduled runs (worker skips non-active statuses) and must be followed by a decision on already-stored data.

# 3. What the statuses mean operationally

status ingestion display example
approved connector may run and persist values shown with source badge + attribution oncotree (CC BY 4.0), cdc-wonder, cdc-uscs, seer, seer-explorer (US Government works)
review connector may exist and run only in dry_run/probe modes; nothing is persisted; the manifest status is review so the scheduler skips it /sources shows "awaiting license review"; every page that would use it shows "Data not yet available (awaiting license review)" — never a placeholder number (none currently; former state of iarc-globocan)
restricted like review, plus a named condition that must be met before approval (permission letter, non-commercial deployment, fee). sync() must fail loudly (iarc-globocan throws license review pending — ingestion disabled) /sources shows "restricted — permission required" iarc-globocan
blocked never ingests: runConnector aborts before sync() ("license status blocked — connector will not ingest") /sources lists the source and the reason; no values anywhere a source whose terms forbid our use and where no permission will be sought

Rules: a source in review/restricted/blocked never contributes to rankings, counters or AI syntheses; already-persisted values from a source later downgraded are hidden (not deleted) until the decision is recorded in docs/adr/.

# 4. Precedence by field (which source wins when two disagree)

Precedence applies to canonical fields; all observations are kept side by side with their own provenance (CLAUDE.md §71), never averaged or merged.

field authority (in order)
cancer entity identity, preferred name, hierarchy NCIt (ncit-evs) → OncoTree (second hierarchy dimension) → curated aliases
gene symbol, gene identity HGNC → NCBI Gene / Ensembl cross-references (never a paper's spelling)
variant nomenclature / clinical significance ClinVar (regulatory-grade curation) → CIViC (curated evidence) → cohort frequencies (GDC)
trial registration, status, phase ClinicalTrials.gov (registry of record) → EU CTR / WHO ICTRP when added
US regulatory status of a therapy FDA (labels, approvals) → NCCN/guidelines for practice context
US registry statistics (counts, age-adjusted rates) SEER / CDC: cdc-uscs (NPCR+SEER, 100% population coverage) for incidence; cdc-wonder (NVSS, final mortality) and cdc-uscs for mortality; seer-explorer for relative survival and SEER-area rates. Within one ranking scope a single source_id is used — sources are never mixed in one ranking
global / country burden IARC GLOBOCAN — once approved; until then no global burden ranking exists and the UI says so
literature counts PubMed (query stored with the count)
age standardization the source's declared standard population is stored on every rate; rates standardized to different populations (US 2000 vs World Segi) are never compared in one ranking

Where two approved US sources cover the same statistic (e.g. USCS mortality count vs WONDER mortality count, which use different site groupings for lung and lymphoma), both are stored with their site_definition, rankings are computed per source (the source is part of the scope), and the UI should prefer by default the source whose site_definition equals the entity's ICD-10 range exactly, listing the other under "also reported as" (UI rule to implement; not enforced in the data layer).

# 5. Attribution requirements per current source

source required text (verbatim where the source prescribes it)
ncit-evs "NCI Thesaurus (NCIt), National Cancer Institute, version " — public domain
oncotree "OncoTree (Memorial Sloan Kettering Cancer Center), CC BY 4.0, https://oncotree.mskcc.org"
cdc-wonder "Centers for Disease Control and Prevention, National Center for Health Statistics. Underlying Cause of Death 2018-2024 on CDC WONDER Online Database. Data are from the Multiple Cause of Death Files, compiled from data provided by the 57 vital statistics jurisdictions through the Vital Statistics Cooperative Program. Accessed at http://wonder.cdc.gov/ucd-icd10-expanded.html." + the query footnotes/caveats; never publish counts ≤ 9 (connector skips suppressed cells)
cdc-uscs "U.S. Cancer Statistics Working Group. U.S. Cancer Statistics Data Visualizations Tool. U.S. Department of Health and Human Services, Centers for Disease Control and Prevention and National Cancer Institute; https://www.cdc.gov/cancer/dataviz, released in June 2026."
seer (API) "appropriate acknowledgement of SEER's contribution": "Surveillance, Epidemiology, and End Results (SEER) Program (www.seer.cancer.gov), National Cancer Institute, DCCPS, Surveillance Research Program."
seer-explorer "SEER*Explorer: An interactive website for SEER cancer statistics [Internet]. Surveillance Research Program, National Cancer Institute; . [cited ]. Available from: https://seer.cancer.gov/statistics-network/explorer/." + the export's Data Source footnote (submission, registries) — labelled as SEER registry areas, not the whole US
iarc-globocan (when approved) "Ferlay J, Ervik M, Lam F, Laversanne M, Colombet M, Mery L, Piñeros M, Znaor A, Soerjomataram I, Bray F (2024). Global Cancer Observatory: Cancer Today (version 1.1). Lyon, France: International Agency for Research on Cancer. Available from: https://gco.iarc.who.int/today, accessed [DD Month YYYY]."
hgnc, clinicaltrials, pubmed, civic, clinvar, gdc see their connector docs; all require the source name + identifier on each value

Every displayed number carries: source badge, unit, population, period, standard population (rates), site_definition, freshness (retrieved_at) and a link to the provenance record (CLAUDE.md UI conventions).

# 6. Compliance checklist (before a connector goes active)

  • Official documentation URL(s) recorded; documentationVerifiedAt set after a live smoke test with the real client (Node fetch — note Akamai rejects curl for CDC hosts).
  • Terms read and quoted verbatim in docs/connectors/<id>.md; termsUrl, termsReviewedAt, termsNotes filled.
  • licenseStatus, commercialUse, redistribution, attribution set and justified; human confirmation named for non-trivial licenses; permission letters filed for restricted → approved transitions.
  • Source-imposed constraints implemented in code, not in prose: rate/frequency limits (WONDER: 1 query / 2 min, single robot), suppression rules (WONDER ≤ 9, USCS < 16 never re-derived), national-only restrictions, key handling (credentialsMissing()), no scraping of undocumented endpoints.
  • Entities located through identifiers (cancer_codes, HGNC ids, NCT ids), never by name; unknown labels go to unresolved_labels.
  • Every row has a provenance row (dataset, dataset_version, evidence_type, access_level, geography, population, methodology) and raw payloads are in the lake when retention is allowed.
  • Fixtures are sanitized/trimmed real responses (or clearly labelled synthetic shapes) and tests cover normal, empty, malformed, rate-limit and server-error cases.
  • Idempotent upserts on the natural key; anomaly guard against shrunken responses; restartable cursor.
  • Time-aware: new years are added, never overwrite older ones; estimates vs observed values labelled (estimate_type).
  • Never present a national/registry statistic as global (SEER/CDC = United States; SEER areas noted in population).
  • pnpm cix sources:sync run; /sources shows the intended status; pnpm cix run <id> --mode dry_run passes before the first full run.