spb/cancerindex

1# Connector `ncit-evs` — NCI Thesaurus (NCIt)23Tier 0 · category `terminology` · anchor of the CancerIndex cancer ontology (CLAUDE.md §4-§5, §209,4§217-§221). Every canonical cancer entity (`CI-CAN-…`) is minted from an NCIt concept under5**Neoplasm (C3262)**; OncoTree and every downstream source reconcile onto these entities.67Code: `packages/connectors/src/connectors/ncit-evs/` — `manifest.ts`, `parse.ts` (FLAT file),8`classify.ts` (pure ontology rules, unit-tested), `index.ts` (DB writes), `fixtures/`,9`ncit-evs.test.ts`.1011## Source and access (documentation verified 2026-09-08)1213| Item | Value |14|---|---|15| Bulk file | `https://evs.nci.nih.gov/ftp1/NCI_Thesaurus/Thesaurus.FLAT.zip` (16.2 MB ZIP → `Thesaurus.txt`, 75.8 MB, 213 083 rows) |16| Format doc | `https://evs.nci.nih.gov/ftp1/NCI_Thesaurus/ReadMe.txt` — tab-delimited: `code, concept IRI, parents (pipe), synonyms (pipe; first = preferred name), definition, display name, concept status, semantic type(s), concept in subset` |17| Version | first line of `ReadMe.txt` → `26.08e` (editing completed 2026-08-31); stored as `ingest_runs.dataset_version`, `cancers.classification_version`, `provenance.dataset_version` |18| Health / spot checks | `GET https://api-evsrest.nci.nih.gov/api/v1/concept/ncit/C9305?include=minimal` → `{ "version": "26.08e", … }`. The REST API is never crawled for the tree. |19| Raw lake | `data/raw/ncit-evs/<date>/Thesaurus.FLAT.zip` (+ `ReadMe.txt`), sha256 + size logged; per-record payloads in `data/raw/ncit-evs/<date>/{concept,qualified_state,qualified_state_map}/…jsonl.gz` |20| Unzip | `unzip -p` via `child_process.spawn` (no ZIP reader in Node); fails with an explicit "install unzip" message when the binary is absent |21| Schedule | monthly, `0 3 1 * *` (NCIt releases monthly) |2223Both `evs.nci.nih.gov` and `api-evsrest.nci.nih.gov` accepted the SDK user agent `CancerIndex/0.1`24and the URL-bearing variant; the connector passes the plain product token anyway (some WAFs25reject UAs containing URLs — see `sdk/http.ts`).2627## License evidence2829The EVS web site is a JavaScript application (WebFetch/Tavily only see the page title), but the FTP30tree carries the authoritative terms page, fetched 2026-09-08:3132> "The NCI Thesaurus™ is released under the Creative Commons Attribution 4.0 International license33> (CC BY 4.0). The NCI Thesaurus™ is produced by the Enterprise Vocabulary Services group of the34> Center for Biomedical Informatics and Information Technology, National Cancer Institute, Maryland,35> USA. The name "NCI Thesaurus" is trademarked. Only the NCI Thesaurus™ published by the NCI can be36> released under this name."37> — https://evs.nci.nih.gov/ftp1/NCI_Thesaurus/ThesaurusTermsofUse.htm3839Corroborated by OBO Foundry (`obofoundry.org/ontology/ncit.html`, "License: CC BY 4.0") and NCBO40BioPortal. The wider NCI reuse policy (https://www.cancer.gov/policies/copyright-reuse: "Unless41otherwise indicated, all text within National Cancer Institute (NCI) products is free of copyright42and may be reused without our permission … Credit the National Cancer Institute as the source")43is compatible; the manifest records the stricter, explicit **CC BY 4.0** statement, so44`licenseStatus: approved`, `redistribution: attribution`, `commercialUse: allowed`. Attribution45string and the trademark caveat are in the manifest (`attribution`, `termsNotes`). It is *not*46recorded as "public domain": the terms page says CC BY 4.0.4748## Fields used4950`code` → `cancers.primary_ncit_code`, `cancer_codes(ncit)`; `parents` → subtree closure, depth,51`cancer_hierarchy(ncit)`; `synonyms[0]` → `canonical_name` (never altered, the top-level display52name goes to an alias); `synonyms[1..]` → `cancer_aliases`; `definition` → `description` +53provenance row; `display name` → alias type `display`; `concept status` → retired/obsolete/header54handling; `semantic type` → `cancers.semantic_types` + disease filter. `concept IRI` and55`concept in subset` are stored in the raw payload only. UMLS CUIs / ICD-O codes are **not** derived56(the FLAT file does not carry them).5758## Construction rules (`classify.ts`)59601. The whole file (213 083 concepts) is loaded; child lists are built for all concepts so subtree61   closures are exact. Descendant sets: Neoplasm `C3262` (15 682), Malignant Neoplasm `C9305`62   (11 123 → `malignant`), Hematopoietic and Lymphoid Cell Neoplasm `C27134` (2 264 →63   `hematologic`, `solid_tumor = !hematologic`), Childhood Malignant Neoplasm `C4005` (638) or64   preferred name starting with Childhood/Pediatric → `pediatric_relevant`, Neoplasm by Site65   `C3263`, Neoplasm by Morphology `C4741`. `depth` = shortest path from `C3262` (BFS).662. **Exclusions** (in order): outside the Neoplasm subtree; `Retired_Concept`; `Obsolete_Concept`;67   animal/model concept (name matches mouse/murine/rat/canine/feline/zebrafish/drosophila/bovine/68   equine/porcine/avian/rodent/hamster/non-human/primate/"animal model"/veterinary, or semantic type69   Experimental Model of Disease/Mammal/Animal/Vertebrate/Organism — **0 hits in 26.08e**, rule kept70   as a guard); semantic type not disease-like (kept set: Neoplastic Process, Disease or Syndrome,71   Finding, Congenital Abnormality, Pathologic Function, Cell or Molecular Dysfunction, Anatomical72   Abnormality, Acquired Abnormality, Sign or Symptom — observed under Neoplasm: 15 656 Neoplastic73   Process, 24 Disease or Syndrome, 3 Congenital Abnormality, 1 Pathologic Function, 1 Finding,74   1 Cell or Molecular Dysfunction; **0 excluded**); qualified disease state via75   `isQualifiedState()` from `@cancerindex/ontology` **plus** a connector-local76   `EXTRA_QUALIFIED_RE` for stage patterns the shared rules miss ("Stage 1/2A/4S Neuroblastoma",77   "Stage A/B/C/D Prostate Cancer", "Stage M0–M4 …", "Stage L1/L2/MS …", "Stage y-I …",78   "… FIGO 2023", "… by Toronto Guidelines v2"). Qualified states are stored as `source_records`79   (`entity_kind = qualified_state`) with `baseLabel = baseDiseaseLabel(name)` and, when the base80   label equals a kept preferred name, `baseCode`/`canonical_id`; the same mapping is written to the81   raw lake (`qualified_state_map`). They never become cancers or aliases.823. **Header concepts** (`Header_Concept`, e.g. "Neoplasm by Site", "Malignant Neoplasm by Grade")83   are kept for connectivity with `entity_type = other` (17 rows) — they are structural, never listed84   as diseases.854. **Retired concepts**: not in the subtree (they hang under "Retired Concept 2025", C223967). The86   565 retired concepts with semantic type Neoplastic Process (664 retired rows have an *empty*87   semantic type despite the ReadMe's guarantee — accepted by the parser) and the 256 obsolete88   subtree concepts are recorded as `source_records` with `status = deprecated`. When a retired89   concept's preferred name equals an active kept concept's name, the retired code is mapped to it90   (`cancer_codes` system `ncit`, `match_type ONTOLOGY_EXACT`, `valid_to = 26.08e`; 48 such91   redirects) — e.g. `C3556 → C61574 "Malignant Uterine Corpus Neoplasm"`.925. **Entity type** (first match): header → `other`; depth ≤ 3 and ≥ 15 neoplasm descendants →93   `cancer_family`; name matches `MOLECULAR_RE` (`EGFR-Mutant`, `HER2-Positive`, `… with NPM194   Mutation`, `t(8;21)`, `Triple-Negative`, `Mismatch Repair-Deficient`…) → `molecular_subtype`;95   not malignant and /in situ|intraepithelial neoplasia|dysplasia|precursor|monoclonal96   gammopathy|myelodysplastic|premalignant/ → `precursor_condition`; under Morphology and not under97   Site → `histology`; hematologic → `hematologic_malignancy`; else depth ≥ 4 → `subtype`, else98   `cancer`. Deviation from the brief's order: molecular and precursor are tested before histology99   so that "Carcinoma In Situ" (a morphology node) is a precursor and molecular subtypes under100   morphology are not typed as histology. Curated top-level entries typed `subtype` are promoted to101   `cancer` (they are ranking scopes).1026. **Aliases** (`cancer_aliases`, `source_terminology = NCIt`): preferred name → `preferred`; other103   synonyms → `synonym`, or `abbreviation` when 2–6 upper-case alphanumerics with ≥ 2 letters (GBM,104   NSCLC, PDAC…); display name → `display`. Deduplicated per concept on `normalizeLabel()` (first105   wins, e.g. "Neoplasms, NOS" collapses into the preferred "Neoplasm"); normalized forms shorter106   than 2 chars skipped. Batched 500/insert with `ON CONFLICT DO NOTHING`.1077. **Codes**: `ncit` `EXACT_IDENTIFIER` for every kept concept; for the 36 `TOP_LEVEL_CANCERS`108   also `icd10` ranges as `CURATED_EXACT`, `top_level = true`, and the registry display name109   (e.g. "Pancreatic Cancer") as a `display` alias — `canonical_name` stays the NCIt preferred name.1108. **Hierarchy** (`cancer_hierarchy`, `hierarchy_type = ncit`): every parent inside the kept set;111   when all parents were excluded, the nearest kept ancestor(s) found by walking up ≤ 10 levels112   (146 concepts re-attached).1139. **Anatomy**: the 15 direct children of "Neoplasm by Site" become `anatomical_sites` (name =114   preferred name minus trailing "Neoplasm", reused by slug when OncoTree already created one) and115   every kept descendant is linked via `cancer_anatomy(relation = primary)`.11610. **Provenance**: one run-level row (dataset "NCI Thesaurus", version 26.08e, `expert_curation`,117    `open`) plus one row per definition (source URL118    `https://ncit.nci.nih.gov/ncitbrowser/ConceptReport.jsp?dictionary=NCI_Thesaurus&code=<code>`)119    referenced by `cancers.description_provenance_id`; rows are only added when the definition is120    new or changed.12111. **Idempotency / guard**: entities are matched on `primary_ncit_code`; slugs never change;122    unchanged rows are not rewritten. Anomaly guard: the run refuses to persist when the FLAT zip is123    < 5 MB, < 100 000 concepts parse, or the kept count drops below 90 % of the previous run124    (`anomalyGuard.minRatioOfPrevious = 0.9`, stored in `connector_cursors.cursor.keptCount`).125    Only after the guard passes are NCIt-anchored cancers that vanished / were retired / were126    reclassified as qualified states set to `status = deprecated` with an explicit127    `deprecated_reason` (`merged_into` set when a same-name replacement exists).128129## Counts observed (NCIt 26.08e, 2026-09-08, database `cancerindex_a`)130131| Metric | Value |132|---|---|133| Concepts in file / parsed | 213 083 / 213 083 (664 retired rows with empty semantic type accepted) |134| Neoplasm subtree | 15 682 |135| Kept → canonical cancers | **9 327** (first run 9 446 before the extra stage rules; 119 then deprecated as qualified states) |136| Qualified states (source records only) | 6 099 (3 844 resolve to a base disease by `baseDiseaseLabel`) |137| Retired (neoplastic) / obsolete recorded as deprecated source records | 565 / 256 (821) |138| Animal / non-disease semantic type excluded | 0 / 0 |139| Malignant / hematologic / pediatric (active, NCIt-anchored) | 5 595 / 1 426 / 526 |140| Entity types | subtype 6 923 · hematologic_malignancy 1 149 · molecular_subtype 384 · histology 300 · cancer_family 229 · cancer 169 · precursor_condition 156 · other 17 |141| Depth distribution | 0:1 · 1:3 · 2:51 · 3:339 · 4:1 269 · 5:2 662 · 6:2 864 · 7:1 560 · 8:578 · 9:90 · 10:25 · 11:4 |142| Aliases (NCIt) | 32 642 (preferred 9 327, synonym ~23 000, abbreviation ~600, display 36 top-level) |143| `cancer_codes` | ncit EXACT 9 327 (+48 retired→replacement ONTOLOGY_EXACT), icd10 38 |144| Hierarchy edges (ncit) | 16 438 |145| Anatomical site groups / `cancer_anatomy` links | 15 / 12 545 |146| Description provenance rows | 9 446 (every kept concept has a definition) |147| Top-level set | **36/36** — `C3556` (corpus-uteri) is *retired* in 26.08e and resolved to its same-name replacement `C61574`; `TOP_LEVEL_CANCERS` should be updated to `C61574` |148| Duplicate canonical names among active malignant cancers at the same depth | 0 (0 across all active cancers) |149| `^(stage|recurrent|metastatic|refractory)` among active NCIt-anchored cancers | 0 (the single remaining hit is OncoTree's "Refractory Cytopenia of Childhood", a WHO MDS entity created by the OncoTree connector) |150| Run time | full run 10 s (download 0.5 s, parse+classify ~1.5 s, writes ~8 s); rerun 7 s (0 created, 30 updated, 9 297 unchanged) |151| OncoTree after NCIt | 897 nodes: 32 tissues, 865 mapped; 682 onto NCIt-anchored entities (798 EXACT_IDENTIFIER + 67 ONTOLOGY_EXACT codes), 183 created new without NCIt anchor |152153## Known limitations154155- `entity_type` is heuristic (depth-based); most organ-level malignancies sit at depth ≥ 4 in NCIt156  and are typed `subtype` (e.g. "Breast Carcinoma", "Glioblastoma"). Curated top-level entries are157  promoted to `cancer`. A curated layer or descendant-count thresholds per depth would refine this.158- `cancer_family` also matches large non-malignant groupers ("Lung Neoplasm", 178 of 229).159- Hierarchy edges are never mass-deleted; edges to nodes deprecated by a later run remain (247 on160  `cancerindex_a` after the reclassification rerun; a fresh pipeline never creates them). Consumers161  must filter on `cancers.status = 'active'`.162- OncoTree runs after NCIt and reuses our site groups by slug (e.g. `breast`), overwriting163  `anatomical_sites.ncit_code` with its own anatomy code (C12971 vs our C2910). Harmless but noted.164- No ICD-O, UMLS or MeSH codes come from the FLAT file; OncoTree contributes UMLS CUIs (650).165- Semantic-type filter and animal regex were validated on 26.08e only; new semantic types under166  Neoplasm are surfaced in the run log (`non-disease concepts under Neoplasm: …`).167- `unresolved_labels` is untouched by this connector (it defines the vocabulary; nothing to resolve).168169## Recommendation for downstream reconciliation (trials, literature, registries)170171Trial conditions and registry labels are frequently *qualified* ("Stage IV Pancreatic Cancer",172"Recurrent Glioblastoma", "Metastatic HER2-Positive Breast Carcinoma"). Those NCIt concepts are173deliberately not canonical entities, so `CancerResolver.byLabel()` will miss them. The trials and174literature connectors should implement this fallback chain (it belongs in `@cancerindex/ontology`,175not in this connector):1761771. `resolve({ codes })` on any NCIt/UMLS/MeSH identifier the source carries (EXACT_IDENTIFIER).1782. `byLabel(label)` (ONTOLOGY_EXACT / ALIAS).1793. If it misses and `isQualifiedState(label) || EXTRA_QUALIFIED_RE.test(label)`: strip qualifiers180   with `baseDiseaseLabel(label)` (iteratively, also removing "Stage <token>" prefixes and staging181   suffixes) and retry `byLabel(base)`; record the mapping with `match_type = CURATED_BROADER`182   (the trial is about a *narrower* state of the base disease) and keep the original label + stage183   in `cohort_definitions.stage` / `attributes`.1844. Otherwise `ctx.recordUnresolved('cancer', label, normalizeLabel(label), ctx)`.185186The raw-lake file `qualified_state_map` (code → `baseLabel` → `baseCode`, 3 844 resolved) can be187loaded as a curated seed for that step. Upstreaming `EXTRA_QUALIFIED_RE` into188`packages/ontology/src/qualifiers.ts` (with tests) is recommended so both sides share one rule set.189