# Connector `ncit-evs` — NCI Thesaurus (NCIt)

Tier 0 · category `terminology` · anchor of the CancerIndex cancer ontology (CLAUDE.md §4-§5, §209,
§217-§221). Every canonical cancer entity (`CI-CAN-…`) is minted from an NCIt concept under
**Neoplasm (C3262)**; OncoTree and every downstream source reconcile onto these entities.

Code: `packages/connectors/src/connectors/ncit-evs/` — `manifest.ts`, `parse.ts` (FLAT file),
`classify.ts` (pure ontology rules, unit-tested), `index.ts` (DB writes), `fixtures/`,
`ncit-evs.test.ts`.

## Source and access (documentation verified 2026-09-08)

| Item | Value |
|---|---|
| Bulk file | `https://evs.nci.nih.gov/ftp1/NCI_Thesaurus/Thesaurus.FLAT.zip` (16.2 MB ZIP → `Thesaurus.txt`, 75.8 MB, 213 083 rows) |
| Format doc | `https://evs.nci.nih.gov/ftp1/NCI_Thesaurus/ReadMe.txt` — tab-delimited: `code, concept IRI, parents (pipe), synonyms (pipe; first = preferred name), definition, display name, concept status, semantic type(s), concept in subset` |
| Version | first line of `ReadMe.txt` → `26.08e` (editing completed 2026-08-31); stored as `ingest_runs.dataset_version`, `cancers.classification_version`, `provenance.dataset_version` |
| Health / spot checks | `GET https://api-evsrest.nci.nih.gov/api/v1/concept/ncit/C9305?include=minimal` → `{ "version": "26.08e", … }`. The REST API is never crawled for the tree. |
| Raw lake | `data/raw/ncit-evs/<date>/Thesaurus.FLAT.zip` (+ `ReadMe.txt`), sha256 + size logged; per-record payloads in `data/raw/ncit-evs/<date>/{concept,qualified_state,qualified_state_map}/…jsonl.gz` |
| Unzip | `unzip -p` via `child_process.spawn` (no ZIP reader in Node); fails with an explicit "install unzip" message when the binary is absent |
| Schedule | monthly, `0 3 1 * *` (NCIt releases monthly) |

Both `evs.nci.nih.gov` and `api-evsrest.nci.nih.gov` accepted the SDK user agent `CancerIndex/0.1`
and the URL-bearing variant; the connector passes the plain product token anyway (some WAFs
reject UAs containing URLs — see `sdk/http.ts`).

## License evidence

The EVS web site is a JavaScript application (WebFetch/Tavily only see the page title), but the FTP
tree carries the authoritative terms page, fetched 2026-09-08:

> "The NCI Thesaurus™ is released under the Creative Commons Attribution 4.0 International license
> (CC BY 4.0). The NCI Thesaurus™ is produced by the Enterprise Vocabulary Services group of the
> Center for Biomedical Informatics and Information Technology, National Cancer Institute, Maryland,
> USA. The name "NCI Thesaurus" is trademarked. Only the NCI Thesaurus™ published by the NCI can be
> released under this name."
> — https://evs.nci.nih.gov/ftp1/NCI_Thesaurus/ThesaurusTermsofUse.htm

Corroborated by OBO Foundry (`obofoundry.org/ontology/ncit.html`, "License: CC BY 4.0") and NCBO
BioPortal. The wider NCI reuse policy (https://www.cancer.gov/policies/copyright-reuse: "Unless
otherwise indicated, all text within National Cancer Institute (NCI) products is free of copyright
and may be reused without our permission … Credit the National Cancer Institute as the source")
is compatible; the manifest records the stricter, explicit **CC BY 4.0** statement, so
`licenseStatus: approved`, `redistribution: attribution`, `commercialUse: allowed`. Attribution
string and the trademark caveat are in the manifest (`attribution`, `termsNotes`). It is *not*
recorded as "public domain": the terms page says CC BY 4.0.

## Fields used

`code` → `cancers.primary_ncit_code`, `cancer_codes(ncit)`; `parents` → subtree closure, depth,
`cancer_hierarchy(ncit)`; `synonyms[0]` → `canonical_name` (never altered, the top-level display
name goes to an alias); `synonyms[1..]` → `cancer_aliases`; `definition` → `description` +
provenance row; `display name` → alias type `display`; `concept status` → retired/obsolete/header
handling; `semantic type` → `cancers.semantic_types` + disease filter. `concept IRI` and
`concept in subset` are stored in the raw payload only. UMLS CUIs / ICD-O codes are **not** derived
(the FLAT file does not carry them).

## Construction rules (`classify.ts`)

1. The whole file (213 083 concepts) is loaded; child lists are built for all concepts so subtree
   closures are exact. Descendant sets: Neoplasm `C3262` (15 682), Malignant Neoplasm `C9305`
   (11 123 → `malignant`), Hematopoietic and Lymphoid Cell Neoplasm `C27134` (2 264 →
   `hematologic`, `solid_tumor = !hematologic`), Childhood Malignant Neoplasm `C4005` (638) or
   preferred name starting with Childhood/Pediatric → `pediatric_relevant`, Neoplasm by Site
   `C3263`, Neoplasm by Morphology `C4741`. `depth` = shortest path from `C3262` (BFS).
2. **Exclusions** (in order): outside the Neoplasm subtree; `Retired_Concept`; `Obsolete_Concept`;
   animal/model concept (name matches mouse/murine/rat/canine/feline/zebrafish/drosophila/bovine/
   equine/porcine/avian/rodent/hamster/non-human/primate/"animal model"/veterinary, or semantic type
   Experimental Model of Disease/Mammal/Animal/Vertebrate/Organism — **0 hits in 26.08e**, rule kept
   as a guard); semantic type not disease-like (kept set: Neoplastic Process, Disease or Syndrome,
   Finding, Congenital Abnormality, Pathologic Function, Cell or Molecular Dysfunction, Anatomical
   Abnormality, Acquired Abnormality, Sign or Symptom — observed under Neoplasm: 15 656 Neoplastic
   Process, 24 Disease or Syndrome, 3 Congenital Abnormality, 1 Pathologic Function, 1 Finding,
   1 Cell or Molecular Dysfunction; **0 excluded**); qualified disease state via
   `isQualifiedState()` from `@cancerindex/ontology` **plus** a connector-local
   `EXTRA_QUALIFIED_RE` for stage patterns the shared rules miss ("Stage 1/2A/4S Neuroblastoma",
   "Stage A/B/C/D Prostate Cancer", "Stage M0–M4 …", "Stage L1/L2/MS …", "Stage y-I …",
   "… FIGO 2023", "… by Toronto Guidelines v2"). Qualified states are stored as `source_records`
   (`entity_kind = qualified_state`) with `baseLabel = baseDiseaseLabel(name)` and, when the base
   label equals a kept preferred name, `baseCode`/`canonical_id`; the same mapping is written to the
   raw lake (`qualified_state_map`). They never become cancers or aliases.
3. **Header concepts** (`Header_Concept`, e.g. "Neoplasm by Site", "Malignant Neoplasm by Grade")
   are kept for connectivity with `entity_type = other` (17 rows) — they are structural, never listed
   as diseases.
4. **Retired concepts**: not in the subtree (they hang under "Retired Concept 2025", C223967). The
   565 retired concepts with semantic type Neoplastic Process (664 retired rows have an *empty*
   semantic type despite the ReadMe's guarantee — accepted by the parser) and the 256 obsolete
   subtree concepts are recorded as `source_records` with `status = deprecated`. When a retired
   concept's preferred name equals an active kept concept's name, the retired code is mapped to it
   (`cancer_codes` system `ncit`, `match_type ONTOLOGY_EXACT`, `valid_to = 26.08e`; 48 such
   redirects) — e.g. `C3556 → C61574 "Malignant Uterine Corpus Neoplasm"`.
5. **Entity type** (first match): header → `other`; depth ≤ 3 and ≥ 15 neoplasm descendants →
   `cancer_family`; name matches `MOLECULAR_RE` (`EGFR-Mutant`, `HER2-Positive`, `… with NPM1
   Mutation`, `t(8;21)`, `Triple-Negative`, `Mismatch Repair-Deficient`…) → `molecular_subtype`;
   not malignant and /in situ|intraepithelial neoplasia|dysplasia|precursor|monoclonal
   gammopathy|myelodysplastic|premalignant/ → `precursor_condition`; under Morphology and not under
   Site → `histology`; hematologic → `hematologic_malignancy`; else depth ≥ 4 → `subtype`, else
   `cancer`. Deviation from the brief's order: molecular and precursor are tested before histology
   so that "Carcinoma In Situ" (a morphology node) is a precursor and molecular subtypes under
   morphology are not typed as histology. Curated top-level entries typed `subtype` are promoted to
   `cancer` (they are ranking scopes).
6. **Aliases** (`cancer_aliases`, `source_terminology = NCIt`): preferred name → `preferred`; other
   synonyms → `synonym`, or `abbreviation` when 2–6 upper-case alphanumerics with ≥ 2 letters (GBM,
   NSCLC, PDAC…); display name → `display`. Deduplicated per concept on `normalizeLabel()` (first
   wins, e.g. "Neoplasms, NOS" collapses into the preferred "Neoplasm"); normalized forms shorter
   than 2 chars skipped. Batched 500/insert with `ON CONFLICT DO NOTHING`.
7. **Codes**: `ncit` `EXACT_IDENTIFIER` for every kept concept; for the 36 `TOP_LEVEL_CANCERS`
   also `icd10` ranges as `CURATED_EXACT`, `top_level = true`, and the registry display name
   (e.g. "Pancreatic Cancer") as a `display` alias — `canonical_name` stays the NCIt preferred name.
8. **Hierarchy** (`cancer_hierarchy`, `hierarchy_type = ncit`): every parent inside the kept set;
   when all parents were excluded, the nearest kept ancestor(s) found by walking up ≤ 10 levels
   (146 concepts re-attached).
9. **Anatomy**: the 15 direct children of "Neoplasm by Site" become `anatomical_sites` (name =
   preferred name minus trailing "Neoplasm", reused by slug when OncoTree already created one) and
   every kept descendant is linked via `cancer_anatomy(relation = primary)`.
10. **Provenance**: one run-level row (dataset "NCI Thesaurus", version 26.08e, `expert_curation`,
    `open`) plus one row per definition (source URL
    `https://ncit.nci.nih.gov/ncitbrowser/ConceptReport.jsp?dictionary=NCI_Thesaurus&code=<code>`)
    referenced by `cancers.description_provenance_id`; rows are only added when the definition is
    new or changed.
11. **Idempotency / guard**: entities are matched on `primary_ncit_code`; slugs never change;
    unchanged rows are not rewritten. Anomaly guard: the run refuses to persist when the FLAT zip is
    < 5 MB, < 100 000 concepts parse, or the kept count drops below 90 % of the previous run
    (`anomalyGuard.minRatioOfPrevious = 0.9`, stored in `connector_cursors.cursor.keptCount`).
    Only after the guard passes are NCIt-anchored cancers that vanished / were retired / were
    reclassified as qualified states set to `status = deprecated` with an explicit
    `deprecated_reason` (`merged_into` set when a same-name replacement exists).

## Counts observed (NCIt 26.08e, 2026-09-08, database `cancerindex_a`)

| Metric | Value |
|---|---|
| Concepts in file / parsed | 213 083 / 213 083 (664 retired rows with empty semantic type accepted) |
| Neoplasm subtree | 15 682 |
| Kept → canonical cancers | **9 327** (first run 9 446 before the extra stage rules; 119 then deprecated as qualified states) |
| Qualified states (source records only) | 6 099 (3 844 resolve to a base disease by `baseDiseaseLabel`) |
| Retired (neoplastic) / obsolete recorded as deprecated source records | 565 / 256 (821) |
| Animal / non-disease semantic type excluded | 0 / 0 |
| Malignant / hematologic / pediatric (active, NCIt-anchored) | 5 595 / 1 426 / 526 |
| Entity types | subtype 6 923 · hematologic_malignancy 1 149 · molecular_subtype 384 · histology 300 · cancer_family 229 · cancer 169 · precursor_condition 156 · other 17 |
| Depth distribution | 0:1 · 1:3 · 2:51 · 3:339 · 4:1 269 · 5:2 662 · 6:2 864 · 7:1 560 · 8:578 · 9:90 · 10:25 · 11:4 |
| Aliases (NCIt) | 32 642 (preferred 9 327, synonym ~23 000, abbreviation ~600, display 36 top-level) |
| `cancer_codes` | ncit EXACT 9 327 (+48 retired→replacement ONTOLOGY_EXACT), icd10 38 |
| Hierarchy edges (ncit) | 16 438 |
| Anatomical site groups / `cancer_anatomy` links | 15 / 12 545 |
| Description provenance rows | 9 446 (every kept concept has a definition) |
| Top-level set | **36/36** — `C3556` (corpus-uteri) is *retired* in 26.08e and resolved to its same-name replacement `C61574`; `TOP_LEVEL_CANCERS` should be updated to `C61574` |
| Duplicate canonical names among active malignant cancers at the same depth | 0 (0 across all active cancers) |
| `^(stage|recurrent|metastatic|refractory)` among active NCIt-anchored cancers | 0 (the single remaining hit is OncoTree's "Refractory Cytopenia of Childhood", a WHO MDS entity created by the OncoTree connector) |
| Run time | full run 10 s (download 0.5 s, parse+classify ~1.5 s, writes ~8 s); rerun 7 s (0 created, 30 updated, 9 297 unchanged) |
| OncoTree after NCIt | 897 nodes: 32 tissues, 865 mapped; 682 onto NCIt-anchored entities (798 EXACT_IDENTIFIER + 67 ONTOLOGY_EXACT codes), 183 created new without NCIt anchor |

## Known limitations

- `entity_type` is heuristic (depth-based); most organ-level malignancies sit at depth ≥ 4 in NCIt
  and are typed `subtype` (e.g. "Breast Carcinoma", "Glioblastoma"). Curated top-level entries are
  promoted to `cancer`. A curated layer or descendant-count thresholds per depth would refine this.
- `cancer_family` also matches large non-malignant groupers ("Lung Neoplasm", 178 of 229).
- Hierarchy edges are never mass-deleted; edges to nodes deprecated by a later run remain (247 on
  `cancerindex_a` after the reclassification rerun; a fresh pipeline never creates them). Consumers
  must filter on `cancers.status = 'active'`.
- OncoTree runs after NCIt and reuses our site groups by slug (e.g. `breast`), overwriting
  `anatomical_sites.ncit_code` with its own anatomy code (C12971 vs our C2910). Harmless but noted.
- No ICD-O, UMLS or MeSH codes come from the FLAT file; OncoTree contributes UMLS CUIs (650).
- Semantic-type filter and animal regex were validated on 26.08e only; new semantic types under
  Neoplasm are surfaced in the run log (`non-disease concepts under Neoplasm: …`).
- `unresolved_labels` is untouched by this connector (it defines the vocabulary; nothing to resolve).

## Recommendation for downstream reconciliation (trials, literature, registries)

Trial conditions and registry labels are frequently *qualified* ("Stage IV Pancreatic Cancer",
"Recurrent Glioblastoma", "Metastatic HER2-Positive Breast Carcinoma"). Those NCIt concepts are
deliberately not canonical entities, so `CancerResolver.byLabel()` will miss them. The trials and
literature connectors should implement this fallback chain (it belongs in `@cancerindex/ontology`,
not in this connector):

1. `resolve({ codes })` on any NCIt/UMLS/MeSH identifier the source carries (EXACT_IDENTIFIER).
2. `byLabel(label)` (ONTOLOGY_EXACT / ALIAS).
3. If it misses and `isQualifiedState(label) || EXTRA_QUALIFIED_RE.test(label)`: strip qualifiers
   with `baseDiseaseLabel(label)` (iteratively, also removing "Stage <token>" prefixes and staging
   suffixes) and retry `byLabel(base)`; record the mapping with `match_type = CURATED_BROADER`
   (the trial is about a *narrower* state of the base disease) and keep the original label + stage
   in `cohort_definitions.stage` / `attributes`.
4. Otherwise `ctx.recordUnresolved('cancer', label, normalizeLabel(label), ctx)`.

The raw-lake file `qualified_state_map` (code → `baseLabel` → `baseCode`, 3 844 resolved) can be
loaded as a curated seed for that step. Upstreaming `EXTRA_QUALIFIED_RE` into
`packages/ontology/src/qualifiers.ts` (with tests) is recommended so both sides share one rule set.