# Connector `mesh` — MeSH (Medical Subject Headings, NLM)

| | |
|---|---|
| Source | https://www.nlm.nih.gov/mesh/ — U.S. National Library of Medicine |
| Tier / category | 1 / terminology |
| Access | SPARQL over HTTP GET, `https://id.nlm.nih.gov/mesh/sparql?query=…&format=JSON&inference=true`, no authentication; fallback `https://nlmpubs.nlm.nih.gov/projects/mesh/MESH_FILES/xmlmesh/desc<year>.xml` |
| Docs verified | 2026-09-08 — https://hhs.github.io/meshrdf/ (RDF model), live predicate probe on D001943, terms page |
| License | **Public domain (U.S. Government work)** under the NLM Terms and Conditions — acknowledgment requested — `licenseStatus: approved`, `redistribution: attribution` |
| Code | `packages/connectors/src/connectors/mesh/` (`manifest.ts`, `sparql.ts`, `xml.ts`, `index.ts`, `mesh.test.ts`, `fixtures/`) |

## Terms (quoted)

NLM Terms and Conditions (https://www.nlm.nih.gov/databases/download/terms_and_conditions.html, fetched 2026-09-08):

> acknowledge NLM as the source of the data by including the phrase "Courtesy of the U.S. National Library of Medicine"

> not indicate or imply that NLM has endorsed its products/services/applications

Redistributors must "maintain the most current version of all distributed data" or make known "in a clear and conspicuous manner" that their product does not reflect NLM's latest data. "No charges, usage fees or royalties are paid to NLM." NLM web policies (https://www.nlm.nih.gov/web_policies.html): "Works produced by the U.S. government are not subject to copyright protection in the United States."

CancerIndex shows "Courtesy of the U.S. National Library of Medicine", the MeSH year (`datasetVersion`, e.g. `MeSH 2026`) and the retrieval date on the `/source/mesh` page and in the API `sources` envelope.

## What MeSH is for in CancerIndex

MeSH is the indexing vocabulary of PubMed, **not** a disease ontology: it never creates canonical cancers (CLAUDE.md §340). The connector gives existing canonical cancers their MeSH descriptor UI so that

1. the PubMed connector can build MeSH-anchored literature queries — `("Breast Neoplasms"[mh])` instead of title/abstract phrases — and
2. publication → cancer edges resolve by UI (`resolver.byCode('mesh', ui)`) or descriptor name (`byLabel(name, {allowMeshInversion: true})`).

## Verified endpoint behaviour (2026-09-08)

- Descriptor model (probed on `mesh:D001943` "Breast Neoplasms"): `rdfs:label`, `meshv:identifier`, `meshv:treeNumber` → resource whose `rdfs:label` is the tree number (`C04.588.180`, `C17.800.090.500`), `meshv:preferredConcept` / `meshv:concept` → Concept (`meshv:scopeNote`, `meshv:term` / `meshv:preferredTerm`) → Term (`meshv:prefLabel`, `meshv:altLabel`), `meshv:dateIntroduced`, `meshv:broaderDescriptor`.
- The default dataset spans every year graph (`<http://id.nlm.nih.gov/mesh/2015>` … `/2026`, 30,512 topical descriptors in 2026): a naive C04 count returns 8,989 rows. Queries are restricted with `FROM <http://id.nlm.nih.gov/mesh>` (current vocabulary) → **703 descriptors under C04**.
- The endpoint answers SPARQL-XML as `text/plain` when the `Accept` header lists `application/json`, ignoring `format=JSON`; with `Accept: */*` it honours the parameter. The connector overrides the SDK default header per request.
- One page (LIMIT 1000) of the grouped descriptor query returns in ~1.2 s (≈ 0.8 MB JSON).
- Year graph detection: `ASK { GRAPH <…/mesh/2027> { … } }` then `/2026` → `datasetVersion = "MeSH 2026"`.
- XML fallback: `desc2026.xml`, Last-Modified 2026-08-12, ~313 MB.

## Scope

Tree-number prefixes fetched (`TREE_PREFIXES` in `sparql.ts`):

| Prefix | Content | Descriptors (MeSH 2026) |
|---|---|---|
| `C04` | Neoplasms — whole tree (by site, by histologic type, syndromes, precancerous conditions, experimental neoplasms) | 703 |
| `C15.378.190.6*` | Bone Marrow Diseases → Myelodysplastic Syndromes (.625), Myeloproliferative Disorders (.636), MDS/MPN (.615) | 31 outside C04 |
| `C20.683` | Immunoproliferative Disorders (lymphoproliferative disorders, paraproteinemias) | — |

Total 734 distinct descriptors. The two extra subtrees contain non-neoplastic entries (Infectious Mononucleosis, Hypergammaglobulinemia, Anemia, Refractory…): those are mapped when the resolver knows them and otherwise **skipped silently** — only C04 misses enter the curation queue.

## Sync design

1. `detectYear` → `datasetVersion`.
2. `fetchViaSparql`: paged grouped query (`descriptorsQuery(offset)`, LIMIT 1000): per descriptor the heading, all tree numbers, the entry terms of the **preferred concept** (`?terms`), the entry terms of the **other concepts** (`?related`), scope note, `dateIntroduced`. Rows failing the `Descriptor` schema are counted (`validationFailures`, `rejected`) and logged, never dropped silently. If the SPARQL route throws, `fetchViaXml` streams the descriptor XML, splits it on `</DescriptorRecord>` boundaries and parses each record with fast-xml-parser (`PreferredConceptYN` separates synonyms from other concepts; records outside the prefixes are discarded). `dry_run` stops after fetching.
3. Anomaly guard: fewer than 500 descriptors → the run fails without writing (CLAUDE.md §171).
4. One `provenance` row per run (dataset "MeSH descriptors — Neoplasms tree …", `evidenceType expert_curation`, `accessLevel open`).
5. Per descriptor: `ctx.upsertSourceRecord('descriptor', <UI>, {ui, label, treeNumbers, terms, relatedTerms, scopeNote, dateIntroduced, meshYear, inC04})` (raw lake `data/raw/mesh/<date>/descriptor/`), then reconciliation (below), then `cancer_codes` / `cancer_aliases` writes.
6. Cursor `{year, route, descriptors, syncedAt}`; the run is idempotent (`onConflictDoNothing` everywhere, source records `unchanged` on re-runs).

### Reconciliation (CLAUDE.md §69, §221) — `mapDescriptor()`

| Step | Input | Accepted | Written |
|---|---|---|---|
| 1 | heading, `resolver.byLabel(label, {allowMeshInversion: true})` ("Carcinoma, Non-Small-Cell Lung" → "Non-Small-Cell Lung Carcinoma") | `ONTOLOGY_EXACT` / `CURATED_EXACT` / `ALIAS` with confidence ≥ 0.9 | `cancer_codes (system 'mesh', code = UI, match_type = resolver type)` + heading alias + entry-term aliases |
| 1b | heading → `CURATED_BROADER` (qualified state / lineage) | kept as fallback | `cancer_codes` with `match_type CURATED_BROADER`, **no aliases** (useful for PubMed queries at the broader level) |
| 2 | each entry term **of the preferred concept**, same call | exact grades ≥ 0.9 | as step 1 but `match_type ALIAS` (confidence capped at 0.9) |
| 3 | nothing | — | C04 descriptors → `unresolved_labels` (`entity_kind cancer`, context `{meshUi, treeNumbers, entryTerms, scopeNote}`, fuzzy suggestion from `resolver.suggest` for the curator); outside C04 → counted as skipped |

Why preferred-concept terms only: MeSH lists the terms of *narrower and related concepts* as entry terms of a descriptor ("Adenocarcinoma, Basal Cell" and "Carcinoma, Tubular" under Adenocarcinoma, "Glioblastoma, Giant Cell" under Glioblastoma, "Breast Cancer" under Breast Neoplasms). Using them for mapping sent D000230 Adenocarcinoma to *Basal Cell Adenocarcinoma* and D005909 Glioblastoma to *Giant Cell Glioblastoma* in a first run. They are now stored in the raw record (`relatedTerms`) and never used for mapping or aliases.

### Aliases

- The heading is stored as `alias_type 'mesh_heading'`, `source_terminology 'MeSH'`. A dedicated type guarantees the row exists even when NCIt already has an identical `synonym` (unique key is cancer + normalized + type): the PubMed connector selects aliases whose `source_terminology` matches `/mesh/i` to build `"<heading>"[mh]` queries, and its `[tiab]` term ranking ignores unknown alias types, so `mesh_heading` never leaks into title/abstract queries.
- Entry terms of the preferred concept are stored as `alias_type 'synonym'`, `source_terminology 'MeSH'`, after `aliasWorthyTerms()`: ≥ 3 characters, not the heading, not already present for that cancer under any type, and not a permutation of a kept term ("Neoplasm, Breast" / "Neoplasms, Breast" / "Breast Neoplasm" all normalize to the same un-inverted string as "Breast Neoplasms").
- `CURATED_BROADER` mappings get the code only.

## Observed run (cancerindex_c, 2026-09-08, NCIt 26.08e + OncoTree + CIViC/ClinVar/GDC loaded — 10,192 cancers)

| | |
|---|---|
| Duration | 7 s (3 SPARQL requests: 2 year probes + 1 page) |
| Descriptors fetched | **734** (703 in C04, 31 in the hematologic/immunoproliferative subtrees) — 0 malformed rows |
| Mapped | **428** — ONTOLOGY_EXACT 219, ALIAS 202, CURATED_EXACT 3, CURATED_BROADER 4 |
| C04 resolution | **419 / 703 (59.6 %)**; 10 cancers received two descriptors (e.g. two MeSH descriptors for one NCIt concept) |
| Aliases written | 424 `mesh_heading` (414 distinct cancers now MeSH-anchorable for PubMed) + 860 `synonym` (source_terminology MeSH) |
| Unresolved (C04, queued) | 284; outside-C04 skipped 22 |
| Spot checks | D009369 Neoplasms → Neoplasm (ONTOLOGY_EXACT); D008175 Lung Neoplasms → Lung Neoplasm; D002289 Carcinoma, Non-Small-Cell Lung → Lung Non-Small Cell Carcinoma (ALIAS via inversion); D015464 → Chronic Myeloid Leukemia, BCR-ABL1 Positive; D001943 Breast Neoplasms → Breast Neoplasm (ALIAS via "Breast Tumor"); D005909 Glioblastoma → Glioblastoma; D018285 Klatskin Tumor → Hilar Cholangiocarcinoma; D001913 Bowen's Disease → Squamous Cell Carcinoma In Situ; D006394 Hemangiosarcoma → Angiosarcoma |

Top 20 unresolved descriptors (alphabetical, each seen once): Abdominal Neoplasms, Aberrant Crypt Foci, ACTH Syndrome Ectopic, Adenocarcinoma in Situ, Adenocarcinoma Bronchiolo-Alveolar, Adenofibroma, Adenoma Acidophil, Adenoma Basophil, Adenoma Chromophobe, Adenomatosis Pulmonary, Adenomatous Polyposis Coli, Anal Gland Neoplasms, Anaplasia, Angiokeratoma, Anti-N-Methyl-D-Aspartate Receptor Encephalitis, Arachnoid Cysts, Atypical Squamous Cells of the Cervix, Avian Leukosis, Barrett Esophagus, Basal Cell Nevus Syndrome. Three groups: (a) non-neoplastic or veterinary descriptors filed under C04 (Anaplasia, Arachnoid Cysts, Avian Leukosis, paraneoplastic syndromes) — correctly unmapped; (b) heading strings that are **ambiguous in this development database** because OncoTree ran before NCIt and minted duplicates ("Adenocarcinoma, NOS" next to NCIt "Adenocarcinoma", "Sarcoma, NOS", "Teratoma", "Paraganglioma"): the resolver refuses ambiguous aliases, so Adenocarcinoma (D000230), Sarcoma (D012509 → resolved via entry term to Soft Tissue Sarcoma) and Teratoma stay open or land on a sibling; with NCIt loaded first the OncoTree nodes map onto NCIt concepts and these headings resolve exactly; (c) genuine vocabulary gaps for the curation queue (Adenoma, Acidophil; Anal Gland Neoplasms; Adenocarcinoma, Bronchiolo-Alveolar — the WHO-retired term).

### Effect on PubMed

Before this connector `literature_counts` queries were all `[tiab]`. After it, `pubmed` `loadTargets()` sees 424 cancers with MeSH names; see `docs/connectors/pubmed.md` and the run report in the integration notes for the `[mh]` share measured on the development database.

## Limitations / notes

- MeSH descriptors cover benign and malignant neoplasms alike; the descriptor is attached to whichever canonical concept the heading names (Breast Neoplasms → Breast Neoplasm, not Malignant Breast Neoplasm). Literature counts anchored on such headings therefore include benign-tumour literature — the query string is stored with every count (CLAUDE.md §251).
- Tree numbers are stored in the source record only; a `hierarchy_type 'mesh'` dimension is a future addition.
- Supplementary Concept Records (C-numbers used by PubMed for rare tumours) are not fetched.
- The XML fallback has been exercised on a fixture, not on the 313 MB file; it exists so a SPARQL outage does not block the monthly run.
- Alert bookkeeping in `scripts/ci.ts` warns when `system_alerts` is absent on a development database created before that table existed (`drizzle-kit push` fixes it locally).