# Connector: `pubmed` — PubMed via NCBI E-utilities

Tier 0 · category `literature` · code `packages/connectors/src/connectors/pubmed/`
Documentation verified **2026-09-08** (einfo: DbBuild `Build-2026.09.07.18.09`, 41.1 M records).

## Source

| | |
|---|---|
| Organization | U.S. National Library of Medicine / NCBI |
| Base URL | `https://eutils.ncbi.nlm.nih.gov/entrez/eutils` |
| Auth | none; `tool` + `email` on every request (env `NCBI_TOOL`, `NCBI_EMAIL`); optional `NCBI_API_KEY` |
| Docs | E-utilities In-Depth https://www.ncbi.nlm.nih.gov/books/NBK25499/ · Usage guidelines https://www.ncbi.nlm.nih.gov/books/NBK25497/ |
| Terms | https://www.nlm.nih.gov/databases/download/terms_and_conditions.html · PubMed README https://ftp.ncbi.nlm.nih.gov/pubmed/README.txt |

### Verified endpoints and parameters

- `esearch.fcgi` — `db=pubmed`, `term`, `rettype=count` (count only) or `uilist`, `retmax` (≤ 10,000;
  "ESearch can only retrieve the first 10,000 records"), `retmode=json`, `sort=pub_date` (live check:
  `sort=date` → "Unknown sort schema 'date' ignored"), date restriction either `datetype=pdat&mindate&maxdate`
  or inside the term as `YYYY/MM/DD:YYYY/MM/DD[dp]` (both verified to give identical counts).
  Response headers: `x-ratelimit-limit: 3`, `x-ratelimit-remaining` (logged by the connector).
- `efetch.fcgi` — `db=pubmed`, `id=<comma list>`, `retmode=xml` (PubmedArticleSet, DTD `pubmed_250101`);
  "If more than about 200 UIDs are to be provided, the request should be made using the HTTP POST method"
  — we send 50 per GET.
- `einfo.fcgi?db=pubmed&retmode=json` → `dbbuild`, `lastupdate`, `count` (health check).
- MeSH UIs are **not** searchable (`D001943[mh]` → 0); descriptor names are (`"Breast Neoplasms"[mh]` → 387,045).

### Usage policy (verbatim, NBK25497)

> "post no more than three URL requests per second" (without key) — with an API key "a site can post up to 10 requests per second by default. Higher rates are available by request"

> "limit large jobs to either weekends or between 9:00 PM and 5:00 AM Eastern time during weekdays"

> for extensive PubMed projects: "download a local copy of the database"

Policy limits are 3 req/s (no key) / 10 req/s (key), decided by `effectiveRequestsPerSecond()`. The
connector runs at **2.5 req/s** (manifest) and switches to a dedicated **8.5 req/s** client when
`NCBI_API_KEY` is set — 15 % headroom because token-bucket bursts at exactly 3 req/s produced 230 HTTP 429s
(all retried with `Retry-After`) in the first real run; concurrency 1. We deliberately do **not** mine the
4,290,331 `neoplasms[mh]` records through the API; that would require the annual baseline files.

### Terms (verbatim)

PubMed README (Last Updated January 30, 2026):

> "NLM freely provides PubMed data. Please note some abstracts may be protected by copyright."

> Users of the data agree to: "acknowledge NLM as the source of the data in a clear and conspicuous manner", "NOT use the PubMed wordmark or the PubMed logo in association or in connection with user's or any other party's product or service", "NOT to indicate or imply that NLM/NIH/HHS has endorsed its products/services/applications."

> Users who republish or redistribute the data agree to: "maintain the most current version of all distributed data, or make known in a clear and conspicuous manner that the products/services/applications do not reflect the most current/accurate data available from NLM."

NLM general terms: "acknowledge NLM as the source of the data by including the phrase 'Courtesy of the U.S. National Library of Medicine' in a clear and conspicuous manner".

**What we store / redisplay.** Bibliographic metadata (title, journal, dates, authors, MeSH, ids, publication
types) is stored and shown. Abstract text is stored in `publications.abstract` for indexing and
reconciliation only; the web layer must display at most the first ~300 characters followed by a link to
`https://pubmed.ncbi.nlm.nih.gov/<pmid>/` (abstracts may be under publisher copyright). Every page showing
PubMed-derived numbers carries the attribution and the retrieval date (`literature_counts.updated_at`,
provenance `retrieved_at`). Manifest: `licenseStatus approved`, `redistribution attribution`.

## Job (a): literature counts per cancer (`literature_counts`)

Research-activity metric (CLAUDE.md §31, §251). For every active, malignant canonical cancer with a
defensible query:

**Query construction** (`query.ts`, the exact string is stored in `literature_counts.query`):

- MeSH-anchored when descriptor *names* are known for the cancer (`cancer_aliases.source_terminology`
  matching `mesh`; a bare MeSH UI in `cancer_codes` is not enough because `[mh]` needs names):
  `("Breast Neoplasms"[mh])`.
- Otherwise title/abstract phrases: preferred name + abbreviation aliases + synonyms, each sanitized
  (quotes/brackets removed), normalized length ≥ 4, not a generic label (`GENERIC_WORDS`), not a
  qualified state (`isQualifiedState`), not a common word, distinct; at most 8 terms:
  `("Pancreatic Ductal Adenocarcinoma"[tiab] OR "PDAC"[tiab])`.
- Cancers whose preferred name is a lone generic word ("Carcinoma", "Neoplasm") are skipped unless
  `top_level` (8 of 798 on the OncoTree-only DB).

**Windows** (`windowsFor(now)`, deterministic, tested with a fixed date; PubMed `[dp]` bounds):

| key | definition (now = 2026-09-08) |
|---|---|
| `all` | no date filter (`window_end` = retrieval day) |
| `12m` | `2025/09/08:2026/09/08` |
| `5y` | last 5 full years + current year `2021/01/01:2026/12/31` |
| `5y_prior` | the 5 years before the last 12 months `2020/09/08:2025/09/07` (denominator of `publication_growth`) |
| `y2016 … y2025` | the ten complete years before now |

Top tier (= `top_level` **or** curated NCIt anchor from `TOP_LEVEL_CANCERS` **or** `entity_type in
(cancer, cancer_family, hematologic_malignancy)`) gets all 14 windows, refreshed when older than 7 days;
subtypes/other get `all, 5y, 12m` (3 requests) refreshed at most every 30 days. Targets are processed in a
deterministic order (top-level → top tier → rest, then id); the cursor `counts.afterKey` remembers the last
processed cancer so a run stopped by the time budget continues where it left off; a completed pass
resets it and increments `counts.passes`.

Each row: `ON CONFLICT (cancer_id, window_key) DO UPDATE`, with a fresh `provenance` row
(dataset `PubMed`, `dataset_version` = run date, `evidence_type database`, `access_level open`,
methodology `esearch count; query stored on row`, `source_url` = the esearch URL **without** the API key).

Feeds `entity_counters.publication_count / _5y / _12m` → rankings `publications_5y`, `publications_12m`,
`publication_growth` (= `12m / (5y_prior / 5)`), `research_gap`.

## Job (b): recent publications per top-level cancer (`publications`, `publication_entity_edges`)

Runs first in every run (bounded). For each top-level cancer with a query: esearch `retmax=50 sort=pub_date`
over the last 30 days → efetch XML in batches of 50 → `parseEfetchXml()` (fast-xml-parser; inline
markup in titles/abstracts stripped) → upsert `publications` on `pmid` (title, abstract, journal,
journalIso, pubDate/pubYear, publicationTypes, meshTerms `{descriptor, ui, major, qualifiers}`, authors
`{name, affiliation, orcid}`, language, doi/pmcid from `ArticleIdList`/`ELocationID`, `nctIds` = `NCT\d{8}`
in DataBankList + abstract + title, `retracted` = PublicationType "Retracted Publication" or
CommentsCorrections `RefType=RetractionIn` (notice kept in `retraction_notice`), `isPreprint` =
PublicationType "Preprint"). Source record `('article', pmid, parsedRecord)`. Cap 2,000 articles per run.

Edges (`ON CONFLICT DO NOTHING` on publication × entity × method):

| method | when | status | confidence |
|---|---|---|---|
| `mesh` | MeSH descriptor resolves via `byCode('mesh', ui)` or `byLabel(descriptor, { allowMeshInversion: true })` | `validated` | 0.95 major topic / 0.85 |
| `dictionary` | the query cancer's preferred name / query terms / abbreviations appear in the title (whole-word, case-insensitive, hyphen-tolerant) | `candidate` | 0.6 |
| `registry_reference` | an NCT id of the article exists in `clinical_trials.nct_id` (missing trials skipped) | `validated` | 1.0 |

## Restartability, health, credentials

- Cursor: `{ pubs: { lastRunAt, lastArticles }, counts: { afterKey, passStartedAt, lastPassCompletedAt, passes } }`,
  persisted every 10 cancers and at the end.
- `healthCheck` → einfo `DbBuild` / `LastUpdate`.
- `credentialsMissing()` → `NCBI_EMAIL` is mandatory (the run is marked `awaiting_credentials` otherwise).
- `mode=dry_run`: one esearch for the first top-level query + 3 cancers' counts logged, nothing written.

## Tests (`pubmed.test.ts`, fixtures only)

Query builder (alias filtering, quotes/brackets escaping, generic skip, MeSH preference), window date math
with a fixed `now`, rate-limit choice (3 vs 10 rps by env) and key-free provenance URLs, esearch JSON
(count / ids / empty), efetch XML sample of 3 articles (one retracted with an NCT id in DataBankList and a
MeSH major topic, one un-indexed with a DOI only, one retracted Nature article), title dictionary match.

## Limitations

- Title/abstract phrase queries over-count when a shorter name is contained in a longer one
  ("Hodgkin Lymphoma"[tiab] also matches "non-Hodgkin lymphoma": 26,033 vs 17,046 in the dry run) and
  under-count cancers known mainly by other phrasings. MeSH-anchored queries fix this as soon as MeSH
  descriptor names are attached to cancers by a terminology connector — the query kind is visible in the
  stored query string.
- With the OncoTree-only ontology only 4 top-level cancers are recognised (via curated NCIt anchors), so
  job (b) is small until NCIt lands; in production the `top_level` flag drives it.
- Counts are PubMed's own (`esearchresult.count`) and reflect indexing lag; `12m` is a moving window.
- No rate-limit key in `.env` yet → 3 req/s; a full top-tier pass (≈ 220 cancers × 14 windows ≈ 3,000
  requests) takes ≈ 25–40 min at the observed ≈ 1.3 req/s effective (network latency, concurrency 1).

## Run report (cancerindex_b, 2026-09-08, OncoTree-only ontology)

- `--mode dry_run`: 57 requests / 34 s; 798 targets (top-level 4 via curated NCIt anchors, top tier 217,
  8 without a defensible query); counts logged for 3 cancers, one recent-publications esearch (50 PMIDs).
- `--max-minutes 10` (`ING-PUBMED-20260908-000002`, status `partial`, cursor saved at `counts.afterKey = 1|CI-CAN-00000049`):
  937 requests, **230 HTTP 429s** (all retried successfully — the reason the keyless rate was lowered to 2.5 req/s
  afterwards), 823 records fetched/created:
  - job (b): 123 `publications` upserted (0 retracted, 4 with NCT ids), 62 `publication_entity_edges`
    (56 `dictionary/candidate`, 6 `registry_reference/validated`, 0 `mesh` — recent articles are not MeSH-indexed
    yet and OncoTree has no MeSH names);
  - job (a): 50 cancers × 14 windows = **700 `literature_counts` rows** with 700 `provenance` rows
    (e.g. Cutaneous Melanoma `all` 10,831 / `12m` 770 / `5y` 3,930 / `5y_prior` 3,459; Hodgkin Lymphoma `all` 26,033).
- `pnpm cix rank` → `publications_5y` eligible 41, `publications_12m` 39, `publication_growth` 31 (level=all).
  Top publications_5y: Hepatocellular Carcinoma 56,139 · Osteosarcoma 9,748 · Hodgkin Lymphoma 8,354 ·
  Head and Neck Squamous Cell Carcinoma 7,917 · Nasopharyngeal Carcinoma 6,142.

Estimated full first pass at 2.5 req/s without a key: top tier 217 × 14 ≈ 3,000 requests ≈ 25 min, then
≈ 580 subtypes × 3 ≈ 1,750 requests ≈ 15 min, spread over nightly runs by the cursor; with `NCBI_API_KEY`
about 3× faster.