# Connector: pubmed — PubMed via NCBI E-utilities

Tier 0 · category literature · code packages/connectors/src/connectors/pubmed/ Documentation verified 2026-09-08 (einfo: DbBuild Build-2026.09.07.18.09, 41.1 M records).

# Source

# Verified endpoints and parameters

• esearch.fcgi — db=pubmed, term, rettype=count (count only) or uilist, retmax (≤ 10,000; "ESearch can only retrieve the first 10,000 records"), retmode=json, sort=pub_date (live check: sort=date → "Unknown sort schema 'date' ignored"), date restriction either datetype=pdat&mindate&maxdate or inside the term as YYYY/MM/DD:YYYY/MM/DD[dp] (both verified to give identical counts). Response headers: x-ratelimit-limit: 3, x-ratelimit-remaining (logged by the connector).
• efetch.fcgi — db=pubmed, id=<comma list>, retmode=xml (PubmedArticleSet, DTD pubmed_250101); "If more than about 200 UIDs are to be provided, the request should be made using the HTTP POST method" — we send 50 per GET.
• einfo.fcgi?db=pubmed&retmode=json → dbbuild, lastupdate, count (health check).
• MeSH UIs are not searchable (D001943[mh] → 0); descriptor names are ("Breast Neoplasms"[mh] → 387,045).

# Usage policy (verbatim, NBK25497)

"post no more than three URL requests per second" (without key) — with an API key "a site can post up to 10 requests per second by default. Higher rates are available by request"

"limit large jobs to either weekends or between 9:00 PM and 5:00 AM Eastern time during weekdays"

for extensive PubMed projects: "download a local copy of the database"

Policy limits are 3 req/s (no key) / 10 req/s (key), decided by effectiveRequestsPerSecond(). The connector runs at 2.5 req/s (manifest) and switches to a dedicated 8.5 req/s client when NCBI_API_KEY is set — 15 % headroom because token-bucket bursts at exactly 3 req/s produced 230 HTTP 429s (all retried with Retry-After) in the first real run; concurrency 1. We deliberately do not mine the 4,290,331 neoplasms[mh] records through the API; that would require the annual baseline files.

# Terms (verbatim)

PubMed README (Last Updated January 30, 2026):

"NLM freely provides PubMed data. Please note some abstracts may be protected by copyright."

Users of the data agree to: "acknowledge NLM as the source of the data in a clear and conspicuous manner", "NOT use the PubMed wordmark or the PubMed logo in association or in connection with user's or any other party's product or service", "NOT to indicate or imply that NLM/NIH/HHS has endorsed its products/services/applications."

Users who republish or redistribute the data agree to: "maintain the most current version of all distributed data, or make known in a clear and conspicuous manner that the products/services/applications do not reflect the most current/accurate data available from NLM."

NLM general terms: "acknowledge NLM as the source of the data by including the phrase 'Courtesy of the U.S. National Library of Medicine' in a clear and conspicuous manner".

What we store / redisplay. Bibliographic metadata (title, journal, dates, authors, MeSH, ids, publication types) is stored and shown. Abstract text is stored in publications.abstract for indexing and reconciliation only; the web layer must display at most the first ~300 characters followed by a link to https://pubmed.ncbi.nlm.nih.gov/<pmid>/ (abstracts may be under publisher copyright). Every page showing PubMed-derived numbers carries the attribution and the retrieval date (literature_counts.updated_at, provenance retrieved_at). Manifest: licenseStatus approved, redistribution attribution.

# Job (a): literature counts per cancer (literature_counts)

Research-activity metric (CLAUDE.md §31, §251). For every active, malignant canonical cancer with a defensible query:

Query construction (query.ts, the exact string is stored in literature_counts.query):

• MeSH-anchored when descriptor names are known for the cancer (cancer_aliases.source_terminology matching mesh; a bare MeSH UI in cancer_codes is not enough because [mh] needs names): ("Breast Neoplasms"[mh]).
• Otherwise title/abstract phrases: preferred name + abbreviation aliases + synonyms, each sanitized (quotes/brackets removed), normalized length ≥ 4, not a generic label (GENERIC_WORDS), not a qualified state (isQualifiedState), not a common word, distinct; at most 8 terms: ("Pancreatic Ductal Adenocarcinoma"[tiab] OR "PDAC"[tiab]).
• Cancers whose preferred name is a lone generic word ("Carcinoma", "Neoplasm") are skipped unless top_level (8 of 798 on the OncoTree-only DB).

Windows (windowsFor(now), deterministic, tested with a fixed date; PubMed [dp] bounds):

Top tier (= top_level or curated NCIt anchor from TOP_LEVEL_CANCERS or entity_type in (cancer, cancer_family, hematologic_malignancy)) gets all 14 windows, refreshed when older than 7 days; subtypes/other get all, 5y, 12m (3 requests) refreshed at most every 30 days. Targets are processed in a deterministic order (top-level → top tier → rest, then id); the cursor counts.afterKey remembers the last processed cancer so a run stopped by the time budget continues where it left off; a completed pass resets it and increments counts.passes.

Each row: ON CONFLICT (cancer_id, window_key) DO UPDATE, with a fresh provenance row (dataset PubMed, dataset_version = run date, evidence_type database, access_level open, methodology esearch count; query stored on row, source_url = the esearch URL without the API key).

Feeds entity_counters.publication_count / _5y / _12m → rankings publications_5y, publications_12m, publication_growth (= 12m / (5y_prior / 5)), research_gap.

# Job (b): recent publications per top-level cancer (publications, publication_entity_edges)

Runs first in every run (bounded). For each top-level cancer with a query: esearch retmax=50 sort=pub_date over the last 30 days → efetch XML in batches of 50 → parseEfetchXml() (fast-xml-parser; inline markup in titles/abstracts stripped) → upsert publications on pmid (title, abstract, journal, journalIso, pubDate/pubYear, publicationTypes, meshTerms {descriptor, ui, major, qualifiers}, authors {name, affiliation, orcid}, language, doi/pmcid from ArticleIdList/ELocationID, nctIds = NCT\d{8} in DataBankList + abstract + title, retracted = PublicationType "Retracted Publication" or CommentsCorrections RefType=RetractionIn (notice kept in retraction_notice), isPreprint = PublicationType "Preprint"). Source record ('article', pmid, parsedRecord). Cap 2,000 articles per run.

Edges (ON CONFLICT DO NOTHING on publication × entity × method):

# Restartability, health, credentials

• Cursor: { pubs: { lastRunAt, lastArticles }, counts: { afterKey, passStartedAt, lastPassCompletedAt, passes } }, persisted every 10 cancers and at the end.
• healthCheck → einfo DbBuild / LastUpdate.
• credentialsMissing() → NCBI_EMAIL is mandatory (the run is marked awaiting_credentials otherwise).
• mode=dry_run: one esearch for the first top-level query + 3 cancers' counts logged, nothing written.

# Tests (pubmed.test.ts, fixtures only)

Query builder (alias filtering, quotes/brackets escaping, generic skip, MeSH preference), window date math with a fixed now, rate-limit choice (3 vs 10 rps by env) and key-free provenance URLs, esearch JSON (count / ids / empty), efetch XML sample of 3 articles (one retracted with an NCT id in DataBankList and a MeSH major topic, one un-indexed with a DOI only, one retracted Nature article), title dictionary match.

# Limitations

• Title/abstract phrase queries over-count when a shorter name is contained in a longer one ("Hodgkin Lymphoma"[tiab] also matches "non-Hodgkin lymphoma": 26,033 vs 17,046 in the dry run) and under-count cancers known mainly by other phrasings. MeSH-anchored queries fix this as soon as MeSH descriptor names are attached to cancers by a terminology connector — the query kind is visible in the stored query string.
• With the OncoTree-only ontology only 4 top-level cancers are recognised (via curated NCIt anchors), so job (b) is small until NCIt lands; in production the top_level flag drives it.
• Counts are PubMed's own (esearchresult.count) and reflect indexing lag; 12m is a moving window.
• No rate-limit key in .env yet → 3 req/s; a full top-tier pass (≈ 220 cancers × 14 windows ≈ 3,000 requests) takes ≈ 25–40 min at the observed ≈ 1.3 req/s effective (network latency, concurrency 1).

# Run report (cancerindex_b, 2026-09-08, OncoTree-only ontology)

• --mode dry_run: 57 requests / 34 s; 798 targets (top-level 4 via curated NCIt anchors, top tier 217, 8 without a defensible query); counts logged for 3 cancers, one recent-publications esearch (50 PMIDs).
• --max-minutes 10 (ING-PUBMED-20260908-000002, status partial, cursor saved at counts.afterKey = 1|CI-CAN-00000049): 937 requests, 230 HTTP 429s (all retried successfully — the reason the keyless rate was lowered to 2.5 req/s afterwards), 823 records fetched/created:
  • job (b): 123 publications upserted (0 retracted, 4 with NCT ids), 62 publication_entity_edges (56 dictionary/candidate, 6 registry_reference/validated, 0 mesh — recent articles are not MeSH-indexed yet and OncoTree has no MeSH names);
  • job (a): 50 cancers × 14 windows = 700 literature_counts rows with 700 provenance rows (e.g. Cutaneous Melanoma all 10,831 / 12m 770 / 5y 3,930 / 5y_prior 3,459; Hodgkin Lymphoma all 26,033).
• pnpm cix rank → publications_5y eligible 41, publications_12m 39, publication_growth 31 (level=all). Top publications_5y: Hepatocellular Carcinoma 56,139 · Osteosarcoma 9,748 · Hodgkin Lymphoma 8,354 · Head and Neck Squamous Cell Carcinoma 7,917 · Nasopharyngeal Carcinoma 6,142.

Estimated full first pass at 2.5 req/s without a key: top tier 217 × 14 ≈ 3,000 requests ≈ 25 min, then ≈ 580 subtypes × 3 ≈ 1,750 requests ≈ 15 min, spread over nightly runs by the cursor; with NCBI_API_KEY about 3× faster.