# Connector: clinicaltrials — ClinicalTrials.gov API v2

Tier 0 · category trials · code packages/connectors/src/connectors/clinicaltrials/ Documentation verified 2026-09-08 (API version 2.0.5, dataTimestamp 2026-09-04T09:00:06).

# Source

# Verified endpoints and parameters

Read from the OpenAPI document and exercised with curl on 2026-09-08:

• GET /studies — query.cond, query.term (Essie syntax), filter.overallStatus, filter.advanced (Essie), fields (comma/pipe list of area/piece/field names; branch pieces expand to descendants), pageSize ("coerced down to 1,000, if greater than that"; default 10), pageToken (opaque; "Do not specify it for first page"), countTotal (returns totalCount with the first page only; "ignored for the subsequent pages"), sort ("only date and numeric fields are allowed for sorting"; Field:asc|desc; studies missing the sort field are last). Response: { totalCount?, studies[], nextPageToken? }.
• GET /studies/{nctId} — single study (not used by the sync; kept for TRACE/repair).
• GET /studies/enums — enum values used by the validator (verified live): Phase NA | EARLY_PHASE1 | PHASE1 | PHASE2 | PHASE3 | PHASE4; Status ACTIVE_NOT_RECRUITING, COMPLETED, ENROLLING_BY_INVITATION, NOT_YET_RECRUITING, RECRUITING, SUSPENDED, TERMINATED, WITHDRAWN, AVAILABLE, NO_LONGER_AVAILABLE, TEMPORARILY_NOT_AVAILABLE, APPROVED_FOR_MARKETING, WITHHELD, UNKNOWN; StudyType EXPANDED_ACCESS | INTERVENTIONAL | OBSERVATIONAL; InterventionType BEHAVIORAL, BIOLOGICAL, COMBINATION_PRODUCT, DEVICE, DIAGNOSTIC_TEST, DIETARY_SUPPLEMENT, DRUG, GENETIC, PROCEDURE, RADIATION, OTHER; AgencyClass NIH, FED, OTHER_GOV, INDIV, INDUSTRY, NETWORK, AMBIG, OTHER, UNKNOWN.
• GET /version → { apiVersion: "2.0.5", dataTimestamp: "2026-09-04T09:00:06" } — used by healthCheck and stored as ingest_runs.dataset_version (the "date the data were processed by ClinicalTrials.gov" that the terms ask us to display).

Rate limits: none published; no x-ratelimit-* or Retry-After headers were observed (the connector logs them if they ever appear). Self-imposed: 3 req/s, concurrency 1, pageSize 1000. A 1,000-study page with the field list below is ≈ 5.8 MB and answered in ≈ 0.9 s.

# Terms and Conditions (read 2026-09-08, https://clinicaltrials.gov/about-site/terms-conditions)

Verbatim excerpts:

"ClinicalTrials.gov data are available to all requesters, both within and outside the United States, at no charge."

"In any publication or distribution of these data, you should: Attribute the source of the data as ClinicalTrials.gov; Update the data such that they are current at all times; Clearly display the date the data were processed by ClinicalTrials.gov; State any modifications made to the content of the data, along with a complete description of the modifications."

"You shall not assert any proprietary rights to any portion of the database, or represent the database or any part thereof to anyone as other than a United States Government database. You shall not use any email addresses extracted from our database for marketing or other promotional purposes."

"The ClinicalTrials.gov data carry an international copyright outside the United States and its Territories or Possessions. Some ClinicalTrials.gov data may be subject to the copyright of third parties; you should consult these entities for any additional terms of use."

Manifest decision: licenseStatus: approved, commercialUse: allowed, redistribution: attribution. Obligations implemented: attribution string in the manifest; dataTimestamp recorded per run; the modifications (field normalization, condition→cancer reconciliation) are described here; contacts (e-mail addresses) are not requested at all (contactsLocationsModule.locations only).

# Oncology scope

filter.advanced expression (stored verbatim in the cursor as expression; changing it resets the cursor):

AREA[ConditionSearch] (cancer OR neoplasm OR carcinoma OR tumor OR tumour OR lymphoma OR leukemia OR leukaemia OR sarcoma OR melanoma OR myeloma OR glioma OR glioblastoma OR mesothelioma OR blastoma OR malignan*)

Live counts on 2026-09-08 (data 2026-09-04):

ConditionSearch covers the free-text conditions, keywords and the derived MeSH condition terms and ancestors, so studies whose only condition is e.g. "Glioblastoma" or "NSCLC" are included through the MeSH ancestor "Neoplasms". Every ingested study is flagged is_oncology = true; conditions that are not cancers (e.g. "HIV Infections" on a Kaposi sarcoma trial) simply stay UNRESOLVED in trial_conditions.

# Fields requested

protocolSection.identificationModule, protocolSection.statusModule, protocolSection.sponsorCollaboratorsModule,
protocolSection.descriptionModule.briefSummary, protocolSection.conditionsModule, protocolSection.designModule,
protocolSection.armsInterventionsModule, protocolSection.outcomesModule, protocolSection.eligibilityModule,
protocolSection.contactsLocationsModule.locations, protocolSection.referencesModule, hasResults,
derivedSection.conditionBrowseModule, derivedSection.interventionBrowseModule

The trimmed payload (exactly these pieces) is what ctx.upsertSourceRecord('study', nctId, payload) hashes and writes to the raw lake (data/raw/clinicaltrials/<date>/study/).

# Crawl strategy (incremental, restartable)

• sort=LastUpdatePostDate:asc; incremental runs add AND AREA[LastUpdatePostDate]RANGE[<lastUpdatePostDate>,MAX].
• Cursor (connector_cursors.cursor, persisted after every page, also on failure): { expression, rangeStart, pageToken, lastUpdatePostDate, fullTotalCount, crawlTotalCount, pagesDone, completedAt, dataTimestamp }. A run interrupted by the time budget (--max-minutes, ctx.shouldStop() between pages) or by --max-records resumes at pageToken; a rejected/expired token (HTTP 400) restarts from RANGE[lastUpdatePostDate,MAX] (idempotent re-processing, same day re-read).
• Anomaly guard: on a full crawl, if totalCount < 50 % of the previous complete full crawl (fullTotalCount, also recovered from ingest_runs.cursor_after) the run aborts before writing. Nothing is ever mass-deleted on a shrunken response; source_records only gain last_seen_run.
• Idempotency: when the trimmed payload hash is unchanged and the trial exists, the study is skipped (unchanged counter). Otherwise the clinical_trials row is upserted on nct_id and the child tables are replaced for that trial (delete + insert): trial_locations (≤ 500 rows, lat/lng from geoPoint), trial_interventions (one row per distinct normalized name, drug_id null, match_type = UNRESOLVED — drug reconciliation is Phase 2), trial_conditions.
• Validation: validateTrial() (NCT id format, phase and status enums) + title presence; rejected records increment rejected/validationFailures and are logged, never thrown.
• mode=dry_run fetches one page (or --max-records), normalizes, validates and reconciles in memory, writes nothing.
• mode=backfill (pnpm cix run clinicaltrials --mode backfill --reset-cursor) re-crawls everything and re-processes studies whose payload is unchanged (child tables rewritten) — the repair path after a reconciliation-rule change; ontology growth alone is covered by the end-of-run re-reconciliation.
• Change events: on incremental crawls each new trial gets a change_events row (trial, created); the initial bulk crawl skips them (volume).

Smoke ladder on cancerindex_b (OncoTree-only ontology, 865 cancers), 2026-09-08:

# Condition reconciliation (trial_conditions)

Implemented in reconcile.ts, using CancerResolver from @cancerindex/ontology only (no OncoTree or NCIt specifics; the alias set grows as terminology connectors run):

1. Label candidates per condition string, most specific first: the string itself; baseDiseaseLabel() (stage/recurrence/resectability qualifiers stripped); registry noise removed ("Patients With", "Histologically Confirmed", "Adult", …); comma forms uninverted ("Lymphoma, Non-Hodgkin" → "Non-Hodgkin Lymphoma") and truncated ("Breast Cancer, Metastatic" → "Breast Cancer"); parenthetical acronyms split ("Non-Small Cell Lung Cancer (NSCLC)"). Each candidate goes through resolver.byLabel(c, { allowMeshInversion: true }) which itself handles "Cancer of the X" reorderings. A hit on a de-qualified candidate keeps the resolver's match_type with confidence capped at 0.85.
2. Study-level MeSH fallback (derivedSection.conditionBrowseModule.meshes): resolver.byCode('mesh', id) first (populated once a terminology connector maps MeSH UIs), then byLabel(term, { allowMeshInversion: true }). Because MeSH terms are attached to the study, not to one condition string, the inference is guarded (each guard was motivated by a wrong mapping observed in the first 1,000-study smoke run): the condition must look oncologic (ONCOLOGY_LABEL_RE; "Pain", "HIV Infections" never inherit) and not be a bare generic word ("Cancer", "Neoplasms" — those resolve by label to the generic NCIt entity or stay unresolved); every MeSH term of the study must resolve (or there is exactly one); exactly one MeSH cancer must remain after removing cancers already assigned to label-resolved conditions ("Breast Cancer" + "Lung Cancer" with MeSH Breast/Lung Neoplasms never maps Lung → Breast); and only one unresolved oncologic condition may claim it. The mapping is stored as PROBABILISTIC (confidence 0.7) unless the study has exactly one condition and one MeSH term (then the resolver's match type, confidence ≤ 0.9). MeSH ancestors are never used for mapping (too generic).
3. Otherwise match_type = UNRESOLVED, cancer_id null, and ctx.recordUnresolved('cancer', text, normalized, { nctId, meshTerms }) once per occurrence (so unresolved_labels.count = number of trial-condition occurrences). Generic labels ("Cancer", "Neoplasms", "Solid Tumor") resolve to the corresponding generic NCIt entity when the ontology provides one — that is the intended behaviour.
4. End of run: resolver.suggest() (pg_trgm) attaches a suggestion to the top 500 unresolved labels of the run (suggested_id/match_type=PROBABILISTIC/score) — curation aid only, never auto-accepted.
5. End of run: re-reconciliation of previously UNRESOLVED rows (≤ 50,000 distinct labels, label candidates only) against the freshly warmed resolver; hits update trial_conditions and mark the unresolved_labels row mapped with resolved_by = clinicaltrials:rereconcile:<match_type>. This is what makes the trial layer catch up automatically after the NCIt ontology lands.

DRUG/BIOLOGICAL interventions are queued once per distinct normalized name per run in unresolved_labels (entity_kind = drug, context { occurrences, sampleNctIds }); other intervention types are not queued.

# Derived table trial_pulse

refreshTrialPulse() recomputes (delete + insert, deterministic, formula ci-trial-pulse-v1) the last 400 days: new_trials per day (= first_posted_date, full YYYY-MM-DD dates only — YYYY-MM is never guessed) × cancer_id (any trial_conditions row with a cancer, distinct per trial) × phase (each value of phases plus the pseudo-phase ALL). Rows with cancer_id NULL are the all-oncology pulse.

# Metrics fed

entity_counters.trial_count / active_trial_count / recruiting_trial_count / phase3_trial_count (packages/ranking/src/counters.ts, aggregated over NCIt descendants) → rankings active_trials, recruiting_trials, phase3_trials, trial_gap.

# Limitations / gotchas

• Resolution rate depends entirely on the alias set. With OncoTree only (865 entities, no "Colorectal Cancer"/"Prostate Cancer"-style registry names) ≈ 40 % of condition occurrences resolve; the top unresolved labels are exactly the registry phrasings NCIt synonyms provide.
• PROBABILISTIC mappings count in entity_counters like any other mapping (the match type is stored on every row, so consumers can filter).
• Conditions of a non-oncology nature on oncology trials (HIV, anemia, pain…) stay unresolved by design; they will keep appearing in the unresolved queue until curators mark them ignored.
• HttpClient sends the plain product token CancerIndex/0.1 (some WAFs reject UAs with URLs).
• ClinicalTrials.gov page tokens are tied to the query; a token from a different pageSize is accepted (page size may change between pages per the docs), a token for a different filter is not (→ 400 → restart).
• Locations are capped at 500 per trial (locations_count keeps the true number).

# Run report (cancerindex_b, 2026-09-08, OncoTree-only ontology: 865 cancers / 865 aliases)

Final state: clinical_trials 126,107 · trial_conditions 278,132 (ONTOLOGY_EXACT 50,890 · PROBABILISTIC 1,025 · UNRESOLVED 226,313 → 18.8 % of condition occurrences resolved, 37,718 trials (29.9 %) with ≥ 1 cancer) · trial_locations 1,211,366 · trial_interventions 246,864 · trial_pulse 6,905 rows (400 days) · source_records(study) 126,107 · cursor { pageToken: null, lastUpdatePostDate: "2026-09-04", fullTotalCount: 126107, dataTimestamp: "2026-09-04T09:00:06" }.

Top 15 unresolved condition labels (occurrences in the last full pass): Prostate Cancer (4,498) · Cancer (4,030) · Breast Cancer (3,846) · Colorectal Cancer (3,508) · Lung Cancer (3,149) · Multiple Myeloma (2,350) · Lymphomas (2,107) · Ovarian Cancer (2,082) · Pancreatic Cancer (1,949) · Solid Tumors (1,930) · Gastric Cancer (1,832) · Leukemia (1,686) · Head and Neck Cancer (1,658) · Advanced Solid Tumor (1,652) · Cervical Cancer (1,348) — all registry phrasings that NCIt synonyms ("Breast Cancer" = Breast Carcinoma, "Malignant Solid Neoplasm", …) will resolve; the end-of-run re-reconciliation then repairs the stored rows automatically.

pnpm cix counters → 865 entities (Σ trial_count 70,211, Σ active_trial_count 17,896); pnpm cix rank → active_trials level=all eligible 383. Top 10 active_trials (level=all): Breast Neoplasm, NOS 1,302 · Non-Small Cell Lung Cancer 1,128 · Lymphoid Neoplasm 1,123 · Non-Hodgkin Lymphoma 1,038 · Myeloid Neoplasm 935 · Mature B-Cell Neoplasms 733 · Acute Myeloid Leukemia 724 · Hepatocellular Carcinoma 668 · Melanoma 519 · Head and Neck Squamous Cell Carcinoma 334.

Gotcha: unresolved_labels.count is incremented on every run that sees the occurrence, so backfill runs inflate it (632,516 after one full crawl + two backfills for 34,808 distinct cancer labels); treat it as a popularity signal, not an exact occurrence count — the exact number is count(*) FROM trial_conditions WHERE normalized = ….