spb/cancerindex

1# Connector `openfda` — Drugs@FDA applications and drug labels via openFDA23| | |4|---|---|5| Source | https://open.fda.gov — U.S. Food and Drug Administration (openFDA), datasets **Drugs@FDA** (`/drug/drugsfda.json`) and **drug labeling / SPL** (`/drug/label.json`) |6| Tier / category | 3 / regulatory (CLAUDE.md §13 — country-aware regulatory status, never a bare `approved = true`) |7| Access | REST `https://api.fda.gov`, no authentication; optional `OPENFDA_API_KEY` (query parameter `api_key`) |8| Docs verified | 2026-09-08 — https://open.fda.gov/apis/ (query syntax), https://open.fda.gov/apis/authentication/ (limits), https://open.fda.gov/apis/drug/drugsfda/, https://open.fda.gov/apis/drug/label/, live API |9| License | **Public domain, CC0 1.0 Universal** (https://open.fda.gov/license/, https://open.fda.gov/terms/) — `licenseStatus: approved`, `commercialUse: allowed`, `redistribution: allowed` |10| Code | `packages/connectors/src/connectors/openfda/` (`manifest.ts`, `normalize.ts`, `index.ts`, `openfda.test.ts`, `fixtures/`) |1112## Terms, license and disclaimer (quoted)1314Terms of service, "Data Rights and Usage" (https://open.fda.gov/terms/):1516> Unless otherwise noted, the content, data, documentation, code, and related materials on openFDA is public domain and made available with a Creative Commons CC0 1.0 Universal dedication. In short, FDA waives all rights to the work worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law. You can copy, modify, distribute, and perform the work, even for commercial purposes, all without asking permission. FDA makes no warranties about the work, and disclaims liability for all uses of the work, to the fullest extent permitted by applicable law. Some data on openFDA may not be public domain, such as copies of copyrightable works made available to the FDA by private entities.1718License page (https://open.fda.gov/license/): same CC0 1.0 dedication, plus "When using or citing the work, you should not imply endorsement by the author or the affirmer." (The GMDN device-terminology restriction on that page does not apply to Drugs@FDA / SPL data.)1920Disclaimer — returned in `meta.disclaimer` of **every** API response and repeated on the Drugs@FDA endpoint page ("Responsible use of the data"); stored verbatim in `manifest.termsNotes`:2122> Do not rely on openFDA to make decisions regarding medical care. While we make every effort to ensure that data is accurate, you should assume all results are unvalidated. We may limit or otherwise restrict your access to the API in line with our Terms of Service.2324> Always speak to your health provider about the risks and benefits of FDA-regulated products.2526CancerIndex mirrors this: regulatory status is shown as a dated, sourced fact per application (evidence label *regulatory*), never as a treatment recommendation.2728Rate limits (https://open.fda.gov/apis/authentication/, verified 2026-09-08):2930> With no API key: 240 requests per minute, per IP address. 1,000 requests per day, per IP address. With an API key: 240 requests per minute, per key. 120,000 requests per day, per key.3132The connector runs at 3 req/s (180/min) sequentially. Without `OPENFDA_API_KEY` it stops itself after ~950 requests in a run (`OPENFDA_DAILY_BUDGET_NO_KEY`) and resumes from its cursor on the next run; a full pass over ~660 drugs needs roughly 900–1,100 requests, so a key (free, https://open.fda.gov/apis/authentication/) is recommended for the weekly schedule. The key is never written to provenance URLs (`applicationProvenanceUrl`).3334## Verified endpoints3536| Endpoint | Verified behaviour (2026-09-08) |37|---|---|38| `GET /drug/drugsfda.json?search=openfda.generic_name:"osimertinib"&limit=100&skip=0` | `results[]` = applications: `application_number` (NDA208065), `sponsor_name`, `openfda{generic_name[], brand_name[], manufacturer_name[], substance_name[], unii[], rxcui[], pharm_class_epc[], pharm_class_moa[]…}`, `products[]{product_number, brand_name, active_ingredients[{name,strength}], dosage_form, route, marketing_status, reference_drug}`, `submissions[]{submission_type ORIG\|SUPPL, submission_number, submission_status AP\|TA\|…, submission_status_date YYYYMMDD, submission_class_code (TYPE 1…, EFFICACY, LABELING, MANUF (CMC)…), review_priority PRIORITY\|STANDARD, submission_property_type[{code Orphan}], application_docs[{url,type Label\|Letter\|Review…}]}`; `meta{disclaimer, terms, license, last_updated "2026-09-04", results{skip,limit,total}}`. Osimertinib: 23 submissions = 1 ORIG (TYPE 1, AP 2015-11-13, PRIORITY), 8 SUPPL/EFFICACY, 13 SUPPL/LABELING, 1 SUPPL/MANUF (CMC). |39| `GET /drug/drugsfda.json?search=openfda.brand_name:"tagrisso"+openfda.substance_name:"tagrisso"&limit=100` | OR search (space/`+` between terms) — fallback for single-token names. |40| `GET /drug/label.json?search=openfda.application_number:"NDA208065"&limit=1` | Current SPL: `set_id`, `id`, `version` (36), `effective_time` (20240925), `indications_and_usage[]` (one text blob: highlights with one indication per `( 1.N )` reference, then the full 1.1…1.N subsections), `openfda.*`; `meta.last_updated 2026-09-07`. |41| No match | HTTP **404** `{"error":{"code":"NOT_FOUND","message":"No matches found!"}}` — treated as an empty result (`isNoMatch`), not as a failure; other statuses go through the SDK retry/backoff. |42| Phrase search noise | `generic_name:"trastuzumab"` also returns ADO-TRASTUZUMAB EMTANSINE (BLA125427), FAM-TRASTUZUMAB DERUXTECAN-NXKI (BLA761139), PERTUZUMAB, TRASTUZUMAB, AND HYALURONIDASE-ZZXF (BLA761170), five biosimilars (TRASTUZUMAB-ANNS…) and HERCEPTIN (BLA103792). `generic_name:"imatinib"` → 12 applications (NDA021588 Gleevec, NDA219097 Imkeldi, 10 ANDAs). `generic_name:"pembrolizumab"` → BLA125514 Keytruda + BLA761467 Keytruda Qlex (pembrolizumab + berahyaluronidase alfa). |4344Health check: `GET /drug/drugsfda.json?limit=1` → healthy when `meta.last_updated` is present.4546## Ingestion4748Scope (bounded, CLAUDE.md §227): every row of the local `drugs` table (CIViC therapies today, trial interventions later), ordered by id; `ctx.cursor.lastDrugId` is saved after each drug so a run stopped by the time budget or the daily quota resumes exactly where it stopped; a completed pass starts a new one on the next schedule.49501. **Skip class / regimen names** (`ATR Inhibitor`, `Adjuvant Chemotherapy`, `Abemaciclib Regimen`…, regex `CLASS_NAME_RE` on multi-word names) — queued in `unresolved_labels` (`entity_kind drug`, `context.reason class_or_regimen_name`) without spending a request.512. **Search** `openfda.generic_name:"<name>"` (paged, ≤ 3 pages of 100); on no match, `brand_name OR substance_name` for single-token names, then ≤ 2 brand-like drug aliases via `openfda.brand_name`. Names with no accepted application → `unresolved_labels` (`reason no_drugsfda_match`, `{source:'openfda'}`).523. **Strict post-filter** (`classifyApplication`): an application is accepted only when its generic name / active ingredient is the molecule itself — exact, salt form (`IMATINIB MESYLATE`), biologic four-letter suffix (`TRASTUZUMAB-ANNS`) or FDA proper-name prefix (`FAM-TRASTUZUMAB DERUXTECAN` for "trastuzumab deruxtecan"). Fixed-dose combinations (several `substance_name`s, or "X, Y, and Z" generic names) and other molecules are `unmatched`. Kinds: `reference` (NDA/BLA of the molecule), `generic_or_biosimilar` (ANDA, or every matching name carries a biologic suffix). A drug named after an ADC (e.g. "Trastuzumab Emtansine") matches only under its own name.534. **Source records** `application` (payload = full application) for every accepted application, `label` (set_id) for approval-bearing ones; `drugs.unii` filled when empty; `openfda.brand_name[]` → `drug_aliases` (`alias_type brand`).545. **Approval rows** (`drug_approvals`, `jurisdiction US`, `authority FDA`, `status approved`) from the **reference** applications; when a molecule only exists as generics/biosimilars, the earliest-approved one bears the rows (others remain source records — ten identical ANDA rows would add no regulatory information). Events (`approvalEvents`): submissions with `submission_status = AP` and `submission_type = ORIG` → `approvalType 'ORIG'`; `SUPPL` with `submission_class_code = EFFICACY` → `approvalType 'SUPPL'`. Labeling / manufacturing supplements and tentative approvals (TA) are not approval events. `approvalDate = submission_status_date` (YYYYMMDD → YYYY-MM-DD). `applicationNumber` on every row; `raw` keeps submission number, review priority, class code, label/letter document URLs and the idempotency `key`.55   - **ORIG rows** carry the label's *Indications and Usage*. The highlights are split into one bullet per `( 1.N )` reference (`splitIndications`; fallbacks: `•` bullets, numbered subsections, whole text). Each bullet goes through the conservative dictionary pass (below): a bullet naming exactly one cancer becomes its own row (`cancerId`, `indication` = bullet text ≤ 4,000 chars, `raw.cancer_match = {matchType:'PROBABILISTIC', via:'label text dictionary', alias}`); the remaining bullets (0 or ≥ 2 cancers, tumor-agnostic) are joined into one row with `cancerId null` (`raw.cancer_match.matchType 'UNRESOLVED'`). When openFDA has no label for the application the ORIG row says so in `indication`.56   - **SUPPL rows**: `indication = "Efficacy supplement <date> (see label)"`, `cancerId null` (the supplement letter, not the current label, defines what changed — parsing it is out of scope).57   - **`accelerated`** stays `null` unless the bullet text says "accelerated approval" (then `true`). `review_priority = PRIORITY` is a *review track* (priority review), **not** accelerated approval (21 CFR 314 subpart H) and is recorded in `raw.reviewPriority` only. Withdrawn accelerated approvals are not detected (limitation).58   - **`tumorAgnostic`** is `true` only when a bullet says "solid tumors" together with a biomarker (NTRK, MSI-H/dMMR, TMB-H, BRAF V600E, RET fusion, HER2, KRAS G12C…); such bullets are never mapped to the generic "Solid Neoplasm" concept.596. **Provenance**: one row per application and run when anything changed (`sourceUrl https://api.fda.gov/drug/drugsfda.json?search=application_number:"…"`, `dataset 'Drugs@FDA via openFDA'`, `datasetVersion 'drugsfda-<meta.last_updated>'`, `evidenceType regulatory`, `accessLevel open`, `publishedAt` = first approval date, `updatedAtSource` = label `meta.last_updated`, methodology text). Unchanged application + unchanged label + all rows present → the previous provenance row is reused (no duplicate evidence per weekly run).607. **Knowledge edges** for rows with a `cancerId`: `drug —APPROVED_FOR→ cancer`, `evidenceCategory regulatory_status`, `evidenceLevel 'FDA ORIG' | 'FDA SUPPL'`, `direction supports`, `cancerContextIds [cancerId]`, `sourceRecordId = application_number`, `provenanceIds [provenance]` (unique per drug/cancer/application, refreshed not incremented).618. **Idempotency**: rows of an application are keyed by `raw.key` (`ORIG|<submission>|b<bullet>`, `ORIG|<submission>|rest`, `SUPPL|<submission>`); existing keys are updated, new keys inserted, keys that no longer derive from the freshly fetched payload are deleted **within that application only** (never a mass delete on a shrunken response — the payload was fetched successfully).6263`dry_run` probes osimertinib, pembrolizumab and imatinib and logs applications, kinds and event counts without writing.6465## Cancer dictionary (conservative, PROBABILISTIC)6667Built once per run from `cancer_aliases` of active, malignant concepts with `top_level = true` or `entity_type IN (cancer, cancer_family, hematologic_malignancy, subtype, histology, molecular_subtype)` — the spec's shorter list (cancer / cancer_family / hematologic_malignancy) was widened because NCIt models NSCLC, HNSCC, urothelial carcinoma, glioblastoma, colorectal cancer, CML (BCR-ABL1+) as `subtype` / `molecular_subtype`; without them most oncology labels would stay unmapped. Rules:6869- alias length ≥ 6 characters, or ≥ 4 for curated `abbreviation` aliases (NSCLC, HNSCC, GIST, DLBCL); 3-letter abbreviations (AML, CML, ALL) are excluded;70- generic words are stop-listed (`cancer`, `tumor`, `carcinoma`, `solid tumor`, `leukemia`, `lymphoma`, `sarcoma`, `adenocarcinoma`, `squamous cell carcinoma`, `carcinoma in situ`, `metastatic disease`…, `GENERIC_ALIAS_STOPLIST`);71- an alias shared by several concepts is kept only when `CancerResolver.byLabel` disambiguates it (preferred name, curated display name, broadest concept of one lineage — e.g. "breast cancer" → Malignant Breast Neoplasm); otherwise it is dropped (`ambiguousDropped`);72- matching is a whole-word n-gram lookup over `normalizeLabel(bullet)`; overlapping mentions resolve to the **longest** span ("metastatic breast cancer" beats "breast cancer"), then the bullet is mapped only if **exactly one** distinct cancer remains. "Ph+ CML in blast crisis" → CML; "HES and/or CEL" (two cancers) → null; "melanoma or breast cancer" → null.7374Every mapped row is `PROBABILISTIC` (text mention, not an identifier) and says so in `raw.cancer_match`; curators can review `drug_approvals WHERE cancer_id IS NULL` (`raw.bullets[].distinctCancers`).7576## Observed run (cancerindex_b, 2026-09-08)7778See the run summary appended below after the first 15-minute run.7980## Limitations / notes8182- One pass costs ~1.5 requests per drug (+1 label per approval-bearing application); without an API key the 1,000/day IP quota is the binding constraint (self-stop at 950, cursor saved).83- Efficacy supplements are dated but not mapped to cancers (no label diff); the current label's bullets are attached to the ORIG event date, so a bullet added by a 2024 supplement is stored under the original approval row set with the ORIG date — the supplement rows carry the actual dates. A future refinement can match `application_docs` label PDFs per supplement.84- `accelerated` relies on label wording ("approved under accelerated approval"); conversions to regular approval and withdrawals are not tracked yet (Project Confirm — spec §13 FDA Oncology, later).85- ANDA generics / biosimilars of a molecule that already has a reference NDA/BLA are kept as source records only.86- Line of therapy, disease stage and biomarker ids (`lineOfTherapy`, `diseaseStage`, `biomarkerIds`) are not extracted (Phase 2).87- Only US / FDA. Health Canada, EMA, MHRA, PMDA are separate connectors (spec §13).88