spb/cancerindex

1# Connector `ema` — European Medicines Agency, centrally authorised medicines (EU)23| | |4|---|---|5| Source | https://www.ema.europa.eu/en/medicines/download-medicine-data — EMA "Download medicine data": the **medicines** table (every medicine evaluated under the centralised procedure, human and veterinary; one row per EMA product number) as an xlsx report |6| Tier / category | 3 / regulatory (CLAUDE.md §13 — jurisdiction **EU**, authority **EMA**, `approval_type 'centralised'`; never a bare `approved = true`) |7| Access | Bulk download, no authentication: `https://www.ema.europa.eu/en/documents/report/medicines-output-medicines-report_en.xlsx` (897,597 bytes, `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`) |8| Docs verified | 2026-09-11 — download page + the live file with `User-Agent: CancerIndex/0.1`; EPAR URL pattern verified (HTTP 200) for xeljanz, keytruda, imfinzi |9| License | EMA legal notice (https://www.ema.europa.eu/en/about-us/legal-notice) — `licenseStatus: approved`, `commercialUse: allowed`, `redistribution: attribution`; attribution: *"Source: European Medicines Agency (www.ema.europa.eu), medicines data download; © EMA, reproduced with acknowledgement"* |10| Code | `packages/connectors/src/connectors/ema/` (`manifest.ts`, `normalize.ts`, `index.ts`, `ema.test.ts`, `fixtures/` incl. `build-fixtures.ts`); dependency `exceljs` (xlsx parsing) |1112## Terms and licence (quoted)1314From the legal notice, section "European Medicines Agency copyright and limited reproduction15notices" (verified 2026-09-11, stored verbatim in `manifest.termsNotes`):1617> Information and documents made available on EMA's webpages are public and may be reproduced and / or distributed, totally or in part, irrespective of the means and / or the formats used, for non-commercial and commercial purposes, provided that EMA is always acknowledged as the source of the material. Such acknowledgement must be included in each copy of the material. Citations may be made from such material without prior permission, provided the source is always acknowledged. The above-mentioned permissions do not apply to content supplied by third parties.1819The medicines table is EMA's own content (no third-party copyright is asserted on it). CancerIndex20reproduces the EMA product number, status, dates, flags and the therapeutic-indication summary as21a dated, sourced regulatory fact per medicine — never as a treatment recommendation.2223Update frequency (download page): *"The website automatically updates these data tables24overnight."* The connector runs weekly (`0 5 * * 3`), one download per run (≈ 3 s end to end),251 request/s, no documented rate limit.2627## Verified file layout (2026-09-11)2829One sheet, **"Medicine"**. Row 1 = `Content type: | Medicine | Output automatically generated from30content on www.ema.europa.eu | 11/09/2026 - 06:00` (the generation timestamp becomes31`datasetVersion = ema-medicines-2026-09-11T06:00`); rows 2–8 empty; **row 9 = header**; data from32row 10 — **2,732 rows** (2,339 Human, 393 Veterinary). Columns are matched by **normalized header33text** (lower-cased, whitespace and embedded newlines collapsed — `"Species\n(veterinary)"`,34`"Pharmacotherapeutic group\n(human)"`), never by position (`COLUMNS` in `normalize.ts`). A missing35**required** column fails the run loudly (`SchemaDriftError: required column(s) missing …`);36unknown columns are kept in `raw.extra` and reported; missing optional columns are warned.3738| Column (header) | Used as |39|---|---|40| `Category` | Human / Veterinary — only Human rows are mapped |41| `Name of medicine` | medicine name → `drug_codes.ema_product.label`, `brand` alias (single-drug medicines), `raw.medicine` |42| `EMA product number` | `EMEA/H/C/004214` → `application_number`, `drug_codes.ema_product`, source record id, edge `source_record_id` |43| `Medicine status` | Authorised, Withdrawn, Application withdrawn, Refused, Lapsed, Opinion, Expired, Revoked, Suspended, Opinion under re-examination, Withdrawn from rolling review → status mapping below |44| `International non-proprietary name (INN) / common name`, `Active substance` | substances (`;` / ` / ` separated; INN preferred, active substance = salt form fallback) |45| `Therapeutic area (MeSH)` | `;`-separated MeSH descriptors → scope + `mesh_therapeutic_area` edges |46| `ATC code (human)`, `Pharmacotherapeutic group (human)` | scope (L01/L02), `drug_codes.atc` (label = group) |47| `Therapeutic indication` | official summary → `indication` (≤ 4,000 chars, full text in `raw.indicationFull`), cancer mapping |48| `Accelerated assessment`, `Conditional approval`, `Biosimilar`, `Generic`, `Orphan medicine`, `PRIME: priority medicine`, `Advanced therapy`, `Exceptional circumstances`, `Additional monitoring` | Yes/No flags → `accelerated`, `conditional`, `raw.flags` |49| `Marketing authorisation developer / applicant / holder` | `raw.holder` |50| `Marketing authorisation date` | **`approval_date`** (initial authorisation) |51| `European Commission decision date` | `raw.ecDecisionDate` only — it is the **latest** Commission decision (Xeljanz: 23/07/2026 vs authorisation 22/03/2017; for withdrawn medicines often the withdrawal decision), so it is never used as approval date |52| `Withdrawal / expiry / revocation / lapse of marketing authorisation date`, `Suspension of marketing authorisation date` | `withdrawal_date` (never derived from anything else) |53| `Last updated date` | `source_records.source_updated_at`, `provenance.updated_at_source` |54| `Medicine URL` | `https://www.ema.europa.eu/en/medicines/human/EPAR/<slug>` → `provenance.source_url` (fallback: the download page) |5556Dates are `DD/MM/YYYY` (`emaDate`: malformed / impossible → null, never fabricated).5758## Ingestion59601. **Download + anomaly guard**: one GET; < 50 KB or < 1,000 parsed rows → `anomaly:` error;61   `ctx.guardCount('medicine', rows)` against the previous successful run (first run = baseline).622. **Raw layer**: `workbook` source record (the file bytes, base64 + sha256, keyed by dataset63   version — `--mode backfill` replays it without HTTP) and one `medicine` record per row (payload =64   the parsed row, every cell verbatim). Rows are processed sorted by product number;65   `ctx.cursor.lastProductNumber` resumes a run stopped by the time budget; pass bookkeeping66   (`passCount`) records `approval_added` / `approval_status_changed` change events from pass 2.673. **Scope** (`classifyOncology`, human rows only) — a medicine is oncology when at least one holds:68   - **`atc_l01_l02`**: an `ATC code (human)` starts with L01 (antineoplastic) or L02 (endocrine therapy);69   - **`mesh_neoplasm`**: a MeSH therapeutic area that (a) carries neoplasm vocabulary70     (`ONCOLOGY_LABEL_RE`) and (b) resolves through `cancer_aliases` (`CancerResolver.byLabel`,71     MeSH inversion; `mesh_heading` aliases from the MeSH connector) to an active cancer entity.72     Generic descriptors (`Neoplasms`, `Cancer`, `Carcinoma`, `Neoplasm Metastasis`…) never count;73     benign conditions the ontology knows as entities (`Leiomyoma`, `Hemangioma`, `Prostatic74     Hyperplasia`, `Dupuytren Contracture`, `Amyloidosis`) fail (a) and are not oncology;75   - **`indication_text`**: the indication matches the ClinicalTrials.gov oncology vocabulary76     (`ONCOLOGY_LABEL_RE`) after neutralising two prose false positives: *tumour necrosis factor*77     (anti-TNF biosimilars) and the lower-case word *all* (the `\ball\b` abbreviation of ALL).78   Everything else is counted (`nonOncology`) and stays in the lake.79   **Cancer mapping and drug minting are only allowed** when the medicine is *oncology by class*80   (ATC L01/L02, or MeSH + oncology text), is **not a diagnostic** (ATC V04 / V08 / V09), and has at81   least one **therapeutic** sentence. Medicines in scope by indication text alone (denosumab82   for bone loss in prostate cancer, G-CSF, epoetins, fentanyl for cancer pain, antiemetics…) get an83   honest EU approval row when their substance is already a known drug, with `cancer_id NULL`,84   no edge and no minted drug.854. **Substances → drugs** (deterministic, no LLM): the INN cell is split on `;` and ` / ` (never on86   commas — vaccine names; a `/` inside brackets is part of the name). Lookup in `drug_aliases.normalized`87   (in-memory index, kept in sync): the full label first, then the molecule with the salt form88   stripped (`parseIngredientName` shared with Health Canada: "abiraterone acetate" → abiraterone,89   "niraparib (tosilate monohydrate)" → niraparib). Exactly one drug → `ALIAS`; several → the90   `generic` alias, else the drug's own name, else **ambiguous** → `unresolved_labels`91   (`ambiguous_alias`: dacomitinib, trastuzumab deruxtecan on 2026-09-11 — two entities each).92   Biosimilars / generics (`Biosimilar` / `Generic` = Yes) resolve to the existing molecule (Herzuma,93   Ontruzant → Trastuzumab; Bortezomib Hospira → Bortezomib) — they never mint.94   - **Unknown INN, single substance, authorised or formerly authorised, oncology by class,95     therapeutic** → a drug is **minted** (`mintId 'DRUG'`; name = INN in Title Case — long common96     names with capitals, brackets or commas are kept verbatim; `slug` with `-ema-<number>` on97     clash; `developmentStatus 'marketed_eu'` when authorised; aliases `generic`, `salt`, `brand`;98     change event `created`).99   - Otherwise → `unresolved_labels` with the reason: `component_of_combination`,100     `not_authorised_medicine` (refused / withdrawn application / pending opinion),101     `diagnostic_agent`, `scope_from_indication_text_only`, `scope_from_mesh_area_only`,102     `non_therapeutic_indication` (epoetin alfa, luspatercept, imetelstat, Gliolan),103     `no_inn_or_active_substance` (older withdrawn entries).104   - Rows are processed in product-number order, so a medicine whose INN is minted by a later row105     of the same pass (Prolia → denosumab minted by Yaxwer) is **retried** at the end of the pass:106     one run converges (`retried 103 … 2 gained rows` on the first run).1075. **Codes and aliases** per resolved drug: `drug_codes` `ema_product` (code = product number,108   label = medicine name) and `atc` (one per code, label = pharmacotherapeutic group), `match_type109   EXACT_IDENTIFIER`; a `brand` alias = medicine name without "(previously …)" for single-drug110   medicines only (a combination name is not an alias of one component).1116. **Status** (`mapEmaStatus`) — `Authorised` → **`approved`** (`conditional = true` when112   Conditional approval = Yes, `accelerated = true` when Accelerated assessment = Yes — a review113   track, noted in `raw.acceleratedNote`); `Withdrawn`, `Lapsed`, `Expired`, `Revoked`, `Suspended`114   → **`withdrawn`** with `withdrawal_date` from the explicit column (null otherwise; a suspension is115   reversible — the verbatim status stays in `raw.emaStatus` and `raw.emaStatusNote` says it is the116   medicine's, not the molecule's); `Refused`, `Application withdrawn`, `Withdrawn from rolling117   review`, `Opinion`, `Opinion under re-examination` → **no approval row** (a refusal or a pending118   opinion is not an approval; counted, lake only; any existing edge of that medicine is superseded).1197. **Approval row** per (medicine, drug) — `drug_approvals`: `jurisdiction 'EU'`, `authority 'EMA'`,120   `approval_type 'centralised'`, `application_number` = product number, `approval_date` =121   marketing authorisation date, `indication` = the official summary (≤ 4,000 chars, "…"), `raw`122   with everything above plus `scope`, `substances`, `flags`, `cancer_match`, `cancerCandidates`,123   `meshCancerIds`, `therapeutic`. Idempotent key `(source, drug, 'EU', product number)`: existing124   rows are updated in place, nothing is deleted. Provenance: one row per medicine and run when the125   source record changed (`sourceUrl` = EPAR page, `dataset 'EMA medicines data (xlsx)'`,126   `datasetVersion`, `evidenceType regulatory`, `accessLevel open`, `geography 'EU'`,127   `publishedAt` = approval date, `updatedAtSource` = last updated date, methodology text); an128   unchanged medicine whose rows exist reuses its previous provenance row.1298. **Cancer mapping** (`resolveIndication`, openFDA `CancerDictionary` / `resolveBullet` /130   `CancerReconciler` imported from `../openfda/normalize.js`): the indication is split into131   paragraphs, then `;` / `:` clauses and sentences (EMA lists indications as "…: as monotherapy132   for …; for treatment of …"). Sentences whose object is supportive care, prevention or diagnosis133   (`isSupportiveCareSentence`: "prevention of skeletal events", "treatment of symptomatic134   anaemia", "patients receiving chemotherapy", "active immunisation…", "visualisation of…",135   "vasomotor symptoms"…) are flagged and never yield a cancer; an introduction ending with ":"136   hands its nature to the clauses it introduces. Each remaining sentence → 0 / 1 cancer137   (`PROBABILISTIC`, several mentions collapse only along one lineage). The whole indication → one138   cancer → **`cancer_id`**; otherwise `cancer_id NULL` and the distinct cancers in139   `raw.cancerCandidates`. `tumor_agnostic` when a sentence pairs "solid tumours" with an agnostic140   biomarker. The dictionary is built **without** the build-time reconciler: for an alias shared by141   a parent and its child ("breast cancer" → Malignant Breast Neoplasm / Breast Carcinoma /142   Childhood Breast Carcinoma) the narrowest-of-lineage rule picked the child, so adult breast-cancer143   indications mapped to *Childhood Breast Carcinoma* (verified 2026-09-11); the resolver's144   preferred / display-name disambiguation decides instead.1459. **Knowledge edges** — for **authorised** medicines only: one `drug APPROVED_FOR cancer` edge per146   cancer named by a therapeutic sentence (`evidenceLevel 'EMA indication text'`), plus one per MeSH147   therapeutic area (`evidenceLevel 'mesh_therapeutic_area'`, only when the text did not already148   name that cancer) — `evidenceCategory regulatory_status`, `cancer_context_ids = [cancer]`,149   `provenance_ids`, `source_record_id` = product number. Edges of a medicine that is no longer150   authorised, or whose indication no longer names the cancer, are set to `status 'superseded'`.151   Multi-indication drugs therefore get one edge per cancer (Keytruda: 15 text edges + 4 MeSH).152153`dry_run` downloads, parses and maps without writing (25 sample lines + totals); `--mode backfill`154replays the lake copy of the workbook (fallback: the per-row `medicine` payloads) — no HTTP, cursor155untouched. Health check: `HEAD` on the xlsx must answer a spreadsheet content-type ≥ 100 KB.156157## Observed runs (local copy of production, 2026-09-11)158159| | |160|---|---|161| Dry run | 2,732 rows (2,339 human, 393 veterinary) in 1.7 s; 620 human medicines in scope — ATC 418, MeSH 462, indication text 573 (text-only 111); out of scope 1,719 |162| First run (`ING-EMA-20260911-000001`) | 3.0 s, 1 download. Statuses among scoped medicines: Authorised 446, Withdrawn 84, Application withdrawn 68, Refused 23, Lapsed 6, Opinion 5, Expired 2, Revoked 1, Opinion under re-examination 1. 622 substances: 475 by alias + 4 after salt stripping, **36 minted**, 2 ambiguous, 113 unresolved (queued), 13 medicines without INN. **459 EU approval rows** on 245 drugs / 456 medicines: 385 `approved` (31 conditional, 12 accelerated), 74 `withdrawn` (Withdrawn 66, Lapsed 5, Expired 2, Revoked 1; 48 with a withdrawal date), 457 with an approval date, **190 with `cancer_id`**, 4 tumour-agnostic, 71 flagged non-therapeutic (supportive care). **859 edges**: 529 `EMA indication text` (188 drugs × 95 cancers), 330 `mesh_therapeutic_area` (126 drugs × 56 cancers). 571 codes (459 `ema_product`, 112 `atc`), 187 brand + 36 generic + 4 salt aliases, 459 provenance rows, 2,732 + 1 source records (2.9 MB in the lake). Retry pass: 103 deferred medicines, 2 gained rows |163| Second run (idempotency) | 2,733 source records unchanged, 0 created / 459 updated approval rows, 0 provenance rows, 0 change events |164| Backfill | replayed `ema-medicines-2026-09-11T06:00` from the lake: same counts, 0 HTTP requests |165| `pnpm cix reconcile-drugs` | linked trial interventions **78,105 → 78,983** (+878, all to the minted drugs: Thalidomide 236, Imiquimod 116, Denosumab 74, Serplulimab 69, Talimogene laherparepvec 43, Duvelisib 38, Tivozanib 32…) |166167Minted (36): 5-Aminolevulinic Acid, Alitretinoin, Bexarotene, Cinacalcet, Denosumab, Dinutuximab168Beta, Duvelisib, Histamine, Hydroxycarbamide, Ibandronic Acid, Imiquimod, Imlunestrant,169Linvoseltamab, Melphalan Flufenamide, Mifamurtide, Moxetumomab Pasudotox, Nogapendekin Alfa170Inbakicept, Obecabtagene Autoleucel, Odronextamab, Padeliporfin, Pixantrone Dimaleate, Radium Ra223171dichloride, Serplulimab, Sugemalimab, Tagraxofusp, Talimogene Laherparepvec, Temoporfin,172Thalidomide, Thyrotropin Alfa, Tisotumab Vedotin, Tivozanib, Toremifene, Vimseltinib, Vinflunine,173two cell therapies kept under their EMA common name (sipuleucel-T "Autologous peripheral-blood174mononuclear cells activated with…", Zalmoxis "Allogeneic T cells genetically modified…").175176Most-connected cancers by EU text edges: NSCLC 33 drugs, malignant breast neoplasm 25, multiple177myeloma 21, RCC 14, ALL 12, colorectal 11, follicular lymphoma 10, prostate 10, AML 9, melanoma 8.178179## Limitations / notes180181- **Medicine-level granularity**: one row per (medicine, active substance) — biosimilars and182  generics each have their own row on the same molecule (trastuzumab: Herceptin + 6 biosimilars).183  The approvals feed counts medicines, not molecules; `raw.flags.biosimilar / generic` say which.184- **Dates**: `approval_date` is the initial EU marketing authorisation of *that medicine* (a185  generic's date for a generic). The Commission decision date is the latest decision, kept in186  `raw` only. Suspensions carry no date in the file (2026-09-11) → `withdrawal_date` null.187- **Refusals / pending opinions** are not approvals: no row, no edge; they are in the lake and188  counted in the run log (`refused`, `notAuthorised`, `pending`) and their unknown INNs are queued189  (`not_authorised_medicine`). Positive CHMP opinions awaiting the Commission decision (`Opinion`,190  5 on 2026-09-11) appear on the next weekly run once authorised.191- **Cancer mapping is conservative**: multi-indication medicines have `cancer_id NULL` (edges carry192  the per-cancer information); a sentence naming ≥ 2 unrelated cancers yields none. MeSH-derived193  edges target the MeSH heading's entity, often the broad "X Neoplasm" umbrella (`malignant =194  false` umbrella of benign + malignant) — hence the separate `evidenceLevel 'mesh_therapeutic_area'`.195- **Gray zone kept, flagged**: medicines oncology by MeSH with a therapeutic sentence outside the196  cancer itself (Cinacalcet — hypercalcaemia of parathyroid carcinoma + hyperparathyroidism;197  Ibandronic acid — one authorised medicine with a non-supportive sentence; Thyrotropin alfa —198  radioiodine ablation adjunct) are minted with their honest indication text and MeSH edge only.199  Conversely luspatercept (MDS anaemia), imetelstat and epoetins are *not* minted because every200  sentence targets anaemia — they land in `unresolved_labels` (`non_therapeutic_indication`).201- **Combinations**: a component whose INN is unknown is queued (`component_of_combination`,202  16 on 2026-09-11); the row is linked to the components that resolved (Akeega → Niraparib and203  Abiraterone Acetate; Phesgo → Pertuzumab and Trastuzumab).204- **Ambiguous aliases** (2): "dacomitinib" and "trastuzumab deruxtecan" each match two drug205  entities — a duplicate-entity question for `drug-duplicates.ts`, not for this connector.206- **Veterinary** rows (393) are recorded in the lake and skipped. `Species`, `ATCvet` and the207  veterinary pharmacotherapeutic group are parsed for the lake only.208- **Finding for `openfda`**: the same `CancerDictionary.build(rows, disambiguate, reconciler)`209  call with the reconciler collapses shared aliases to the child concept (adult "breast cancer" →210  Childhood Breast Carcinoma, "diffuse large B-cell lymphoma" → Childhood DLBCL); the EMA211  connector passes `undefined` at build time. Not changed in the openFDA connector by this work.212- Fixtures: `fixtures/ema-sample.xlsx` (38 real rows — oncology L01/L02, biosimilars, a generic,213  withdrawn / lapsed / expired / revoked / suspended / refused / application-withdrawn / pending214  rows, MeSH-only and text-only rows, combinations, vaccines, veterinary rows; one marketing215  authorisation date corrupted to `99/99/2016`), `ema-header-drift.xlsx` (required column renamed),216  `ema-empty.xlsx` (header only). Regenerate with `pnpm exec tsx217  packages/connectors/src/connectors/ema/fixtures/build-fixtures.ts /tmp/ema.xlsx`.218