spb/cancerindex

1# Connector `health-canada-dpd` — Health Canada Drug Product Database (DPD)23| | |4|---|---|5| Source | https://health-products.canada.ca/dpd-bdpp/ — Health Canada, **Drug Product Database** (human, veterinary, radiopharmaceutical and disinfectant products authorized in Canada; one record per DIN) |6| Tier / category | 3 / regulatory (CLAUDE.md §13 — jurisdiction **CA**, authority **Health Canada**; never a bare `approved = true`) |7| Access | REST `https://health-products.canada.ca/api/drug/`, no authentication, JSON (`type=json`) or XML, `lang=en|fr` |8| Docs verified | 2026-09-11 — https://health-products.canada.ca/api/documentation/dpd-documentation-en.html ("DPD API Guide") + live API with `User-Agent: CancerIndex/0.1` |9| License | **Open Government Licence – Canada** (https://open.canada.ca/en/open-government-licence-canada) — `licenseStatus: approved`, `commercialUse: allowed`, `redistribution: attribution`; attribution: *"Contains information licensed under the Open Government Licence – Canada. Source: Health Canada, Drug Product Database."* |10| Code | `packages/connectors/src/connectors/health-canada-dpd/` (`manifest.ts`, `normalize.ts`, `index.ts`, `health-canada-dpd.test.ts`, `fixtures/`) |1112## Terms and licence (quoted)1314The DPD web application carries no licence statement of its own (its footer links to the Canada.ca15terms and conditions, verified 2026-09-11). The DPD **data extract** is published on the Open16Government portal (dataset `bf55e42a-63cb-4556-bfd8-44f26e5a36fe`, "Drug Product Database - All17Files") with **"Licence: Open Government Licence - Canada"**. From that licence18(https://open.canada.ca/en/open-government-licence-canada), stored verbatim in `manifest.termsNotes`:1920> The Information Provider grants you a worldwide, royalty-free, perpetual, non-exclusive licence to use the Information, including for commercial purposes, subject to the terms below. You are free to: Copy, modify, publish, translate, adapt, distribute or otherwise use the Information in any medium, mode or format for any lawful purpose. You must, where you do any of the above: Acknowledge the source of the Information by including any attribution statement specified by the Information Provider(s) and, where possible, provide a link to this licence. […] you must use the following attribution statement: Contains information licensed under the Open Government Licence – Canada.2122CancerIndex shows every DPD record as a dated, sourced **market authorization for one product23(DIN)** — never as an indication (the DPD publishes none) and never as a treatment recommendation.2425Rate limits: none documented (2026-09-11). The connector self-limits to **2 requests/s, one at a26time** (`rateLimits`), i.e. ≈ 1,500 products × 4 requests ≈ 50 min per full pass; the pass is27restartable per product (below). No 429 was observed during the first full pass.2829## Verified endpoints (2026-09-11)3031| Endpoint | Verified behaviour |32|---|---|33| `GET /therapeuticclass/?lang=en&type=json` | Bulk list, **48,042 rows** `{drug_code, tc_atc_number, tc_atc}` (3.7 MB, 0.2 s), one row per `drug_code`. Oncology groups kept: **L01** antineoplastic agents 1,156, **L02** endocrine therapy 255, **L03** immunostimulants 114 (filgrastim, interferons, BCG…), **V10** therapeutic radiopharmaceuticals 14 → **1,539 products**; the ATC group is tagged in `raw.atc.group`. L04 (immunosuppressants, 675) is out of scope. |34| `GET /therapeuticclass/?lang=en&type=json&id=92551` | `[{drug_code: 92551, tc_atc_number: "L01FF02", tc_atc: "PEMBROLIZUMAB"}]` — **health check** (must return L01FF02). `tc_atc` is the ATC 5th-level substance name, not a class label. |35| `GET /drugproduct/?lang=en&type=json&id=92551` | `{drug_code, class_name "Human" \| "Radiopharmaceutical" \| "Veterinary" \| "Disinfectant", drug_identification_number "02441152", brand_name "KEYTRUDA", descriptor, number_of_ais "1", ai_group_no, company_name, last_update_date "2026-08-13"}`. Brand names often embed the presentation ("PROCYTOX TABLETS 50MG", "VELBE 1MG/ML"). |36| `GET /activeingredient/?lang=en&type=json&id=` | `[{ingredient_name "PEMBROLIZUMAB" \| "IMATINIB (IMATINIB MESYLATE)" \| "BORTEZOMIB (BORTEZOMIB MANNITOL BORONIC ESTER)", strength, strength_unit, dosage_value, dosage_unit}]`. Guide: *"Information enclosed within brackets represents the salt and identifies how the ingredient is supplied."* Kits list diluents (WATER, SODIUM CHLORIDE, BUFFER SOLUTION) as ingredients. |37| `GET /status/?lang=en&type=json&id=` | `{status, history_date, original_market_date, external_status_code, expiration_date, lot_number}`. Status vocabulary (guide, `drugproduct/?status=`): 1 Approved, 2 Marketed, 3 Cancelled Pre Market, 4 Cancelled Post Market, 6 Dormant, 9 Cancelled (Unreturned Annual), 10 Cancelled (Safety Issue), 11 Authorized By Interim Order, 12 Authorized… `original_market_date` is **null** for "Approved" (not yet marketed) products. |38| `GET /route/?lang=en&type=json&id=` | `[{route_of_administration_code, route_of_administration_name "Intravenous"}]` |39| Unknown `drug_code` | Object endpoints (`drugproduct`, `status`) answer **HTTP 200 with zeros/nulls** (`{drug_code: 0, brand_name: null, …}`); array endpoints (`activeingredient`, `route`, `therapeuticclass`) answer **HTTP 404 with an empty body**. Both are treated as *missing*, never as a failure (`isMissingProduct`, `getArrayOrEmpty`). |40| Public product page | `https://health-products.canada.ca/dpd-bdpp/info?lang=eng&code=<drug_code>` (HTTP 200 verified) — used as `provenance.source_url`. |4142Schedule/form endpoints exist but are not fetched (4 requests per product: product, ingredients,43status, routes).4445## Ingestion4647Scope (bounded, CLAUDE.md §227): the oncology therapeutic-class rows, **sorted by `drug_code`**.48`ctx.cursor.lastDrugCode` is advanced after each fully persisted product (checkpoint every 5049records or 60 s, `manifest.checkpointEvery`), so a run stopped by the time budget (`--max-minutes`)50or a signal resumes exactly where it stopped; a completed pass (`cursor.completedAt`) starts a51fresh one on the next schedule (`passCount` increments; change events are recorded from pass 2).52531. **Anomaly guard**: `ctx.guardCount('product', n)` on the filtered bulk list (1,539) against the54   previous successful run; plus a hard floor of 10,000 bulk rows (`anomaly:` error) — a truncated55   list never triggers deletions (nothing is ever deleted by this connector).562. **Per product** (`fetchBundle`): product → skip if missing; ingredients, status, routes. Each57   sub-object is validated (zod) and observed for schema drift (`product`, `active_ingredient`,58   `status`, `route`). One **`product` source record** per `drug_code` with the composite payload59   `{therapeuticClass, product, ingredients, status, routes}` — self-contained, so `--mode backfill`60   can replay it. Non-human classes (Veterinary, Disinfectant) are recorded and then skipped.613. **Ingredient → drug** (deterministic, no LLM; `parseIngredientName`): the bracketed salt is62   split off ("IMATINIB (IMATINIB MESYLATE)" → molecule IMATINIB, salt form IMATINIB MESYLATE), a63   trailing salt token is stripped ("DOXORUBICIN HYDROCHLORIDE" → DOXORUBICIN; the salt list is64   `SALT_TOKENS` shared with trial reconciliation). Excipients listed as kit components (WATER,65   SODIUM CHLORIDE, BUFFER SOLUTION…) are skipped. Lookup in `drug_aliases.normalized` (index66   loaded once per run and kept in sync): the full name first, then the molecule. Exactly one67   drug → resolved (`match_type ALIAS`); several → the one whose alias is `generic`, else the one68   whose own name matches, else **ambiguous** → `unresolved_labels` (`reason ambiguous_alias`).69   - **Unknown molecule, single-ingredient product** → a drug is **minted** (`mintId 'DRUG'`,70     `name` = molecule in Title Case — "Vinblastine", not "Vinblastine Sulfate" (CLAUDE.md §7:71     the base molecule is the entity, salts are aliases), `slug` = slugify(name) with a72     `-hc-<drug_code>` suffix on clash, `kind` null, `developmentStatus 'marketed_ca'` when the DPD73     status maps to approved), aliases `generic` (molecule), `salt` (salt form) and `brand`74     (cleaned brand), change event `created`.75   - **Unknown molecule, multi-ingredient product** → `unresolved_labels` (`reason76     component_of_multi_ingredient_product`, with DIN, brand, ATC); the approval row is linked to77     every component that *did* resolve.784. **Codes and aliases** for every resolved drug: `drug_codes` `atc` (code `tc_atc_number`, label79   `tc_atc`), `din` (label = brand as published), `hc_drug_code` (label = brand), `match_type80   EXACT_IDENTIFIER`; a `brand` alias with the presentation noise removed (`cleanBrandName`:81   "PROCYTOX TABLETS 50MG" → "PROCYTOX", "THIO TEPA INJ 15MG/VIAL" → "THIO TEPA") and a `salt`82   alias when the ingredient carried one.835. **Approval row** per (product, drug) — `drug_approvals`: `jurisdiction 'CA'`, `authority84   'Health Canada'`, `approval_type 'DIN'`, `application_number` = DIN (8 digits), `approval_date`85   = `original_market_date` (null when the product is approved but not marketed — never86   fabricated), `status`: Marketed / Approved / Authorized… → **`approved`**; Cancelled Post Market87   / Pre Market / (Safety Issue) / (Unreturned Annual) / Dormant → **`withdrawn`** with88   `withdrawal_date = history_date`; anything else → no row + warning (`statusUnknown`). The DPD89   status is kept verbatim in `raw.dpdStatus` and surfaced on the pages: **a cancelled or dormant90   DIN is one product's status, not a withdrawal of the molecule** (KEYTRUDA DIN 02441152 is91   "Cancelled Post Market" since 2019-12-04 while other pembrolizumab DINs are marketed).92   `indication` is honest text: *"Marketed in Canada as KEYTRUDA (DIN 02441152) under ATC L01FF0293   PEMBROLIZUMAB. Indications are not published in the Drug Product Database — see the Health Canada94   Product Monograph."*; `cancer_id NULL`, `tumor_agnostic false` — **no cancer is ever inferred95   from an ATC class**. `raw` keeps drug code, DIN, brand, statuses/dates, ATC (code, label, group),96   class, company, descriptor, all ingredients with strengths, routes, `last_update_date`.976. **Provenance**: one row per product and run when the source record changed (`sourceUrl` =98   public product page, `dataset 'Health Canada DPD'`, `datasetVersion dpd-<run date>`,99   `evidenceType regulatory`, `accessLevel open`, `geography 'CA'`, `publishedAt` =100   original market date, `updatedAtSource` = `last_update_date`, methodology text naming the101   ingredient and how it was reconciled). Unchanged product + existing row → the previous102   provenance row is reused (no duplicate evidence per weekly run).1037. **Idempotency**: rows are keyed by `(source_id, drug_id, jurisdiction 'CA', application_number =104   DIN)` — existing rows are updated in place (status changes are recorded as105   `approval_status_changed` from pass 2), new rows inserted (`approval_added` from pass 2). Nothing106   is deleted: a DIN that disappears from the bulk list keeps its last known status.107108`dry_run` walks the first 20 products of the list without writing (resolution against the alias109index is logged as "→ CI-DRUG-…", "would mint …" or "AMBIGUOUS"). `--mode backfill` re-derives drugs,110codes, aliases and approval rows from the composite payloads in the raw lake — no HTTP, cursor111untouched.112113Health check: `therapeuticclass?id=92551` must contain `L01FF02`.114115## Observed runs (local copy of production, 2026-09-11)116117| | |118|---|---|119| Dry run | 20 products, 81 requests, 40 s; 10 resolved by alias, 10 "would mint" (Vinblastine, Diethylstilbestrol, Dactinomycin, Procarbazine, Busulfan, Thiotepa, Bleomycin…) |120| 10-record smoke (`--max-records 10`) | 41 requests, 19 s, status `partial`, cursor `lastDrugCode 960`; 5 drugs minted, 10 approval rows, 27 codes; rerun resumed from the cursor |121| Full pass (`--reset-cursor --max-minutes 58`, run ING-HEALTHCANADADPD-20260911-000003) | Still running at reporting time — **455 / 1,539 products** processed (cursor `lastDrugCode 80770`, ≈ 2.1 s per product = 4 requests at 2 req/s, 0 HTTP failures, 0 warnings). The cursor resumes the pass on the next `pnpm cix run health-canada-dpd`. |122123Counts after those 455 products (plus the 10-record smoke): 507 `product` source records; **52 drugs minted**124(Vinblastine, Busulfan, Dactinomycin, Procarbazine, Thiotepa, Bleomycin, Interferon Alfa-2b…); ingredients125resolved by alias 303 + 106 after salt stripping; **502 CA approval rows** on 161 drugs (165 `approved`,126337 `withdrawn` — old DINs of long-marketed molecules dominate the low drug codes); `drug_codes` 502 DIN +127502 hc_drug_code + 119 ATC; aliases 186 brand + 52 generic + 47 salt; 8 labels in `unresolved_labels`128(components of multi-ingredient kits: RIBAVIRIN of REBETRON, ANCESTIM of STEMGEN, FORMESTANE, MELACINE129lysate components…), 0 ambiguous. `pnpm cix reconcile-drugs` afterwards linked **69,207 → 74,435 trial130intervention rows** (+5,228; 5,155 of them to the minted drugs — e.g. Busulfan, Bleomycin, Vinblastine131trials were unresolved before).132133## Limitations / notes134135- **No indications**: the DPD is a product registry; indications live in the Product Monograph136  (PDF, not in the API). Canadian rows therefore never carry a cancer and never feed cancer-scoped137  pipeline stages or `approved_drug_count` per cancer; they do make the unscoped pipeline row138  "approved" and appear in the approvals feed and on the drug page.139- **DIN-level granularity**: one molecule has many DINs (brands, generics, strengths); the feed140  counts records, not molecules. Status is per DIN.141- **Dates**: `original_market_date` is the first marketing date of *that DIN*, often a generic's142  date; `approval_date` on a CA row is therefore not "the Canadian approval of the molecule" — the143  earliest CA row of a drug is the best available proxy. Historic products carry year-end144  placeholders from the registry itself ("1954-12-31") — kept as published.145- **Scope by ATC**: products without an ATC assignment in the DPD, or classed outside L01/L02/L03/146  V10 (e.g. supportive-care antiemetics, L04 immunosuppressants), are not ingested. L03 brings147  non-oncology immunostimulants (interferon beta for MS) — they are drugs, honestly recorded with148  their ATC group, but should not be read as oncology approvals.149- **Minted drugs** start with no `kind`, mechanism or targets; ChEMBL/openFDA enrichment and150  `pnpm cix reconcile-drugs` (trial interventions) run afterwards. Salt-form duplicates against151  existing entities are avoided by minting the base molecule; remaining duplicates are proposed by152  `packages/ranking/src/drug-duplicates.ts` (`entity_merges`, status `proposed`).153- Only human and radiopharmaceutical classes are ingested; veterinary/disinfectant products are154  recorded as source records and skipped.155