SPB Git forge

spb/cancerindex

Public
37commits 1branches 0releases
2.9 MBsize
maindefault branch
10 days agolast push
TypeScript 97.2% SQL 1.5% CSS 0.6% JavaScript 0.5%
14.6 KB

# Connector health-canada-dpd — Health Canada Drug Product Database (DPD)

Source https://health-products.canada.ca/dpd-bdpp/ — Health Canada, Drug Product Database (human, veterinary, radiopharmaceutical and disinfectant products authorized in Canada; one record per DIN)
Tier / category 3 / regulatory (CLAUDE.md §13 — jurisdiction CA, authority Health Canada; never a bare approved = true)
Access REST https://health-products.canada.ca/api/drug/, no authentication, JSON (type=json) or XML, `lang=en
Docs verified 2026-09-11 — https://health-products.canada.ca/api/documentation/dpd-documentation-en.html ("DPD API Guide") + live API with User-Agent: CancerIndex/0.1
License Open Government Licence – Canada (https://open.canada.ca/en/open-government-licence-canada) — licenseStatus: approved, commercialUse: allowed, redistribution: attribution; attribution: "Contains information licensed under the Open Government Licence – Canada. Source: Health Canada, Drug Product Database."
Code packages/connectors/src/connectors/health-canada-dpd/ (manifest.ts, normalize.ts, index.ts, health-canada-dpd.test.ts, fixtures/)

# Terms and licence (quoted)

The DPD web application carries no licence statement of its own (its footer links to the Canada.ca terms and conditions, verified 2026-09-11). The DPD data extract is published on the Open Government portal (dataset bf55e42a-63cb-4556-bfd8-44f26e5a36fe, "Drug Product Database - All Files") with "Licence: Open Government Licence - Canada". From that licence (https://open.canada.ca/en/open-government-licence-canada), stored verbatim in manifest.termsNotes:

The Information Provider grants you a worldwide, royalty-free, perpetual, non-exclusive licence to use the Information, including for commercial purposes, subject to the terms below. You are free to: Copy, modify, publish, translate, adapt, distribute or otherwise use the Information in any medium, mode or format for any lawful purpose. You must, where you do any of the above: Acknowledge the source of the Information by including any attribution statement specified by the Information Provider(s) and, where possible, provide a link to this licence. […] you must use the following attribution statement: Contains information licensed under the Open Government Licence – Canada.

CancerIndex shows every DPD record as a dated, sourced market authorization for one product (DIN) — never as an indication (the DPD publishes none) and never as a treatment recommendation.

Rate limits: none documented (2026-09-11). The connector self-limits to 2 requests/s, one at a time (rateLimits), i.e. ≈ 1,500 products × 4 requests ≈ 50 min per full pass; the pass is restartable per product (below). No 429 was observed during the first full pass.

# Verified endpoints (2026-09-11)

Endpoint Verified behaviour
GET /therapeuticclass/?lang=en&type=json Bulk list, 48,042 rows {drug_code, tc_atc_number, tc_atc} (3.7 MB, 0.2 s), one row per drug_code. Oncology groups kept: L01 antineoplastic agents 1,156, L02 endocrine therapy 255, L03 immunostimulants 114 (filgrastim, interferons, BCG…), V10 therapeutic radiopharmaceuticals 14 → 1,539 products; the ATC group is tagged in raw.atc.group. L04 (immunosuppressants, 675) is out of scope.
GET /therapeuticclass/?lang=en&type=json&id=92551 [{drug_code: 92551, tc_atc_number: "L01FF02", tc_atc: "PEMBROLIZUMAB"}] — health check (must return L01FF02). tc_atc is the ATC 5th-level substance name, not a class label.
GET /drugproduct/?lang=en&type=json&id=92551 {drug_code, class_name "Human" | "Radiopharmaceutical" | "Veterinary" | "Disinfectant", drug_identification_number "02441152", brand_name "KEYTRUDA", descriptor, number_of_ais "1", ai_group_no, company_name, last_update_date "2026-08-13"}. Brand names often embed the presentation ("PROCYTOX TABLETS 50MG", "VELBE 1MG/ML").
GET /activeingredient/?lang=en&type=json&id= [{ingredient_name "PEMBROLIZUMAB" | "IMATINIB (IMATINIB MESYLATE)" | "BORTEZOMIB (BORTEZOMIB MANNITOL BORONIC ESTER)", strength, strength_unit, dosage_value, dosage_unit}]. Guide: "Information enclosed within brackets represents the salt and identifies how the ingredient is supplied." Kits list diluents (WATER, SODIUM CHLORIDE, BUFFER SOLUTION) as ingredients.
GET /status/?lang=en&type=json&id= {status, history_date, original_market_date, external_status_code, expiration_date, lot_number}. Status vocabulary (guide, drugproduct/?status=): 1 Approved, 2 Marketed, 3 Cancelled Pre Market, 4 Cancelled Post Market, 6 Dormant, 9 Cancelled (Unreturned Annual), 10 Cancelled (Safety Issue), 11 Authorized By Interim Order, 12 Authorized… original_market_date is null for "Approved" (not yet marketed) products.
GET /route/?lang=en&type=json&id= [{route_of_administration_code, route_of_administration_name "Intravenous"}]
Unknown drug_code Object endpoints (drugproduct, status) answer HTTP 200 with zeros/nulls ({drug_code: 0, brand_name: null, …}); array endpoints (activeingredient, route, therapeuticclass) answer HTTP 404 with an empty body. Both are treated as missing, never as a failure (isMissingProduct, getArrayOrEmpty).
Public product page https://health-products.canada.ca/dpd-bdpp/info?lang=eng&code=<drug_code> (HTTP 200 verified) — used as provenance.source_url.

Schedule/form endpoints exist but are not fetched (4 requests per product: product, ingredients, status, routes).

# Ingestion

Scope (bounded, CLAUDE.md §227): the oncology therapeutic-class rows, sorted by drug_code. ctx.cursor.lastDrugCode is advanced after each fully persisted product (checkpoint every 50 records or 60 s, manifest.checkpointEvery), so a run stopped by the time budget (--max-minutes) or a signal resumes exactly where it stopped; a completed pass (cursor.completedAt) starts a fresh one on the next schedule (passCount increments; change events are recorded from pass 2).

  1. Anomaly guard: ctx.guardCount('product', n) on the filtered bulk list (1,539) against the previous successful run; plus a hard floor of 10,000 bulk rows (anomaly: error) — a truncated list never triggers deletions (nothing is ever deleted by this connector).
  2. Per product (fetchBundle): product → skip if missing; ingredients, status, routes. Each sub-object is validated (zod) and observed for schema drift (product, active_ingredient, status, route). One product source record per drug_code with the composite payload {therapeuticClass, product, ingredients, status, routes} — self-contained, so --mode backfill can replay it. Non-human classes (Veterinary, Disinfectant) are recorded and then skipped.
  3. Ingredient → drug (deterministic, no LLM; parseIngredientName): the bracketed salt is split off ("IMATINIB (IMATINIB MESYLATE)" → molecule IMATINIB, salt form IMATINIB MESYLATE), a trailing salt token is stripped ("DOXORUBICIN HYDROCHLORIDE" → DOXORUBICIN; the salt list is SALT_TOKENS shared with trial reconciliation). Excipients listed as kit components (WATER, SODIUM CHLORIDE, BUFFER SOLUTION…) are skipped. Lookup in drug_aliases.normalized (index loaded once per run and kept in sync): the full name first, then the molecule. Exactly one drug → resolved (match_type ALIAS); several → the one whose alias is generic, else the one whose own name matches, else ambiguous → unresolved_labels (reason ambiguous_alias).
    • Unknown molecule, single-ingredient product → a drug is minted (mintId 'DRUG', name = molecule in Title Case — "Vinblastine", not "Vinblastine Sulfate" (CLAUDE.md §7: the base molecule is the entity, salts are aliases), slug = slugify(name) with a -hc-<drug_code> suffix on clash, kind null, developmentStatus 'marketed_ca' when the DPD status maps to approved), aliases generic (molecule), salt (salt form) and brand (cleaned brand), change event created.
    • Unknown molecule, multi-ingredient product → unresolved_labels (reason component_of_multi_ingredient_product, with DIN, brand, ATC); the approval row is linked to every component that did resolve.
  4. Codes and aliases for every resolved drug: drug_codes atc (code tc_atc_number, label tc_atc), din (label = brand as published), hc_drug_code (label = brand), match_type EXACT_IDENTIFIER; a brand alias with the presentation noise removed (cleanBrandName: "PROCYTOX TABLETS 50MG" → "PROCYTOX", "THIO TEPA INJ 15MG/VIAL" → "THIO TEPA") and a salt alias when the ingredient carried one.
  5. Approval row per (product, drug) — drug_approvals: jurisdiction 'CA', authority 'Health Canada', approval_type 'DIN', application_number = DIN (8 digits), approval_date = original_market_date (null when the product is approved but not marketed — never fabricated), status: Marketed / Approved / Authorized… → approved; Cancelled Post Market / Pre Market / (Safety Issue) / (Unreturned Annual) / Dormant → withdrawn with withdrawal_date = history_date; anything else → no row + warning (statusUnknown). The DPD status is kept verbatim in raw.dpdStatus and surfaced on the pages: a cancelled or dormant DIN is one product's status, not a withdrawal of the molecule (KEYTRUDA DIN 02441152 is "Cancelled Post Market" since 2019-12-04 while other pembrolizumab DINs are marketed). indication is honest text: "Marketed in Canada as KEYTRUDA (DIN 02441152) under ATC L01FF02 PEMBROLIZUMAB. Indications are not published in the Drug Product Database — see the Health Canada Product Monograph."; cancer_id NULL, tumor_agnostic false — no cancer is ever inferred from an ATC class. raw keeps drug code, DIN, brand, statuses/dates, ATC (code, label, group), class, company, descriptor, all ingredients with strengths, routes, last_update_date.
  6. Provenance: one row per product and run when the source record changed (sourceUrl = public product page, dataset 'Health Canada DPD', datasetVersion dpd-<run date>, evidenceType regulatory, accessLevel open, geography 'CA', publishedAt = original market date, updatedAtSource = last_update_date, methodology text naming the ingredient and how it was reconciled). Unchanged product + existing row → the previous provenance row is reused (no duplicate evidence per weekly run).
  7. Idempotency: rows are keyed by (source_id, drug_id, jurisdiction 'CA', application_number = DIN) — existing rows are updated in place (status changes are recorded as approval_status_changed from pass 2), new rows inserted (approval_added from pass 2). Nothing is deleted: a DIN that disappears from the bulk list keeps its last known status.

dry_run walks the first 20 products of the list without writing (resolution against the alias index is logged as "→ CI-DRUG-…", "would mint …" or "AMBIGUOUS"). --mode backfill re-derives drugs, codes, aliases and approval rows from the composite payloads in the raw lake — no HTTP, cursor untouched.

Health check: therapeuticclass?id=92551 must contain L01FF02.

# Observed runs (local copy of production, 2026-09-11)

Dry run 20 products, 81 requests, 40 s; 10 resolved by alias, 10 "would mint" (Vinblastine, Diethylstilbestrol, Dactinomycin, Procarbazine, Busulfan, Thiotepa, Bleomycin…)
10-record smoke (--max-records 10) 41 requests, 19 s, status partial, cursor lastDrugCode 960; 5 drugs minted, 10 approval rows, 27 codes; rerun resumed from the cursor
Full pass (--reset-cursor --max-minutes 58, run ING-HEALTHCANADADPD-20260911-000003) Still running at reporting time — 455 / 1,539 products processed (cursor lastDrugCode 80770, ≈ 2.1 s per product = 4 requests at 2 req/s, 0 HTTP failures, 0 warnings). The cursor resumes the pass on the next pnpm cix run health-canada-dpd.

Counts after those 455 products (plus the 10-record smoke): 507 product source records; 52 drugs minted (Vinblastine, Busulfan, Dactinomycin, Procarbazine, Thiotepa, Bleomycin, Interferon Alfa-2b…); ingredients resolved by alias 303 + 106 after salt stripping; 502 CA approval rows on 161 drugs (165 approved, 337 withdrawn — old DINs of long-marketed molecules dominate the low drug codes); drug_codes 502 DIN + 502 hc_drug_code + 119 ATC; aliases 186 brand + 52 generic + 47 salt; 8 labels in unresolved_labels (components of multi-ingredient kits: RIBAVIRIN of REBETRON, ANCESTIM of STEMGEN, FORMESTANE, MELACINE lysate components…), 0 ambiguous. pnpm cix reconcile-drugs afterwards linked 69,207 → 74,435 trial intervention rows (+5,228; 5,155 of them to the minted drugs — e.g. Busulfan, Bleomycin, Vinblastine trials were unresolved before).

# Limitations / notes

  • No indications: the DPD is a product registry; indications live in the Product Monograph (PDF, not in the API). Canadian rows therefore never carry a cancer and never feed cancer-scoped pipeline stages or approved_drug_count per cancer; they do make the unscoped pipeline row "approved" and appear in the approvals feed and on the drug page.
  • DIN-level granularity: one molecule has many DINs (brands, generics, strengths); the feed counts records, not molecules. Status is per DIN.
  • Dates: original_market_date is the first marketing date of that DIN, often a generic's date; approval_date on a CA row is therefore not "the Canadian approval of the molecule" — the earliest CA row of a drug is the best available proxy. Historic products carry year-end placeholders from the registry itself ("1954-12-31") — kept as published.
  • Scope by ATC: products without an ATC assignment in the DPD, or classed outside L01/L02/L03/ V10 (e.g. supportive-care antiemetics, L04 immunosuppressants), are not ingested. L03 brings non-oncology immunostimulants (interferon beta for MS) — they are drugs, honestly recorded with their ATC group, but should not be read as oncology approvals.
  • Minted drugs start with no kind, mechanism or targets; ChEMBL/openFDA enrichment and pnpm cix reconcile-drugs (trial interventions) run afterwards. Salt-form duplicates against existing entities are avoided by minting the base molecule; remaining duplicates are proposed by packages/ranking/src/drug-duplicates.ts (entity_merges, status proposed).
  • Only human and radiopharmaceutical classes are ingested; veterinary/disinfectant products are recorded as source records and skipped.