# ADR-002 — TypeScript connectors in Phase 1; Python/DuckDB workers later

Status: accepted (2026-09-08)

## Context

Connectors ingest from OncoTree, NCIt EVS, HGNC, CIViC, GDC, ClinVar, ClinicalTrials.gov, PubMed and
CDC WONDER. In Phase 1 all of these are JSON or TSV over HTTP with pagination, rate limits and
modest volumes (≤ a few million small records). The rest of the platform (schema, API, web, worker)
is TypeScript in a pnpm monorepo; the ingestion SDK needs typed manifests, shared normalisation,
the `CancerResolver`, drizzle access to the same schema, and fixtures-based tests in the same test
runner.

The bioinformatics ecosystem, however, is Python-first for HGVS parsing/normalisation, liftover
between GRCh37/GRCh38, VCF handling and large columnar processing (spec §231-234).

## Decision

Phase 1 connectors are TypeScript classes extending `Connector` in `packages/connectors`, sharing one
SDK (manifest validation, rate-limited `HttpClient`, `RawLake`, `RunContext`, validators) and one
registry. The worker (`workers/main.ts`) runs them in-process with pg-boss.

Heavy genomics work is explicitly deferred to **Python workers** (planned package `workers-py/`):
HGVS normalisation (`hgvs`/`biocommons`), liftover (`pyliftover`/CrossMap), VCF/ClinVar bulk parsing,
and cohort-scale statistics with **DuckDB over Parquet** files in the data lake. They will consume
the same raw lake, write summarised rows into PostgreSQL through the same provenance model, and be
scheduled by the same pg-boss queue (pg-boss is SQL-compatible; a Python worker can claim jobs
through the documented SQL API or via an HTTP proxy).

## Consequences

- One language for the whole Phase 1 stack: shared types (`match_type`, provenance input),
  shared normalisation, one test runner, one lockfile.
- Variant coordinates in Phase 1 are stored as delivered by the source with the assembly recorded
  (`variants.assembly`, `coordinates[]`), original and normalised strings kept, no liftover
  performed. Cross-assembly comparisons are not offered until the Python worker ships.
- The SDK must stay language-neutral at the boundaries: manifests are plain JSON-serialisable
  objects (stored in `sources.manifest`), lake files are JSON Lines gzip, job payloads are JSON.
- Revisit when a source requires bulk binary formats (VCF, BAM-derived) or when a connector's run
  time exceeds the worker time budget because of CPU-bound normalisation.