SPB Git forge
spboucher.ai ↗

spb/cancerindex

Public
37commits 1branches 0releases
2.9 MBsize
maindefault branch
10 days agolast push
TypeScript 97.2% SQL 1.5% CSS 0.6% JavaScript 0.5%
2.4 KB
View source Raw Blame History

# ADR-002 — TypeScript connectors in Phase 1; Python/DuckDB workers later

Status: accepted (2026-09-08)

# Context

Connectors ingest from OncoTree, NCIt EVS, HGNC, CIViC, GDC, ClinVar, ClinicalTrials.gov, PubMed and CDC WONDER. In Phase 1 all of these are JSON or TSV over HTTP with pagination, rate limits and modest volumes (≤ a few million small records). The rest of the platform (schema, API, web, worker) is TypeScript in a pnpm monorepo; the ingestion SDK needs typed manifests, shared normalisation, the CancerResolver, drizzle access to the same schema, and fixtures-based tests in the same test runner.

The bioinformatics ecosystem, however, is Python-first for HGVS parsing/normalisation, liftover between GRCh37/GRCh38, VCF handling and large columnar processing (spec §231-234).

# Decision

Phase 1 connectors are TypeScript classes extending Connector in packages/connectors, sharing one SDK (manifest validation, rate-limited HttpClient, RawLake, RunContext, validators) and one registry. The worker (workers/main.ts) runs them in-process with pg-boss.

Heavy genomics work is explicitly deferred to Python workers (planned package workers-py/): HGVS normalisation (hgvs/biocommons), liftover (pyliftover/CrossMap), VCF/ClinVar bulk parsing, and cohort-scale statistics with DuckDB over Parquet files in the data lake. They will consume the same raw lake, write summarised rows into PostgreSQL through the same provenance model, and be scheduled by the same pg-boss queue (pg-boss is SQL-compatible; a Python worker can claim jobs through the documented SQL API or via an HTTP proxy).

# Consequences

  • One language for the whole Phase 1 stack: shared types (match_type, provenance input), shared normalisation, one test runner, one lockfile.
  • Variant coordinates in Phase 1 are stored as delivered by the source with the assembly recorded (variants.assembly, coordinates[]), original and normalised strings kept, no liftover performed. Cross-assembly comparisons are not offered until the Python worker ships.
  • The SDK must stay language-neutral at the boundaries: manifests are plain JSON-serialisable objects (stored in sources.manifest), lake files are JSON Lines gzip, job payloads are JSON.
  • Revisit when a source requires bulk binary formats (VCF, BAM-derived) or when a connector's run time exceeds the worker time budget because of CPU-bound normalisation.