SPB Git forge

spb/cancerindex

Public
37commits 1branches 0releases
2.9 MBsize
maindefault branch
10 days agolast push
TypeScript 97.2% SQL 1.5% CSS 0.6% JavaScript 0.5%
7.5 KB

# Connector hgnc — HUGO Gene Nomenclature Committee (complete set)

Tier 0 · category genes · canonical human gene layer (CLAUDE.md §11, §143). Every genomic source (CIViC, GDC, ClinVar…) must reconcile gene symbols against genes/gene_aliases filled by this connector.

Code: packages/connectors/src/connectors/hgnc/ — manifest.ts, parse.ts (pure TSV/row parsing), index.ts (batched DB writes), fixtures/hgnc_complete_set.sample.tsv, hgnc.test.ts.

# Source and access (documentation verified 2026-09-08)

Item Value
Bulk file https://storage.googleapis.com/public-download-files/hgnc/tsv/tsv/hgnc_complete_set.txt — HTTP 200, 16 903 161 bytes, Last-Modified: Fri, 04 Sep 2026 15:22:27 GMT, text/plain, 45 045 data rows, 53 tab-delimited columns with header
Download docs https://www.genenames.org/download/archive/
Format TSV; multi-valued fields pipe-delimited and wrapped in double quotes when they contain a pipe (`"NCRNA00181
Version ISO date of the Last-Modified header (2026-09-04) → ingest_runs.dataset_version, provenance.dataset_version, cursor
Health HEAD on the file URL → healthy with last-modified + content-length
Raw lake data/raw/hgnc/<date>/hgnc_complete_set.txt (sha256 + size logged) and one JSON payload per row (gene entity, all 53 columns)
Schedule weekly, 0 5 * * 2

Both the SDK plain user agent CancerIndex/0.1 and the URL-bearing variant get HTTP 200 from Google Cloud Storage; the connector sends the plain token.

# License evidence

  • https://www.genenames.org/about/ (fetched 2026-09-08): "No restrictions are imposed on access to, or use of, the data provided by the HGNC" … "provided in good faith, but make no warranty, express or implied, nor assume any legal liability or responsibility for any purpose for which they are used" … users are asked to reference "the 'HUGO Gene Nomenclature Committee at the University of Cambridge' and the website URL where possible" or cite "HGNC (RRID:SCR_002827)".
  • https://www.genenames.org/about/license/ (fetched 2026-09-08): "all data is released under the Creative Commons Public Domain (CC0) License" (links to the Creative Commons CC0 page; no version number stated on the page — recorded as CC0 1.0, the only CC0 version).

Manifest: license: CC0 1.0 (public domain dedication), licenseStatus: approved, redistribution: allowed, commercialUse: allowed, attribution string kept anyway as requested.

# Columns used

Required (the run fails loudly with schema-drift log entries if any is missing from the header): hgnc_id, symbol, name, locus_group, locus_type, status, location, alias_symbol, alias_name, prev_symbol, prev_name, gene_group, entrez_id, ensembl_gene_id, refseq_accession, uniprot_ids, omim_id. New header columns are detected through ctx.observe('hgnc_header', …) → schema_drift on the run / degraded health. The other 36 columns are kept in the raw payload only.

Mapping to genes: hgnc_id → hgnc_id (unique key), symbol, name, locus_type, locus_group, location, chromosome (parsed from location: leading 1–22 | X | Y, MT for "mitochondria", null for "reserved"/"unplaced"/"not on reference assembly"/empty), ensembl_gene_id, entrez_id → ncbi_gene_id, omim_id → omim_ids[], uniprot_ids[], refseq_accession (first value), prev_symbol → prev_symbols[], alias_symbol → alias_symbols[], gene_group → gene_families[], status. gene_aliases: prev_symbol → prev_symbol, alias_symbol → alias_symbol, prev_name → prev_name, alias_name → alias_name.

# Rules

  • Only status = Approved rows become genes. Any other status ("Entry Withdrawn") is recorded as a source_records row with status = deprecated and never inserted into genes. The complete set of 2026-09-04 contains 45 045 Approved rows and 0 withdrawn (withdrawn entries live in a separate HGNC file).
  • Genes are keyed by hgnc_id (genes_hgnc_uq). symbol is also unique: when an incoming row's symbol already belongs to a different hgnc_id, the row is skipped with a warning and rejected++ (0 cases on a fresh load; guards against symbol transfers between IDs).
  • Symbols failing GENE_SYMBOL_RE (sdk/validate.ts) are still ingested — HGNC is the authority — but counted as validation_failures and logged (11 in this release, all readthrough / duplicated-segment symbols such as GTF2H2C_2, ANKRD62P1-PARP4P3, C4A_2).
  • Batched writes: existing genes are loaded once (id, hgnc_id, symbol); new ids are minted in one mintIds() call; genes are inserted in chunks of 1 000 with ON CONFLICT (hgnc_id) DO UPDATE SET … = excluded.…; gene_aliases in chunks of 1 000 with ON CONFLICT DO NOTHING (deduplicated on (gene_id, alias, alias_type)).
  • Batched source records (upsertSourceRecordsBatch in index.ts): same semantics as ctx.upsertSourceRecord (sha256 payload hash, raw-lake write, first_seen_run/last_seen_run, canonical_type/id, run counters fetched/created/updated/unchanged, ctx.observe for drift) but one SELECT … WHERE source_record_id IN (…) per 1 000 rows, one bulk UPDATE last_seen_run for unchanged rows and one bulk INSERT … RETURNING for new rows. Documented here because it duplicates SDK logic on purpose (45 k per-row round trips would take minutes).
  • Provenance: one row per run (dataset "HGNC complete set", version = Last-Modified date, evidence_type = database, access_level = open, updated_at_source = Last-Modified).
  • Anomaly guard: refuse to persist when fewer than 30 000 rows parse or when the approved count drops below 90 % of the genes already in the table. Existing genes absent from a new file are not deprecated automatically (HGNC never removes approved entries from the complete set; a withdrawal shows up as a status change) — see limitations.
  • Cursor: { version, lastModified, approvedCount, syncedAt }.

# Counts observed (2026-09-08, file of 2026-09-04, database cancerindex_a)

Metric Value
Rows / approved / withdrawn / malformed 45 045 / 45 045 / 0 / 0
genes 45 045 (distinct symbols 45 045, distinct hgnc_id 45 045)
locus groups protein-coding 19 297 · pseudogene 14 672 · non-coding RNA 10 074 · other 1 002
chromosome parsed 45 009 (null 36: reserved/unplaced/empty); MT 56, X 1 967, Y 483
cross-refs Ensembl 42 361 · NCBI Gene 44 405
gene_aliases 109 373 (alias_symbol 44 787 · prev_name 36 982 · prev_symbol 15 882 · alias_name 11 722)
source_records (gene) 45 045
validation warnings 11 symbols outside GENE_SYMBOL_RE (ingested)
Run time first run 10 s (fetched 45 045, created 45 045); rerun 8 s (unchanged 45 045, created 0, updated 0)

# Known limitations

  • Withdrawn/merged HGNC entries are not in hgnc_complete_set.txt; a gene that disappears from the file keeps status = Approved in genes until a curator or a future withdrawn.txt connector handles it (no automatic deprecation, per the never-mass-delete rule).
  • gene_aliases are never deleted when HGNC drops an alias (idempotent inserts only).
  • refseq_accession keeps only the first accession; omim_ids/uniprot_ids keep all values.
  • is_cancer_gene is derived later by evidence connectors, not here.
  • Alias symbols are not unique across genes (e.g. p53); resolvers must prefer symbol and prev_symbol over alias_symbol and treat ambiguous alias hits as unresolved.