spb/cancerindex

1# Connector `hgnc` — HUGO Gene Nomenclature Committee (complete set)23Tier 0 · category `genes` · canonical human gene layer (CLAUDE.md §11, §143). Every genomic4source (CIViC, GDC, ClinVar…) must reconcile gene symbols against `genes`/`gene_aliases` filled by5this connector.67Code: `packages/connectors/src/connectors/hgnc/` — `manifest.ts`, `parse.ts` (pure TSV/row8parsing), `index.ts` (batched DB writes), `fixtures/hgnc_complete_set.sample.tsv`, `hgnc.test.ts`.910## Source and access (documentation verified 2026-09-08)1112| Item | Value |13|---|---|14| Bulk file | `https://storage.googleapis.com/public-download-files/hgnc/tsv/tsv/hgnc_complete_set.txt` — HTTP 200, 16 903 161 bytes, `Last-Modified: Fri, 04 Sep 2026 15:22:27 GMT`, `text/plain`, 45 045 data rows, 53 tab-delimited columns with header |15| Download docs | https://www.genenames.org/download/archive/ |16| Format | TSV; multi-valued fields pipe-delimited and wrapped in double quotes when they contain a pipe (`"NCRNA00181|A1BGAS|A1BG-AS"`); many empty columns |17| Version | ISO date of the `Last-Modified` header (`2026-09-04`) → `ingest_runs.dataset_version`, `provenance.dataset_version`, cursor |18| Health | `HEAD` on the file URL → `healthy` with `last-modified` + `content-length` |19| Raw lake | `data/raw/hgnc/<date>/hgnc_complete_set.txt` (sha256 + size logged) and one JSON payload per row (`gene` entity, all 53 columns) |20| Schedule | weekly, `0 5 * * 2` |2122Both the SDK plain user agent `CancerIndex/0.1` and the URL-bearing variant get HTTP 200 from23Google Cloud Storage; the connector sends the plain token.2425## License evidence2627- https://www.genenames.org/about/ (fetched 2026-09-08): "No restrictions are imposed on access28  to, or use of, the data provided by the HGNC" … "provided in good faith, but make no warranty,29  express or implied, nor assume any legal liability or responsibility for any purpose for which30  they are used" … users are asked to reference "the 'HUGO Gene Nomenclature Committee at the31  University of Cambridge' and the website URL where possible" or cite "HGNC (RRID:SCR_002827)".32- https://www.genenames.org/about/license/ (fetched 2026-09-08): "all data is released under the33  Creative Commons Public Domain (CC0) License" (links to the Creative Commons CC0 page; no version34  number stated on the page — recorded as CC0 1.0, the only CC0 version).3536Manifest: `license: CC0 1.0 (public domain dedication)`, `licenseStatus: approved`,37`redistribution: allowed`, `commercialUse: allowed`, attribution string kept anyway as requested.3839## Columns used4041Required (the run fails loudly with schema-drift log entries if any is missing from the header):42`hgnc_id, symbol, name, locus_group, locus_type, status, location, alias_symbol, alias_name,43prev_symbol, prev_name, gene_group, entrez_id, ensembl_gene_id, refseq_accession, uniprot_ids,44omim_id`. New header columns are detected through `ctx.observe('hgnc_header', …)` → `schema_drift`45on the run / `degraded` health. The other 36 columns are kept in the raw payload only.4647Mapping to `genes`: `hgnc_id → hgnc_id` (unique key), `symbol`, `name`, `locus_type`,48`locus_group`, `location`, `chromosome` (parsed from `location`: leading `1–22 | X | Y`, `MT` for49"mitochondria", null for "reserved"/"unplaced"/"not on reference assembly"/empty),50`ensembl_gene_id`, `entrez_id → ncbi_gene_id`, `omim_id → omim_ids[]`, `uniprot_ids[]`,51`refseq_accession` (first value), `prev_symbol → prev_symbols[]`, `alias_symbol → alias_symbols[]`,52`gene_group → gene_families[]`, `status`. `gene_aliases`: `prev_symbol → prev_symbol`,53`alias_symbol → alias_symbol`, `prev_name → prev_name`, `alias_name → alias_name`.5455## Rules5657- Only `status = Approved` rows become `genes`. Any other status ("Entry Withdrawn") is recorded as58  a `source_records` row with `status = deprecated` and never inserted into `genes`. The complete59  set of 2026-09-04 contains **45 045 Approved rows and 0 withdrawn** (withdrawn entries live in a60  separate HGNC file).61- Genes are keyed by `hgnc_id` (`genes_hgnc_uq`). `symbol` is also unique: when an incoming row's62  symbol already belongs to a *different* `hgnc_id`, the row is skipped with a warning and63  `rejected++` (0 cases on a fresh load; guards against symbol transfers between IDs).64- Symbols failing `GENE_SYMBOL_RE` (`sdk/validate.ts`) are still ingested — HGNC is the65  authority — but counted as `validation_failures` and logged (11 in this release, all readthrough66  / duplicated-segment symbols such as `GTF2H2C_2`, `ANKRD62P1-PARP4P3`, `C4A_2`).67- **Batched writes**: existing genes are loaded once (id, hgnc_id, symbol); new ids are minted in68  one `mintIds()` call; `genes` are inserted in chunks of 1 000 with `ON CONFLICT (hgnc_id) DO69  UPDATE SET … = excluded.…`; `gene_aliases` in chunks of 1 000 with `ON CONFLICT DO NOTHING`70  (deduplicated on `(gene_id, alias, alias_type)`).71- **Batched source records** (`upsertSourceRecordsBatch` in `index.ts`): same semantics as72  `ctx.upsertSourceRecord` (sha256 payload hash, raw-lake write, `first_seen_run`/`last_seen_run`,73  `canonical_type/id`, run counters fetched/created/updated/unchanged, `ctx.observe` for drift) but74  one `SELECT … WHERE source_record_id IN (…)` per 1 000 rows, one bulk `UPDATE last_seen_run` for75  unchanged rows and one bulk `INSERT … RETURNING` for new rows. Documented here because it76  duplicates SDK logic on purpose (45 k per-row round trips would take minutes).77- Provenance: one row per run (dataset "HGNC complete set", version = Last-Modified date,78  `evidence_type = database`, `access_level = open`, `updated_at_source` = Last-Modified).79- Anomaly guard: refuse to persist when fewer than 30 000 rows parse or when the approved count80  drops below 90 % of the genes already in the table. Existing genes absent from a new file are81  *not* deprecated automatically (HGNC never removes approved entries from the complete set; a82  withdrawal shows up as a status change) — see limitations.83- Cursor: `{ version, lastModified, approvedCount, syncedAt }`.8485## Counts observed (2026-09-08, file of 2026-09-04, database `cancerindex_a`)8687| Metric | Value |88|---|---|89| Rows / approved / withdrawn / malformed | 45 045 / 45 045 / 0 / 0 |90| `genes` | 45 045 (distinct symbols 45 045, distinct hgnc_id 45 045) |91| locus groups | protein-coding 19 297 · pseudogene 14 672 · non-coding RNA 10 074 · other 1 002 |92| chromosome parsed | 45 009 (null 36: reserved/unplaced/empty); MT 56, X 1 967, Y 483 |93| cross-refs | Ensembl 42 361 · NCBI Gene 44 405 |94| `gene_aliases` | 109 373 (alias_symbol 44 787 · prev_name 36 982 · prev_symbol 15 882 · alias_name 11 722) |95| `source_records` (gene) | 45 045 |96| validation warnings | 11 symbols outside `GENE_SYMBOL_RE` (ingested) |97| Run time | first run 10 s (fetched 45 045, created 45 045); rerun 8 s (unchanged 45 045, created 0, updated 0) |9899## Known limitations100101- Withdrawn/merged HGNC entries are not in `hgnc_complete_set.txt`; a gene that disappears from the102  file keeps `status = Approved` in `genes` until a curator or a future `withdrawn.txt` connector103  handles it (no automatic deprecation, per the never-mass-delete rule).104- `gene_aliases` are never deleted when HGNC drops an alias (idempotent inserts only).105- `refseq_accession` keeps only the first accession; `omim_ids`/`uniprot_ids` keep all values.106- `is_cancer_gene` is derived later by evidence connectors, not here.107- Alias symbols are not unique across genes (e.g. `p53`); resolvers must prefer `symbol` and108  `prev_symbol` over `alias_symbol` and treat ambiguous alias hits as unresolved.109