# Connector clinvar — NCBI ClinVar (variant_summary bulk file)

# License / data-use policy (quoted)

NCBI Website and Data Usage Policies — https://www.ncbi.nlm.nih.gov/home/about/policies/ :

Information that is created by or for the US government on this site is within the public domain. Public domain information on the National Library of Medicine (NLM) Web pages may be freely distributed and copied. However, it is requested that in any subsequent use of this work, NLM be given appropriate acknowledgment.

Molecular Data Usage: "NCBI itself places no restrictions on the use or distribution of the data contained therein." Referenced vocabularies (OMIM, Orphanet) have their own terms — only their identifiers are stored. Citation: Landrum MJ et al., ClinVar: improving access to variant interpretations and supporting evidence, Nucleic Acids Res 2018 (PMID 29165669).

# Verified file and columns

HEAD on 2026-09-08: Content-Length 442645117, Last-Modified Sun, 06 Sep 2026 14:42:05 GMT → dataset version 2026-09-06; sha256 0946d1661f102f39e4a5b37db1e04be2990478eb547cceb21ffb0ba23c7ee78f; 9,050,979 data lines (one line per variant per assembly). Header — 43 tab-separated columns:

#AlleleID, Type, Name, GeneID, GeneSymbol, HGNC_ID, ClinicalSignificance, ClinSigSimple, LastEvaluated, RS# (dbSNP), nsv/esv (dbVar), RCVaccession, PhenotypeIDS, PhenotypeList, Origin, OriginSimple, Assembly, ChromosomeAccession, Chromosome, Start, Stop, ReferenceAllele, AlternateAllele, Cytogenetic, ReviewStatus, NumberSubmitters, Guidelines, TestedInGTR, OtherIDs, SubmitterCategories, VariationID, PositionVCF, ReferenceAlleleVCF, AlternateAlleleVCF, SomaticClinicalImpact, SomaticClinicalImpactLastEvaluated, ReviewStatusClinicalImpact, Oncogenicity, OncogenicityLastEvaluated, ReviewStatusOncogenicity, SCVsForAggregateGermlineClassification, SCVsForAggregateSomaticClinicalImpact, SCVsForAggregateOncogenicityClassification

parseHeader maps names → indexes and throws when a required column disappears (schema-drift guard); the six somatic/oncogenicity columns are optional. Placeholders observed in the live file: -, na, -1 (GeneID, RS#, PositionVCF), "N conditions" in PhenotypeIDS/PhenotypeList when an RCV has > 5 conditions, and "no classification for the single variant" inside the somatic/oncogenicity columns (treated as empty). GeneSymbol may be a comma-separated list or free text ("covers 14 genes, none of which curated to show dosage sensitivity") — only single-gene rows (GeneID ≠ −1) get a gene.

Health check: HEAD of the file URL → healthy with Last-Modified and size.

# Filter (documented)

A row is kept when all hold:

1. Assembly == 'GRCh38' (GRCh37 rows are the same variants; no lift-over is ever performed).
2. Either the row has a non-empty SomaticClinicalImpact or Oncogenicity classification, or the row has some germline classification (rows with ClinSigSimple = -1 and ClinicalSignificance = '-' are "included" variants that only exist as members of a haplotype/genotype record and carry no interpretation — ≈236k GRCh38 rows in the 2026-09-06 file; skipped) and PhenotypeList matches the cancer scope regular expression:

cancer|carcinoma|neoplas|tumor|tumour|sarcoma|lymphoma|leukemia|leukaemia|melanoma|myeloma|glioma|blastoma|adenoma|polyposis|li-fraumeni|lynch|cowden|von hippel|neurofibromatosis|tuberous sclerosis|fanconi|bloom syndrome|ataxia-telangiectasia|peutz|birt-hogg|gorlin|werner|xeroderma|dyskeratosis|hereditary (breast|diffuse|leiomyomatosis|paraganglioma|papillary)|multiple endocrine neoplasia|familial adenomatous|juvenile polyposis|retinoblastoma|wilms|malignan   (case-insensitive)

Identifiers alone (MedGen/OMIM/MONDO in PhenotypeIDS) never qualify a row; no gene list is hard-coded. Non-cancer co-conditions of a kept variant (e.g. Hemochromatosis type 1 next to Neuroendocrine neoplasm) are stored in conditions but are not queued as unresolved cancer labels.

Result on the 2026-09-06 file: 412,489 rows kept of 4.5M GRCh38 rows (phenotype 409,475; somatic classification 1,319; oncogenicity 1,695). Hereditary cancer-predisposing syndrome (a generic panel phenotype) alone accounts for ~207k rows.

# Normalization

• variants keyed by clinvarVariationId (= VariationID): existing rows (e.g. created by CIViC with the same ClinVar id — 793 on 2026-09-08) are reused and only enriched (coordinates entry, dbSNP, HGVS when empty; CIViC keeps naming/typing); new rows get slug = clinvar-<VariationID>, name = ClinVar Name, geneSymbol/geneId (gene ensured by symbol via the shared GeneCache, minimal row when absent; ncbiGeneId filled when empty), hgvsC/hgvsP/hgvsG parsed from Name (NM_007294.4(BRCA1):c.181T>G (p.Cys61Gly) → NM_007294.4:c.181T>G, p.Cys61Gly; NC_…:g.… → hgvsG), assembly 'GRCh38', dbsnpIds ['rs<RS#>'] when RS# ≠ −1.
• Variant type map (Type → variant_type): single nucleotide variant → SNV; Deletion → deletion; Duplication / Insertion / Tandem duplication → insertion; Indel → indel; Microsatellite → other; copy number gain → amplification; copy number loss → deletion_cna; Inversion / Translocation / Complex → structural; fusion → fusion; Variation / protein only / Haplotype / CompoundHeterozygote / Diplotype / Phase unknown / Distinct chromosomes → other.
• Coordinates: when PositionVCF/ReferenceAlleleVCF/AlternateAlleleVCF are present they populate start/end/ref/alt (end = PositionVCF + len(ref) − 1, convention vcf_left_shifted); otherwise ClinVar's right-shifted Start/Stop with ReferenceAllele/AlternateAllele (na → null). Both representations, the chromosome accession and the cytogenetic band are preserved verbatim in the coordinates JSON entry (source: 'clinvar').
• variant_clinical_significance (unique on clinvar_variation_id): clinicalSignificance (aggregate germline classification as published; - when only a somatic/oncogenicity classification exists), reviewStatus, starRating (practice guideline 4 · reviewed by expert panel 3 · criteria provided, multiple submitters, no conflicts 2 · criteria provided, single submitter 1 · criteria provided, conflicting classifications 1 · no assertion / no classification 0 · otherwise null), lastEvaluated, conditions (PhenotypeList split on | and ;, placeholders removed), conditionCancerIds (resolver byLabel per condition + byCode for MONDO/MedGen/OMIM/Orphanet/MeSH identifiers — no-ops until those code systems are loaded), originSimple, numberSubmitters, provenanceId.
• Provenance: one run-level row per dataset (dataset 'ClinVar variant_summary', datasetVersion = Last-Modified date, sourceUrl = file URL, evidenceType database, accessLevel open, methodology incl. sha256/size) shared by benign/VUS/conflicting rows; a per-variant provenance row (sourceRecordId = VariationID, sourceUrl https://www.ncbi.nlm.nih.gov/clinvar/variation/<id>/, updatedAtSource = LastEvaluated, methodology = classification + review status + submitter count) for pathogenic / likely pathogenic rows and for rows with a somatic/oncogenicity classification — bounds volume (≈61k rows) while keeping a direct link for the clinically loaded records.
• Unresolved conditions: cancer-looking labels that byLabel cannot resolve are aggregated in memory and flushed once (unresolved_labels, count = number of rows observing the label, context {clinvarVariationId sample, occurrences}).

# Restartability, idempotency, volume

• Download skipped when data/raw/clinvar/variant_summary-<version>.txt.gz exists with the announced size; otherwise downloadBulk (sha256 recorded, truncated download refused).
• Streaming gunzip → readline; rows persisted in batches of 500 (batched source_records upsert with hash comparison + raw lake, batched variants insert with mintIds, batched provenance, batched variant_clinical_significance upsert). Rows whose source record is unchanged since the last load are skipped (only last_seen_run refreshed) unless --mode backfill.
• Cursor {datasetVersion, file, sha256, bytes, lineOffset, kept, completed}: ctx.shouldStop() is honoured between batches, lineOffset lets the next run resume (skip lines < offset); a new Last-Modified resets the offset. When the same file is already fully loaded the run is a no-op.
• --max-records N persists exactly N rows (batch size capped).

Observed on cancerindex_c (Apple Silicon laptop, local Postgres):

Final state: 411,467 significance rows / 412,692 ClinVar-anchored variants (793 shared with CIViC), 61,207 distinct provenance rows; star ratings 0 ★ 9,113 · 1 ★ 234,610 · 2 ★ 153,476 · 3 ★ 12,663 · 4 ★ 1 · null 1,604 (somatic/oncogenicity-only rows); origin germline 392,774 · unknown 12,292 · somatic 3,973 · germline/somatic 2,397; classifications: Uncertain significance 174k, Likely benign 96k, Pathogenic 41.5k, Conflicting 40k, Benign/Likely benign 22.6k, Benign 15k, Likely pathogenic 10k; 34,763 rows with ≥ 1 condition resolved to a canonical cancer and 703 distinct unresolved cancer-looking condition labels queued (OncoTree-only ontology in the development DB, no MedGen/MONDO/OMIM codes yet). Top unresolved condition labels: Hereditary cancer-predisposing syndrome (207k rows), Ovarian serous cystadenocarcinoma, Thyroid cancer, nonmedullary, 1, Familial cancer of breast, Nonpapillary renal cell carcinoma, Hereditary breast ovarian cancer syndrome, Uterine corpus endometrial carcinoma, Cervical cancer, Hereditary nonpolyposis colorectal neoplasms, Gastric cancer, Malignant tumor of esophagus, Fanconi anemia, Lung cancer, Ataxia-telangiectasia syndrome, Neurofibromatosis, type 1 — most resolve once NCIt synonyms/MedGen/MONDO codes are loaded (byCode calls are already in place).

# Limitations / notes

• Germline-centric: variant_clinical_significance has no columns for the somatic clinical impact / oncogenicity classifications; they are kept in the raw payload (source_records → lake) and flagged by the keep reason only. A follow-up schema extension (variant_somatic_classifications) would expose them.
• Hereditary-syndrome phenotypes (Fanconi anemia, NF1, ataxia-telangiectasia…) are in scope by design (cancer predisposition) although the conditions themselves are not malignancies.
• ~16k minimal genes rows are created for symbols not yet present (panel genes); HGNC enriches them.
• The SDK persists connector_cursors only at the end of a run; a hard kill (SIGKILL) loses the in-memory line offset, a time-budget stop does not.
• Raw lake: data/raw/clinvar/ holds the bulk file (442 MB) plus ~36 MB of gzip JSONL rows.