SPB Git
spboucher.ai

spb/cancerindex

Public

Foundation: monorepo, schema, provenance SDK, ontology resolver, ranking engine, OncoTree connector

Simon-Pierre Boucher committed (Sep 8, 2026) Browse files

74 changed files +13,851 −0

74 changed files
Load all 74 files Unified Split
added .env.example +34 −0
@@ -0,0 +1,34 @@
1 +# CancerIndex — environment (never commit real values)
2 +NODE_ENV=development
3 +DATABASE_URL=postgres://localhost:5432/cancerindex
4 +DB_POOL_MAX=10
5 +
6 +WEB_PORT=8250
7 +API_PORT=8251
8 +API_HOST=127.0.0.1
9 +CI_API_URL=http://127.0.0.1:8251
10 +NEXT_PUBLIC_SITE_URL=http://localhost:8250
11 +NEXT_TELEMETRY_DISABLED=1
12 +
13 +# Data lake (raw connector payloads, gzip JSON) — outside git
14 +CI_DATA_DIR=./data
15 +
16 +# Admin console (/admin) bearer token
17 +ADMIN_TOKEN=change-me
18 +
19 +# NCBI E-utilities (PubMed, ClinVar) — tool/email are mandatory, api key raises the limit to 10 req/s
20 +NCBI_TOOL=cancerindex
21 +NCBI_EMAIL=
22 +NCBI_API_KEY=
23 +
24 +# SEER API (free key from https://api.seer.cancer.gov/) — connector stays "awaiting_credentials" without it
25 +SEER_API_KEY=
26 +
27 +# Optional AI provider for /ask (Phase 5) — the database works without it
28 +ANTHROPIC_API_KEY=
29 +OPENAI_API_KEY=
30 +OPENAI_BASE_URL=
31 +
32 +WORKER_CONCURRENCY=4
33 +CI_MAX_RUN_MINUTES=45
34 +LOG_LEVEL=info
added .gitignore +15 −0
@@ -0,0 +1,15 @@
1 +node_modules/
2 +.next/
3 +dist/
4 +*.tsbuildinfo
5 +.env
6 +.env.*
7 +!.env.example
8 +data/raw/
9 +data/cache/
10 +data/tmp/
11 +logs/
12 +tmp/
13 +coverage/
14 +.DS_Store
15 +.claude/
added .npmrc +2 −0
@@ -0,0 +1,2 @@
1 +auto-install-peers=true
2 +strict-peer-dependencies=false
added CLAUDE.md +82 −0
@@ -0,0 +1,82 @@
1 +# CLAUDE.md — CancerIndex.io (repository guide)
2 +
3 +The full product specification (364 sections) lives in `docs/SPEC-original.md`. This file is the
4 +condensed, operational version every contributor (human or agent) must follow.
5 +
6 +## Mission
7 +Build **the global index of cancer**: a provenance-first, continuously updated, transparently
8 +sourced oncology ontology connecting epidemiology, genomics, biomarkers, therapies, trials,
9 +regulatory evidence and literature — for every recognized malignant disease entity, not a list of
10 +30 common cancers. Not a physician, not a diagnostic tool, no treatment recommendations.
11 +
12 +## Non-negotiables (from the spec)
13 +1. **Provenance first** — no scientific number without a `provenance` row; derived values carry a
14 + `formula_version` and their inputs (§2, §250-253). Raw payloads go to the data lake (§26).
15 +2. **Layers stay separable**: RAW → NORMALIZED → CANONICAL → DERIVED → RANKED → AI (§2).
16 +3. **Never fake data**: no mock/hardcoded statistics in UI; show "Data not yet available" (§281-283).
17 +4. **Identifiers are first-class**: keep every upstream ID in `*_codes` / dedicated columns (§347);
18 + public IDs are `CI-<NS>-00000001` minted via `mintId` (§6), never DB integers.
19 +5. **Reconciliation before ingestion**: IDs → curated aliases → normalized strings; LLM only as a
20 + *candidate* generator; unknown labels go to `unresolved_labels`, never dropped (§69, §222, §340).
21 +6. **Time-aware observations**: never overwrite year X with year Y; store both (§71).
22 +7. **Context on every edge**: cancer context, direction, evidence level, provenance (§29, §244).
23 +8. **Licensing gate**: no connector goes live before manifest `licenseStatus` is reviewed; IARC/
24 + GLOBOCAN stays `review`, SEER `awaiting_credentials` until keys/terms are settled (§10.4, §142).
25 +9. **Verify docs before coding a connector** — official docs, smoke test with curl, fixture, tests,
26 + prove on 10/100/1000 records before a full sync (§224-227). Record `documentationVerifiedAt`.
27 +10. **Rate limits, idempotency, restartability, anomaly guard** (never mass-delete on a shrunken
28 + response) (§90-92, §171, §229).
29 +11. **Rankings need scope + year + source + formula version + "Why this rank?"** (§33, §183).
30 + Composite scores are Phase 2+ (§353).
31 +12. **Scientific safety labels**: observed / published / curated / regulatory / guideline /
32 + computed / AI-generated — never merged (§3). Population survival ≠ individual prognosis (§325).
33 +
34 +## Repository layout
35 +```
36 +apps/web Next.js 16 (webpack build, --webpack), Tailwind v4, server components, /api/v1 proxied to apps/api
37 +apps/api Fastify /v1 public API (JSON envelope { data, sources, dataRelease })
38 +workers/ pg-boss scheduler: connector schedules, counters, rankings
39 +packages/shared ids, provenance types, normalization, logger
40 +packages/database Drizzle schema (snake_case casing), migrations, seed (metrics + geographies only)
41 +packages/ontology qualifier rules, CancerResolver (alias/code reconciliation), TOP_LEVEL_CANCERS
42 +packages/connectors SDK (manifest, HttpClient, RawLake, RunContext, validators) + connectors/<id>/{manifest.ts,index.ts,fixtures/,*.test.ts}
43 +packages/ranking counters (entity_counters), ranking engine (snapshots + lineage), trace
44 +scripts/ci.ts operator CLI: `pnpm cix connectors|run <id>|run-all|health|sources:sync|counters|rank|stats|trace`
45 +docs/ architecture, data model, methodology, source policy, ADRs, connector docs
46 +data/raw data lake (gitignored) — {source}/{date}/{entity}/{runId}-{part}.jsonl.gz
47 +```
48 +
49 +## Conventions
50 +- TypeScript strict, ESM, NodeNext imports with `.js` suffix inside packages; web app uses
51 + `moduleResolution: bundler` + webpack `extensionAlias` (never import `@cancerindex/connectors` in web).
52 +- Database: `getDb()` singleton; Drizzle `casing: 'snake_case'`; raw SQL via `sql` template for
53 + aggregates. Schema changes = new migration via `pnpm db:generate` (one per integration), never
54 + manual prod mutation. Agents developing in parallel use `drizzle-kit push` on their own DB and do
55 + **not** commit migration files; the integrator generates the migration.
56 +- Connector = `class extends Connector` with `manifest`, `healthCheck(ctx)`, `sync(ctx)`; register
57 + in `packages/connectors/src/registry.ts`. Use `ctx.http` (rate-limited), `ctx.upsertSourceRecord`
58 + (idempotent + raw lake), `ctx.addProvenance`, `ctx.recordUnresolved`, `ctx.cursor` (restart),
59 + `ctx.shouldStop()` between pages (time budget), validators from `sdk/validate.ts`.
60 +- Reconciliation: `new CancerResolver(db)` → `warm()` → `byCode()/byLabel()/resolve()`; store
61 + `match_type` on every mapping; enqueue misses with `ctx.recordUnresolved`.
62 +- Tests: vitest; connector tests run against **fixtures** (sanitized JSON in `fixtures/`), never
63 + live APIs; cover normal / empty / pagination / rate-limit / server error / malformed record.
64 +- UI: scientific-editorial aesthetic (off-white, charcoal, muted neutrals, dense tables, no
65 + gradients/cards-everywhere), WCAG-minded, mobile first-class. Every number shows source badge,
66 + unit, population, period and a freshness line. Empty state = "Data not yet available".
67 +- Language: English UI; code comments English; commit messages English.
68 +
69 +## Running locally
70 +```
71 +createdb cancerindex && cp .env.example .env
72 +pnpm install && pnpm db:migrate && pnpm db:seed && pnpm cix sources:sync
73 +pnpm cix run oncotree --mode dry_run # smoke; then without --mode
74 +pnpm cix counters && pnpm cix rank
75 +pnpm dev:api & pnpm dev:web
76 +```
77 +
78 +## Deployment
79 +MacLustr cluster via `mld` (gateway M1M32): manifest `~/dispatch/apps/cancerindex.json`,
80 +`mld stage <dir> cancerindex` then `mld deploy cancerindex`. Web :8250, API :8251, PM2 processes
81 +`cancerindex-web|api|worker|ngrok`, domain www.cancerindex.io (ngrok reserved). Postgres 17 +
82 +pgvector on the node, database `cancerindex`. Source of truth for the code: spbgit `cancerindex.git`.
added package.json +40 −0
@@ -0,0 +1,40 @@
1 +{
2 + "name": "cancerindex",
3 + "version": "0.1.0",
4 + "private": true,
5 + "description": "CancerIndex.io — the global index of cancer: provenance-first oncology ontology, epidemiology, genomics, trials, literature, rankings",
6 + "type": "module",
7 + "packageManager": "pnpm@11.1.2",
8 + "engines": {
9 + "node": ">=22"
10 + },
11 + "scripts": {
12 + "build": "pnpm -r --filter './packages/**' run build && pnpm --filter @cancerindex/api run build && pnpm --filter @cancerindex/web run build",
13 + "typecheck": "pnpm -r run typecheck",
14 + "test": "pnpm -r run test",
15 + "db:generate": "pnpm --filter @cancerindex/database run generate",
16 + "db:migrate": "pnpm --filter @cancerindex/database run migrate",
17 + "db:seed": "pnpm --filter @cancerindex/database run seed",
18 + "dev:web": "pnpm --filter @cancerindex/web run dev",
19 + "dev:api": "pnpm --filter @cancerindex/api run dev",
20 + "worker": "tsx workers/main.ts",
21 + "cix": "tsx scripts/ci.ts",
22 + "registry": "tsx scripts/build-registry.ts",
23 + "registry:check": "tsx scripts/build-registry.ts --check"
24 + },
25 + "dependencies": {
26 + "@cancerindex/connectors": "workspace:*",
27 + "@cancerindex/database": "workspace:*",
28 + "@cancerindex/ontology": "workspace:*",
29 + "@cancerindex/ranking": "workspace:*",
30 + "@cancerindex/shared": "workspace:*",
31 + "drizzle-orm": "^0.45.0",
32 + "postgres": "^3.4.7"
33 + },
34 + "devDependencies": {
35 + "@types/node": "^24.0.0",
36 + "tsx": "^4.20.0",
37 + "typescript": "^5.9.3",
38 + "vitest": "^3.2.0"
39 + }
40 +}
added packages/connectors/package.json +36 −0
@@ -0,0 +1,36 @@
1 +{
2 + "name": "@cancerindex/connectors",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "exports": {
7 + ".": {
8 + "types": "./src/index.ts",
9 + "default": "./src/index.ts"
10 + },
11 + "./sdk": {
12 + "types": "./src/sdk/index.ts",
13 + "default": "./src/sdk/index.ts"
14 + }
15 + },
16 + "scripts": {
17 + "build": "tsc -p tsconfig.json --noEmit",
18 + "typecheck": "tsc -p tsconfig.json --noEmit",
19 + "test": "vitest run --passWithNoTests"
20 + },
21 + "dependencies": {
22 + "@cancerindex/database": "workspace:*",
23 + "@cancerindex/ontology": "workspace:*",
24 + "@cancerindex/shared": "workspace:*",
25 + "drizzle-orm": "^0.45.0",
26 + "fast-xml-parser": "^5.2.0",
27 + "postgres": "^3.4.7",
28 + "yaml": "^2.8.0",
29 + "zod": "^4.0.0"
30 + },
31 + "devDependencies": {
32 + "@types/node": "^24.0.0",
33 + "typescript": "^5.9.3",
34 + "vitest": "^3.2.0"
35 + }
36 +}
added packages/connectors/src/connectors/oncotree/index.ts +185 −0
@@ -0,0 +1,185 @@
1 +import { eq, sql } from 'drizzle-orm';
2 +import { z } from 'zod';
3 +import { normalizeLabel, slugify } from '@cancerindex/shared';
4 +import { anatomicalSites, cancerAliases, cancerAnatomy, cancerCodes, cancerHierarchy, cancers, mintId } from '@cancerindex/database';
5 +import { CancerResolver } from '@cancerindex/ontology';
6 +import { Connector, type ConnectorHealth, type RunContext } from '../../sdk/run.js';
7 +import { manifest } from './manifest.js';
8 +
9 +const TumorType = z.object({
10 + code: z.string(),
11 + name: z.string(),
12 + mainType: z.string().nullable().optional(),
13 + tissue: z.string().nullable().optional(),
14 + parent: z.string().nullable().optional(),
15 + level: z.number().int(),
16 + externalReferences: z.record(z.string(), z.array(z.string())).default({}),
17 + history: z.array(z.string()).default([]),
18 + revocations: z.array(z.string()).default([]),
19 + precursors: z.array(z.string()).default([]),
20 + color: z.string().nullable().optional(),
21 +});
22 +type TumorType = z.infer<typeof TumorType>;
23 +
24 +const HEME_RE = /lymphoid|myeloid|blood|leukemia|lymphoma|myeloma|histiocyt|mastocyt|plasma cell/i;
25 +const BENIGN_RE = /\bbenign\b|borderline|\bnevus\b|\bnevi\b|\badenoma\b|hyperplasia|\bcyst\b|hamartoma|\blipoma\b|leiomyoma\b|\bfibroma\b|hemangioma\b|papilloma\b|dysplasia|in situ|\bpolyp\b|schwannoma\b|meningioma\b|paraganglioma\b|pheochromocytoma\b|teratoma\b|thymoma\b/i;
26 +
27 +/**
28 + * OncoTree connector: maps every OncoTree node onto the canonical cancer set.
29 + * - level-1 "tissue" nodes (parent TISSUE) are anatomical sites, not cancers → anatomical_sites
30 + * - NCIt cross-reference → EXACT_IDENTIFIER match to the NCIt-derived canonical entity
31 + * - otherwise preferred-name match → ONTOLOGY_EXACT
32 + * - otherwise a new canonical entity is created *only* because OncoTree is a recognized ontology
33 + * (CLAUDE.md §340) — flagged with primaryOncotreeCode and no NCIt code.
34 + * The OncoTree parent/child structure is stored as hierarchy_type = "oncotree" (levels ≥ 2), and
35 + * level-2 nodes are linked to their tissue via cancer_anatomy.
36 + */
37 +export class OncotreeConnector extends Connector {
38 + readonly manifest = manifest;
39 +
40 + async healthCheck(ctx: RunContext): Promise<ConnectorHealth> {
41 + try {
42 + const versions = await ctx.http.json<Array<{ api_identifier: string; release_date: string }>>(`${manifest.access.baseUrl}/versions`);
43 + const stable = versions.find((v) => v.api_identifier === 'oncotree_latest_stable');
44 + return { status: 'healthy', detail: `latest stable ${stable?.release_date ?? '?'}` };
45 + } catch (e) {
46 + return { status: 'failing', detail: (e as Error).message };
47 + }
48 + }
49 +
50 + async sync(ctx: RunContext): Promise<void> {
51 + const versions = await ctx.http.json<Array<{ api_identifier: string; release_date: string; visible: boolean }>>(`${manifest.access.baseUrl}/versions`);
52 + const stable = versions.find((v) => v.api_identifier === 'oncotree_latest_stable');
53 + // Pin to the dated release identifier so the dataset version is reproducible.
54 + const dated = versions.find((v) => v.release_date === stable?.release_date && v.api_identifier !== 'oncotree_latest_stable');
55 + const version = dated?.api_identifier ?? 'oncotree_latest_stable';
56 + ctx.datasetVersion = version;
57 + ctx.info(`OncoTree version ${version}`);
58 +
59 + const raw = await ctx.http.json<unknown[]>(`${manifest.access.baseUrl}/tumorTypes?version=${encodeURIComponent(version)}`);
60 + const nodes: TumorType[] = [];
61 + for (const r of raw) {
62 + const parsed = TumorType.safeParse(r);
63 + if (!parsed.success) {
64 + ctx.counters.validationFailures++;
65 + ctx.counters.rejected++;
66 + ctx.warn(`invalid tumor type record: ${parsed.error.issues[0]?.message}`);
67 + continue;
68 + }
69 + nodes.push(parsed.data);
70 + }
71 + ctx.info(`fetched ${nodes.length} tumor types`);
72 + if (nodes.length < 500) throw new Error(`anomaly: only ${nodes.length} OncoTree nodes returned (expected ~900) — refusing to persist (CLAUDE.md §171)`);
73 + if (ctx.mode === 'dry_run') {
74 + ctx.counters.fetched = nodes.length;
75 + return;
76 + }
77 +
78 + const resolver = new CancerResolver(ctx.db);
79 + await resolver.warm();
80 + const byCode = new Map(nodes.map((n) => [n.code, n]));
81 + const canonicalByOncotree = new Map<string, string>();
82 + const siteByOncotree = new Map<string, string>();
83 + await ctx.addProvenance({ dataset: 'OncoTree tumor types', datasetVersion: version, sourceUrl: `${manifest.access.baseUrl}/tumorTypes?version=${version}`, evidenceType: 'expert_curation', accessLevel: 'open', methodology: 'OncoTree curated tumor type hierarchy' });
84 +
85 + // Pass 0: tissue roots → anatomical sites
86 + for (const node of nodes.filter((n) => n.parent === 'TISSUE')) {
87 + await ctx.upsertSourceRecord('tissue', node.code, node);
88 + const slug = slugify(node.name);
89 + const ncit = node.externalReferences['NCI']?.[0] ?? null;
90 + const [existing] = await ctx.db.select({ id: anatomicalSites.id }).from(anatomicalSites).where(eq(anatomicalSites.slug, slug)).limit(1);
91 + let siteId = existing?.id;
92 + if (!siteId) {
93 + siteId = await mintId(ctx.db, 'ANAT');
94 + await ctx.db.insert(anatomicalSites).values({ id: siteId, name: node.name, slug, ncitCode: ncit, system: node.tissue ?? node.name });
95 + } else await ctx.db.update(anatomicalSites).set({ name: node.name, ncitCode: ncit }).where(eq(anatomicalSites.id, siteId));
96 + siteByOncotree.set(node.code, siteId);
97 + }
98 +
99 + // Pass 1: resolve / create canonical entities (parents before children thanks to level ordering).
100 + for (const node of [...nodes].filter((n) => n.parent !== 'TISSUE').sort((a, b) => a.level - b.level)) {
101 + const rec = await ctx.upsertSourceRecord('tumor_type', node.code, node);
102 + let cancerId: string | null = null;
103 + let matchType = 'UNRESOLVED';
104 + const ncitCodes = node.externalReferences['NCI'] ?? [];
105 + for (const c of ncitCodes) {
106 + const hit = resolver.byCode('ncit', c);
107 + if (hit) {
108 + cancerId = hit.cancerId;
109 + matchType = hit.matchType;
110 + break;
111 + }
112 + }
113 + if (!cancerId) {
114 + const hit = resolver.byCode('oncotree', node.code);
115 + if (hit) {
116 + cancerId = hit.cancerId;
117 + matchType = hit.matchType;
118 + }
119 + }
120 + if (!cancerId) {
121 + const hit = resolver.byLabel(node.name);
122 + if (hit && hit.matchType === 'ONTOLOGY_EXACT') {
123 + cancerId = hit.cancerId;
124 + matchType = hit.matchType;
125 + }
126 + }
127 + if (!cancerId) {
128 + // Recognized ontology node without NCIt anchor → new canonical entity (CLAUDE.md §340 satisfied: OncoTree is a recognized ontology).
129 + const slugBase = slugify(node.name);
130 + let slug = slugBase;
131 + for (let i = 2; ; i++) {
132 + const [clash] = await ctx.db.select({ id: cancers.id }).from(cancers).where(eq(cancers.slug, slug)).limit(1);
133 + if (!clash) break;
134 + slug = `${slugBase}-${i}`;
135 + }
136 + const heme = HEME_RE.test(`${node.tissue ?? ''} ${node.mainType ?? ''} ${node.name}`);
137 + cancerId = await mintId(ctx.db, 'CAN');
138 + await ctx.db.insert(cancers).values({
139 + id: cancerId,
140 + slug,
141 + canonicalName: node.name,
142 + entityType: node.level >= 3 ? 'subtype' : heme ? 'hematologic_malignancy' : 'cancer',
143 + malignant: !BENIGN_RE.test(node.name),
144 + solidTumor: !heme,
145 + hematologic: heme,
146 + primaryOncotreeCode: node.code,
147 + depth: node.level,
148 + classificationVersion: version,
149 + });
150 + await ctx.db.insert(cancerAliases).values({ cancerId, alias: node.name, normalized: normalizeLabel(node.name), aliasType: 'preferred', sourceId: ctx.sourceId, sourceTerminology: 'OncoTree' }).onConflictDoNothing();
151 + matchType = 'EXACT_IDENTIFIER';
152 + await ctx.recordChange('cancer', cancerId, 'created', `Created from OncoTree ${node.code} (${node.name})`);
153 + } else {
154 + await ctx.db.insert(cancerAliases).values({ cancerId, alias: node.name, normalized: normalizeLabel(node.name), aliasType: 'synonym', sourceId: ctx.sourceId, sourceTerminology: 'OncoTree' }).onConflictDoNothing();
155 + }
156 + canonicalByOncotree.set(node.code, cancerId);
157 + await ctx.db.insert(cancerCodes).values({ cancerId, system: 'oncotree', code: node.code, matchType, sourceId: ctx.sourceId, validFrom: stable?.release_date }).onConflictDoNothing();
158 + for (const u of node.externalReferences['UMLS'] ?? []) await ctx.db.insert(cancerCodes).values({ cancerId, system: 'umls', code: u, matchType: 'EXACT_IDENTIFIER', sourceId: ctx.sourceId }).onConflictDoNothing();
159 + for (const c of ncitCodes) await ctx.db.insert(cancerCodes).values({ cancerId, system: 'ncit', code: c, matchType: 'EXACT_IDENTIFIER', sourceId: ctx.sourceId }).onConflictDoNothing();
160 + // Record the OncoTree code on the entity only when it has none yet (first mapping wins; others stay in cancer_codes).
161 + await ctx.db.execute(sql`UPDATE cancers SET primary_oncotree_code = ${node.code} WHERE id = ${cancerId} AND primary_oncotree_code IS NULL`);
162 + // Level-2 nodes hang directly under a tissue → anatomical relation.
163 + if (node.parent && siteByOncotree.has(node.parent)) {
164 + await ctx.db.insert(cancerAnatomy).values({ cancerId, siteId: siteByOncotree.get(node.parent)!, relation: 'primary', sourceId: ctx.sourceId }).onConflictDoNothing();
165 + }
166 + void rec;
167 + }
168 +
169 + // Pass 2: hierarchy edges (oncotree dimension), levels ≥ 2 only.
170 + let edges = 0;
171 + for (const node of nodes) {
172 + if (!node.parent || node.parent === 'TISSUE' || siteByOncotree.has(node.parent)) continue;
173 + const childId = canonicalByOncotree.get(node.code);
174 + const parentId = canonicalByOncotree.get(node.parent);
175 + if (!childId || !parentId || childId === parentId) continue;
176 + if (!byCode.has(node.parent)) continue;
177 + await ctx.db.insert(cancerHierarchy).values({ parentId, childId, hierarchyType: 'oncotree', sourceId: ctx.sourceId }).onConflictDoNothing();
178 + edges++;
179 + }
180 + ctx.info(`tissues ${siteByOncotree.size}; canonical mapped ${canonicalByOncotree.size}; hierarchy edges ${edges}`);
181 + ctx.cursor = { version, syncedAt: new Date().toISOString() };
182 + }
183 +}
184 +
185 +export const connector = new OncotreeConnector();
added packages/connectors/src/connectors/oncotree/manifest.ts +34 −0
@@ -0,0 +1,34 @@
1 +import { defineManifest } from '../../sdk/manifest.js';
2 +
3 +/**
4 + * Docs verified 2026-09-08: https://oncotree.mskcc.org (API tab) — GET /api/tumorTypes?version=…,
5 + * GET /api/versions, GET /api/mainTypes. Response fields: code, name, mainType, tissue, parent,
6 + * level, externalReferences{UMLS[],NCI[]}, history[], revocations[], precursors[], color.
7 + * License (repo README, cBioPortal/oncotree): CC BY 4.0.
8 + */
9 +export const manifest = defineManifest({
10 + id: 'oncotree',
11 + name: 'OncoTree',
12 + organization: 'Memorial Sloan Kettering Cancer Center / cBioPortal',
13 + category: 'terminology',
14 + tier: 0,
15 + description: 'Open cancer classification used by cBioPortal and clinical sequencing programs. Provides a tissue-based tumor type hierarchy with NCIt and UMLS cross-references. Used as a second hierarchy dimension and to anchor cohort/genomic sources.',
16 + homepage: 'https://oncotree.mskcc.org',
17 + docsUrl: 'https://oncotree.mskcc.org/#/home?tab=api',
18 + termsUrl: 'https://github.com/cBioPortal/oncotree/blob/master/LICENSE',
19 + access: { type: 'api', auth: 'none', baseUrl: 'https://oncotree.mskcc.org/api' },
20 + license: 'CC BY 4.0',
21 + licenseStatus: 'approved',
22 + commercialUse: 'allowed',
23 + redistribution: 'attribution',
24 + attribution: 'OncoTree (Memorial Sloan Kettering Cancer Center), CC BY 4.0, https://oncotree.mskcc.org',
25 + termsReviewedAt: '2026-09-08',
26 + updateFrequency: 'Stable releases a few times per year',
27 + supportsIncrementalSync: false,
28 + entities: ['cancer', 'cancer_hierarchy', 'cancer_codes'],
29 + rateLimits: { requestsPerSecond: 2, maxConcurrency: 1 },
30 + rawRetention: 'full',
31 + documentationVerifiedAt: '2026-09-08',
32 + status: 'active',
33 + schedule: '0 4 * * 1',
34 +});
added packages/connectors/src/index.ts +4 −0
@@ -0,0 +1,4 @@
1 +export * from './sdk/index.js';
2 +export * from './registry.js';
3 +export * from './sources-sync.js';
4 +export * from './planned.js';
added packages/connectors/src/planned.ts +60 −0
@@ -0,0 +1,60 @@
1 +import { defineManifest, type ConnectorManifest } from './sdk/manifest.js';
2 +
3 +/**
4 + * Sources that are registered (visible on /sources with their compliance status) but have no
5 + * ingesting connector yet — either because license review is pending (CLAUDE.md §10.4, §142) or
6 + * because credentials are required. Nothing here ingests data.
7 + */
8 +export const PLANNED_MANIFESTS: ConnectorManifest[] = [
9 + defineManifest({
10 + id: 'iarc-globocan',
11 + name: 'IARC Global Cancer Observatory (GLOBOCAN)',
12 + organization: 'International Agency for Research on Cancer (WHO)',
13 + category: 'epidemiology',
14 + tier: 0,
15 + description: 'Global estimates of cancer incidence, mortality and prevalence by country, sex, age and cancer site. The intended global epidemiology layer of CancerIndex.',
16 + homepage: 'https://gco.iarc.who.int/',
17 + docsUrl: 'https://gco.iarc.who.int/today/en/about',
18 + termsUrl: 'https://gco.iarc.who.int/en/terms-of-use',
19 + access: { type: 'api', auth: 'none', baseUrl: 'https://gco.iarc.who.int/' },
20 + license: 'IARC/WHO terms of use — redistribution rights under review',
21 + licenseStatus: 'review',
22 + commercialUse: 'unknown',
23 + redistribution: 'unknown',
24 + attribution: 'Ferlay J, Ervik M, Lam F, et al. Global Cancer Observatory: Cancer Today. Lyon, France: IARC.',
25 + termsNotes:
26 + 'Terms page is a JavaScript application; automated review on 2026-09-08 could not extract the text. Per CLAUDE.md §10.4 no automated bulk ingestion or republication of GLOBOCAN estimates until a human has reviewed the terms and, if required, obtained permission from IARC. Country/global rankings therefore remain unavailable and are shown as "awaiting license review".',
27 + updateFrequency: 'Major releases every ~2 years (GLOBOCAN 2022 current)',
28 + supportsIncrementalSync: false,
29 + entities: ['epidemiology_observations'],
30 + metrics: ['incidence_count', 'mortality_count', 'as_incidence_rate', 'as_mortality_rate', 'prevalence_5y'],
31 + rateLimits: { requestsPerSecond: 1, maxConcurrency: 1 },
32 + documentationVerifiedAt: '2026-09-08',
33 + status: 'review',
34 + }),
35 + defineManifest({
36 + id: 'seer',
37 + name: 'SEER (Surveillance, Epidemiology, and End Results)',
38 + organization: 'National Cancer Institute',
39 + category: 'epidemiology',
40 + tier: 0,
41 + description: 'US population-based registry statistics: incidence, mortality, survival, stage, age, sex, race/ethnicity. Authoritative for US relative survival.',
42 + homepage: 'https://seer.cancer.gov/',
43 + docsUrl: 'https://api.seer.cancer.gov/',
44 + termsUrl: 'https://seer.cancer.gov/about/terms.html',
45 + access: { type: 'api', auth: 'api_key', baseUrl: 'https://api.seer.cancer.gov/rest/' },
46 + license: 'US Government work (public domain); SEER*Explorer statistics citable with attribution; API requires a free key',
47 + licenseStatus: 'review',
48 + commercialUse: 'unknown',
49 + redistribution: 'attribution',
50 + attribution: 'Surveillance, Epidemiology, and End Results (SEER) Program (www.seer.cancer.gov), National Cancer Institute.',
51 + termsNotes: 'Connector is implemented against the SEER API only once a key is present (SEER_API_KEY). Status awaiting_credentials until then. Never present SEER (US) estimates as global.',
52 + updateFrequency: 'Annual (April)',
53 + supportsIncrementalSync: false,
54 + entities: ['epidemiology_observations', 'survival_observations'],
55 + metrics: ['incidence_count', 'as_incidence_rate', 'mortality_count', 'as_mortality_rate', 'five_year_survival'],
56 + rateLimits: { requestsPerSecond: 2, maxConcurrency: 1 },
57 + documentationVerifiedAt: '2026-09-08',
58 + status: 'awaiting_credentials',
59 + }),
60 +];
added packages/connectors/src/registry.ts +13 −0
@@ -0,0 +1,13 @@
1 +import type { Connector } from './sdk/run.js';
2 +import { connector as oncotree } from './connectors/oncotree/index.js';
3 +
4 +/**
5 + * Connector registry. Order = recommended first-run order (CLAUDE.md §351): terminology first,
6 + * then genes, then genomics/trials/literature/variants/evidence, then epidemiology.
7 + * Add new connectors here; `pnpm cix sources:sync` seeds their manifests into `sources`.
8 + */
9 +export const CONNECTORS: Connector[] = [oncotree];
10 +
11 +export function getConnector(id: string): Connector | undefined {
12 + return CONNECTORS.find((c) => c.manifest.id === id);
13 +}
added packages/connectors/src/sdk/http.ts +162 −0
@@ -0,0 +1,162 @@
1 +import { setTimeout as sleep } from 'node:timers/promises';
2 +import type { ConnectorManifest } from './manifest.js';
3 +
4 +export interface HttpStats {
5 + requests: number;
6 + failures: number;
7 + rateLimitEvents: number;
8 + retries: number;
9 +}
10 +
11 +export class HttpError extends Error {
12 + constructor(
13 + public readonly status: number,
14 + public readonly url: string,
15 + public readonly bodySnippet: string,
16 + ) {
17 + super(`HTTP ${status} for ${url}: ${bodySnippet.slice(0, 200)}`);
18 + }
19 +}
20 +
21 +/** Token bucket rate limiter (CLAUDE.md §229). */
22 +class TokenBucket {
23 + private tokens: number;
24 + private last = Date.now();
25 + constructor(
26 + private readonly ratePerSec: number,
27 + private readonly burst: number,
28 + ) {
29 + this.tokens = burst;
30 + }
31 + async take(): Promise<void> {
32 + for (;;) {
33 + const now = Date.now();
34 + this.tokens = Math.min(this.burst, this.tokens + ((now - this.last) / 1000) * this.ratePerSec);
35 + this.last = now;
36 + if (this.tokens >= 1) {
37 + this.tokens -= 1;
38 + return;
39 + }
40 + await sleep(Math.ceil(((1 - this.tokens) / this.ratePerSec) * 1000));
41 + }
42 + }
43 +}
44 +
45 +class Semaphore {
46 + private queue: Array<() => void> = [];
47 + private active = 0;
48 + constructor(private readonly max: number) {}
49 + async acquire(): Promise<() => void> {
50 + if (this.active >= this.max) await new Promise<void>((resolve) => this.queue.push(resolve));
51 + this.active++;
52 + let released = false;
53 + return () => {
54 + if (released) return;
55 + released = true;
56 + this.active--;
57 + const next = this.queue.shift();
58 + if (next) next();
59 + };
60 + }
61 +}
62 +
63 +export interface HttpClientOptions {
64 + userAgent?: string;
65 + timeoutMs?: number;
66 + headers?: Record<string, string>;
67 +}
68 +
69 +/**
70 + * Source-aware HTTP client: token bucket, bounded concurrency, exponential backoff with jitter,
71 + * Retry-After respect, request accounting for the ingest run.
72 + */
73 +export class HttpClient {
74 + readonly stats: HttpStats = { requests: 0, failures: 0, rateLimitEvents: 0, retries: 0 };
75 + private readonly bucket: TokenBucket | null;
76 + private readonly sem: Semaphore;
77 + private readonly retry: NonNullable<ConnectorManifest['retryPolicy']>;
78 + private readonly ua: string;
79 + private readonly timeoutMs: number;
80 + private readonly headers: Record<string, string>;
81 +
82 + constructor(manifest: ConnectorManifest, opts: HttpClientOptions = {}) {
83 + const rps = manifest.rateLimits.requestsPerSecond ?? (manifest.rateLimits.requestsPerMinute ? manifest.rateLimits.requestsPerMinute / 60 : undefined);
84 + this.bucket = rps ? new TokenBucket(rps, Math.max(1, Math.ceil(rps))) : null;
85 + this.sem = new Semaphore(manifest.rateLimits.maxConcurrency);
86 + this.retry = manifest.retryPolicy;
87 + // Plain product token only: some WAFs (e.g. oncotree.mskcc.org, Akamai) reject UAs containing URLs/parentheses.
88 + this.ua = opts.userAgent ?? `CancerIndex/0.1`;
89 + this.timeoutMs = opts.timeoutMs ?? 60_000;
90 + this.headers = opts.headers ?? {};
91 + }
92 +
93 + /** One attempt: returns the Response (any status) or throws on network error. */
94 + private async attempt(url: string, init: RequestInit): Promise<Response> {
95 + const release = await this.sem.acquire();
96 + try {
97 + if (this.bucket) await this.bucket.take();
98 + this.stats.requests++;
99 + const controller = new AbortController();
100 + const timer = setTimeout(() => controller.abort(), this.timeoutMs);
101 + try {
102 + return await fetch(url, {
103 + ...init,
104 + headers: { 'user-agent': this.ua, accept: 'application/json, text/plain, */*', ...this.headers, ...(init.headers as Record<string, string> | undefined) },
105 + signal: controller.signal,
106 + });
107 + } finally {
108 + clearTimeout(timer);
109 + }
110 + } finally {
111 + release();
112 + }
113 + }
114 +
115 + async request(url: string, init: RequestInit = {}): Promise<Response> {
116 + for (let attempt = 0; ; attempt++) {
117 + let res: Response;
118 + try {
119 + res = await this.attempt(url, init);
120 + } catch (err) {
121 + if (attempt >= this.retry.maxRetries) {
122 + this.stats.failures++;
123 + throw err;
124 + }
125 + this.stats.retries++;
126 + await sleep(this.backoff(attempt));
127 + continue;
128 + }
129 + if (res.ok) return res;
130 + const body = await res.text().catch(() => '');
131 + if (res.status === 429) this.stats.rateLimitEvents++;
132 + const retryable = res.status === 429 || res.status >= 500 || res.status === 408;
133 + if (!retryable || attempt >= this.retry.maxRetries) {
134 + this.stats.failures++;
135 + throw new HttpError(res.status, url, body);
136 + }
137 + const retryAfter = Number(res.headers.get('retry-after'));
138 + const delay = Number.isFinite(retryAfter) && retryAfter > 0 ? Math.min(retryAfter * 1000, 120_000) : this.backoff(attempt);
139 + this.stats.retries++;
140 + await sleep(delay);
141 + }
142 + }
143 +
144 + async json<T = unknown>(url: string, init?: RequestInit): Promise<T> {
145 + const res = await this.request(url, init);
146 + return (await res.json()) as T;
147 + }
148 +
149 + async text(url: string, init?: RequestInit): Promise<string> {
150 + const res = await this.request(url, init);
151 + return await res.text();
152 + }
153 +
154 + async postJson<T = unknown>(url: string, body: unknown, init: RequestInit = {}): Promise<T> {
155 + return this.json<T>(url, { ...init, method: 'POST', headers: { 'content-type': 'application/json', ...(init.headers as Record<string, string> | undefined) }, body: JSON.stringify(body) });
156 + }
157 +
158 + private backoff(attempt: number): number {
159 + const base = Math.min(this.retry.maxDelayMs, this.retry.baseDelayMs * 2 ** attempt);
160 + return Math.round(base / 2 + Math.random() * (base / 2));
161 + }
162 +}
added packages/connectors/src/sdk/index.ts +5 −0
@@ -0,0 +1,5 @@
1 +export * from './manifest.js';
2 +export * from './http.js';
3 +export * from './lake.js';
4 +export * from './run.js';
5 +export * from './validate.js';
added packages/connectors/src/sdk/lake.ts +105 −0
@@ -0,0 +1,105 @@
1 +import { createWriteStream, existsSync, mkdirSync, createReadStream } from 'node:fs';
2 +import { mkdir, stat } from 'node:fs/promises';
3 +import path from 'node:path';
4 +import { createGzip, createGunzip } from 'node:zlib';
5 +import { pipeline } from 'node:stream/promises';
6 +import { createInterface } from 'node:readline';
7 +import { dataDir } from '@cancerindex/shared';
8 +
9 +/**
10 + * Raw data lake (CLAUDE.md §26): every source payload retained when licensing allows.
11 + * Layout: {CI_DATA_DIR}/raw/{source}/{YYYY-MM-DD}/{entity}/{runId}-{part}.jsonl.gz
12 + * A record's `rawPath` is "<file>#<line>" so any canonical value can be traced to its raw payload.
13 + */
14 +export class RawLake {
15 + private readonly root: string;
16 + private writers = new Map<string, { path: string; stream: ReturnType<typeof createWriteStream>; gz: ReturnType<typeof createGzip>; lines: number; part: number; bytes: number }>();
17 + constructor(
18 + private readonly source: string,
19 + private readonly runId: string,
20 + private readonly date = new Date(),
21 + root?: string,
22 + ) {
23 + this.root = root ?? path.join(dataDir(), 'raw');
24 + }
25 +
26 + dir(entity: string): string {
27 + const day = this.date.toISOString().slice(0, 10);
28 + return path.join(this.root, this.source, day, entity);
29 + }
30 +
31 + /** Append one raw JSON payload; returns its rawPath reference. */
32 + async put(entity: string, payload: unknown): Promise<string> {
33 + let w = this.writers.get(entity);
34 + if (!w || w.bytes > 64 * 1024 * 1024) {
35 + if (w) await this.closeWriter(entity);
36 + const part = (w?.part ?? 0) + 1;
37 + const dir = this.dir(entity);
38 + if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
39 + const file = path.join(dir, `${this.runId}-${String(part).padStart(3, '0')}.jsonl.gz`);
40 + const stream = createWriteStream(file);
41 + const gz = createGzip({ level: 6 });
42 + gz.pipe(stream);
43 + w = { path: file, stream, gz, lines: 0, part, bytes: 0 };
44 + this.writers.set(entity, w);
45 + }
46 + const line = JSON.stringify(payload) + '\n';
47 + w.bytes += line.length;
48 + const ref = `${w.path}#${w.lines}`;
49 + w.lines++;
50 + if (!w.gz.write(line)) await new Promise<void>((r) => w!.gz.once('drain', () => r()));
51 + return ref;
52 + }
53 +
54 + private async closeWriter(entity: string): Promise<void> {
55 + const w = this.writers.get(entity);
56 + if (!w) return;
57 + await new Promise<void>((resolve, reject) => {
58 + w.stream.once('finish', () => resolve());
59 + w.stream.once('error', reject);
60 + w.gz.end();
61 + });
62 + this.writers.delete(entity);
63 + }
64 +
65 + async close(): Promise<void> {
66 + for (const entity of [...this.writers.keys()]) await this.closeWriter(entity);
67 + }
68 +
69 + /** Read a raw payload back by reference (admin TRACE, reprocessing). */
70 + static async read(ref: string): Promise<unknown | null> {
71 + const [file, lineStr] = ref.split('#');
72 + if (!file || lineStr === undefined) return null;
73 + const target = Number(lineStr);
74 + const rl = createInterface({ input: createReadStream(file).pipe(createGunzip()) });
75 + let i = 0;
76 + for await (const line of rl) {
77 + if (i === target) {
78 + rl.close();
79 + return JSON.parse(line);
80 + }
81 + i++;
82 + }
83 + return null;
84 + }
85 +}
86 +
87 +/** Download a bulk file to the lake (CLAUDE.md §228-230) with checksum + size recorded. */
88 +export async function downloadBulk(url: string, dest: string, fetchImpl: typeof fetch = fetch, headers: Record<string, string> = {}): Promise<{ path: string; bytes: number; sha256: string }> {
89 + await mkdir(path.dirname(dest), { recursive: true });
90 + const res = await fetchImpl(url, { headers: { 'user-agent': 'CancerIndex/0.1 (+https://www.cancerindex.io)', ...headers } });
91 + if (!res.ok || !res.body) throw new Error(`bulk download failed ${res.status} ${url}`);
92 + const { createHash } = await import('node:crypto');
93 + const hash = createHash('sha256');
94 + const { Transform } = await import('node:stream');
95 + const tap = new Transform({
96 + transform(chunk, _enc, cb) {
97 + hash.update(chunk);
98 + cb(null, chunk);
99 + },
100 + });
101 + const { Readable } = await import('node:stream');
102 + await pipeline(Readable.fromWeb(res.body as never), tap, createWriteStream(dest));
103 + const s = await stat(dest);
104 + return { path: dest, bytes: s.size, sha256: hash.digest('hex') };
105 +}
added packages/connectors/src/sdk/manifest.ts +61 −0
@@ -0,0 +1,61 @@
1 +import { z } from 'zod';
2 +
3 +/** Connector manifest (CLAUDE.md §9). Seeds the `sources` table; drives the public /source pages. */
4 +export const ConnectorManifest = z.object({
5 + id: z.string().regex(/^[a-z0-9-]+$/),
6 + name: z.string(),
7 + organization: z.string(),
8 + category: z.enum(['terminology', 'genes', 'genomics', 'variants', 'epidemiology', 'trials', 'literature', 'drugs', 'regulatory', 'evidence', 'pathways', 'funding', 'guidelines']),
9 + tier: z.number().int().min(0).max(11),
10 + description: z.string(),
11 + homepage: z.string().url(),
12 + docsUrl: z.string().url().optional(),
13 + termsUrl: z.string().url().optional(),
14 + access: z.object({
15 + type: z.enum(['api', 'bulk', 'rss', 'ftp', 'graphql', 'rest', 'scrape', 'manual']),
16 + auth: z.enum(['none', 'api_key', 'oauth', 'account', 'controlled']),
17 + baseUrl: z.string().url().optional(),
18 + }),
19 + license: z.string(),
20 + licenseStatus: z.enum(['review', 'approved', 'restricted', 'blocked']),
21 + commercialUse: z.enum(['allowed', 'restricted', 'prohibited', 'unknown']),
22 + redistribution: z.enum(['allowed', 'attribution', 'restricted', 'prohibited', 'unknown']),
23 + attribution: z.string().optional(),
24 + termsReviewedAt: z.string().optional(), // YYYY-MM-DD, human review date
25 + termsNotes: z.string().optional(),
26 + updateFrequency: z.string(),
27 + expectedLatency: z.string().optional(),
28 + supportsIncrementalSync: z.boolean(),
29 + entities: z.array(z.string()),
30 + metrics: z.array(z.string()).default([]),
31 + rateLimits: z.object({
32 + requestsPerSecond: z.number().positive().optional(),
33 + requestsPerMinute: z.number().positive().optional(),
34 + maxConcurrency: z.number().int().positive().default(1),
35 + notes: z.string().optional(),
36 + }),
37 + retryPolicy: z
38 + .object({
39 + maxRetries: z.number().int().min(0).default(4),
40 + baseDelayMs: z.number().int().min(0).default(500),
41 + maxDelayMs: z.number().int().min(0).default(30_000),
42 + })
43 + .default({ maxRetries: 4, baseDelayMs: 500, maxDelayMs: 30_000 }),
44 + rawRetention: z.enum(['full', 'summary', 'none']).default('full'),
45 + schemaVersion: z.string().default('1'),
46 + documentationVerifiedAt: z.string(), // YYYY-MM-DD, the date official docs were checked (CLAUDE.md §224-225)
47 + owner: z.string().default('cancerindex'),
48 + status: z.enum(['planned', 'active', 'paused', 'awaiting_credentials', 'review', 'retired']),
49 + schedule: z.string().optional(), // cron expression for the worker scheduler
50 + anomalyGuard: z
51 + .object({
52 + /** Refuse to mark records source_missing when the fetched count drops below this ratio of the previous run. */
53 + minRatioOfPrevious: z.number().min(0).max(1).default(0.5),
54 + })
55 + .default({ minRatioOfPrevious: 0.5 }),
56 +});
57 +export type ConnectorManifest = z.infer<typeof ConnectorManifest>;
58 +
59 +export function defineManifest(m: z.input<typeof ConnectorManifest>): ConnectorManifest {
60 + return ConnectorManifest.parse(m);
61 +}
added packages/connectors/src/sdk/run.ts +0 −0

Binary file not shown.

added packages/connectors/src/sdk/validate.ts +49 −0
@@ -0,0 +1,49 @@
1 +/** Scientific invariants applied to normalized records (CLAUDE.md §110, §154). */
2 +
3 +export interface ValidationIssue {
4 + field: string;
5 + message: string;
6 +}
7 +
8 +export function validateEpidemiology(o: { metric: string; value: number; year: number; lowerCi?: number | null; upperCi?: number | null; unit: string }): ValidationIssue[] {
9 + const issues: ValidationIssue[] = [];
10 + if (!Number.isFinite(o.value)) issues.push({ field: 'value', message: 'not finite' });
11 + if (o.value < 0) issues.push({ field: 'value', message: 'negative count/rate' });
12 + if (o.year < 1900 || o.year > new Date().getFullYear() + 30) issues.push({ field: 'year', message: `implausible year ${o.year}` });
13 + if (o.lowerCi != null && o.lowerCi > o.value) issues.push({ field: 'lowerCi', message: 'lowerCI > estimate' });
14 + if (o.upperCi != null && o.upperCi < o.value) issues.push({ field: 'upperCi', message: 'upperCI < estimate' });
15 + if (o.metric.endsWith('_count') && o.unit !== 'count') issues.push({ field: 'unit', message: 'count metric must use unit=count' });
16 + if (o.metric.endsWith('_rate') && o.unit !== 'per_100k') issues.push({ field: 'unit', message: 'rate metric must use unit=per_100k' });
17 + return issues;
18 +}
19 +
20 +export function validateSurvival(o: { probability?: number | null; durationMonths: number; lowerCi?: number | null; upperCi?: number | null; cohortSize?: number | null }): ValidationIssue[] {
21 + const issues: ValidationIssue[] = [];
22 + if (o.probability != null && (o.probability < 0 || o.probability > 1)) issues.push({ field: 'probability', message: 'survival must be within [0,1]' });
23 + if (o.durationMonths <= 0) issues.push({ field: 'durationMonths', message: 'duration must be positive' });
24 + if (o.probability != null && o.lowerCi != null && o.lowerCi > o.probability) issues.push({ field: 'lowerCi', message: 'lowerCI > estimate' });
25 + if (o.probability != null && o.upperCi != null && o.upperCi < o.probability) issues.push({ field: 'upperCi', message: 'upperCI < estimate' });
26 + if (o.cohortSize != null && o.cohortSize < 0) issues.push({ field: 'cohortSize', message: 'negative cohort' });
27 + return issues;
28 +}
29 +
30 +export const TRIAL_PHASES = new Set(['EARLY_PHASE1', 'PHASE1', 'PHASE2', 'PHASE3', 'PHASE4', 'NA']);
31 +export const TRIAL_STATUSES = new Set(['ACTIVE_NOT_RECRUITING', 'COMPLETED', 'ENROLLING_BY_INVITATION', 'NOT_YET_RECRUITING', 'RECRUITING', 'SUSPENDED', 'TERMINATED', 'WITHDRAWN', 'AVAILABLE', 'NO_LONGER_AVAILABLE', 'TEMPORARILY_NOT_AVAILABLE', 'APPROVED_FOR_MARKETING', 'WITHHELD', 'UNKNOWN']);
32 +
33 +export function validateTrial(o: { nctId: string; phases: string[]; overallStatus?: string | null }): ValidationIssue[] {
34 + const issues: ValidationIssue[] = [];
35 + if (!/^NCT\d{8}$/.test(o.nctId)) issues.push({ field: 'nctId', message: 'malformed NCT id' });
36 + for (const p of o.phases) if (!TRIAL_PHASES.has(p)) issues.push({ field: 'phases', message: `unknown phase enum ${p}` });
37 + if (o.overallStatus && !TRIAL_STATUSES.has(o.overallStatus)) issues.push({ field: 'overallStatus', message: `unknown status enum ${o.overallStatus}` });
38 + return issues;
39 +}
40 +
41 +export function validateFrequency(o: { casesAffected: number; casesProfiled: number }): ValidationIssue[] {
42 + const issues: ValidationIssue[] = [];
43 + if (o.casesProfiled <= 0) issues.push({ field: 'casesProfiled', message: 'denominator must be positive' });
44 + if (o.casesAffected < 0 || o.casesAffected > o.casesProfiled) issues.push({ field: 'casesAffected', message: 'numerator out of range' });
45 + return issues;
46 +}
47 +
48 +/** Gene symbol sanity (HGNC style). */
49 +export const GENE_SYMBOL_RE = /^[A-Z][A-Z0-9-]{0,15}(?:@|orf\d+)?$/i;
added packages/connectors/src/sources-sync.ts +51 −0
@@ -0,0 +1,51 @@
1 +import { eq } from 'drizzle-orm';
2 +import { type Database, sources, mintId } from '@cancerindex/database';
3 +import { CONNECTORS } from './registry.js';
4 +import type { ConnectorManifest } from './sdk/manifest.js';
5 +
6 +/** Sync connector manifests into the `sources` registry (CLAUDE.md §95, §142, §307). Idempotent. */
7 +export async function syncSources(db: Database, extra: ConnectorManifest[] = []): Promise<{ created: number; updated: number }> {
8 + let created = 0;
9 + let updated = 0;
10 + const manifests = [...CONNECTORS.map((c) => c.manifest), ...extra];
11 + for (const m of manifests) {
12 + const values = {
13 + slug: m.id,
14 + name: m.name,
15 + organization: m.organization,
16 + category: m.category,
17 + description: m.description,
18 + homepage: m.homepage,
19 + docsUrl: m.docsUrl ?? null,
20 + termsUrl: m.termsUrl ?? null,
21 + accessType: m.access.type,
22 + accessAuth: m.access.auth,
23 + license: m.license,
24 + licenseStatus: m.licenseStatus,
25 + commercialUse: m.commercialUse,
26 + redistribution: m.redistribution,
27 + attribution: m.attribution ?? null,
28 + licenseReviewedAt: m.termsReviewedAt ? new Date(m.termsReviewedAt) : null,
29 + approvedForProduction: m.licenseStatus === 'approved',
30 + updateFrequency: m.updateFrequency,
31 + supportsIncremental: m.supportsIncrementalSync,
32 + entities: m.entities,
33 + metrics: m.metrics,
34 + rateLimit: m.rateLimits.requestsPerSecond ? `${m.rateLimits.requestsPerSecond} req/s` : m.rateLimits.requestsPerMinute ? `${m.rateLimits.requestsPerMinute} req/min` : null,
35 + tier: m.tier,
36 + manifest: m as unknown as Record<string, unknown>,
37 + updatedAt: new Date(),
38 + };
39 + const [existing] = await db.select({ id: sources.id, status: sources.status }).from(sources).where(eq(sources.slug, m.id)).limit(1);
40 + if (existing) {
41 + // Keep runtime status (active/degraded) unless manifest declares a gating status.
42 + const gating = ['awaiting_credentials', 'review', 'retired', 'paused'].includes(m.status);
43 + await db.update(sources).set({ ...values, ...(gating ? { status: m.status } : {}) }).where(eq(sources.id, existing.id));
44 + updated++;
45 + } else {
46 + await db.insert(sources).values({ id: await mintId(db, 'SOURCE'), ...values, status: m.status });
47 + created++;
48 + }
49 + }
50 + return { created, updated };
51 +}
added packages/connectors/tsconfig.json +8 −0
@@ -0,0 +1,8 @@
1 +{
2 + "extends": "../../tsconfig.base.json",
3 + "compilerOptions": {
4 + "rootDir": ".",
5 + "noEmit": true
6 + },
7 + "include": ["src", "test"]
8 +}
added packages/connectors/vitest.config.ts +8 −0
@@ -0,0 +1,8 @@
1 +import { defineConfig } from 'vitest/config';
2 +
3 +export default defineConfig({
4 + test: {
5 + include: ['src/**/*.test.ts', 'test/**/*.test.ts'],
6 + testTimeout: 20_000,
7 + },
8 +});
added packages/database/drizzle.config.ts +11 −0
@@ -0,0 +1,11 @@
1 +import { defineConfig } from 'drizzle-kit';
2 +
3 +export default defineConfig({
4 + dialect: 'postgresql',
5 + schema: './src/schema/index.ts',
6 + out: './migrations',
7 + dbCredentials: { url: process.env.DATABASE_URL ?? 'postgres://localhost:5432/cancerindex' },
8 + casing: 'snake_case',
9 + strict: true,
10 + verbose: true,
11 +});
added packages/database/migrations/0000_small_blazing_skull.sql +936 −0
@@ -0,0 +1,936 @@
1 +CREATE TABLE "audit_log" (
2 + "id" bigserial PRIMARY KEY NOT NULL,
3 + "actor" text NOT NULL,
4 + "action" text NOT NULL,
5 + "entity_type" text,
6 + "entity_id" text,
7 + "before" jsonb,
8 + "after" jsonb,
9 + "reason" text,
10 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
11 +);
12 +--> statement-breakpoint
13 +CREATE TABLE "change_events" (
14 + "id" bigserial PRIMARY KEY NOT NULL,
15 + "entity_type" text NOT NULL,
16 + "entity_id" text NOT NULL,
17 + "kind" text NOT NULL,
18 + "summary" text NOT NULL,
19 + "before" jsonb,
20 + "after" jsonb,
21 + "ingest_run_id" text,
22 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
23 +);
24 +--> statement-breakpoint
25 +CREATE TABLE "connector_cursors" (
26 + "connector_id" text PRIMARY KEY NOT NULL,
27 + "cursor" jsonb DEFAULT '{}'::jsonb NOT NULL,
28 + "last_success_at" timestamp with time zone,
29 + "last_attempt_at" timestamp with time zone,
30 + "paused" boolean DEFAULT false NOT NULL,
31 + "health" text DEFAULT 'unknown' NOT NULL,
32 + "health_detail" text,
33 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
34 +);
35 +--> statement-breakpoint
36 +CREATE TABLE "connector_field_stats" (
37 + "id" bigserial PRIMARY KEY NOT NULL,
38 + "connector_id" text NOT NULL,
39 + "entity" text NOT NULL,
40 + "field" text NOT NULL,
41 + "types" text[] DEFAULT '{}' NOT NULL,
42 + "seen_count" integer DEFAULT 0 NOT NULL,
43 + "null_count" integer DEFAULT 0 NOT NULL,
44 + "first_seen_run" text,
45 + "last_seen_run" text,
46 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
47 +);
48 +--> statement-breakpoint
49 +CREATE TABLE "entity_merges" (
50 + "id" bigserial PRIMARY KEY NOT NULL,
51 + "entity_type" text NOT NULL,
52 + "keep_id" text NOT NULL,
53 + "merge_id" text NOT NULL,
54 + "evidence" jsonb DEFAULT '{}'::jsonb NOT NULL,
55 + "status" text DEFAULT 'proposed' NOT NULL,
56 + "decided_by" text,
57 + "decided_at" timestamp with time zone,
58 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
59 +);
60 +--> statement-breakpoint
61 +CREATE TABLE "id_sequences" (
62 + "namespace" varchar(16) PRIMARY KEY NOT NULL,
63 + "next" bigint DEFAULT 1 NOT NULL
64 +);
65 +--> statement-breakpoint
66 +CREATE TABLE "ingest_runs" (
67 + "id" text PRIMARY KEY NOT NULL,
68 + "connector_id" text NOT NULL,
69 + "source_id" varchar(32) NOT NULL,
70 + "mode" text DEFAULT 'incremental' NOT NULL,
71 + "status" text DEFAULT 'running' NOT NULL,
72 + "started_at" timestamp with time zone DEFAULT now() NOT NULL,
73 + "finished_at" timestamp with time zone,
74 + "duration_ms" integer,
75 + "records_fetched" integer DEFAULT 0 NOT NULL,
76 + "records_created" integer DEFAULT 0 NOT NULL,
77 + "records_updated" integer DEFAULT 0 NOT NULL,
78 + "records_unchanged" integer DEFAULT 0 NOT NULL,
79 + "records_rejected" integer DEFAULT 0 NOT NULL,
80 + "http_requests" integer DEFAULT 0 NOT NULL,
81 + "http_failures" integer DEFAULT 0 NOT NULL,
82 + "rate_limit_events" integer DEFAULT 0 NOT NULL,
83 + "validation_failures" integer DEFAULT 0 NOT NULL,
84 + "schema_drift" jsonb DEFAULT '[]'::jsonb NOT NULL,
85 + "cursor_before" jsonb,
86 + "cursor_after" jsonb,
87 + "error" text,
88 + "log" jsonb DEFAULT '[]'::jsonb NOT NULL,
89 + "dataset_version" text,
90 + "anomaly" text
91 +);
92 +--> statement-breakpoint
93 +CREATE TABLE "provenance" (
94 + "id" bigserial PRIMARY KEY NOT NULL,
95 + "public_id" varchar(32),
96 + "source_id" varchar(32) NOT NULL,
97 + "source_record_id" text,
98 + "source_url" text,
99 + "dataset" text,
100 + "dataset_version" text,
101 + "publication_id" varchar(32),
102 + "pmid" text,
103 + "doi" text,
104 + "retrieved_at" timestamp with time zone NOT NULL,
105 + "published_at" text,
106 + "updated_at_source" text,
107 + "geography" text,
108 + "population" text,
109 + "cohort_size" integer,
110 + "methodology" text,
111 + "evidence_type" text NOT NULL,
112 + "access_level" text DEFAULT 'open' NOT NULL,
113 + "confidence" real,
114 + "license" text,
115 + "ingest_run_id" text,
116 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
117 +);
118 +--> statement-breakpoint
119 +CREATE TABLE "source_records" (
120 + "id" bigserial PRIMARY KEY NOT NULL,
121 + "source_id" varchar(32) NOT NULL,
122 + "entity_kind" text NOT NULL,
123 + "source_record_id" text NOT NULL,
124 + "payload_hash" text NOT NULL,
125 + "raw_path" text,
126 + "status" text DEFAULT 'active' NOT NULL,
127 + "first_seen_run" text,
128 + "last_seen_run" text,
129 + "retrieved_at" timestamp with time zone DEFAULT now() NOT NULL,
130 + "source_updated_at" timestamp with time zone,
131 + "canonical_type" text,
132 + "canonical_id" text,
133 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
134 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
135 +);
136 +--> statement-breakpoint
137 +CREATE TABLE "sources" (
138 + "id" varchar(32) PRIMARY KEY NOT NULL,
139 + "slug" text NOT NULL,
140 + "name" text NOT NULL,
141 + "organization" text,
142 + "category" text NOT NULL,
143 + "description" text,
144 + "homepage" text,
145 + "docs_url" text,
146 + "terms_url" text,
147 + "access_type" text NOT NULL,
148 + "access_auth" text NOT NULL,
149 + "license" text,
150 + "license_status" text DEFAULT 'review' NOT NULL,
151 + "commercial_use" text DEFAULT 'unknown' NOT NULL,
152 + "redistribution" text DEFAULT 'unknown' NOT NULL,
153 + "attribution" text,
154 + "license_reviewed_at" timestamp with time zone,
155 + "approved_for_production" boolean DEFAULT false NOT NULL,
156 + "update_frequency" text,
157 + "supports_incremental" boolean DEFAULT false NOT NULL,
158 + "entities" text[] DEFAULT '{}' NOT NULL,
159 + "metrics" text[] DEFAULT '{}' NOT NULL,
160 + "rate_limit" text,
161 + "status" text DEFAULT 'planned' NOT NULL,
162 + "tier" integer DEFAULT 0 NOT NULL,
163 + "manifest" jsonb DEFAULT '{}'::jsonb NOT NULL,
164 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
165 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
166 +);
167 +--> statement-breakpoint
168 +CREATE TABLE "unresolved_labels" (
169 + "id" bigserial PRIMARY KEY NOT NULL,
170 + "source_id" varchar(32) NOT NULL,
171 + "entity_kind" text NOT NULL,
172 + "source_text" text NOT NULL,
173 + "normalized" text NOT NULL,
174 + "context" jsonb DEFAULT '{}'::jsonb NOT NULL,
175 + "count" integer DEFAULT 1 NOT NULL,
176 + "status" text DEFAULT 'open' NOT NULL,
177 + "suggested_id" text,
178 + "suggested_match_type" text,
179 + "suggested_score" real,
180 + "resolved_id" text,
181 + "resolved_by" text,
182 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
183 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
184 +);
185 +--> statement-breakpoint
186 +CREATE TABLE "anatomical_sites" (
187 + "id" varchar(32) PRIMARY KEY NOT NULL,
188 + "name" text NOT NULL,
189 + "slug" text NOT NULL,
190 + "ncit_code" text,
191 + "uberon_id" text,
192 + "parent_id" varchar(32),
193 + "system" text
194 +);
195 +--> statement-breakpoint
196 +CREATE TABLE "cancer_aliases" (
197 + "id" bigserial PRIMARY KEY NOT NULL,
198 + "cancer_id" varchar(32) NOT NULL,
199 + "alias" text NOT NULL,
200 + "normalized" text NOT NULL,
201 + "alias_type" text DEFAULT 'synonym' NOT NULL,
202 + "source_id" varchar(32),
203 + "source_terminology" text,
204 + "language" text DEFAULT 'en' NOT NULL
205 +);
206 +--> statement-breakpoint
207 +CREATE TABLE "cancer_anatomy" (
208 + "id" bigserial PRIMARY KEY NOT NULL,
209 + "cancer_id" varchar(32) NOT NULL,
210 + "site_id" varchar(32) NOT NULL,
211 + "relation" text DEFAULT 'primary' NOT NULL,
212 + "source_id" varchar(32)
213 +);
214 +--> statement-breakpoint
215 +CREATE TABLE "cancer_codes" (
216 + "id" bigserial PRIMARY KEY NOT NULL,
217 + "cancer_id" varchar(32) NOT NULL,
218 + "system" text NOT NULL,
219 + "code" text NOT NULL,
220 + "match_type" text DEFAULT 'EXACT_IDENTIFIER' NOT NULL,
221 + "source_id" varchar(32),
222 + "valid_from" text,
223 + "valid_to" text
224 +);
225 +--> statement-breakpoint
226 +CREATE TABLE "cancer_hierarchy" (
227 + "id" bigserial PRIMARY KEY NOT NULL,
228 + "parent_id" varchar(32) NOT NULL,
229 + "child_id" varchar(32) NOT NULL,
230 + "hierarchy_type" text NOT NULL,
231 + "source_id" varchar(32)
232 +);
233 +--> statement-breakpoint
234 +CREATE TABLE "cancers" (
235 + "id" varchar(32) PRIMARY KEY NOT NULL,
236 + "slug" text NOT NULL,
237 + "canonical_name" text NOT NULL,
238 + "short_name" text,
239 + "entity_type" text DEFAULT 'cancer' NOT NULL,
240 + "malignant" boolean DEFAULT true NOT NULL,
241 + "solid_tumor" boolean DEFAULT true NOT NULL,
242 + "hematologic" boolean DEFAULT false NOT NULL,
243 + "pediatric_relevant" boolean DEFAULT false NOT NULL,
244 + "rare_cancer" boolean,
245 + "top_level" boolean DEFAULT false NOT NULL,
246 + "description" text,
247 + "description_provenance_id" integer,
248 + "primary_ncit_code" text,
249 + "primary_oncotree_code" text,
250 + "depth" integer DEFAULT 0 NOT NULL,
251 + "status" text DEFAULT 'active' NOT NULL,
252 + "merged_into" varchar(32),
253 + "deprecated_reason" text,
254 + "classification_version" text,
255 + "semantic_types" text[] DEFAULT '{}' NOT NULL,
256 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
257 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
258 +);
259 +--> statement-breakpoint
260 +CREATE TABLE "cohort_definitions" (
261 + "id" bigserial PRIMARY KEY NOT NULL,
262 + "name" text NOT NULL,
263 + "cancer_id" varchar(32) NOT NULL,
264 + "biomarker_ids" text[] DEFAULT '{}' NOT NULL,
265 + "variant_ids" text[] DEFAULT '{}' NOT NULL,
266 + "stage" text,
267 + "attributes" jsonb DEFAULT '{}'::jsonb NOT NULL,
268 + "confidence" real,
269 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
270 +);
271 +--> statement-breakpoint
272 +CREATE TABLE "geographies" (
273 + "id" varchar(32) PRIMARY KEY NOT NULL,
274 + "slug" text NOT NULL,
275 + "name" text NOT NULL,
276 + "kind" text NOT NULL,
277 + "iso2" text,
278 + "iso3" text,
279 + "parent_id" varchar(32),
280 + "who_region" text,
281 + "population" integer,
282 + "population_year" integer
283 +);
284 +--> statement-breakpoint
285 +CREATE TABLE "biomarkers" (
286 + "id" varchar(32) PRIMARY KEY NOT NULL,
287 + "slug" text NOT NULL,
288 + "name" text NOT NULL,
289 + "kind" text NOT NULL,
290 + "gene_id" varchar(32),
291 + "ncit_code" text,
292 + "description" text,
293 + "measurement" jsonb DEFAULT '{}'::jsonb NOT NULL,
294 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
295 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
296 +);
297 +--> statement-breakpoint
298 +CREATE TABLE "cancer_gene_frequencies" (
299 + "id" bigserial PRIMARY KEY NOT NULL,
300 + "cohort_id" varchar(32) NOT NULL,
301 + "cancer_id" varchar(32),
302 + "gene_id" varchar(32),
303 + "gene_symbol" text NOT NULL,
304 + "alteration_type" text DEFAULT 'ssm' NOT NULL,
305 + "cases_affected" integer NOT NULL,
306 + "cases_profiled" integer NOT NULL,
307 + "frequency" real NOT NULL,
308 + "rank" integer,
309 + "data_release" text,
310 + "provenance_id" integer NOT NULL,
311 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
312 +);
313 +--> statement-breakpoint
314 +CREATE TABLE "entity_embeddings" (
315 + "id" bigserial PRIMARY KEY NOT NULL,
316 + "entity_type" text NOT NULL,
317 + "entity_id" text NOT NULL,
318 + "model" text NOT NULL,
319 + "dimensions" integer NOT NULL,
320 + "text_hash" text NOT NULL,
321 + "embedding" text,
322 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
323 +);
324 +--> statement-breakpoint
325 +CREATE TABLE "gene_aliases" (
326 + "id" bigserial PRIMARY KEY NOT NULL,
327 + "gene_id" varchar(32) NOT NULL,
328 + "alias" text NOT NULL,
329 + "alias_type" text NOT NULL,
330 + "source_id" varchar(32)
331 +);
332 +--> statement-breakpoint
333 +CREATE TABLE "genes" (
334 + "id" varchar(32) PRIMARY KEY NOT NULL,
335 + "hgnc_id" text,
336 + "symbol" text NOT NULL,
337 + "name" text,
338 + "locus_type" text,
339 + "locus_group" text,
340 + "location" text,
341 + "chromosome" text,
342 + "ensembl_gene_id" text,
343 + "ncbi_gene_id" text,
344 + "omim_ids" text[] DEFAULT '{}' NOT NULL,
345 + "uniprot_ids" text[] DEFAULT '{}' NOT NULL,
346 + "refseq_accession" text,
347 + "prev_symbols" text[] DEFAULT '{}' NOT NULL,
348 + "alias_symbols" text[] DEFAULT '{}' NOT NULL,
349 + "gene_families" text[] DEFAULT '{}' NOT NULL,
350 + "status" text DEFAULT 'Approved' NOT NULL,
351 + "is_cancer_gene" boolean DEFAULT false NOT NULL,
352 + "civic_gene_id" integer,
353 + "description" text,
354 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
355 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
356 +);
357 +--> statement-breakpoint
358 +CREATE TABLE "genomic_cohorts" (
359 + "id" varchar(32) PRIMARY KEY NOT NULL,
360 + "source_id" varchar(32) NOT NULL,
361 + "study_id" text NOT NULL,
362 + "name" text NOT NULL,
363 + "program" text,
364 + "primary_sites" text[] DEFAULT '{}' NOT NULL,
365 + "disease_types" text[] DEFAULT '{}' NOT NULL,
366 + "cancer_id" varchar(32),
367 + "cancer_match_type" text,
368 + "case_count" integer,
369 + "cases_with_ssm" integer,
370 + "data_release" text,
371 + "access_level" text DEFAULT 'open' NOT NULL,
372 + "url" text,
373 + "provenance_id" integer,
374 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
375 +);
376 +--> statement-breakpoint
377 +CREATE TABLE "variant_aliases" (
378 + "id" bigserial PRIMARY KEY NOT NULL,
379 + "variant_id" varchar(32) NOT NULL,
380 + "alias" text NOT NULL,
381 + "source_id" varchar(32)
382 +);
383 +--> statement-breakpoint
384 +CREATE TABLE "variant_clinical_significance" (
385 + "id" bigserial PRIMARY KEY NOT NULL,
386 + "variant_id" varchar(32) NOT NULL,
387 + "clinvar_variation_id" text NOT NULL,
388 + "clinical_significance" text NOT NULL,
389 + "review_status" text,
390 + "star_rating" integer,
391 + "last_evaluated" text,
392 + "conditions" text[] DEFAULT '{}' NOT NULL,
393 + "condition_cancer_ids" text[] DEFAULT '{}' NOT NULL,
394 + "origin_simple" text,
395 + "number_submitters" integer,
396 + "provenance_id" integer NOT NULL,
397 + "ingest_run_id" text,
398 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
399 +);
400 +--> statement-breakpoint
401 +CREATE TABLE "variants" (
402 + "id" varchar(32) PRIMARY KEY NOT NULL,
403 + "slug" text NOT NULL,
404 + "gene_id" varchar(32),
405 + "gene_symbol" text,
406 + "name" text NOT NULL,
407 + "variant_type" text,
408 + "hgvs_g" text,
409 + "hgvs_c" text,
410 + "hgvs_p" text,
411 + "assembly" text,
412 + "chromosome" text,
413 + "start" integer,
414 + "end" integer,
415 + "reference_bases" text,
416 + "alternate_bases" text,
417 + "coordinates" jsonb DEFAULT '[]'::jsonb NOT NULL,
418 + "clinvar_variation_id" text,
419 + "civic_variant_id" integer,
420 + "dbsnp_ids" text[] DEFAULT '{}' NOT NULL,
421 + "fusion_partners" text[] DEFAULT '{}' NOT NULL,
422 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
423 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
424 +);
425 +--> statement-breakpoint
426 +CREATE TABLE "drug_aliases" (
427 + "id" bigserial PRIMARY KEY NOT NULL,
428 + "drug_id" varchar(32) NOT NULL,
429 + "alias" text NOT NULL,
430 + "normalized" text NOT NULL,
431 + "alias_type" text DEFAULT 'synonym' NOT NULL,
432 + "source_id" varchar(32)
433 +);
434 +--> statement-breakpoint
435 +CREATE TABLE "drug_approvals" (
436 + "id" bigserial PRIMARY KEY NOT NULL,
437 + "drug_id" varchar(32) NOT NULL,
438 + "cancer_id" varchar(32),
439 + "biomarker_ids" text[] DEFAULT '{}' NOT NULL,
440 + "tumor_agnostic" boolean DEFAULT false NOT NULL,
441 + "jurisdiction" text NOT NULL,
442 + "authority" text NOT NULL,
443 + "indication" text NOT NULL,
444 + "line_of_therapy" text,
445 + "disease_stage" text,
446 + "approval_type" text,
447 + "accelerated" boolean,
448 + "conditional" boolean,
449 + "approval_date" text,
450 + "withdrawal_date" text,
451 + "status" text NOT NULL,
452 + "application_number" text,
453 + "source_id" varchar(32) NOT NULL,
454 + "provenance_id" integer NOT NULL,
455 + "raw" jsonb,
456 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
457 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
458 +);
459 +--> statement-breakpoint
460 +CREATE TABLE "drugs" (
461 + "id" varchar(32) PRIMARY KEY NOT NULL,
462 + "slug" text NOT NULL,
463 + "name" text NOT NULL,
464 + "kind" text,
465 + "ncit_code" text,
466 + "chembl_id" text,
467 + "civic_therapy_id" integer,
468 + "drugbank_id" text,
469 + "pubchem_cid" text,
470 + "unii" text,
471 + "mechanism" text,
472 + "target_gene_ids" text[] DEFAULT '{}' NOT NULL,
473 + "development_status" text,
474 + "description" text,
475 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
476 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
477 +);
478 +--> statement-breakpoint
479 +CREATE TABLE "treatment_regimens" (
480 + "id" varchar(32) PRIMARY KEY NOT NULL,
481 + "slug" text NOT NULL,
482 + "name" text NOT NULL,
483 + "component_drug_ids" text[] DEFAULT '{}' NOT NULL,
484 + "modality" text DEFAULT 'drug_combination' NOT NULL,
485 + "description" text,
486 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
487 +);
488 +--> statement-breakpoint
489 +CREATE TABLE "clinical_trials" (
490 + "id" varchar(32) PRIMARY KEY NOT NULL,
491 + "nct_id" text NOT NULL,
492 + "brief_title" text NOT NULL,
493 + "official_title" text,
494 + "acronym" text,
495 + "study_type" text,
496 + "phases" text[] DEFAULT '{}' NOT NULL,
497 + "overall_status" text,
498 + "why_stopped" text,
499 + "start_date" text,
500 + "primary_completion_date" text,
501 + "completion_date" text,
502 + "first_posted_date" text,
503 + "last_update_posted_date" text,
504 + "results_first_posted_date" text,
505 + "has_results" boolean DEFAULT false NOT NULL,
506 + "enrollment_count" integer,
507 + "enrollment_type" text,
508 + "lead_sponsor" text,
509 + "lead_sponsor_class" text,
510 + "collaborators" text[] DEFAULT '{}' NOT NULL,
511 + "conditions" text[] DEFAULT '{}' NOT NULL,
512 + "keywords" text[] DEFAULT '{}' NOT NULL,
513 + "interventions" jsonb DEFAULT '[]'::jsonb NOT NULL,
514 + "arms" jsonb DEFAULT '[]'::jsonb NOT NULL,
515 + "primary_outcomes" jsonb DEFAULT '[]'::jsonb NOT NULL,
516 + "secondary_outcomes" jsonb DEFAULT '[]'::jsonb NOT NULL,
517 + "eligibility" jsonb DEFAULT '{}'::jsonb NOT NULL,
518 + "sex" text,
519 + "minimum_age" text,
520 + "maximum_age" text,
521 + "countries" text[] DEFAULT '{}' NOT NULL,
522 + "locations_count" integer DEFAULT 0 NOT NULL,
523 + "references" jsonb DEFAULT '[]'::jsonb NOT NULL,
524 + "brief_summary" text,
525 + "is_oncology" boolean DEFAULT true NOT NULL,
526 + "source_record_id" integer,
527 + "ingest_run_id" text,
528 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
529 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
530 +);
531 +--> statement-breakpoint
532 +CREATE TABLE "trial_conditions" (
533 + "id" bigserial PRIMARY KEY NOT NULL,
534 + "trial_id" varchar(32) NOT NULL,
535 + "condition_text" text NOT NULL,
536 + "normalized" text NOT NULL,
537 + "cancer_id" varchar(32),
538 + "match_type" text DEFAULT 'UNRESOLVED' NOT NULL,
539 + "confidence" real
540 +);
541 +--> statement-breakpoint
542 +CREATE TABLE "trial_interventions" (
543 + "id" bigserial PRIMARY KEY NOT NULL,
544 + "trial_id" varchar(32) NOT NULL,
545 + "name" text NOT NULL,
546 + "normalized" text NOT NULL,
547 + "intervention_type" text,
548 + "drug_id" varchar(32),
549 + "match_type" text DEFAULT 'UNRESOLVED' NOT NULL
550 +);
551 +--> statement-breakpoint
552 +CREATE TABLE "trial_locations" (
553 + "id" bigserial PRIMARY KEY NOT NULL,
554 + "trial_id" varchar(32) NOT NULL,
555 + "facility" text,
556 + "city" text,
557 + "state" text,
558 + "zip" text,
559 + "country" text,
560 + "status" text,
561 + "lat" real,
562 + "lng" real
563 +);
564 +--> statement-breakpoint
565 +CREATE TABLE "trial_pulse" (
566 + "id" bigserial PRIMARY KEY NOT NULL,
567 + "day" date NOT NULL,
568 + "cancer_id" varchar(32),
569 + "phase" text,
570 + "new_trials" integer NOT NULL,
571 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
572 +);
573 +--> statement-breakpoint
574 +CREATE TABLE "literature_counts" (
575 + "id" bigserial PRIMARY KEY NOT NULL,
576 + "cancer_id" varchar(32) NOT NULL,
577 + "window_key" text NOT NULL,
578 + "window_start" text,
579 + "window_end" text,
580 + "query" text NOT NULL,
581 + "count" integer NOT NULL,
582 + "provenance_id" integer NOT NULL,
583 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
584 +);
585 +--> statement-breakpoint
586 +CREATE TABLE "publication_entity_edges" (
587 + "id" bigserial PRIMARY KEY NOT NULL,
588 + "publication_id" varchar(32) NOT NULL,
589 + "entity_type" text NOT NULL,
590 + "entity_id" text NOT NULL,
591 + "method" text NOT NULL,
592 + "confidence" real,
593 + "status" text DEFAULT 'candidate' NOT NULL,
594 + "source_id" varchar(32),
595 + "ingest_run_id" text,
596 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
597 +);
598 +--> statement-breakpoint
599 +CREATE TABLE "publications" (
600 + "id" varchar(32) PRIMARY KEY NOT NULL,
601 + "pmid" text,
602 + "doi" text,
603 + "pmcid" text,
604 + "title" text NOT NULL,
605 + "abstract" text,
606 + "journal" text,
607 + "journal_iso" text,
608 + "pub_date" text,
609 + "pub_year" integer,
610 + "publication_types" text[] DEFAULT '{}' NOT NULL,
611 + "mesh_terms" jsonb DEFAULT '[]'::jsonb NOT NULL,
612 + "authors" jsonb DEFAULT '[]'::jsonb NOT NULL,
613 + "language" text,
614 + "is_preprint" boolean DEFAULT false NOT NULL,
615 + "retracted" boolean DEFAULT false NOT NULL,
616 + "retraction_notice" text,
617 + "nct_ids" text[] DEFAULT '{}' NOT NULL,
618 + "cited_by_count" integer,
619 + "source_record_id" integer,
620 + "ingest_run_id" text,
621 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
622 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
623 +);
624 +--> statement-breakpoint
625 +CREATE TABLE "civic_evidence_items" (
626 + "id" bigserial PRIMARY KEY NOT NULL,
627 + "civic_id" integer NOT NULL,
628 + "name" text,
629 + "molecular_profile_id" integer,
630 + "molecular_profile_name" text,
631 + "gene_symbols" text[] DEFAULT '{}' NOT NULL,
632 + "gene_ids" text[] DEFAULT '{}' NOT NULL,
633 + "variant_ids" text[] DEFAULT '{}' NOT NULL,
634 + "civic_variant_ids" integer[] DEFAULT '{}' NOT NULL,
635 + "disease_name" text,
636 + "doid" text,
637 + "cancer_id" varchar(32),
638 + "cancer_match_type" text,
639 + "therapy_names" text[] DEFAULT '{}' NOT NULL,
640 + "therapy_ids" text[] DEFAULT '{}' NOT NULL,
641 + "therapy_interaction_type" text,
642 + "evidence_type" text,
643 + "evidence_level" text,
644 + "evidence_direction" text,
645 + "significance" text,
646 + "evidence_rating" integer,
647 + "status" text,
648 + "description" text,
649 + "pmid" text,
650 + "source_citation" text,
651 + "phenotypes" text[] DEFAULT '{}' NOT NULL,
652 + "provenance_id" integer NOT NULL,
653 + "ingest_run_id" text,
654 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
655 +);
656 +--> statement-breakpoint
657 +CREATE TABLE "knowledge_edges" (
658 + "id" bigserial PRIMARY KEY NOT NULL,
659 + "source_entity_type" text NOT NULL,
660 + "source_entity_id" text NOT NULL,
661 + "target_entity_type" text NOT NULL,
662 + "target_entity_id" text NOT NULL,
663 + "relationship_type" text NOT NULL,
664 + "cancer_context_ids" text[] DEFAULT '{}' NOT NULL,
665 + "predictive" boolean,
666 + "prognostic" boolean,
667 + "diagnostic" boolean,
668 + "predisposing" boolean,
669 + "direction" text,
670 + "evidence_level" text,
671 + "evidence_score" real,
672 + "evidence_category" text DEFAULT 'curated_evidence' NOT NULL,
673 + "status" text DEFAULT 'active' NOT NULL,
674 + "source_id" varchar(32) NOT NULL,
675 + "source_record_id" text,
676 + "provenance_ids" integer[] DEFAULT '{}' NOT NULL,
677 + "support_count" integer DEFAULT 1 NOT NULL,
678 + "first_seen_at" timestamp with time zone DEFAULT now() NOT NULL,
679 + "last_seen_at" timestamp with time zone DEFAULT now() NOT NULL
680 +);
681 +--> statement-breakpoint
682 +CREATE TABLE "risk_factors" (
683 + "id" bigserial PRIMARY KEY NOT NULL,
684 + "slug" text NOT NULL,
685 + "name" text NOT NULL,
686 + "kind" text NOT NULL,
687 + "classification_authority" text,
688 + "classification" text,
689 + "description" text,
690 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
691 +);
692 +--> statement-breakpoint
693 +CREATE TABLE "epidemiology_observations" (
694 + "id" bigserial PRIMARY KEY NOT NULL,
695 + "cancer_id" varchar(32) NOT NULL,
696 + "geography_id" varchar(32) NOT NULL,
697 + "year" integer NOT NULL,
698 + "year_end" integer,
699 + "sex" text DEFAULT 'all' NOT NULL,
700 + "age_group" text DEFAULT 'all' NOT NULL,
701 + "metric" text NOT NULL,
702 + "value" double precision NOT NULL,
703 + "unit" text NOT NULL,
704 + "lower_ci" double precision,
705 + "upper_ci" double precision,
706 + "standard_population" text,
707 + "estimate_type" text DEFAULT 'observed' NOT NULL,
708 + "site_definition" text,
709 + "source_id" varchar(32) NOT NULL,
710 + "provenance_id" integer NOT NULL,
711 + "ingest_run_id" text,
712 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
713 +);
714 +--> statement-breakpoint
715 +CREATE TABLE "survival_observations" (
716 + "id" bigserial PRIMARY KEY NOT NULL,
717 + "cancer_id" varchar(32) NOT NULL,
718 + "geography_id" varchar(32),
719 + "stage" text,
720 + "staging_system" text,
721 + "sex" text DEFAULT 'all' NOT NULL,
722 + "age_group" text DEFAULT 'all' NOT NULL,
723 + "diagnosis_period" text,
724 + "survival_type" text NOT NULL,
725 + "duration_months" integer NOT NULL,
726 + "probability" real,
727 + "median_months" real,
728 + "cohort_size" integer,
729 + "lower_ci" real,
730 + "upper_ci" real,
731 + "method" text,
732 + "source_id" varchar(32) NOT NULL,
733 + "provenance_id" integer NOT NULL,
734 + "ingest_run_id" text,
735 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
736 +);
737 +--> statement-breakpoint
738 +CREATE TABLE "ai_answers" (
739 + "id" bigserial PRIMARY KEY NOT NULL,
740 + "kind" text NOT NULL,
741 + "subject_id" text,
742 + "question_hash" text NOT NULL,
743 + "question" text,
744 + "answer" jsonb NOT NULL,
745 + "model" text NOT NULL,
746 + "prompt_version" text NOT NULL,
747 + "source_snapshot" jsonb DEFAULT '{}'::jsonb NOT NULL,
748 + "data_as_of" timestamp with time zone NOT NULL,
749 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
750 +);
751 +--> statement-breakpoint
752 +CREATE TABLE "api_keys" (
753 + "id" bigserial PRIMARY KEY NOT NULL,
754 + "key_hash" text NOT NULL,
755 + "prefix" text NOT NULL,
756 + "label" text,
757 + "owner_email" text,
758 + "tier" text DEFAULT 'free' NOT NULL,
759 + "rate_limit_per_minute" integer DEFAULT 60 NOT NULL,
760 + "active" boolean DEFAULT true NOT NULL,
761 + "last_used_at" timestamp with time zone,
762 + "created_at" timestamp with time zone DEFAULT now() NOT NULL
763 +);
764 +--> statement-breakpoint
765 +CREATE TABLE "entity_counters" (
766 + "id" bigserial PRIMARY KEY NOT NULL,
767 + "entity_type" text NOT NULL,
768 + "entity_id" text NOT NULL,
769 + "trial_count" integer DEFAULT 0 NOT NULL,
770 + "active_trial_count" integer DEFAULT 0 NOT NULL,
771 + "recruiting_trial_count" integer DEFAULT 0 NOT NULL,
772 + "phase3_trial_count" integer DEFAULT 0 NOT NULL,
773 + "publication_count" integer DEFAULT 0 NOT NULL,
774 + "publication_count_5y" integer DEFAULT 0 NOT NULL,
775 + "publication_count_12m" integer DEFAULT 0 NOT NULL,
776 + "gene_count" integer DEFAULT 0 NOT NULL,
777 + "variant_count" integer DEFAULT 0 NOT NULL,
778 + "drug_count" integer DEFAULT 0 NOT NULL,
779 + "approved_drug_count" integer DEFAULT 0 NOT NULL,
780 + "evidence_count" integer DEFAULT 0 NOT NULL,
781 + "cohort_count" integer DEFAULT 0 NOT NULL,
782 + "subtype_count" integer DEFAULT 0 NOT NULL,
783 + "descendant_count" integer DEFAULT 0 NOT NULL,
784 + "epidemiology_obs_count" integer DEFAULT 0 NOT NULL,
785 + "survival_obs_count" integer DEFAULT 0 NOT NULL,
786 + "completeness" jsonb DEFAULT '{}'::jsonb NOT NULL,
787 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
788 +);
789 +--> statement-breakpoint
790 +CREATE TABLE "metric_definitions" (
791 + "id" varchar(32) PRIMARY KEY NOT NULL,
792 + "slug" text NOT NULL,
793 + "name" text NOT NULL,
794 + "description" text NOT NULL,
795 + "formula" text NOT NULL,
796 + "formula_version" text NOT NULL,
797 + "unit" text NOT NULL,
798 + "higher_is_worse" boolean,
799 + "aggregation" text,
800 + "valid_dimensions" text[] DEFAULT '{}' NOT NULL,
801 + "source_slugs" text[] DEFAULT '{}' NOT NULL,
802 + "category" text NOT NULL,
803 + "eligibility" jsonb DEFAULT '{}'::jsonb NOT NULL,
804 + "experimental" boolean DEFAULT false NOT NULL,
805 + "created_at" timestamp with time zone DEFAULT now() NOT NULL,
806 + "updated_at" timestamp with time zone DEFAULT now() NOT NULL
807 +);
808 +--> statement-breakpoint
809 +CREATE TABLE "ranking_snapshots" (
810 + "id" bigserial PRIMARY KEY NOT NULL,
811 + "metric_id" varchar(32) NOT NULL,
812 + "metric_slug" text NOT NULL,
813 + "scope_key" text NOT NULL,
814 + "geography" text DEFAULT 'WORLD' NOT NULL,
815 + "sex" text DEFAULT 'all' NOT NULL,
816 + "age_group" text DEFAULT 'all' NOT NULL,
817 + "year" integer,
818 + "entity_level" text DEFAULT 'top' NOT NULL,
819 + "formula_version" text NOT NULL,
820 + "eligible_entities" integer NOT NULL,
821 + "inputs_hash" text NOT NULL,
822 + "source_ids" text[] DEFAULT '{}' NOT NULL,
823 + "is_current" boolean DEFAULT true NOT NULL,
824 + "generated_at" timestamp with time zone DEFAULT now() NOT NULL
825 +);
826 +--> statement-breakpoint
827 +CREATE TABLE "rankings" (
828 + "id" bigserial PRIMARY KEY NOT NULL,
829 + "snapshot_id" integer NOT NULL,
830 + "metric_slug" text NOT NULL,
831 + "scope_key" text NOT NULL,
832 + "cancer_id" varchar(32) NOT NULL,
833 + "rank" integer NOT NULL,
834 + "eligible_entities" integer NOT NULL,
835 + "percentile" real NOT NULL,
836 + "value" double precision NOT NULL,
837 + "unit" text NOT NULL,
838 + "confidence" text DEFAULT 'MEDIUM' NOT NULL,
839 + "inputs" jsonb DEFAULT '{}'::jsonb NOT NULL,
840 + "breakdown" jsonb,
841 + "previous_rank" integer,
842 + "generated_at" timestamp with time zone DEFAULT now() NOT NULL
843 +);
844 +--> statement-breakpoint
845 +CREATE INDEX "change_events_entity_idx" ON "change_events" USING btree ("entity_type","entity_id","created_at");--> statement-breakpoint
846 +CREATE UNIQUE INDEX "connector_field_stats_uq" ON "connector_field_stats" USING btree ("connector_id","entity","field");--> statement-breakpoint
847 +CREATE INDEX "ingest_runs_connector_idx" ON "ingest_runs" USING btree ("connector_id","started_at");--> statement-breakpoint
848 +CREATE INDEX "provenance_source_idx" ON "provenance" USING btree ("source_id","source_record_id");--> statement-breakpoint
849 +CREATE INDEX "provenance_pmid_idx" ON "provenance" USING btree ("pmid");--> statement-breakpoint
850 +CREATE UNIQUE INDEX "source_records_uq" ON "source_records" USING btree ("source_id","entity_kind","source_record_id");--> statement-breakpoint
851 +CREATE INDEX "source_records_canonical_idx" ON "source_records" USING btree ("canonical_type","canonical_id");--> statement-breakpoint
852 +CREATE UNIQUE INDEX "sources_slug_uq" ON "sources" USING btree ("slug");--> statement-breakpoint
853 +CREATE UNIQUE INDEX "unresolved_labels_uq" ON "unresolved_labels" USING btree ("source_id","entity_kind","normalized");--> statement-breakpoint
854 +CREATE INDEX "unresolved_labels_count_idx" ON "unresolved_labels" USING btree ("status","count");--> statement-breakpoint
855 +CREATE UNIQUE INDEX "anatomical_sites_slug_uq" ON "anatomical_sites" USING btree ("slug");--> statement-breakpoint
856 +CREATE UNIQUE INDEX "cancer_aliases_uq" ON "cancer_aliases" USING btree ("cancer_id","normalized","alias_type");--> statement-breakpoint
857 +CREATE INDEX "cancer_aliases_norm_idx" ON "cancer_aliases" USING btree ("normalized");--> statement-breakpoint
858 +CREATE UNIQUE INDEX "cancer_anatomy_uq" ON "cancer_anatomy" USING btree ("cancer_id","site_id","relation");--> statement-breakpoint
859 +CREATE UNIQUE INDEX "cancer_codes_uq" ON "cancer_codes" USING btree ("cancer_id","system","code");--> statement-breakpoint
860 +CREATE INDEX "cancer_codes_lookup_idx" ON "cancer_codes" USING btree ("system","code");--> statement-breakpoint
861 +CREATE UNIQUE INDEX "cancer_hierarchy_uq" ON "cancer_hierarchy" USING btree ("parent_id","child_id","hierarchy_type");--> statement-breakpoint
862 +CREATE INDEX "cancer_hierarchy_child_idx" ON "cancer_hierarchy" USING btree ("child_id");--> statement-breakpoint
863 +CREATE UNIQUE INDEX "cancers_slug_uq" ON "cancers" USING btree ("slug");--> statement-breakpoint
864 +CREATE UNIQUE INDEX "cancers_ncit_uq" ON "cancers" USING btree ("primary_ncit_code");--> statement-breakpoint
865 +CREATE INDEX "cancers_name_idx" ON "cancers" USING btree ("canonical_name");--> statement-breakpoint
866 +CREATE INDEX "cancers_type_idx" ON "cancers" USING btree ("entity_type","malignant","top_level");--> statement-breakpoint
867 +CREATE UNIQUE INDEX "geographies_slug_uq" ON "geographies" USING btree ("slug");--> statement-breakpoint
868 +CREATE INDEX "geographies_iso3_idx" ON "geographies" USING btree ("iso3");--> statement-breakpoint
869 +CREATE UNIQUE INDEX "biomarkers_slug_uq" ON "biomarkers" USING btree ("slug");--> statement-breakpoint
870 +CREATE UNIQUE INDEX "cancer_gene_freq_uq" ON "cancer_gene_frequencies" USING btree ("cohort_id","gene_symbol","alteration_type");--> statement-breakpoint
871 +CREATE INDEX "cancer_gene_freq_cancer_idx" ON "cancer_gene_frequencies" USING btree ("cancer_id","frequency");--> statement-breakpoint
872 +CREATE INDEX "cancer_gene_freq_gene_idx" ON "cancer_gene_frequencies" USING btree ("gene_id");--> statement-breakpoint
873 +CREATE UNIQUE INDEX "entity_embeddings_uq" ON "entity_embeddings" USING btree ("entity_type","entity_id","model");--> statement-breakpoint
874 +CREATE UNIQUE INDEX "gene_aliases_uq" ON "gene_aliases" USING btree ("gene_id","alias","alias_type");--> statement-breakpoint
875 +CREATE INDEX "gene_aliases_alias_idx" ON "gene_aliases" USING btree ("alias");--> statement-breakpoint
876 +CREATE UNIQUE INDEX "genes_symbol_uq" ON "genes" USING btree ("symbol");--> statement-breakpoint
877 +CREATE UNIQUE INDEX "genes_hgnc_uq" ON "genes" USING btree ("hgnc_id");--> statement-breakpoint
878 +CREATE INDEX "genes_ensembl_idx" ON "genes" USING btree ("ensembl_gene_id");--> statement-breakpoint
879 +CREATE INDEX "genes_ncbi_idx" ON "genes" USING btree ("ncbi_gene_id");--> statement-breakpoint
880 +CREATE UNIQUE INDEX "genomic_cohorts_uq" ON "genomic_cohorts" USING btree ("source_id","study_id");--> statement-breakpoint
881 +CREATE INDEX "genomic_cohorts_cancer_idx" ON "genomic_cohorts" USING btree ("cancer_id");--> statement-breakpoint
882 +CREATE UNIQUE INDEX "variant_aliases_uq" ON "variant_aliases" USING btree ("variant_id","alias");--> statement-breakpoint
883 +CREATE UNIQUE INDEX "variant_clinsig_uq" ON "variant_clinical_significance" USING btree ("clinvar_variation_id");--> statement-breakpoint
884 +CREATE UNIQUE INDEX "variants_slug_uq" ON "variants" USING btree ("slug");--> statement-breakpoint
885 +CREATE INDEX "variants_gene_idx" ON "variants" USING btree ("gene_id");--> statement-breakpoint
886 +CREATE INDEX "variants_clinvar_idx" ON "variants" USING btree ("clinvar_variation_id");--> statement-breakpoint
887 +CREATE INDEX "variants_civic_idx" ON "variants" USING btree ("civic_variant_id");--> statement-breakpoint
888 +CREATE UNIQUE INDEX "drug_aliases_uq" ON "drug_aliases" USING btree ("drug_id","normalized","alias_type");--> statement-breakpoint
889 +CREATE INDEX "drug_aliases_norm_idx" ON "drug_aliases" USING btree ("normalized");--> statement-breakpoint
890 +CREATE INDEX "drug_approvals_drug_idx" ON "drug_approvals" USING btree ("drug_id");--> statement-breakpoint
891 +CREATE INDEX "drug_approvals_cancer_idx" ON "drug_approvals" USING btree ("cancer_id");--> statement-breakpoint
892 +CREATE UNIQUE INDEX "drugs_slug_uq" ON "drugs" USING btree ("slug");--> statement-breakpoint
893 +CREATE INDEX "drugs_ncit_idx" ON "drugs" USING btree ("ncit_code");--> statement-breakpoint
894 +CREATE INDEX "drugs_civic_idx" ON "drugs" USING btree ("civic_therapy_id");--> statement-breakpoint
895 +CREATE INDEX "drugs_chembl_idx" ON "drugs" USING btree ("chembl_id");--> statement-breakpoint
896 +CREATE UNIQUE INDEX "treatment_regimens_slug_uq" ON "treatment_regimens" USING btree ("slug");--> statement-breakpoint
897 +CREATE UNIQUE INDEX "clinical_trials_nct_uq" ON "clinical_trials" USING btree ("nct_id");--> statement-breakpoint
898 +CREATE INDEX "clinical_trials_status_idx" ON "clinical_trials" USING btree ("overall_status");--> statement-breakpoint
899 +CREATE INDEX "clinical_trials_updated_idx" ON "clinical_trials" USING btree ("last_update_posted_date");--> statement-breakpoint
900 +CREATE INDEX "clinical_trials_sponsor_idx" ON "clinical_trials" USING btree ("lead_sponsor");--> statement-breakpoint
901 +CREATE UNIQUE INDEX "trial_conditions_uq" ON "trial_conditions" USING btree ("trial_id","normalized");--> statement-breakpoint
902 +CREATE INDEX "trial_conditions_cancer_idx" ON "trial_conditions" USING btree ("cancer_id");--> statement-breakpoint
903 +CREATE INDEX "trial_conditions_norm_idx" ON "trial_conditions" USING btree ("normalized");--> statement-breakpoint
904 +CREATE UNIQUE INDEX "trial_interventions_uq" ON "trial_interventions" USING btree ("trial_id","normalized");--> statement-breakpoint
905 +CREATE INDEX "trial_interventions_drug_idx" ON "trial_interventions" USING btree ("drug_id");--> statement-breakpoint
906 +CREATE INDEX "trial_locations_trial_idx" ON "trial_locations" USING btree ("trial_id");--> statement-breakpoint
907 +CREATE INDEX "trial_locations_country_idx" ON "trial_locations" USING btree ("country");--> statement-breakpoint
908 +CREATE UNIQUE INDEX "trial_pulse_uq" ON "trial_pulse" USING btree ("day","cancer_id","phase");--> statement-breakpoint
909 +CREATE UNIQUE INDEX "literature_counts_uq" ON "literature_counts" USING btree ("cancer_id","window_key");--> statement-breakpoint
910 +CREATE INDEX "literature_counts_window_idx" ON "literature_counts" USING btree ("window_key","count");--> statement-breakpoint
911 +CREATE UNIQUE INDEX "pub_entity_edges_uq" ON "publication_entity_edges" USING btree ("publication_id","entity_type","entity_id","method");--> statement-breakpoint
912 +CREATE INDEX "pub_entity_edges_entity_idx" ON "publication_entity_edges" USING btree ("entity_type","entity_id");--> statement-breakpoint
913 +CREATE UNIQUE INDEX "publications_pmid_uq" ON "publications" USING btree ("pmid");--> statement-breakpoint
914 +CREATE INDEX "publications_doi_idx" ON "publications" USING btree ("doi");--> statement-breakpoint
915 +CREATE INDEX "publications_year_idx" ON "publications" USING btree ("pub_year");--> statement-breakpoint
916 +CREATE INDEX "publications_retracted_idx" ON "publications" USING btree ("retracted");--> statement-breakpoint
917 +CREATE UNIQUE INDEX "civic_evidence_uq" ON "civic_evidence_items" USING btree ("civic_id");--> statement-breakpoint
918 +CREATE INDEX "civic_evidence_cancer_idx" ON "civic_evidence_items" USING btree ("cancer_id");--> statement-breakpoint
919 +CREATE INDEX "civic_evidence_gene_idx" ON "civic_evidence_items" USING btree ("gene_symbols");--> statement-breakpoint
920 +CREATE UNIQUE INDEX "knowledge_edges_uq" ON "knowledge_edges" USING btree ("source_entity_type","source_entity_id","target_entity_type","target_entity_id","relationship_type","source_id","source_record_id");--> statement-breakpoint
921 +CREATE INDEX "knowledge_edges_source_idx" ON "knowledge_edges" USING btree ("source_entity_type","source_entity_id","relationship_type");--> statement-breakpoint
922 +CREATE INDEX "knowledge_edges_target_idx" ON "knowledge_edges" USING btree ("target_entity_type","target_entity_id","relationship_type");--> statement-breakpoint
923 +CREATE UNIQUE INDEX "risk_factors_slug_uq" ON "risk_factors" USING btree ("slug");--> statement-breakpoint
924 +CREATE UNIQUE INDEX "epi_obs_uq" ON "epidemiology_observations" USING btree ("cancer_id","geography_id","year","sex","age_group","metric","source_id","site_definition");--> statement-breakpoint
925 +CREATE INDEX "epi_obs_lookup_idx" ON "epidemiology_observations" USING btree ("metric","geography_id","year","sex");--> statement-breakpoint
926 +CREATE INDEX "epi_obs_cancer_idx" ON "epidemiology_observations" USING btree ("cancer_id","metric");--> statement-breakpoint
927 +CREATE INDEX "survival_obs_cancer_idx" ON "survival_observations" USING btree ("cancer_id","survival_type","duration_months");--> statement-breakpoint
928 +CREATE UNIQUE INDEX "ai_answers_uq" ON "ai_answers" USING btree ("kind","question_hash","prompt_version");--> statement-breakpoint
929 +CREATE UNIQUE INDEX "api_keys_hash_uq" ON "api_keys" USING btree ("key_hash");--> statement-breakpoint
930 +CREATE UNIQUE INDEX "entity_counters_uq" ON "entity_counters" USING btree ("entity_type","entity_id");--> statement-breakpoint
931 +CREATE INDEX "entity_counters_trials_idx" ON "entity_counters" USING btree ("entity_type","active_trial_count");--> statement-breakpoint
932 +CREATE UNIQUE INDEX "metric_definitions_slug_uq" ON "metric_definitions" USING btree ("slug");--> statement-breakpoint
933 +CREATE INDEX "ranking_snapshots_lookup_idx" ON "ranking_snapshots" USING btree ("metric_slug","scope_key","is_current");--> statement-breakpoint
934 +CREATE UNIQUE INDEX "rankings_uq" ON "rankings" USING btree ("snapshot_id","cancer_id");--> statement-breakpoint
935 +CREATE INDEX "rankings_cancer_idx" ON "rankings" USING btree ("cancer_id","metric_slug");--> statement-breakpoint
936 +CREATE INDEX "rankings_lookup_idx" ON "rankings" USING btree ("metric_slug","scope_key","rank");
\ No newline at end of file
added packages/database/migrations/meta/0000_snapshot.json +6918 −0
Large diff (6,919 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/migrations/meta/_journal.json +13 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/package.json +37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/client.ts +42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/ids.ts +25 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/index.ts +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/migrate.ts +40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/_common.ts +23 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/cancers.ts +139 −0
Large diff (139 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/drugs.ts +84 −0
Large diff (84 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/epidemiology.ts +61 −0
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/evidence.ts +89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/genomics.ts +184 −0
Large diff (184 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/index.ts +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/publications.ts +68 −0
Large diff (68 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/rankings.ts +138 −0
Large diff (138 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/sources.ts +228 −0
Large diff (228 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/schema/trials.ts +115 −0
Large diff (115 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/seed-data/geographies.ts +101 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/seed-data/metrics.ts +285 −0
Large diff (285 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/src/seed.ts +49 −0
Large diff (49 lines) is not loaded by default to keep the page fast.Load diff
added packages/database/tsconfig.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added packages/ontology/package.json +27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added packages/ontology/src/index.ts +3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added packages/ontology/src/qualifiers.ts +76 −0
Large diff (76 lines) is not loaded by default to keep the page fast.Load diff
added packages/ontology/src/reconcile.ts +106 −0
Large diff (106 lines) is not loaded by default to keep the page fast.Load diff
added packages/ontology/src/top-level.ts +75 −0
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
added packages/ontology/test/qualifiers.test.ts +55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added packages/ontology/tsconfig.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added packages/ontology/vitest.config.ts +3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added packages/ranking/package.json +28 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
added packages/ranking/src/counters.ts +140 −0
Large diff (140 lines) is not loaded by default to keep the page fast.Load diff
added packages/ranking/src/engine.ts +173 −0
Large diff (173 lines) is not loaded by default to keep the page fast.Load diff
added packages/ranking/src/index.ts +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added packages/ranking/src/rank.ts +50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added packages/ranking/src/trace.ts +51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added packages/ranking/test/rank.test.ts +33 −0
Large diff (33 lines) is not loaded by default to keep the page fast.Load diff
added packages/ranking/tsconfig.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added packages/ranking/vitest.config.ts +3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/package.json +26 −0
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/env.ts +16 −0
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/ids.ts +70 −0
Large diff (70 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/index.ts +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/logger.ts +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/normalize.ts +40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/src/provenance.ts +85 −0
Large diff (85 lines) is not loaded by default to keep the page fast.Load diff
added packages/shared/tsconfig.json +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added pnpm-lock.yaml +2023 −0
Large diff (2,023 lines) is not loaded by default to keep the page fast.Load diff
added pnpm-workspace.yaml +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added scripts/ci.ts +136 −0
Large diff (136 lines) is not loaded by default to keep the page fast.Load diff
added tsconfig.base.json +22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added vitest.workspace.ts +1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere