Foundation: monorepo, schema, provenance SDK, ontology resolver, ranking engine, OncoTree connector

Simon-Pierre Boucher committed 2 mo ago (Sep 8, 2026) Browse files

74 changed files +13,851 −0

added .env.example +34 −0

++# CancerIndex — environment (never commit real values)
++NODE_ENV=development
++DATABASE_URL=postgres://localhost:5432/cancerindex
++DB_POOL_MAX=10
++
++WEB_PORT=8250
++API_PORT=8251
++API_HOST=127.0.0.1
++CI_API_URL=http://127.0.0.1:8251
++NEXT_PUBLIC_SITE_URL=http://localhost:8250
++NEXT_TELEMETRY_DISABLED=1
++
++# Data lake (raw connector payloads, gzip JSON) — outside git
++CI_DATA_DIR=./data
++
++# Admin console (/admin) bearer token
++ADMIN_TOKEN=change-me
++
++# NCBI E-utilities (PubMed, ClinVar) — tool/email are mandatory, api key raises the limit to 10 req/s
++NCBI_TOOL=cancerindex
++NCBI_EMAIL=
++NCBI_API_KEY=
++
++# SEER API (free key from https://api.seer.cancer.gov/) — connector stays "awaiting_credentials" without it
++SEER_API_KEY=
++
++# Optional AI provider for /ask (Phase 5) — the database works without it
++ANTHROPIC_API_KEY=
++OPENAI_API_KEY=
++OPENAI_BASE_URL=
++
++WORKER_CONCURRENCY=4
++CI_MAX_RUN_MINUTES=45
++LOG_LEVEL=info

added .gitignore +15 −0

++node_modules/
++.next/
++dist/
++*.tsbuildinfo
++.env
++.env.*
++!.env.example
++data/raw/
++data/cache/
++data/tmp/
++logs/
++tmp/
++coverage/
++.DS_Store
++.claude/

added .npmrc +2 −0

		`@@ -0,0 +1,2 @@`
1	+auto-install-peers=true
2	+strict-peer-dependencies=false

added CLAUDE.md +82 −0

++# CLAUDE.md — CancerIndex.io (repository guide)
++
++The full product specification (364 sections) lives in `docs/SPEC-original.md`. This file is the
++condensed, operational version every contributor (human or agent) must follow.
++
++## Mission
++Build **the global index of cancer**: a provenance-first, continuously updated, transparently
++sourced oncology ontology connecting epidemiology, genomics, biomarkers, therapies, trials,
++regulatory evidence and literature — for every recognized malignant disease entity, not a list of
++30 common cancers. Not a physician, not a diagnostic tool, no treatment recommendations.
++
++## Non-negotiables (from the spec)
++1. **Provenance first** — no scientific number without a `provenance` row; derived values carry a
++   `formula_version` and their inputs (§2, §250-253). Raw payloads go to the data lake (§26).
++2. **Layers stay separable**: RAW → NORMALIZED → CANONICAL → DERIVED → RANKED → AI (§2).
++3. **Never fake data**: no mock/hardcoded statistics in UI; show "Data not yet available" (§281-283).
++4. **Identifiers are first-class**: keep every upstream ID in `*_codes` / dedicated columns (§347);
++   public IDs are `CI-<NS>-00000001` minted via `mintId` (§6), never DB integers.
++5. **Reconciliation before ingestion**: IDs → curated aliases → normalized strings; LLM only as a
++   *candidate* generator; unknown labels go to `unresolved_labels`, never dropped (§69, §222, §340).
++6. **Time-aware observations**: never overwrite year X with year Y; store both (§71).
++7. **Context on every edge**: cancer context, direction, evidence level, provenance (§29, §244).
++8. **Licensing gate**: no connector goes live before manifest `licenseStatus` is reviewed; IARC/
++   GLOBOCAN stays `review`, SEER `awaiting_credentials` until keys/terms are settled (§10.4, §142).
++9. **Verify docs before coding a connector** — official docs, smoke test with curl, fixture, tests,
++   prove on 10/100/1000 records before a full sync (§224-227). Record `documentationVerifiedAt`.
++10. **Rate limits, idempotency, restartability, anomaly guard** (never mass-delete on a shrunken
++    response) (§90-92, §171, §229).
++11. **Rankings need scope + year + source + formula version + "Why this rank?"** (§33, §183).
++    Composite scores are Phase 2+ (§353).
++12. **Scientific safety labels**: observed / published / curated / regulatory / guideline /
++    computed / AI-generated — never merged (§3). Population survival ≠ individual prognosis (§325).
++
++## Repository layout
++```
++apps/web        Next.js 16 (webpack build, --webpack), Tailwind v4, server components, /api/v1 proxied to apps/api
++apps/api        Fastify /v1 public API (JSON envelope { data, sources, dataRelease })
++workers/        pg-boss scheduler: connector schedules, counters, rankings
++packages/shared     ids, provenance types, normalization, logger
++packages/database   Drizzle schema (snake_case casing), migrations, seed (metrics + geographies only)
++packages/ontology   qualifier rules, CancerResolver (alias/code reconciliation), TOP_LEVEL_CANCERS
++packages/connectors SDK (manifest, HttpClient, RawLake, RunContext, validators) + connectors/<id>/{manifest.ts,index.ts,fixtures/,*.test.ts}
++packages/ranking    counters (entity_counters), ranking engine (snapshots + lineage), trace
++scripts/ci.ts       operator CLI: `pnpm cix connectors|run <id>|run-all|health|sources:sync|counters|rank|stats|trace`
++docs/               architecture, data model, methodology, source policy, ADRs, connector docs
++data/raw            data lake (gitignored) — {source}/{date}/{entity}/{runId}-{part}.jsonl.gz
++```
++
++## Conventions
++- TypeScript strict, ESM, NodeNext imports with `.js` suffix inside packages; web app uses
++  `moduleResolution: bundler` + webpack `extensionAlias` (never import `@cancerindex/connectors` in web).
++- Database: `getDb()` singleton; Drizzle `casing: 'snake_case'`; raw SQL via `sql` template for
++  aggregates. Schema changes = new migration via `pnpm db:generate` (one per integration), never
++  manual prod mutation. Agents developing in parallel use `drizzle-kit push` on their own DB and do
++  **not** commit migration files; the integrator generates the migration.
++- Connector = `class extends Connector` with `manifest`, `healthCheck(ctx)`, `sync(ctx)`; register
++  in `packages/connectors/src/registry.ts`. Use `ctx.http` (rate-limited), `ctx.upsertSourceRecord`
++  (idempotent + raw lake), `ctx.addProvenance`, `ctx.recordUnresolved`, `ctx.cursor` (restart),
++  `ctx.shouldStop()` between pages (time budget), validators from `sdk/validate.ts`.
++- Reconciliation: `new CancerResolver(db)` → `warm()` → `byCode()/byLabel()/resolve()`; store
++  `match_type` on every mapping; enqueue misses with `ctx.recordUnresolved`.
++- Tests: vitest; connector tests run against **fixtures** (sanitized JSON in `fixtures/`), never
++  live APIs; cover normal / empty / pagination / rate-limit / server error / malformed record.
++- UI: scientific-editorial aesthetic (off-white, charcoal, muted neutrals, dense tables, no
++  gradients/cards-everywhere), WCAG-minded, mobile first-class. Every number shows source badge,
++  unit, population, period and a freshness line. Empty state = "Data not yet available".
++- Language: English UI; code comments English; commit messages English.
++
++## Running locally
++```
++createdb cancerindex && cp .env.example .env
++pnpm install && pnpm db:migrate && pnpm db:seed && pnpm cix sources:sync
++pnpm cix run oncotree --mode dry_run   # smoke; then without --mode
++pnpm cix counters && pnpm cix rank
++pnpm dev:api & pnpm dev:web
++```
++
++## Deployment
++MacLustr cluster via `mld` (gateway M1M32): manifest `~/dispatch/apps/cancerindex.json`,
++`mld stage <dir> cancerindex` then `mld deploy cancerindex`. Web :8250, API :8251, PM2 processes
++`cancerindex-web|api|worker|ngrok`, domain www.cancerindex.io (ngrok reserved). Postgres 17 +
++pgvector on the node, database `cancerindex`. Source of truth for the code: spbgit `cancerindex.git`.

added package.json +40 −0

++{
++  "name": "cancerindex",
++  "version": "0.1.0",
++  "private": true,
++  "description": "CancerIndex.io — the global index of cancer: provenance-first oncology ontology, epidemiology, genomics, trials, literature, rankings",
++  "type": "module",
++  "packageManager": "pnpm@11.1.2",
++  "engines": {
++    "node": ">=22"
++  },
++  "scripts": {
++    "build": "pnpm -r --filter './packages/**' run build && pnpm --filter @cancerindex/api run build && pnpm --filter @cancerindex/web run build",
++    "typecheck": "pnpm -r run typecheck",
++    "test": "pnpm -r run test",
++    "db:generate": "pnpm --filter @cancerindex/database run generate",
++    "db:migrate": "pnpm --filter @cancerindex/database run migrate",
++    "db:seed": "pnpm --filter @cancerindex/database run seed",
++    "dev:web": "pnpm --filter @cancerindex/web run dev",
++    "dev:api": "pnpm --filter @cancerindex/api run dev",
++    "worker": "tsx workers/main.ts",
++    "cix": "tsx scripts/ci.ts",
++    "registry": "tsx scripts/build-registry.ts",
++    "registry:check": "tsx scripts/build-registry.ts --check"
++  },
++  "dependencies": {
++    "@cancerindex/connectors": "workspace:*",
++    "@cancerindex/database": "workspace:*",
++    "@cancerindex/ontology": "workspace:*",
++    "@cancerindex/ranking": "workspace:*",
++    "@cancerindex/shared": "workspace:*",
++    "drizzle-orm": "^0.45.0",
++    "postgres": "^3.4.7"
++  },
++  "devDependencies": {
++    "@types/node": "^24.0.0",
++    "tsx": "^4.20.0",
++    "typescript": "^5.9.3",
++    "vitest": "^3.2.0"
++  }
++}

added packages/connectors/package.json +36 −0

++{
++  "name": "@cancerindex/connectors",
++  "version": "0.1.0",
++  "private": true,
++  "type": "module",
++  "exports": {
++    ".": {
++      "types": "./src/index.ts",
++      "default": "./src/index.ts"
++    },
++    "./sdk": {
++      "types": "./src/sdk/index.ts",
++      "default": "./src/sdk/index.ts"
++    }
++  },
++  "scripts": {
++    "build": "tsc -p tsconfig.json --noEmit",
++    "typecheck": "tsc -p tsconfig.json --noEmit",
++    "test": "vitest run --passWithNoTests"
++  },
++  "dependencies": {
++    "@cancerindex/database": "workspace:*",
++    "@cancerindex/ontology": "workspace:*",
++    "@cancerindex/shared": "workspace:*",
++    "drizzle-orm": "^0.45.0",
++    "fast-xml-parser": "^5.2.0",
++    "postgres": "^3.4.7",
++    "yaml": "^2.8.0",
++    "zod": "^4.0.0"
++  },
++  "devDependencies": {
++    "@types/node": "^24.0.0",
++    "typescript": "^5.9.3",
++    "vitest": "^3.2.0"
++  }
++}

added packages/connectors/src/connectors/oncotree/index.ts +185 −0

++import { eq, sql } from 'drizzle-orm';
++import { z } from 'zod';
++import { normalizeLabel, slugify } from '@cancerindex/shared';
++import { anatomicalSites, cancerAliases, cancerAnatomy, cancerCodes, cancerHierarchy, cancers, mintId } from '@cancerindex/database';
++import { CancerResolver } from '@cancerindex/ontology';
++import { Connector, type ConnectorHealth, type RunContext } from '../../sdk/run.js';
++import { manifest } from './manifest.js';
++
++const TumorType = z.object({
++  code: z.string(),
++  name: z.string(),
++  mainType: z.string().nullable().optional(),
++  tissue: z.string().nullable().optional(),
++  parent: z.string().nullable().optional(),
++  level: z.number().int(),
++  externalReferences: z.record(z.string(), z.array(z.string())).default({}),
++  history: z.array(z.string()).default([]),
++  revocations: z.array(z.string()).default([]),
++  precursors: z.array(z.string()).default([]),
++  color: z.string().nullable().optional(),
++});
++type TumorType = z.infer<typeof TumorType>;
++
++const HEME_RE = /lymphoid|myeloid|blood|leukemia|lymphoma|myeloma|histiocyt|mastocyt|plasma cell/i;
++const BENIGN_RE = /\bbenign\b|borderline|\bnevus\b|\bnevi\b|\badenoma\b|hyperplasia|\bcyst\b|hamartoma|\blipoma\b|leiomyoma\b|\bfibroma\b|hemangioma\b|papilloma\b|dysplasia|in situ|\bpolyp\b|schwannoma\b|meningioma\b|paraganglioma\b|pheochromocytoma\b|teratoma\b|thymoma\b/i;
++
++/**
++ * OncoTree connector: maps every OncoTree node onto the canonical cancer set.
++ *  - level-1 "tissue" nodes (parent TISSUE) are anatomical sites, not cancers → anatomical_sites
++ *  - NCIt cross-reference → EXACT_IDENTIFIER match to the NCIt-derived canonical entity
++ *  - otherwise preferred-name match → ONTOLOGY_EXACT
++ *  - otherwise a new canonical entity is created *only* because OncoTree is a recognized ontology
++ *    (CLAUDE.md §340) — flagged with primaryOncotreeCode and no NCIt code.
++ * The OncoTree parent/child structure is stored as hierarchy_type = "oncotree" (levels ≥ 2), and
++ * level-2 nodes are linked to their tissue via cancer_anatomy.
++ */
++export class OncotreeConnector extends Connector {
++  readonly manifest = manifest;
++
++  async healthCheck(ctx: RunContext): Promise<ConnectorHealth> {
++    try {
++      const versions = await ctx.http.json<Array<{ api_identifier: string; release_date: string }>>(`${manifest.access.baseUrl}/versions`);
++      const stable = versions.find((v) => v.api_identifier === 'oncotree_latest_stable');
++      return { status: 'healthy', detail: `latest stable ${stable?.release_date ?? '?'}` };
++    } catch (e) {
++      return { status: 'failing', detail: (e as Error).message };
++    }
++  }
++
++  async sync(ctx: RunContext): Promise<void> {
++    const versions = await ctx.http.json<Array<{ api_identifier: string; release_date: string; visible: boolean }>>(`${manifest.access.baseUrl}/versions`);
++    const stable = versions.find((v) => v.api_identifier === 'oncotree_latest_stable');
++    // Pin to the dated release identifier so the dataset version is reproducible.
++    const dated = versions.find((v) => v.release_date === stable?.release_date && v.api_identifier !== 'oncotree_latest_stable');
++    const version = dated?.api_identifier ?? 'oncotree_latest_stable';
++    ctx.datasetVersion = version;
++    ctx.info(`OncoTree version ${version}`);
++
++    const raw = await ctx.http.json<unknown[]>(`${manifest.access.baseUrl}/tumorTypes?version=${encodeURIComponent(version)}`);
++    const nodes: TumorType[] = [];
++    for (const r of raw) {
++      const parsed = TumorType.safeParse(r);
++      if (!parsed.success) {
++        ctx.counters.validationFailures++;
++        ctx.counters.rejected++;
++        ctx.warn(`invalid tumor type record: ${parsed.error.issues[0]?.message}`);
++        continue;
++      }
++      nodes.push(parsed.data);
++    }
++    ctx.info(`fetched ${nodes.length} tumor types`);
++    if (nodes.length < 500) throw new Error(`anomaly: only ${nodes.length} OncoTree nodes returned (expected ~900) — refusing to persist (CLAUDE.md §171)`);
++    if (ctx.mode === 'dry_run') {
++      ctx.counters.fetched = nodes.length;
++      return;
++    }
++
++    const resolver = new CancerResolver(ctx.db);
++    await resolver.warm();
++    const byCode = new Map(nodes.map((n) => [n.code, n]));
++    const canonicalByOncotree = new Map<string, string>();
++    const siteByOncotree = new Map<string, string>();
++    await ctx.addProvenance({ dataset: 'OncoTree tumor types', datasetVersion: version, sourceUrl: `${manifest.access.baseUrl}/tumorTypes?version=${version}`, evidenceType: 'expert_curation', accessLevel: 'open', methodology: 'OncoTree curated tumor type hierarchy' });
++
++    // Pass 0: tissue roots → anatomical sites
++    for (const node of nodes.filter((n) => n.parent === 'TISSUE')) {
++      await ctx.upsertSourceRecord('tissue', node.code, node);
++      const slug = slugify(node.name);
++      const ncit = node.externalReferences['NCI']?.[0] ?? null;
++      const [existing] = await ctx.db.select({ id: anatomicalSites.id }).from(anatomicalSites).where(eq(anatomicalSites.slug, slug)).limit(1);
++      let siteId = existing?.id;
++      if (!siteId) {
++        siteId = await mintId(ctx.db, 'ANAT');
++        await ctx.db.insert(anatomicalSites).values({ id: siteId, name: node.name, slug, ncitCode: ncit, system: node.tissue ?? node.name });
++      } else await ctx.db.update(anatomicalSites).set({ name: node.name, ncitCode: ncit }).where(eq(anatomicalSites.id, siteId));
++      siteByOncotree.set(node.code, siteId);
++    }
++
++    // Pass 1: resolve / create canonical entities (parents before children thanks to level ordering).
++    for (const node of [...nodes].filter((n) => n.parent !== 'TISSUE').sort((a, b) => a.level - b.level)) {
++      const rec = await ctx.upsertSourceRecord('tumor_type', node.code, node);
++      let cancerId: string | null = null;
++      let matchType = 'UNRESOLVED';
++      const ncitCodes = node.externalReferences['NCI'] ?? [];
++      for (const c of ncitCodes) {
++        const hit = resolver.byCode('ncit', c);
++        if (hit) {
++          cancerId = hit.cancerId;
++          matchType = hit.matchType;
++          break;
++        }
++      }
++      if (!cancerId) {
++        const hit = resolver.byCode('oncotree', node.code);
++        if (hit) {
++          cancerId = hit.cancerId;
++          matchType = hit.matchType;
++        }
++      }
++      if (!cancerId) {
++        const hit = resolver.byLabel(node.name);
++        if (hit && hit.matchType === 'ONTOLOGY_EXACT') {
++          cancerId = hit.cancerId;
++          matchType = hit.matchType;
++        }
++      }
++      if (!cancerId) {
++        // Recognized ontology node without NCIt anchor → new canonical entity (CLAUDE.md §340 satisfied: OncoTree is a recognized ontology).
++        const slugBase = slugify(node.name);
++        let slug = slugBase;
++        for (let i = 2; ; i++) {
++          const [clash] = await ctx.db.select({ id: cancers.id }).from(cancers).where(eq(cancers.slug, slug)).limit(1);
++          if (!clash) break;
++          slug = `${slugBase}-${i}`;
++        }
++        const heme = HEME_RE.test(`${node.tissue ?? ''} ${node.mainType ?? ''} ${node.name}`);
++        cancerId = await mintId(ctx.db, 'CAN');
++        await ctx.db.insert(cancers).values({
++          id: cancerId,
++          slug,
++          canonicalName: node.name,
++          entityType: node.level >= 3 ? 'subtype' : heme ? 'hematologic_malignancy' : 'cancer',
++          malignant: !BENIGN_RE.test(node.name),
++          solidTumor: !heme,
++          hematologic: heme,
++          primaryOncotreeCode: node.code,
++          depth: node.level,
++          classificationVersion: version,
++        });
++        await ctx.db.insert(cancerAliases).values({ cancerId, alias: node.name, normalized: normalizeLabel(node.name), aliasType: 'preferred', sourceId: ctx.sourceId, sourceTerminology: 'OncoTree' }).onConflictDoNothing();
++        matchType = 'EXACT_IDENTIFIER';
++        await ctx.recordChange('cancer', cancerId, 'created', `Created from OncoTree ${node.code} (${node.name})`);
++      } else {
++        await ctx.db.insert(cancerAliases).values({ cancerId, alias: node.name, normalized: normalizeLabel(node.name), aliasType: 'synonym', sourceId: ctx.sourceId, sourceTerminology: 'OncoTree' }).onConflictDoNothing();
++      }
++      canonicalByOncotree.set(node.code, cancerId);
++      await ctx.db.insert(cancerCodes).values({ cancerId, system: 'oncotree', code: node.code, matchType, sourceId: ctx.sourceId, validFrom: stable?.release_date }).onConflictDoNothing();
++      for (const u of node.externalReferences['UMLS'] ?? []) await ctx.db.insert(cancerCodes).values({ cancerId, system: 'umls', code: u, matchType: 'EXACT_IDENTIFIER', sourceId: ctx.sourceId }).onConflictDoNothing();
++      for (const c of ncitCodes) await ctx.db.insert(cancerCodes).values({ cancerId, system: 'ncit', code: c, matchType: 'EXACT_IDENTIFIER', sourceId: ctx.sourceId }).onConflictDoNothing();
++      // Record the OncoTree code on the entity only when it has none yet (first mapping wins; others stay in cancer_codes).
++      await ctx.db.execute(sql`UPDATE cancers SET primary_oncotree_code = ${node.code} WHERE id = ${cancerId} AND primary_oncotree_code IS NULL`);
++      // Level-2 nodes hang directly under a tissue → anatomical relation.
++      if (node.parent && siteByOncotree.has(node.parent)) {
++        await ctx.db.insert(cancerAnatomy).values({ cancerId, siteId: siteByOncotree.get(node.parent)!, relation: 'primary', sourceId: ctx.sourceId }).onConflictDoNothing();
++      }
++      void rec;
++    }
++
++    // Pass 2: hierarchy edges (oncotree dimension), levels ≥ 2 only.
++    let edges = 0;
++    for (const node of nodes) {
++      if (!node.parent || node.parent === 'TISSUE' || siteByOncotree.has(node.parent)) continue;
++      const childId = canonicalByOncotree.get(node.code);
++      const parentId = canonicalByOncotree.get(node.parent);
++      if (!childId || !parentId || childId === parentId) continue;
++      if (!byCode.has(node.parent)) continue;
++      await ctx.db.insert(cancerHierarchy).values({ parentId, childId, hierarchyType: 'oncotree', sourceId: ctx.sourceId }).onConflictDoNothing();
++      edges++;
++    }
++    ctx.info(`tissues ${siteByOncotree.size}; canonical mapped ${canonicalByOncotree.size}; hierarchy edges ${edges}`);
++    ctx.cursor = { version, syncedAt: new Date().toISOString() };
++  }
++}
++
++export const connector = new OncotreeConnector();

added packages/connectors/src/connectors/oncotree/manifest.ts +34 −0

++import { defineManifest } from '../../sdk/manifest.js';
++
++/**
++ * Docs verified 2026-09-08: https://oncotree.mskcc.org (API tab) — GET /api/tumorTypes?version=…,
++ * GET /api/versions, GET /api/mainTypes. Response fields: code, name, mainType, tissue, parent,
++ * level, externalReferences{UMLS[],NCI[]}, history[], revocations[], precursors[], color.
++ * License (repo README, cBioPortal/oncotree): CC BY 4.0.
++ */
++export const manifest = defineManifest({
++  id: 'oncotree',
++  name: 'OncoTree',
++  organization: 'Memorial Sloan Kettering Cancer Center / cBioPortal',
++  category: 'terminology',
++  tier: 0,
++  description: 'Open cancer classification used by cBioPortal and clinical sequencing programs. Provides a tissue-based tumor type hierarchy with NCIt and UMLS cross-references. Used as a second hierarchy dimension and to anchor cohort/genomic sources.',
++  homepage: 'https://oncotree.mskcc.org',
++  docsUrl: 'https://oncotree.mskcc.org/#/home?tab=api',
++  termsUrl: 'https://github.com/cBioPortal/oncotree/blob/master/LICENSE',
++  access: { type: 'api', auth: 'none', baseUrl: 'https://oncotree.mskcc.org/api' },
++  license: 'CC BY 4.0',
++  licenseStatus: 'approved',
++  commercialUse: 'allowed',
++  redistribution: 'attribution',
++  attribution: 'OncoTree (Memorial Sloan Kettering Cancer Center), CC BY 4.0, https://oncotree.mskcc.org',
++  termsReviewedAt: '2026-09-08',
++  updateFrequency: 'Stable releases a few times per year',
++  supportsIncrementalSync: false,
++  entities: ['cancer', 'cancer_hierarchy', 'cancer_codes'],
++  rateLimits: { requestsPerSecond: 2, maxConcurrency: 1 },
++  rawRetention: 'full',
++  documentationVerifiedAt: '2026-09-08',
++  status: 'active',
++  schedule: '0 4 * * 1',
++});

added packages/connectors/src/index.ts +4 −0

++export * from './sdk/index.js';
++export * from './registry.js';
++export * from './sources-sync.js';
++export * from './planned.js';

added packages/connectors/src/planned.ts +60 −0

++import { defineManifest, type ConnectorManifest } from './sdk/manifest.js';
++
++/**
++ * Sources that are registered (visible on /sources with their compliance status) but have no
++ * ingesting connector yet — either because license review is pending (CLAUDE.md §10.4, §142) or
++ * because credentials are required. Nothing here ingests data.
++ */
++export const PLANNED_MANIFESTS: ConnectorManifest[] = [
++  defineManifest({
++    id: 'iarc-globocan',
++    name: 'IARC Global Cancer Observatory (GLOBOCAN)',
++    organization: 'International Agency for Research on Cancer (WHO)',
++    category: 'epidemiology',
++    tier: 0,
++    description: 'Global estimates of cancer incidence, mortality and prevalence by country, sex, age and cancer site. The intended global epidemiology layer of CancerIndex.',
++    homepage: 'https://gco.iarc.who.int/',
++    docsUrl: 'https://gco.iarc.who.int/today/en/about',
++    termsUrl: 'https://gco.iarc.who.int/en/terms-of-use',
++    access: { type: 'api', auth: 'none', baseUrl: 'https://gco.iarc.who.int/' },
++    license: 'IARC/WHO terms of use — redistribution rights under review',
++    licenseStatus: 'review',
++    commercialUse: 'unknown',
++    redistribution: 'unknown',
++    attribution: 'Ferlay J, Ervik M, Lam F, et al. Global Cancer Observatory: Cancer Today. Lyon, France: IARC.',
++    termsNotes:
++      'Terms page is a JavaScript application; automated review on 2026-09-08 could not extract the text. Per CLAUDE.md §10.4 no automated bulk ingestion or republication of GLOBOCAN estimates until a human has reviewed the terms and, if required, obtained permission from IARC. Country/global rankings therefore remain unavailable and are shown as "awaiting license review".',
++    updateFrequency: 'Major releases every ~2 years (GLOBOCAN 2022 current)',
++    supportsIncrementalSync: false,
++    entities: ['epidemiology_observations'],
++    metrics: ['incidence_count', 'mortality_count', 'as_incidence_rate', 'as_mortality_rate', 'prevalence_5y'],
++    rateLimits: { requestsPerSecond: 1, maxConcurrency: 1 },
++    documentationVerifiedAt: '2026-09-08',
++    status: 'review',
++  }),
++  defineManifest({
++    id: 'seer',
++    name: 'SEER (Surveillance, Epidemiology, and End Results)',
++    organization: 'National Cancer Institute',
++    category: 'epidemiology',
++    tier: 0,
++    description: 'US population-based registry statistics: incidence, mortality, survival, stage, age, sex, race/ethnicity. Authoritative for US relative survival.',
++    homepage: 'https://seer.cancer.gov/',
++    docsUrl: 'https://api.seer.cancer.gov/',
++    termsUrl: 'https://seer.cancer.gov/about/terms.html',
++    access: { type: 'api', auth: 'api_key', baseUrl: 'https://api.seer.cancer.gov/rest/' },
++    license: 'US Government work (public domain); SEER*Explorer statistics citable with attribution; API requires a free key',
++    licenseStatus: 'review',
++    commercialUse: 'unknown',
++    redistribution: 'attribution',
++    attribution: 'Surveillance, Epidemiology, and End Results (SEER) Program (www.seer.cancer.gov), National Cancer Institute.',
++    termsNotes: 'Connector is implemented against the SEER API only once a key is present (SEER_API_KEY). Status awaiting_credentials until then. Never present SEER (US) estimates as global.',
++    updateFrequency: 'Annual (April)',
++    supportsIncrementalSync: false,
++    entities: ['epidemiology_observations', 'survival_observations'],
++    metrics: ['incidence_count', 'as_incidence_rate', 'mortality_count', 'as_mortality_rate', 'five_year_survival'],
++    rateLimits: { requestsPerSecond: 2, maxConcurrency: 1 },
++    documentationVerifiedAt: '2026-09-08',
++    status: 'awaiting_credentials',
++  }),
++];

added packages/connectors/src/registry.ts +13 −0

++import type { Connector } from './sdk/run.js';
++import { connector as oncotree } from './connectors/oncotree/index.js';
++
++/**
++ * Connector registry. Order = recommended first-run order (CLAUDE.md §351): terminology first,
++ * then genes, then genomics/trials/literature/variants/evidence, then epidemiology.
++ * Add new connectors here; `pnpm cix sources:sync` seeds their manifests into `sources`.
++ */
++export const CONNECTORS: Connector[] = [oncotree];
++
++export function getConnector(id: string): Connector | undefined {
++  return CONNECTORS.find((c) => c.manifest.id === id);
++}

added packages/connectors/src/sdk/http.ts +162 −0

++import { setTimeout as sleep } from 'node:timers/promises';
++import type { ConnectorManifest } from './manifest.js';
++
++export interface HttpStats {
++  requests: number;
++  failures: number;
++  rateLimitEvents: number;
++  retries: number;
++}
++
++export class HttpError extends Error {
++  constructor(
++    public readonly status: number,
++    public readonly url: string,
++    public readonly bodySnippet: string,
++  ) {
++    super(`HTTP ${status} for ${url}: ${bodySnippet.slice(0, 200)}`);
++  }
++}
++
++/** Token bucket rate limiter (CLAUDE.md §229). */
++class TokenBucket {
++  private tokens: number;
++  private last = Date.now();
++  constructor(
++    private readonly ratePerSec: number,
++    private readonly burst: number,
++  ) {
++    this.tokens = burst;
++  }
++  async take(): Promise<void> {
++    for (;;) {
++      const now = Date.now();
++      this.tokens = Math.min(this.burst, this.tokens + ((now - this.last) / 1000) * this.ratePerSec);
++      this.last = now;
++      if (this.tokens >= 1) {
++        this.tokens -= 1;
++        return;
++      }
++      await sleep(Math.ceil(((1 - this.tokens) / this.ratePerSec) * 1000));
++    }
++  }
++}
++
++class Semaphore {
++  private queue: Array<() => void> = [];
++  private active = 0;
++  constructor(private readonly max: number) {}
++  async acquire(): Promise<() => void> {
++    if (this.active >= this.max) await new Promise<void>((resolve) => this.queue.push(resolve));
++    this.active++;
++    let released = false;
++    return () => {
++      if (released) return;
++      released = true;
++      this.active--;
++      const next = this.queue.shift();
++      if (next) next();
++    };
++  }
++}
++
++export interface HttpClientOptions {
++  userAgent?: string;
++  timeoutMs?: number;
++  headers?: Record<string, string>;
++}
++
++/**
++ * Source-aware HTTP client: token bucket, bounded concurrency, exponential backoff with jitter,
++ * Retry-After respect, request accounting for the ingest run.
++ */
++export class HttpClient {
++  readonly stats: HttpStats = { requests: 0, failures: 0, rateLimitEvents: 0, retries: 0 };
++  private readonly bucket: TokenBucket | null;
++  private readonly sem: Semaphore;
++  private readonly retry: NonNullable<ConnectorManifest['retryPolicy']>;
++  private readonly ua: string;
++  private readonly timeoutMs: number;
++  private readonly headers: Record<string, string>;
++
++  constructor(manifest: ConnectorManifest, opts: HttpClientOptions = {}) {
++    const rps = manifest.rateLimits.requestsPerSecond ?? (manifest.rateLimits.requestsPerMinute ? manifest.rateLimits.requestsPerMinute / 60 : undefined);
++    this.bucket = rps ? new TokenBucket(rps, Math.max(1, Math.ceil(rps))) : null;
++    this.sem = new Semaphore(manifest.rateLimits.maxConcurrency);
++    this.retry = manifest.retryPolicy;
++    // Plain product token only: some WAFs (e.g. oncotree.mskcc.org, Akamai) reject UAs containing URLs/parentheses.
++    this.ua = opts.userAgent ?? `CancerIndex/0.1`;
++    this.timeoutMs = opts.timeoutMs ?? 60_000;
++    this.headers = opts.headers ?? {};
++  }
++
++  /** One attempt: returns the Response (any status) or throws on network error. */
++  private async attempt(url: string, init: RequestInit): Promise<Response> {
++    const release = await this.sem.acquire();
++    try {
++      if (this.bucket) await this.bucket.take();
++      this.stats.requests++;
++      const controller = new AbortController();
++      const timer = setTimeout(() => controller.abort(), this.timeoutMs);
++      try {
++        return await fetch(url, {
++          ...init,
++          headers: { 'user-agent': this.ua, accept: 'application/json, text/plain, */*', ...this.headers, ...(init.headers as Record<string, string> | undefined) },
++          signal: controller.signal,
++        });
++      } finally {
++        clearTimeout(timer);
++      }
++    } finally {
++      release();
++    }
++  }
++
++  async request(url: string, init: RequestInit = {}): Promise<Response> {
++    for (let attempt = 0; ; attempt++) {
++      let res: Response;
++      try {
++        res = await this.attempt(url, init);
++      } catch (err) {
++        if (attempt >= this.retry.maxRetries) {
++          this.stats.failures++;
++          throw err;
++        }
++        this.stats.retries++;
++        await sleep(this.backoff(attempt));
++        continue;
++      }
++      if (res.ok) return res;
++      const body = await res.text().catch(() => '');
++      if (res.status === 429) this.stats.rateLimitEvents++;
++      const retryable = res.status === 429 || res.status >= 500 || res.status === 408;
++      if (!retryable || attempt >= this.retry.maxRetries) {
++        this.stats.failures++;
++        throw new HttpError(res.status, url, body);
++      }
++      const retryAfter = Number(res.headers.get('retry-after'));
++      const delay = Number.isFinite(retryAfter) && retryAfter > 0 ? Math.min(retryAfter * 1000, 120_000) : this.backoff(attempt);
++      this.stats.retries++;
++      await sleep(delay);
++    }
++  }
++
++  async json<T = unknown>(url: string, init?: RequestInit): Promise<T> {
++    const res = await this.request(url, init);
++    return (await res.json()) as T;
++  }
++
++  async text(url: string, init?: RequestInit): Promise<string> {
++    const res = await this.request(url, init);
++    return await res.text();
++  }
++
++  async postJson<T = unknown>(url: string, body: unknown, init: RequestInit = {}): Promise<T> {
++    return this.json<T>(url, { ...init, method: 'POST', headers: { 'content-type': 'application/json', ...(init.headers as Record<string, string> | undefined) }, body: JSON.stringify(body) });
++  }
++
++  private backoff(attempt: number): number {
++    const base = Math.min(this.retry.maxDelayMs, this.retry.baseDelayMs * 2 ** attempt);
++    return Math.round(base / 2 + Math.random() * (base / 2));
++  }
++}

added packages/connectors/src/sdk/index.ts +5 −0

++export * from './manifest.js';
++export * from './http.js';
++export * from './lake.js';
++export * from './run.js';
++export * from './validate.js';

added packages/connectors/src/sdk/lake.ts +105 −0

++import { createWriteStream, existsSync, mkdirSync, createReadStream } from 'node:fs';
++import { mkdir, stat } from 'node:fs/promises';
++import path from 'node:path';
++import { createGzip, createGunzip } from 'node:zlib';
++import { pipeline } from 'node:stream/promises';
++import { createInterface } from 'node:readline';
++import { dataDir } from '@cancerindex/shared';
++
++/**
++ * Raw data lake (CLAUDE.md §26): every source payload retained when licensing allows.
++ * Layout: {CI_DATA_DIR}/raw/{source}/{YYYY-MM-DD}/{entity}/{runId}-{part}.jsonl.gz
++ * A record's `rawPath` is "<file>#<line>" so any canonical value can be traced to its raw payload.
++ */
++export class RawLake {
++  private readonly root: string;
++  private writers = new Map<string, { path: string; stream: ReturnType<typeof createWriteStream>; gz: ReturnType<typeof createGzip>; lines: number; part: number; bytes: number }>();
++  constructor(
++    private readonly source: string,
++    private readonly runId: string,
++    private readonly date = new Date(),
++    root?: string,
++  ) {
++    this.root = root ?? path.join(dataDir(), 'raw');
++  }
++
++  dir(entity: string): string {
++    const day = this.date.toISOString().slice(0, 10);
++    return path.join(this.root, this.source, day, entity);
++  }
++
++  /** Append one raw JSON payload; returns its rawPath reference. */
++  async put(entity: string, payload: unknown): Promise<string> {
++    let w = this.writers.get(entity);
++    if (!w || w.bytes > 64 * 1024 * 1024) {
++      if (w) await this.closeWriter(entity);
++      const part = (w?.part ?? 0) + 1;
++      const dir = this.dir(entity);
++      if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
++      const file = path.join(dir, `${this.runId}-${String(part).padStart(3, '0')}.jsonl.gz`);
++      const stream = createWriteStream(file);
++      const gz = createGzip({ level: 6 });
++      gz.pipe(stream);
++      w = { path: file, stream, gz, lines: 0, part, bytes: 0 };
++      this.writers.set(entity, w);
++    }
++    const line = JSON.stringify(payload) + '\n';
++    w.bytes += line.length;
++    const ref = `${w.path}#${w.lines}`;
++    w.lines++;
++    if (!w.gz.write(line)) await new Promise<void>((r) => w!.gz.once('drain', () => r()));
++    return ref;
++  }
++
++  private async closeWriter(entity: string): Promise<void> {
++    const w = this.writers.get(entity);
++    if (!w) return;
++    await new Promise<void>((resolve, reject) => {
++      w.stream.once('finish', () => resolve());
++      w.stream.once('error', reject);
++      w.gz.end();
++    });
++    this.writers.delete(entity);
++  }
++
++  async close(): Promise<void> {
++    for (const entity of [...this.writers.keys()]) await this.closeWriter(entity);
++  }
++
++  /** Read a raw payload back by reference (admin TRACE, reprocessing). */
++  static async read(ref: string): Promise<unknown | null> {
++    const [file, lineStr] = ref.split('#');
++    if (!file || lineStr === undefined) return null;
++    const target = Number(lineStr);
++    const rl = createInterface({ input: createReadStream(file).pipe(createGunzip()) });
++    let i = 0;
++    for await (const line of rl) {
++      if (i === target) {
++        rl.close();
++        return JSON.parse(line);
++      }
++      i++;
++    }
++    return null;
++  }
++}
++
++/** Download a bulk file to the lake (CLAUDE.md §228-230) with checksum + size recorded. */
++export async function downloadBulk(url: string, dest: string, fetchImpl: typeof fetch = fetch, headers: Record<string, string> = {}): Promise<{ path: string; bytes: number; sha256: string }> {
++  await mkdir(path.dirname(dest), { recursive: true });
++  const res = await fetchImpl(url, { headers: { 'user-agent': 'CancerIndex/0.1 (+https://www.cancerindex.io)', ...headers } });
++  if (!res.ok || !res.body) throw new Error(`bulk download failed ${res.status} ${url}`);
++  const { createHash } = await import('node:crypto');
++  const hash = createHash('sha256');
++  const { Transform } = await import('node:stream');
++  const tap = new Transform({
++    transform(chunk, _enc, cb) {
++      hash.update(chunk);
++      cb(null, chunk);
++    },
++  });
++  const { Readable } = await import('node:stream');
++  await pipeline(Readable.fromWeb(res.body as never), tap, createWriteStream(dest));
++  const s = await stat(dest);
++  return { path: dest, bytes: s.size, sha256: hash.digest('hex') };
++}

added packages/connectors/src/sdk/manifest.ts +61 −0

++import { z } from 'zod';
++
++/** Connector manifest (CLAUDE.md §9). Seeds the `sources` table; drives the public /source pages. */
++export const ConnectorManifest = z.object({
++  id: z.string().regex(/^[a-z0-9-]+$/),
++  name: z.string(),
++  organization: z.string(),
++  category: z.enum(['terminology', 'genes', 'genomics', 'variants', 'epidemiology', 'trials', 'literature', 'drugs', 'regulatory', 'evidence', 'pathways', 'funding', 'guidelines']),
++  tier: z.number().int().min(0).max(11),
++  description: z.string(),
++  homepage: z.string().url(),
++  docsUrl: z.string().url().optional(),
++  termsUrl: z.string().url().optional(),
++  access: z.object({
++    type: z.enum(['api', 'bulk', 'rss', 'ftp', 'graphql', 'rest', 'scrape', 'manual']),
++    auth: z.enum(['none', 'api_key', 'oauth', 'account', 'controlled']),
++    baseUrl: z.string().url().optional(),
++  }),
++  license: z.string(),
++  licenseStatus: z.enum(['review', 'approved', 'restricted', 'blocked']),
++  commercialUse: z.enum(['allowed', 'restricted', 'prohibited', 'unknown']),
++  redistribution: z.enum(['allowed', 'attribution', 'restricted', 'prohibited', 'unknown']),
++  attribution: z.string().optional(),
++  termsReviewedAt: z.string().optional(), // YYYY-MM-DD, human review date
++  termsNotes: z.string().optional(),
++  updateFrequency: z.string(),
++  expectedLatency: z.string().optional(),
++  supportsIncrementalSync: z.boolean(),
++  entities: z.array(z.string()),
++  metrics: z.array(z.string()).default([]),
++  rateLimits: z.object({
++    requestsPerSecond: z.number().positive().optional(),
++    requestsPerMinute: z.number().positive().optional(),
++    maxConcurrency: z.number().int().positive().default(1),
++    notes: z.string().optional(),
++  }),
++  retryPolicy: z
++    .object({
++      maxRetries: z.number().int().min(0).default(4),
++      baseDelayMs: z.number().int().min(0).default(500),
++      maxDelayMs: z.number().int().min(0).default(30_000),
++    })
++    .default({ maxRetries: 4, baseDelayMs: 500, maxDelayMs: 30_000 }),
++  rawRetention: z.enum(['full', 'summary', 'none']).default('full'),
++  schemaVersion: z.string().default('1'),
++  documentationVerifiedAt: z.string(), // YYYY-MM-DD, the date official docs were checked (CLAUDE.md §224-225)
++  owner: z.string().default('cancerindex'),
++  status: z.enum(['planned', 'active', 'paused', 'awaiting_credentials', 'review', 'retired']),
++  schedule: z.string().optional(), // cron expression for the worker scheduler
++  anomalyGuard: z
++    .object({
++      /** Refuse to mark records source_missing when the fetched count drops below this ratio of the previous run. */
++      minRatioOfPrevious: z.number().min(0).max(1).default(0.5),
++    })
++    .default({ minRatioOfPrevious: 0.5 }),
++});
++export type ConnectorManifest = z.infer<typeof ConnectorManifest>;
++
++export function defineManifest(m: z.input<typeof ConnectorManifest>): ConnectorManifest {
++  return ConnectorManifest.parse(m);
++}

added packages/connectors/src/sdk/run.ts +0 −0

Binary file not shown.

added packages/connectors/src/sdk/validate.ts +49 −0

++/** Scientific invariants applied to normalized records (CLAUDE.md §110, §154). */
++
++export interface ValidationIssue {
++  field: string;
++  message: string;
++}
++
++export function validateEpidemiology(o: { metric: string; value: number; year: number; lowerCi?: number | null; upperCi?: number | null; unit: string }): ValidationIssue[] {
++  const issues: ValidationIssue[] = [];
++  if (!Number.isFinite(o.value)) issues.push({ field: 'value', message: 'not finite' });
++  if (o.value < 0) issues.push({ field: 'value', message: 'negative count/rate' });
++  if (o.year < 1900 || o.year > new Date().getFullYear() + 30) issues.push({ field: 'year', message: `implausible year ${o.year}` });
++  if (o.lowerCi != null && o.lowerCi > o.value) issues.push({ field: 'lowerCi', message: 'lowerCI > estimate' });
++  if (o.upperCi != null && o.upperCi < o.value) issues.push({ field: 'upperCi', message: 'upperCI < estimate' });
++  if (o.metric.endsWith('_count') && o.unit !== 'count') issues.push({ field: 'unit', message: 'count metric must use unit=count' });
++  if (o.metric.endsWith('_rate') && o.unit !== 'per_100k') issues.push({ field: 'unit', message: 'rate metric must use unit=per_100k' });
++  return issues;
++}
++
++export function validateSurvival(o: { probability?: number | null; durationMonths: number; lowerCi?: number | null; upperCi?: number | null; cohortSize?: number | null }): ValidationIssue[] {
++  const issues: ValidationIssue[] = [];
++  if (o.probability != null && (o.probability < 0 || o.probability > 1)) issues.push({ field: 'probability', message: 'survival must be within [0,1]' });
++  if (o.durationMonths <= 0) issues.push({ field: 'durationMonths', message: 'duration must be positive' });
++  if (o.probability != null && o.lowerCi != null && o.lowerCi > o.probability) issues.push({ field: 'lowerCi', message: 'lowerCI > estimate' });
++  if (o.probability != null && o.upperCi != null && o.upperCi < o.probability) issues.push({ field: 'upperCi', message: 'upperCI < estimate' });
++  if (o.cohortSize != null && o.cohortSize < 0) issues.push({ field: 'cohortSize', message: 'negative cohort' });
++  return issues;
++}
++
++export const TRIAL_PHASES = new Set(['EARLY_PHASE1', 'PHASE1', 'PHASE2', 'PHASE3', 'PHASE4', 'NA']);
++export const TRIAL_STATUSES = new Set(['ACTIVE_NOT_RECRUITING', 'COMPLETED', 'ENROLLING_BY_INVITATION', 'NOT_YET_RECRUITING', 'RECRUITING', 'SUSPENDED', 'TERMINATED', 'WITHDRAWN', 'AVAILABLE', 'NO_LONGER_AVAILABLE', 'TEMPORARILY_NOT_AVAILABLE', 'APPROVED_FOR_MARKETING', 'WITHHELD', 'UNKNOWN']);
++
++export function validateTrial(o: { nctId: string; phases: string[]; overallStatus?: string | null }): ValidationIssue[] {
++  const issues: ValidationIssue[] = [];
++  if (!/^NCT\d{8}$/.test(o.nctId)) issues.push({ field: 'nctId', message: 'malformed NCT id' });
++  for (const p of o.phases) if (!TRIAL_PHASES.has(p)) issues.push({ field: 'phases', message: `unknown phase enum ${p}` });
++  if (o.overallStatus && !TRIAL_STATUSES.has(o.overallStatus)) issues.push({ field: 'overallStatus', message: `unknown status enum ${o.overallStatus}` });
++  return issues;
++}
++
++export function validateFrequency(o: { casesAffected: number; casesProfiled: number }): ValidationIssue[] {
++  const issues: ValidationIssue[] = [];
++  if (o.casesProfiled <= 0) issues.push({ field: 'casesProfiled', message: 'denominator must be positive' });
++  if (o.casesAffected < 0 || o.casesAffected > o.casesProfiled) issues.push({ field: 'casesAffected', message: 'numerator out of range' });
++  return issues;
++}
++
++/** Gene symbol sanity (HGNC style). */
++export const GENE_SYMBOL_RE = /^[A-Z][A-Z0-9-]{0,15}(?:@|orf\d+)?$/i;

added packages/connectors/src/sources-sync.ts +51 −0

++import { eq } from 'drizzle-orm';
++import { type Database, sources, mintId } from '@cancerindex/database';
++import { CONNECTORS } from './registry.js';
++import type { ConnectorManifest } from './sdk/manifest.js';
++
++/** Sync connector manifests into the `sources` registry (CLAUDE.md §95, §142, §307). Idempotent. */
++export async function syncSources(db: Database, extra: ConnectorManifest[] = []): Promise<{ created: number; updated: number }> {
++  let created = 0;
++  let updated = 0;
++  const manifests = [...CONNECTORS.map((c) => c.manifest), ...extra];
++  for (const m of manifests) {
++    const values = {
++      slug: m.id,
++      name: m.name,
++      organization: m.organization,
++      category: m.category,
++      description: m.description,
++      homepage: m.homepage,
++      docsUrl: m.docsUrl ?? null,
++      termsUrl: m.termsUrl ?? null,
++      accessType: m.access.type,
++      accessAuth: m.access.auth,
++      license: m.license,
++      licenseStatus: m.licenseStatus,
++      commercialUse: m.commercialUse,
++      redistribution: m.redistribution,
++      attribution: m.attribution ?? null,
++      licenseReviewedAt: m.termsReviewedAt ? new Date(m.termsReviewedAt) : null,
++      approvedForProduction: m.licenseStatus === 'approved',
++      updateFrequency: m.updateFrequency,
++      supportsIncremental: m.supportsIncrementalSync,
++      entities: m.entities,
++      metrics: m.metrics,
++      rateLimit: m.rateLimits.requestsPerSecond ? `${m.rateLimits.requestsPerSecond} req/s` : m.rateLimits.requestsPerMinute ? `${m.rateLimits.requestsPerMinute} req/min` : null,
++      tier: m.tier,
++      manifest: m as unknown as Record<string, unknown>,
++      updatedAt: new Date(),
++    };
++    const [existing] = await db.select({ id: sources.id, status: sources.status }).from(sources).where(eq(sources.slug, m.id)).limit(1);
++    if (existing) {
++      // Keep runtime status (active/degraded) unless manifest declares a gating status.
++      const gating = ['awaiting_credentials', 'review', 'retired', 'paused'].includes(m.status);
++      await db.update(sources).set({ ...values, ...(gating ? { status: m.status } : {}) }).where(eq(sources.id, existing.id));
++      updated++;
++    } else {
++      await db.insert(sources).values({ id: await mintId(db, 'SOURCE'), ...values, status: m.status });
++      created++;
++    }
++  }
++  return { created, updated };
++}

added packages/connectors/tsconfig.json +8 −0

++{
++  "extends": "../../tsconfig.base.json",
++  "compilerOptions": {
++    "rootDir": ".",
++    "noEmit": true
++  },
++  "include": ["src", "test"]
++}

added packages/connectors/vitest.config.ts +8 −0

++import { defineConfig } from 'vitest/config';
++
++export default defineConfig({
++  test: {
++    include: ['src/**/*.test.ts', 'test/**/*.test.ts'],
++    testTimeout: 20_000,
++  },
++});

added packages/database/drizzle.config.ts +11 −0

++import { defineConfig } from 'drizzle-kit';
++
++export default defineConfig({
++  dialect: 'postgresql',
++  schema: './src/schema/index.ts',
++  out: './migrations',
++  dbCredentials: { url: process.env.DATABASE_URL ?? 'postgres://localhost:5432/cancerindex' },
++  casing: 'snake_case',
++  strict: true,
++  verbose: true,
++});

added packages/database/migrations/0000_small_blazing_skull.sql +936 −0

++CREATE TABLE "audit_log" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"actor" text NOT NULL,
++	"action" text NOT NULL,
++	"entity_type" text,
++	"entity_id" text,
++	"before" jsonb,
++	"after" jsonb,
++	"reason" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "change_events" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"entity_type" text NOT NULL,
++	"entity_id" text NOT NULL,
++	"kind" text NOT NULL,
++	"summary" text NOT NULL,
++	"before" jsonb,
++	"after" jsonb,
++	"ingest_run_id" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "connector_cursors" (
++	"connector_id" text PRIMARY KEY NOT NULL,
++	"cursor" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"last_success_at" timestamp with time zone,
++	"last_attempt_at" timestamp with time zone,
++	"paused" boolean DEFAULT false NOT NULL,
++	"health" text DEFAULT 'unknown' NOT NULL,
++	"health_detail" text,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "connector_field_stats" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"connector_id" text NOT NULL,
++	"entity" text NOT NULL,
++	"field" text NOT NULL,
++	"types" text[] DEFAULT '{}' NOT NULL,
++	"seen_count" integer DEFAULT 0 NOT NULL,
++	"null_count" integer DEFAULT 0 NOT NULL,
++	"first_seen_run" text,
++	"last_seen_run" text,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "entity_merges" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"entity_type" text NOT NULL,
++	"keep_id" text NOT NULL,
++	"merge_id" text NOT NULL,
++	"evidence" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"status" text DEFAULT 'proposed' NOT NULL,
++	"decided_by" text,
++	"decided_at" timestamp with time zone,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "id_sequences" (
++	"namespace" varchar(16) PRIMARY KEY NOT NULL,
++	"next" bigint DEFAULT 1 NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "ingest_runs" (
++	"id" text PRIMARY KEY NOT NULL,
++	"connector_id" text NOT NULL,
++	"source_id" varchar(32) NOT NULL,
++	"mode" text DEFAULT 'incremental' NOT NULL,
++	"status" text DEFAULT 'running' NOT NULL,
++	"started_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"finished_at" timestamp with time zone,
++	"duration_ms" integer,
++	"records_fetched" integer DEFAULT 0 NOT NULL,
++	"records_created" integer DEFAULT 0 NOT NULL,
++	"records_updated" integer DEFAULT 0 NOT NULL,
++	"records_unchanged" integer DEFAULT 0 NOT NULL,
++	"records_rejected" integer DEFAULT 0 NOT NULL,
++	"http_requests" integer DEFAULT 0 NOT NULL,
++	"http_failures" integer DEFAULT 0 NOT NULL,
++	"rate_limit_events" integer DEFAULT 0 NOT NULL,
++	"validation_failures" integer DEFAULT 0 NOT NULL,
++	"schema_drift" jsonb DEFAULT '[]'::jsonb NOT NULL,
++	"cursor_before" jsonb,
++	"cursor_after" jsonb,
++	"error" text,
++	"log" jsonb DEFAULT '[]'::jsonb NOT NULL,
++	"dataset_version" text,
++	"anomaly" text
++);
++--> statement-breakpoint
++CREATE TABLE "provenance" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"public_id" varchar(32),
++	"source_id" varchar(32) NOT NULL,
++	"source_record_id" text,
++	"source_url" text,
++	"dataset" text,
++	"dataset_version" text,
++	"publication_id" varchar(32),
++	"pmid" text,
++	"doi" text,
++	"retrieved_at" timestamp with time zone NOT NULL,
++	"published_at" text,
++	"updated_at_source" text,
++	"geography" text,
++	"population" text,
++	"cohort_size" integer,
++	"methodology" text,
++	"evidence_type" text NOT NULL,
++	"access_level" text DEFAULT 'open' NOT NULL,
++	"confidence" real,
++	"license" text,
++	"ingest_run_id" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "source_records" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"source_id" varchar(32) NOT NULL,
++	"entity_kind" text NOT NULL,
++	"source_record_id" text NOT NULL,
++	"payload_hash" text NOT NULL,
++	"raw_path" text,
++	"status" text DEFAULT 'active' NOT NULL,
++	"first_seen_run" text,
++	"last_seen_run" text,
++	"retrieved_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"source_updated_at" timestamp with time zone,
++	"canonical_type" text,
++	"canonical_id" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "sources" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"slug" text NOT NULL,
++	"name" text NOT NULL,
++	"organization" text,
++	"category" text NOT NULL,
++	"description" text,
++	"homepage" text,
++	"docs_url" text,
++	"terms_url" text,
++	"access_type" text NOT NULL,
++	"access_auth" text NOT NULL,
++	"license" text,
++	"license_status" text DEFAULT 'review' NOT NULL,
++	"commercial_use" text DEFAULT 'unknown' NOT NULL,
++	"redistribution" text DEFAULT 'unknown' NOT NULL,
++	"attribution" text,
++	"license_reviewed_at" timestamp with time zone,
++	"approved_for_production" boolean DEFAULT false NOT NULL,
++	"update_frequency" text,
++	"supports_incremental" boolean DEFAULT false NOT NULL,
++	"entities" text[] DEFAULT '{}' NOT NULL,
++	"metrics" text[] DEFAULT '{}' NOT NULL,
++	"rate_limit" text,
++	"status" text DEFAULT 'planned' NOT NULL,
++	"tier" integer DEFAULT 0 NOT NULL,
++	"manifest" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "unresolved_labels" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"source_id" varchar(32) NOT NULL,
++	"entity_kind" text NOT NULL,
++	"source_text" text NOT NULL,
++	"normalized" text NOT NULL,
++	"context" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"count" integer DEFAULT 1 NOT NULL,
++	"status" text DEFAULT 'open' NOT NULL,
++	"suggested_id" text,
++	"suggested_match_type" text,
++	"suggested_score" real,
++	"resolved_id" text,
++	"resolved_by" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "anatomical_sites" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"name" text NOT NULL,
++	"slug" text NOT NULL,
++	"ncit_code" text,
++	"uberon_id" text,
++	"parent_id" varchar(32),
++	"system" text
++);
++--> statement-breakpoint
++CREATE TABLE "cancer_aliases" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"cancer_id" varchar(32) NOT NULL,
++	"alias" text NOT NULL,
++	"normalized" text NOT NULL,
++	"alias_type" text DEFAULT 'synonym' NOT NULL,
++	"source_id" varchar(32),
++	"source_terminology" text,
++	"language" text DEFAULT 'en' NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "cancer_anatomy" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"cancer_id" varchar(32) NOT NULL,
++	"site_id" varchar(32) NOT NULL,
++	"relation" text DEFAULT 'primary' NOT NULL,
++	"source_id" varchar(32)
++);
++--> statement-breakpoint
++CREATE TABLE "cancer_codes" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"cancer_id" varchar(32) NOT NULL,
++	"system" text NOT NULL,
++	"code" text NOT NULL,
++	"match_type" text DEFAULT 'EXACT_IDENTIFIER' NOT NULL,
++	"source_id" varchar(32),
++	"valid_from" text,
++	"valid_to" text
++);
++--> statement-breakpoint
++CREATE TABLE "cancer_hierarchy" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"parent_id" varchar(32) NOT NULL,
++	"child_id" varchar(32) NOT NULL,
++	"hierarchy_type" text NOT NULL,
++	"source_id" varchar(32)
++);
++--> statement-breakpoint
++CREATE TABLE "cancers" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"slug" text NOT NULL,
++	"canonical_name" text NOT NULL,
++	"short_name" text,
++	"entity_type" text DEFAULT 'cancer' NOT NULL,
++	"malignant" boolean DEFAULT true NOT NULL,
++	"solid_tumor" boolean DEFAULT true NOT NULL,
++	"hematologic" boolean DEFAULT false NOT NULL,
++	"pediatric_relevant" boolean DEFAULT false NOT NULL,
++	"rare_cancer" boolean,
++	"top_level" boolean DEFAULT false NOT NULL,
++	"description" text,
++	"description_provenance_id" integer,
++	"primary_ncit_code" text,
++	"primary_oncotree_code" text,
++	"depth" integer DEFAULT 0 NOT NULL,
++	"status" text DEFAULT 'active' NOT NULL,
++	"merged_into" varchar(32),
++	"deprecated_reason" text,
++	"classification_version" text,
++	"semantic_types" text[] DEFAULT '{}' NOT NULL,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "cohort_definitions" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"name" text NOT NULL,
++	"cancer_id" varchar(32) NOT NULL,
++	"biomarker_ids" text[] DEFAULT '{}' NOT NULL,
++	"variant_ids" text[] DEFAULT '{}' NOT NULL,
++	"stage" text,
++	"attributes" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"confidence" real,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "geographies" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"slug" text NOT NULL,
++	"name" text NOT NULL,
++	"kind" text NOT NULL,
++	"iso2" text,
++	"iso3" text,
++	"parent_id" varchar(32),
++	"who_region" text,
++	"population" integer,
++	"population_year" integer
++);
++--> statement-breakpoint
++CREATE TABLE "biomarkers" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"slug" text NOT NULL,
++	"name" text NOT NULL,
++	"kind" text NOT NULL,
++	"gene_id" varchar(32),
++	"ncit_code" text,
++	"description" text,
++	"measurement" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "cancer_gene_frequencies" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"cohort_id" varchar(32) NOT NULL,
++	"cancer_id" varchar(32),
++	"gene_id" varchar(32),
++	"gene_symbol" text NOT NULL,
++	"alteration_type" text DEFAULT 'ssm' NOT NULL,
++	"cases_affected" integer NOT NULL,
++	"cases_profiled" integer NOT NULL,
++	"frequency" real NOT NULL,
++	"rank" integer,
++	"data_release" text,
++	"provenance_id" integer NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "entity_embeddings" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"entity_type" text NOT NULL,
++	"entity_id" text NOT NULL,
++	"model" text NOT NULL,
++	"dimensions" integer NOT NULL,
++	"text_hash" text NOT NULL,
++	"embedding" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "gene_aliases" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"gene_id" varchar(32) NOT NULL,
++	"alias" text NOT NULL,
++	"alias_type" text NOT NULL,
++	"source_id" varchar(32)
++);
++--> statement-breakpoint
++CREATE TABLE "genes" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"hgnc_id" text,
++	"symbol" text NOT NULL,
++	"name" text,
++	"locus_type" text,
++	"locus_group" text,
++	"location" text,
++	"chromosome" text,
++	"ensembl_gene_id" text,
++	"ncbi_gene_id" text,
++	"omim_ids" text[] DEFAULT '{}' NOT NULL,
++	"uniprot_ids" text[] DEFAULT '{}' NOT NULL,
++	"refseq_accession" text,
++	"prev_symbols" text[] DEFAULT '{}' NOT NULL,
++	"alias_symbols" text[] DEFAULT '{}' NOT NULL,
++	"gene_families" text[] DEFAULT '{}' NOT NULL,
++	"status" text DEFAULT 'Approved' NOT NULL,
++	"is_cancer_gene" boolean DEFAULT false NOT NULL,
++	"civic_gene_id" integer,
++	"description" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "genomic_cohorts" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"source_id" varchar(32) NOT NULL,
++	"study_id" text NOT NULL,
++	"name" text NOT NULL,
++	"program" text,
++	"primary_sites" text[] DEFAULT '{}' NOT NULL,
++	"disease_types" text[] DEFAULT '{}' NOT NULL,
++	"cancer_id" varchar(32),
++	"cancer_match_type" text,
++	"case_count" integer,
++	"cases_with_ssm" integer,
++	"data_release" text,
++	"access_level" text DEFAULT 'open' NOT NULL,
++	"url" text,
++	"provenance_id" integer,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "variant_aliases" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"variant_id" varchar(32) NOT NULL,
++	"alias" text NOT NULL,
++	"source_id" varchar(32)
++);
++--> statement-breakpoint
++CREATE TABLE "variant_clinical_significance" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"variant_id" varchar(32) NOT NULL,
++	"clinvar_variation_id" text NOT NULL,
++	"clinical_significance" text NOT NULL,
++	"review_status" text,
++	"star_rating" integer,
++	"last_evaluated" text,
++	"conditions" text[] DEFAULT '{}' NOT NULL,
++	"condition_cancer_ids" text[] DEFAULT '{}' NOT NULL,
++	"origin_simple" text,
++	"number_submitters" integer,
++	"provenance_id" integer NOT NULL,
++	"ingest_run_id" text,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "variants" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"slug" text NOT NULL,
++	"gene_id" varchar(32),
++	"gene_symbol" text,
++	"name" text NOT NULL,
++	"variant_type" text,
++	"hgvs_g" text,
++	"hgvs_c" text,
++	"hgvs_p" text,
++	"assembly" text,
++	"chromosome" text,
++	"start" integer,
++	"end" integer,
++	"reference_bases" text,
++	"alternate_bases" text,
++	"coordinates" jsonb DEFAULT '[]'::jsonb NOT NULL,
++	"clinvar_variation_id" text,
++	"civic_variant_id" integer,
++	"dbsnp_ids" text[] DEFAULT '{}' NOT NULL,
++	"fusion_partners" text[] DEFAULT '{}' NOT NULL,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "drug_aliases" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"drug_id" varchar(32) NOT NULL,
++	"alias" text NOT NULL,
++	"normalized" text NOT NULL,
++	"alias_type" text DEFAULT 'synonym' NOT NULL,
++	"source_id" varchar(32)
++);
++--> statement-breakpoint
++CREATE TABLE "drug_approvals" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"drug_id" varchar(32) NOT NULL,
++	"cancer_id" varchar(32),
++	"biomarker_ids" text[] DEFAULT '{}' NOT NULL,
++	"tumor_agnostic" boolean DEFAULT false NOT NULL,
++	"jurisdiction" text NOT NULL,
++	"authority" text NOT NULL,
++	"indication" text NOT NULL,
++	"line_of_therapy" text,
++	"disease_stage" text,
++	"approval_type" text,
++	"accelerated" boolean,
++	"conditional" boolean,
++	"approval_date" text,
++	"withdrawal_date" text,
++	"status" text NOT NULL,
++	"application_number" text,
++	"source_id" varchar(32) NOT NULL,
++	"provenance_id" integer NOT NULL,
++	"raw" jsonb,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "drugs" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"slug" text NOT NULL,
++	"name" text NOT NULL,
++	"kind" text,
++	"ncit_code" text,
++	"chembl_id" text,
++	"civic_therapy_id" integer,
++	"drugbank_id" text,
++	"pubchem_cid" text,
++	"unii" text,
++	"mechanism" text,
++	"target_gene_ids" text[] DEFAULT '{}' NOT NULL,
++	"development_status" text,
++	"description" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "treatment_regimens" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"slug" text NOT NULL,
++	"name" text NOT NULL,
++	"component_drug_ids" text[] DEFAULT '{}' NOT NULL,
++	"modality" text DEFAULT 'drug_combination' NOT NULL,
++	"description" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "clinical_trials" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"nct_id" text NOT NULL,
++	"brief_title" text NOT NULL,
++	"official_title" text,
++	"acronym" text,
++	"study_type" text,
++	"phases" text[] DEFAULT '{}' NOT NULL,
++	"overall_status" text,
++	"why_stopped" text,
++	"start_date" text,
++	"primary_completion_date" text,
++	"completion_date" text,
++	"first_posted_date" text,
++	"last_update_posted_date" text,
++	"results_first_posted_date" text,
++	"has_results" boolean DEFAULT false NOT NULL,
++	"enrollment_count" integer,
++	"enrollment_type" text,
++	"lead_sponsor" text,
++	"lead_sponsor_class" text,
++	"collaborators" text[] DEFAULT '{}' NOT NULL,
++	"conditions" text[] DEFAULT '{}' NOT NULL,
++	"keywords" text[] DEFAULT '{}' NOT NULL,
++	"interventions" jsonb DEFAULT '[]'::jsonb NOT NULL,
++	"arms" jsonb DEFAULT '[]'::jsonb NOT NULL,
++	"primary_outcomes" jsonb DEFAULT '[]'::jsonb NOT NULL,
++	"secondary_outcomes" jsonb DEFAULT '[]'::jsonb NOT NULL,
++	"eligibility" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"sex" text,
++	"minimum_age" text,
++	"maximum_age" text,
++	"countries" text[] DEFAULT '{}' NOT NULL,
++	"locations_count" integer DEFAULT 0 NOT NULL,
++	"references" jsonb DEFAULT '[]'::jsonb NOT NULL,
++	"brief_summary" text,
++	"is_oncology" boolean DEFAULT true NOT NULL,
++	"source_record_id" integer,
++	"ingest_run_id" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "trial_conditions" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"trial_id" varchar(32) NOT NULL,
++	"condition_text" text NOT NULL,
++	"normalized" text NOT NULL,
++	"cancer_id" varchar(32),
++	"match_type" text DEFAULT 'UNRESOLVED' NOT NULL,
++	"confidence" real
++);
++--> statement-breakpoint
++CREATE TABLE "trial_interventions" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"trial_id" varchar(32) NOT NULL,
++	"name" text NOT NULL,
++	"normalized" text NOT NULL,
++	"intervention_type" text,
++	"drug_id" varchar(32),
++	"match_type" text DEFAULT 'UNRESOLVED' NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "trial_locations" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"trial_id" varchar(32) NOT NULL,
++	"facility" text,
++	"city" text,
++	"state" text,
++	"zip" text,
++	"country" text,
++	"status" text,
++	"lat" real,
++	"lng" real
++);
++--> statement-breakpoint
++CREATE TABLE "trial_pulse" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"day" date NOT NULL,
++	"cancer_id" varchar(32),
++	"phase" text,
++	"new_trials" integer NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "literature_counts" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"cancer_id" varchar(32) NOT NULL,
++	"window_key" text NOT NULL,
++	"window_start" text,
++	"window_end" text,
++	"query" text NOT NULL,
++	"count" integer NOT NULL,
++	"provenance_id" integer NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "publication_entity_edges" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"publication_id" varchar(32) NOT NULL,
++	"entity_type" text NOT NULL,
++	"entity_id" text NOT NULL,
++	"method" text NOT NULL,
++	"confidence" real,
++	"status" text DEFAULT 'candidate' NOT NULL,
++	"source_id" varchar(32),
++	"ingest_run_id" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "publications" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"pmid" text,
++	"doi" text,
++	"pmcid" text,
++	"title" text NOT NULL,
++	"abstract" text,
++	"journal" text,
++	"journal_iso" text,
++	"pub_date" text,
++	"pub_year" integer,
++	"publication_types" text[] DEFAULT '{}' NOT NULL,
++	"mesh_terms" jsonb DEFAULT '[]'::jsonb NOT NULL,
++	"authors" jsonb DEFAULT '[]'::jsonb NOT NULL,
++	"language" text,
++	"is_preprint" boolean DEFAULT false NOT NULL,
++	"retracted" boolean DEFAULT false NOT NULL,
++	"retraction_notice" text,
++	"nct_ids" text[] DEFAULT '{}' NOT NULL,
++	"cited_by_count" integer,
++	"source_record_id" integer,
++	"ingest_run_id" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "civic_evidence_items" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"civic_id" integer NOT NULL,
++	"name" text,
++	"molecular_profile_id" integer,
++	"molecular_profile_name" text,
++	"gene_symbols" text[] DEFAULT '{}' NOT NULL,
++	"gene_ids" text[] DEFAULT '{}' NOT NULL,
++	"variant_ids" text[] DEFAULT '{}' NOT NULL,
++	"civic_variant_ids" integer[] DEFAULT '{}' NOT NULL,
++	"disease_name" text,
++	"doid" text,
++	"cancer_id" varchar(32),
++	"cancer_match_type" text,
++	"therapy_names" text[] DEFAULT '{}' NOT NULL,
++	"therapy_ids" text[] DEFAULT '{}' NOT NULL,
++	"therapy_interaction_type" text,
++	"evidence_type" text,
++	"evidence_level" text,
++	"evidence_direction" text,
++	"significance" text,
++	"evidence_rating" integer,
++	"status" text,
++	"description" text,
++	"pmid" text,
++	"source_citation" text,
++	"phenotypes" text[] DEFAULT '{}' NOT NULL,
++	"provenance_id" integer NOT NULL,
++	"ingest_run_id" text,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "knowledge_edges" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"source_entity_type" text NOT NULL,
++	"source_entity_id" text NOT NULL,
++	"target_entity_type" text NOT NULL,
++	"target_entity_id" text NOT NULL,
++	"relationship_type" text NOT NULL,
++	"cancer_context_ids" text[] DEFAULT '{}' NOT NULL,
++	"predictive" boolean,
++	"prognostic" boolean,
++	"diagnostic" boolean,
++	"predisposing" boolean,
++	"direction" text,
++	"evidence_level" text,
++	"evidence_score" real,
++	"evidence_category" text DEFAULT 'curated_evidence' NOT NULL,
++	"status" text DEFAULT 'active' NOT NULL,
++	"source_id" varchar(32) NOT NULL,
++	"source_record_id" text,
++	"provenance_ids" integer[] DEFAULT '{}' NOT NULL,
++	"support_count" integer DEFAULT 1 NOT NULL,
++	"first_seen_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"last_seen_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "risk_factors" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"slug" text NOT NULL,
++	"name" text NOT NULL,
++	"kind" text NOT NULL,
++	"classification_authority" text,
++	"classification" text,
++	"description" text,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "epidemiology_observations" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"cancer_id" varchar(32) NOT NULL,
++	"geography_id" varchar(32) NOT NULL,
++	"year" integer NOT NULL,
++	"year_end" integer,
++	"sex" text DEFAULT 'all' NOT NULL,
++	"age_group" text DEFAULT 'all' NOT NULL,
++	"metric" text NOT NULL,
++	"value" double precision NOT NULL,
++	"unit" text NOT NULL,
++	"lower_ci" double precision,
++	"upper_ci" double precision,
++	"standard_population" text,
++	"estimate_type" text DEFAULT 'observed' NOT NULL,
++	"site_definition" text,
++	"source_id" varchar(32) NOT NULL,
++	"provenance_id" integer NOT NULL,
++	"ingest_run_id" text,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "survival_observations" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"cancer_id" varchar(32) NOT NULL,
++	"geography_id" varchar(32),
++	"stage" text,
++	"staging_system" text,
++	"sex" text DEFAULT 'all' NOT NULL,
++	"age_group" text DEFAULT 'all' NOT NULL,
++	"diagnosis_period" text,
++	"survival_type" text NOT NULL,
++	"duration_months" integer NOT NULL,
++	"probability" real,
++	"median_months" real,
++	"cohort_size" integer,
++	"lower_ci" real,
++	"upper_ci" real,
++	"method" text,
++	"source_id" varchar(32) NOT NULL,
++	"provenance_id" integer NOT NULL,
++	"ingest_run_id" text,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "ai_answers" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"kind" text NOT NULL,
++	"subject_id" text,
++	"question_hash" text NOT NULL,
++	"question" text,
++	"answer" jsonb NOT NULL,
++	"model" text NOT NULL,
++	"prompt_version" text NOT NULL,
++	"source_snapshot" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"data_as_of" timestamp with time zone NOT NULL,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "api_keys" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"key_hash" text NOT NULL,
++	"prefix" text NOT NULL,
++	"label" text,
++	"owner_email" text,
++	"tier" text DEFAULT 'free' NOT NULL,
++	"rate_limit_per_minute" integer DEFAULT 60 NOT NULL,
++	"active" boolean DEFAULT true NOT NULL,
++	"last_used_at" timestamp with time zone,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "entity_counters" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"entity_type" text NOT NULL,
++	"entity_id" text NOT NULL,
++	"trial_count" integer DEFAULT 0 NOT NULL,
++	"active_trial_count" integer DEFAULT 0 NOT NULL,
++	"recruiting_trial_count" integer DEFAULT 0 NOT NULL,
++	"phase3_trial_count" integer DEFAULT 0 NOT NULL,
++	"publication_count" integer DEFAULT 0 NOT NULL,
++	"publication_count_5y" integer DEFAULT 0 NOT NULL,
++	"publication_count_12m" integer DEFAULT 0 NOT NULL,
++	"gene_count" integer DEFAULT 0 NOT NULL,
++	"variant_count" integer DEFAULT 0 NOT NULL,
++	"drug_count" integer DEFAULT 0 NOT NULL,
++	"approved_drug_count" integer DEFAULT 0 NOT NULL,
++	"evidence_count" integer DEFAULT 0 NOT NULL,
++	"cohort_count" integer DEFAULT 0 NOT NULL,
++	"subtype_count" integer DEFAULT 0 NOT NULL,
++	"descendant_count" integer DEFAULT 0 NOT NULL,
++	"epidemiology_obs_count" integer DEFAULT 0 NOT NULL,
++	"survival_obs_count" integer DEFAULT 0 NOT NULL,
++	"completeness" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "metric_definitions" (
++	"id" varchar(32) PRIMARY KEY NOT NULL,
++	"slug" text NOT NULL,
++	"name" text NOT NULL,
++	"description" text NOT NULL,
++	"formula" text NOT NULL,
++	"formula_version" text NOT NULL,
++	"unit" text NOT NULL,
++	"higher_is_worse" boolean,
++	"aggregation" text,
++	"valid_dimensions" text[] DEFAULT '{}' NOT NULL,
++	"source_slugs" text[] DEFAULT '{}' NOT NULL,
++	"category" text NOT NULL,
++	"eligibility" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"experimental" boolean DEFAULT false NOT NULL,
++	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
++	"updated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "ranking_snapshots" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"metric_id" varchar(32) NOT NULL,
++	"metric_slug" text NOT NULL,
++	"scope_key" text NOT NULL,
++	"geography" text DEFAULT 'WORLD' NOT NULL,
++	"sex" text DEFAULT 'all' NOT NULL,
++	"age_group" text DEFAULT 'all' NOT NULL,
++	"year" integer,
++	"entity_level" text DEFAULT 'top' NOT NULL,
++	"formula_version" text NOT NULL,
++	"eligible_entities" integer NOT NULL,
++	"inputs_hash" text NOT NULL,
++	"source_ids" text[] DEFAULT '{}' NOT NULL,
++	"is_current" boolean DEFAULT true NOT NULL,
++	"generated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE TABLE "rankings" (
++	"id" bigserial PRIMARY KEY NOT NULL,
++	"snapshot_id" integer NOT NULL,
++	"metric_slug" text NOT NULL,
++	"scope_key" text NOT NULL,
++	"cancer_id" varchar(32) NOT NULL,
++	"rank" integer NOT NULL,
++	"eligible_entities" integer NOT NULL,
++	"percentile" real NOT NULL,
++	"value" double precision NOT NULL,
++	"unit" text NOT NULL,
++	"confidence" text DEFAULT 'MEDIUM' NOT NULL,
++	"inputs" jsonb DEFAULT '{}'::jsonb NOT NULL,
++	"breakdown" jsonb,
++	"previous_rank" integer,
++	"generated_at" timestamp with time zone DEFAULT now() NOT NULL
++);
++--> statement-breakpoint
++CREATE INDEX "change_events_entity_idx" ON "change_events" USING btree ("entity_type","entity_id","created_at");--> statement-breakpoint
++CREATE UNIQUE INDEX "connector_field_stats_uq" ON "connector_field_stats" USING btree ("connector_id","entity","field");--> statement-breakpoint
++CREATE INDEX "ingest_runs_connector_idx" ON "ingest_runs" USING btree ("connector_id","started_at");--> statement-breakpoint
++CREATE INDEX "provenance_source_idx" ON "provenance" USING btree ("source_id","source_record_id");--> statement-breakpoint
++CREATE INDEX "provenance_pmid_idx" ON "provenance" USING btree ("pmid");--> statement-breakpoint
++CREATE UNIQUE INDEX "source_records_uq" ON "source_records" USING btree ("source_id","entity_kind","source_record_id");--> statement-breakpoint
++CREATE INDEX "source_records_canonical_idx" ON "source_records" USING btree ("canonical_type","canonical_id");--> statement-breakpoint
++CREATE UNIQUE INDEX "sources_slug_uq" ON "sources" USING btree ("slug");--> statement-breakpoint
++CREATE UNIQUE INDEX "unresolved_labels_uq" ON "unresolved_labels" USING btree ("source_id","entity_kind","normalized");--> statement-breakpoint
++CREATE INDEX "unresolved_labels_count_idx" ON "unresolved_labels" USING btree ("status","count");--> statement-breakpoint
++CREATE UNIQUE INDEX "anatomical_sites_slug_uq" ON "anatomical_sites" USING btree ("slug");--> statement-breakpoint
++CREATE UNIQUE INDEX "cancer_aliases_uq" ON "cancer_aliases" USING btree ("cancer_id","normalized","alias_type");--> statement-breakpoint
++CREATE INDEX "cancer_aliases_norm_idx" ON "cancer_aliases" USING btree ("normalized");--> statement-breakpoint
++CREATE UNIQUE INDEX "cancer_anatomy_uq" ON "cancer_anatomy" USING btree ("cancer_id","site_id","relation");--> statement-breakpoint
++CREATE UNIQUE INDEX "cancer_codes_uq" ON "cancer_codes" USING btree ("cancer_id","system","code");--> statement-breakpoint
++CREATE INDEX "cancer_codes_lookup_idx" ON "cancer_codes" USING btree ("system","code");--> statement-breakpoint
++CREATE UNIQUE INDEX "cancer_hierarchy_uq" ON "cancer_hierarchy" USING btree ("parent_id","child_id","hierarchy_type");--> statement-breakpoint
++CREATE INDEX "cancer_hierarchy_child_idx" ON "cancer_hierarchy" USING btree ("child_id");--> statement-breakpoint
++CREATE UNIQUE INDEX "cancers_slug_uq" ON "cancers" USING btree ("slug");--> statement-breakpoint
++CREATE UNIQUE INDEX "cancers_ncit_uq" ON "cancers" USING btree ("primary_ncit_code");--> statement-breakpoint
++CREATE INDEX "cancers_name_idx" ON "cancers" USING btree ("canonical_name");--> statement-breakpoint
++CREATE INDEX "cancers_type_idx" ON "cancers" USING btree ("entity_type","malignant","top_level");--> statement-breakpoint
++CREATE UNIQUE INDEX "geographies_slug_uq" ON "geographies" USING btree ("slug");--> statement-breakpoint
++CREATE INDEX "geographies_iso3_idx" ON "geographies" USING btree ("iso3");--> statement-breakpoint
++CREATE UNIQUE INDEX "biomarkers_slug_uq" ON "biomarkers" USING btree ("slug");--> statement-breakpoint
++CREATE UNIQUE INDEX "cancer_gene_freq_uq" ON "cancer_gene_frequencies" USING btree ("cohort_id","gene_symbol","alteration_type");--> statement-breakpoint
++CREATE INDEX "cancer_gene_freq_cancer_idx" ON "cancer_gene_frequencies" USING btree ("cancer_id","frequency");--> statement-breakpoint
++CREATE INDEX "cancer_gene_freq_gene_idx" ON "cancer_gene_frequencies" USING btree ("gene_id");--> statement-breakpoint
++CREATE UNIQUE INDEX "entity_embeddings_uq" ON "entity_embeddings" USING btree ("entity_type","entity_id","model");--> statement-breakpoint
++CREATE UNIQUE INDEX "gene_aliases_uq" ON "gene_aliases" USING btree ("gene_id","alias","alias_type");--> statement-breakpoint
++CREATE INDEX "gene_aliases_alias_idx" ON "gene_aliases" USING btree ("alias");--> statement-breakpoint
++CREATE UNIQUE INDEX "genes_symbol_uq" ON "genes" USING btree ("symbol");--> statement-breakpoint
++CREATE UNIQUE INDEX "genes_hgnc_uq" ON "genes" USING btree ("hgnc_id");--> statement-breakpoint
++CREATE INDEX "genes_ensembl_idx" ON "genes" USING btree ("ensembl_gene_id");--> statement-breakpoint
++CREATE INDEX "genes_ncbi_idx" ON "genes" USING btree ("ncbi_gene_id");--> statement-breakpoint
++CREATE UNIQUE INDEX "genomic_cohorts_uq" ON "genomic_cohorts" USING btree ("source_id","study_id");--> statement-breakpoint
++CREATE INDEX "genomic_cohorts_cancer_idx" ON "genomic_cohorts" USING btree ("cancer_id");--> statement-breakpoint
++CREATE UNIQUE INDEX "variant_aliases_uq" ON "variant_aliases" USING btree ("variant_id","alias");--> statement-breakpoint
++CREATE UNIQUE INDEX "variant_clinsig_uq" ON "variant_clinical_significance" USING btree ("clinvar_variation_id");--> statement-breakpoint
++CREATE UNIQUE INDEX "variants_slug_uq" ON "variants" USING btree ("slug");--> statement-breakpoint
++CREATE INDEX "variants_gene_idx" ON "variants" USING btree ("gene_id");--> statement-breakpoint
++CREATE INDEX "variants_clinvar_idx" ON "variants" USING btree ("clinvar_variation_id");--> statement-breakpoint
++CREATE INDEX "variants_civic_idx" ON "variants" USING btree ("civic_variant_id");--> statement-breakpoint
++CREATE UNIQUE INDEX "drug_aliases_uq" ON "drug_aliases" USING btree ("drug_id","normalized","alias_type");--> statement-breakpoint
++CREATE INDEX "drug_aliases_norm_idx" ON "drug_aliases" USING btree ("normalized");--> statement-breakpoint
++CREATE INDEX "drug_approvals_drug_idx" ON "drug_approvals" USING btree ("drug_id");--> statement-breakpoint
++CREATE INDEX "drug_approvals_cancer_idx" ON "drug_approvals" USING btree ("cancer_id");--> statement-breakpoint
++CREATE UNIQUE INDEX "drugs_slug_uq" ON "drugs" USING btree ("slug");--> statement-breakpoint
++CREATE INDEX "drugs_ncit_idx" ON "drugs" USING btree ("ncit_code");--> statement-breakpoint
++CREATE INDEX "drugs_civic_idx" ON "drugs" USING btree ("civic_therapy_id");--> statement-breakpoint
++CREATE INDEX "drugs_chembl_idx" ON "drugs" USING btree ("chembl_id");--> statement-breakpoint
++CREATE UNIQUE INDEX "treatment_regimens_slug_uq" ON "treatment_regimens" USING btree ("slug");--> statement-breakpoint
++CREATE UNIQUE INDEX "clinical_trials_nct_uq" ON "clinical_trials" USING btree ("nct_id");--> statement-breakpoint
++CREATE INDEX "clinical_trials_status_idx" ON "clinical_trials" USING btree ("overall_status");--> statement-breakpoint
++CREATE INDEX "clinical_trials_updated_idx" ON "clinical_trials" USING btree ("last_update_posted_date");--> statement-breakpoint
++CREATE INDEX "clinical_trials_sponsor_idx" ON "clinical_trials" USING btree ("lead_sponsor");--> statement-breakpoint
++CREATE UNIQUE INDEX "trial_conditions_uq" ON "trial_conditions" USING btree ("trial_id","normalized");--> statement-breakpoint
++CREATE INDEX "trial_conditions_cancer_idx" ON "trial_conditions" USING btree ("cancer_id");--> statement-breakpoint
++CREATE INDEX "trial_conditions_norm_idx" ON "trial_conditions" USING btree ("normalized");--> statement-breakpoint
++CREATE UNIQUE INDEX "trial_interventions_uq" ON "trial_interventions" USING btree ("trial_id","normalized");--> statement-breakpoint
++CREATE INDEX "trial_interventions_drug_idx" ON "trial_interventions" USING btree ("drug_id");--> statement-breakpoint
++CREATE INDEX "trial_locations_trial_idx" ON "trial_locations" USING btree ("trial_id");--> statement-breakpoint
++CREATE INDEX "trial_locations_country_idx" ON "trial_locations" USING btree ("country");--> statement-breakpoint
++CREATE UNIQUE INDEX "trial_pulse_uq" ON "trial_pulse" USING btree ("day","cancer_id","phase");--> statement-breakpoint
++CREATE UNIQUE INDEX "literature_counts_uq" ON "literature_counts" USING btree ("cancer_id","window_key");--> statement-breakpoint
++CREATE INDEX "literature_counts_window_idx" ON "literature_counts" USING btree ("window_key","count");--> statement-breakpoint
++CREATE UNIQUE INDEX "pub_entity_edges_uq" ON "publication_entity_edges" USING btree ("publication_id","entity_type","entity_id","method");--> statement-breakpoint
++CREATE INDEX "pub_entity_edges_entity_idx" ON "publication_entity_edges" USING btree ("entity_type","entity_id");--> statement-breakpoint
++CREATE UNIQUE INDEX "publications_pmid_uq" ON "publications" USING btree ("pmid");--> statement-breakpoint
++CREATE INDEX "publications_doi_idx" ON "publications" USING btree ("doi");--> statement-breakpoint
++CREATE INDEX "publications_year_idx" ON "publications" USING btree ("pub_year");--> statement-breakpoint
++CREATE INDEX "publications_retracted_idx" ON "publications" USING btree ("retracted");--> statement-breakpoint
++CREATE UNIQUE INDEX "civic_evidence_uq" ON "civic_evidence_items" USING btree ("civic_id");--> statement-breakpoint
++CREATE INDEX "civic_evidence_cancer_idx" ON "civic_evidence_items" USING btree ("cancer_id");--> statement-breakpoint
++CREATE INDEX "civic_evidence_gene_idx" ON "civic_evidence_items" USING btree ("gene_symbols");--> statement-breakpoint
++CREATE UNIQUE INDEX "knowledge_edges_uq" ON "knowledge_edges" USING btree ("source_entity_type","source_entity_id","target_entity_type","target_entity_id","relationship_type","source_id","source_record_id");--> statement-breakpoint
++CREATE INDEX "knowledge_edges_source_idx" ON "knowledge_edges" USING btree ("source_entity_type","source_entity_id","relationship_type");--> statement-breakpoint
++CREATE INDEX "knowledge_edges_target_idx" ON "knowledge_edges" USING btree ("target_entity_type","target_entity_id","relationship_type");--> statement-breakpoint
++CREATE UNIQUE INDEX "risk_factors_slug_uq" ON "risk_factors" USING btree ("slug");--> statement-breakpoint
++CREATE UNIQUE INDEX "epi_obs_uq" ON "epidemiology_observations" USING btree ("cancer_id","geography_id","year","sex","age_group","metric","source_id","site_definition");--> statement-breakpoint
++CREATE INDEX "epi_obs_lookup_idx" ON "epidemiology_observations" USING btree ("metric","geography_id","year","sex");--> statement-breakpoint
++CREATE INDEX "epi_obs_cancer_idx" ON "epidemiology_observations" USING btree ("cancer_id","metric");--> statement-breakpoint
++CREATE INDEX "survival_obs_cancer_idx" ON "survival_observations" USING btree ("cancer_id","survival_type","duration_months");--> statement-breakpoint
++CREATE UNIQUE INDEX "ai_answers_uq" ON "ai_answers" USING btree ("kind","question_hash","prompt_version");--> statement-breakpoint
++CREATE UNIQUE INDEX "api_keys_hash_uq" ON "api_keys" USING btree ("key_hash");--> statement-breakpoint
++CREATE UNIQUE INDEX "entity_counters_uq" ON "entity_counters" USING btree ("entity_type","entity_id");--> statement-breakpoint
++CREATE INDEX "entity_counters_trials_idx" ON "entity_counters" USING btree ("entity_type","active_trial_count");--> statement-breakpoint
++CREATE UNIQUE INDEX "metric_definitions_slug_uq" ON "metric_definitions" USING btree ("slug");--> statement-breakpoint
++CREATE INDEX "ranking_snapshots_lookup_idx" ON "ranking_snapshots" USING btree ("metric_slug","scope_key","is_current");--> statement-breakpoint
++CREATE UNIQUE INDEX "rankings_uq" ON "rankings" USING btree ("snapshot_id","cancer_id");--> statement-breakpoint
++CREATE INDEX "rankings_cancer_idx" ON "rankings" USING btree ("cancer_id","metric_slug");--> statement-breakpoint
++CREATE INDEX "rankings_lookup_idx" ON "rankings" USING btree ("metric_slug","scope_key","rank");
+ \ No newline at end of file

added packages/database/migrations/meta/0000_snapshot.json +6918 −0

Large diff (6,919 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/migrations/meta/_journal.json +13 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/package.json +37 −0

Large diff (37 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/client.ts +42 −0

Large diff (42 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/ids.ts +25 −0

Large diff (25 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/index.ts +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/migrate.ts +40 −0

Large diff (40 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/_common.ts +23 −0

Large diff (23 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/cancers.ts +139 −0

Large diff (139 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/drugs.ts +84 −0

Large diff (84 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/epidemiology.ts +61 −0

Large diff (61 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/evidence.ts +89 −0

Large diff (89 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/genomics.ts +184 −0

Large diff (184 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/index.ts +9 −0

Large diff (9 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/publications.ts +68 −0

Large diff (68 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/rankings.ts +138 −0

Large diff (138 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/sources.ts +228 −0

Large diff (228 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/schema/trials.ts +115 −0

Large diff (115 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/seed-data/geographies.ts +101 −0

Large diff (101 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/seed-data/metrics.ts +285 −0

Large diff (285 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/src/seed.ts +49 −0

Large diff (49 lines) is not loaded by default to keep the page fast.Load diff

added packages/database/tsconfig.json +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added packages/ontology/package.json +27 −0

Large diff (27 lines) is not loaded by default to keep the page fast.Load diff

added packages/ontology/src/index.ts +3 −0

Large diff (3 lines) is not loaded by default to keep the page fast.Load diff

added packages/ontology/src/qualifiers.ts +76 −0

Large diff (76 lines) is not loaded by default to keep the page fast.Load diff

added packages/ontology/src/reconcile.ts +106 −0

Large diff (106 lines) is not loaded by default to keep the page fast.Load diff

added packages/ontology/src/top-level.ts +75 −0

Large diff (75 lines) is not loaded by default to keep the page fast.Load diff

added packages/ontology/test/qualifiers.test.ts +55 −0

Large diff (55 lines) is not loaded by default to keep the page fast.Load diff

added packages/ontology/tsconfig.json +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added packages/ontology/vitest.config.ts +3 −0

Large diff (3 lines) is not loaded by default to keep the page fast.Load diff

added packages/ranking/package.json +28 −0

Large diff (28 lines) is not loaded by default to keep the page fast.Load diff

added packages/ranking/src/counters.ts +140 −0

Large diff (140 lines) is not loaded by default to keep the page fast.Load diff

added packages/ranking/src/engine.ts +173 −0

Large diff (173 lines) is not loaded by default to keep the page fast.Load diff

added packages/ranking/src/index.ts +4 −0

Large diff (4 lines) is not loaded by default to keep the page fast.Load diff

added packages/ranking/src/rank.ts +50 −0

Large diff (50 lines) is not loaded by default to keep the page fast.Load diff

added packages/ranking/src/trace.ts +51 −0

Large diff (51 lines) is not loaded by default to keep the page fast.Load diff

added packages/ranking/test/rank.test.ts +33 −0

Large diff (33 lines) is not loaded by default to keep the page fast.Load diff

added packages/ranking/tsconfig.json +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added packages/ranking/vitest.config.ts +3 −0

Large diff (3 lines) is not loaded by default to keep the page fast.Load diff

added packages/shared/package.json +26 −0

Large diff (26 lines) is not loaded by default to keep the page fast.Load diff

added packages/shared/src/env.ts +16 −0

Large diff (16 lines) is not loaded by default to keep the page fast.Load diff

added packages/shared/src/ids.ts +70 −0

Large diff (70 lines) is not loaded by default to keep the page fast.Load diff

added packages/shared/src/index.ts +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added packages/shared/src/logger.ts +9 −0

Large diff (9 lines) is not loaded by default to keep the page fast.Load diff

added packages/shared/src/normalize.ts +40 −0

Large diff (40 lines) is not loaded by default to keep the page fast.Load diff

added packages/shared/src/provenance.ts +85 −0

Large diff (85 lines) is not loaded by default to keep the page fast.Load diff

added packages/shared/tsconfig.json +9 −0

Large diff (9 lines) is not loaded by default to keep the page fast.Load diff

added pnpm-lock.yaml +2023 −0

Large diff (2,023 lines) is not loaded by default to keep the page fast.Load diff

added pnpm-workspace.yaml +4 −0

Large diff (4 lines) is not loaded by default to keep the page fast.Load diff

added scripts/ci.ts +136 −0

Large diff (136 lines) is not loaded by default to keep the page fast.Load diff

added tsconfig.base.json +22 −0

Large diff (22 lines) is not loaded by default to keep the page fast.Load diff

added vitest.workspace.ts +1 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

spb/cancerindex

Foundation: monorepo, schema, provenance SDK, ontology resolver, ranking engine, OncoTree connector

Theme