# CLAUDE.md — CancerIndex.io (repository guide)

The full product specification (364 sections) lives in docs/SPEC-original.md. This file is the condensed, operational version every contributor (human or agent) must follow.

# Mission

Build the global index of cancer: a provenance-first, continuously updated, transparently sourced oncology ontology connecting epidemiology, genomics, biomarkers, therapies, trials, regulatory evidence and literature — for every recognized malignant disease entity, not a list of 30 common cancers. Not a physician, not a diagnostic tool, no treatment recommendations.

# Non-negotiables (from the spec)

Provenance first — no scientific number without a provenance row; derived values carry a formula_version and their inputs (§2, §250-253). Raw payloads go to the data lake (§26).
Layers stay separable: RAW → NORMALIZED → CANONICAL → DERIVED → RANKED → AI (§2).
Never fake data: no mock/hardcoded statistics in UI; show "Data not yet available" (§281-283).
Identifiers are first-class: keep every upstream ID in *_codes / dedicated columns (§347); public IDs are CI-<NS>-00000001 minted via mintId (§6), never DB integers.
Reconciliation before ingestion: IDs → curated aliases → normalized strings; LLM only as a candidate generator; unknown labels go to unresolved_labels, never dropped (§69, §222, §340).
Time-aware observations: never overwrite year X with year Y; store both (§71).
Context on every edge: cancer context, direction, evidence level, provenance (§29, §244).
Licensing gate: no connector goes live before manifest licenseStatus is reviewed; IARC/ GLOBOCAN stays review, SEER awaiting_credentials until keys/terms are settled (§10.4, §142).
Verify docs before coding a connector — official docs, smoke test with curl, fixture, tests, prove on 10/100/1000 records before a full sync (§224-227). Record documentationVerifiedAt.
Rate limits, idempotency, restartability, anomaly guard (never mass-delete on a shrunken response) (§90-92, §171, §229).
Rankings need scope + year + source + formula version + "Why this rank?" (§33, §183). Composite scores are Phase 2+ (§353).
Scientific safety labels: observed / published / curated / regulatory / guideline / computed / AI-generated — never merged (§3). Population survival ≠ individual prognosis (§325).

# Repository layout

text

apps/web        Next.js 16 (webpack build, --webpack), Tailwind v4, server components, /api/v1 proxied to apps/api; pages: cancers, explore (data explorer), trials (+intelligence, map, terminated), drugs, genes, approvals, pipeline, rankings, research-gap, graph, pulse, data-updates, countries, compare, taxonomy, sources, methodology
apps/api        Fastify /v1 public API (JSON envelope { data, sources, dataRelease }); routes/*.ts incl. epidemiology, intelligence, trial-sites, research-gap, approvals (+pipeline), graph
workers/        pg-boss scheduler: connector schedules, counters, rankings
packages/shared     ids, provenance types, normalization, logger
packages/database   Drizzle schema (snake_case casing), migrations, seed (metrics + geographies only)
packages/ontology   qualifier rules, CancerResolver (alias/code reconciliation), TOP_LEVEL_CANCERS
packages/connectors SDK (manifest, HttpClient, RawLake, RunContext, validators) + connectors/<id>/{manifest.ts,index.ts,fixtures/,*.test.ts}
packages/ranking    counters (entity_counters), intelligence (trial_intelligence, trial sites, drug_pipeline, research_gap_components), ranking engine (snapshots + lineage), trace
scripts/ci.ts       operator CLI: `pnpm cix connectors|run <id>|run-all|health|sources:sync|counters|intel|reconcile-drugs|rank|stats|trace|doctor|alerts`
docs/               architecture, data model, methodology, source policy, ADRs, connector docs
data/raw            data lake (gitignored) — {source}/{date}/{entity}/{runId}-{part}.jsonl.gz

# Conventions

TypeScript strict, ESM, NodeNext imports with .js suffix inside packages; web app uses moduleResolution: bundler + webpack extensionAlias (never import @cancerindex/connectors in web).
Database: getDb() singleton; Drizzle casing: 'snake_case'; raw SQL via sql template for aggregates. Schema changes = new migration via pnpm db:generate (one per integration), never manual prod mutation. Agents developing in parallel use drizzle-kit push on their own DB and do not commit migration files; the integrator generates the migration.
Connector = class extends Connector with manifest, healthCheck(ctx), sync(ctx); register in packages/connectors/src/registry.ts. Use ctx.http (rate-limited), ctx.upsertSourceRecord (idempotent + raw lake), ctx.addProvenance, ctx.recordUnresolved, ctx.cursor (restart), ctx.shouldStop() between pages (time budget), validators from sdk/validate.ts.
Reconciliation: new CancerResolver(db) → warm() → byCode()/byLabel()/resolve(); store match_type on every mapping; enqueue misses with ctx.recordUnresolved.
Tests: vitest; connector tests run against fixtures (sanitized JSON in fixtures/), never live APIs; cover normal / empty / pagination / rate-limit / server error / malformed record.
UI: scientific-editorial aesthetic (off-white, charcoal, muted neutrals, dense tables, no gradients/cards-everywhere), WCAG-minded, mobile first-class, light + dark themes (tokens only: --color-* and --color-series-* in globals.css; never hard-code hex in components). Server components must not receive a prop named ref (React reserves it; rendering fails silently in prod). Every number shows source badge, unit, population, period and a freshness line. Empty state = "Data not yet available".
Language: English UI; code comments English; commit messages English.

# Running locally

text

createdb cancerindex && cp .env.example .env
pnpm install && pnpm db:migrate && pnpm db:seed && pnpm cix sources:sync
pnpm cix run oncotree --mode dry_run   # smoke; then without --mode
pnpm cix counters && pnpm cix rank
pnpm dev:api & pnpm dev:web

# Deployment

Since 2026-09-14: OVH server BHS128b (ssh BHS128b, Ubuntu 24.04), outside mld — code ~/apps/cancerindex, Postgres 17 + pgvector native (db/role cancerindex), PM2 under systemd (~/apps/pm2.cancerindex.config.cjs, generated from the mld manifest kept in ~/apps/.manifests/). Web :8250, API :8251, PM2 processes cancerindex-web|api|worker|backup, domain www.cancerindex.io served by the MacLustr Tunnel (BHS64 Caddy → wg1 10.67.0.62:8250). Release procedure: deploy/README.md (rsync + pnpm install + migrate + build + pm2 restart). Former home: MacLustr node M4M64b via mld (manifest ~/dispatch/apps/cancerindex.json), kept as a cold copy. Source of truth for the code: spbgit cancerindex.git. Daily maintenance (UTC, worker): counters 06:00 → intelligence 06:15 → rankings 06:30; pnpm cix intel recomputes the derived layer by hand.