spb/cancerindex

1# CLAUDE.md — CancerIndex.io (repository guide)23The full product specification (364 sections) lives in `docs/SPEC-original.md`. This file is the4condensed, operational version every contributor (human or agent) must follow.56## Mission7Build **the global index of cancer**: a provenance-first, continuously updated, transparently8sourced oncology ontology connecting epidemiology, genomics, biomarkers, therapies, trials,9regulatory evidence and literature — for every recognized malignant disease entity, not a list of1030 common cancers. Not a physician, not a diagnostic tool, no treatment recommendations.1112## Non-negotiables (from the spec)131. **Provenance first** — no scientific number without a `provenance` row; derived values carry a14   `formula_version` and their inputs (§2, §250-253). Raw payloads go to the data lake (§26).152. **Layers stay separable**: RAW → NORMALIZED → CANONICAL → DERIVED → RANKED → AI (§2).163. **Never fake data**: no mock/hardcoded statistics in UI; show "Data not yet available" (§281-283).174. **Identifiers are first-class**: keep every upstream ID in `*_codes` / dedicated columns (§347);18   public IDs are `CI-<NS>-00000001` minted via `mintId` (§6), never DB integers.195. **Reconciliation before ingestion**: IDs → curated aliases → normalized strings; LLM only as a20   *candidate* generator; unknown labels go to `unresolved_labels`, never dropped (§69, §222, §340).216. **Time-aware observations**: never overwrite year X with year Y; store both (§71).227. **Context on every edge**: cancer context, direction, evidence level, provenance (§29, §244).238. **Licensing gate**: no connector goes live before manifest `licenseStatus` is reviewed; IARC/24   GLOBOCAN stays `review`, SEER `awaiting_credentials` until keys/terms are settled (§10.4, §142).259. **Verify docs before coding a connector** — official docs, smoke test with curl, fixture, tests,26   prove on 10/100/1000 records before a full sync (§224-227). Record `documentationVerifiedAt`.2710. **Rate limits, idempotency, restartability, anomaly guard** (never mass-delete on a shrunken28    response) (§90-92, §171, §229).2911. **Rankings need scope + year + source + formula version + "Why this rank?"** (§33, §183).30    Composite scores are Phase 2+ (§353).3112. **Scientific safety labels**: observed / published / curated / regulatory / guideline /32    computed / AI-generated — never merged (§3). Population survival ≠ individual prognosis (§325).3334## Repository layout35```36apps/web        Next.js 16 (webpack build, --webpack), Tailwind v4, server components, /api/v1 proxied to apps/api; pages: cancers, explore (data explorer), trials (+intelligence, map, terminated), drugs, genes, approvals, pipeline, rankings, research-gap, graph, pulse, data-updates, countries, compare, taxonomy, sources, methodology37apps/api        Fastify /v1 public API (JSON envelope { data, sources, dataRelease }); routes/*.ts incl. epidemiology, intelligence, trial-sites, research-gap, approvals (+pipeline), graph38workers/        pg-boss scheduler: connector schedules, counters, rankings39packages/shared     ids, provenance types, normalization, logger40packages/database   Drizzle schema (snake_case casing), migrations, seed (metrics + geographies only)41packages/ontology   qualifier rules, CancerResolver (alias/code reconciliation), TOP_LEVEL_CANCERS42packages/connectors SDK (manifest, HttpClient, RawLake, RunContext, validators) + connectors/<id>/{manifest.ts,index.ts,fixtures/,*.test.ts}43packages/ranking    counters (entity_counters), intelligence (trial_intelligence, trial sites, drug_pipeline, research_gap_components), ranking engine (snapshots + lineage), trace44scripts/ci.ts       operator CLI: `pnpm cix connectors|run <id>|run-all|health|sources:sync|counters|intel|reconcile-drugs|rank|stats|trace|doctor|alerts`45docs/               architecture, data model, methodology, source policy, ADRs, connector docs46data/raw            data lake (gitignored) — {source}/{date}/{entity}/{runId}-{part}.jsonl.gz47```4849## Conventions50- TypeScript strict, ESM, NodeNext imports with `.js` suffix inside packages; web app uses51  `moduleResolution: bundler` + webpack `extensionAlias` (never import `@cancerindex/connectors` in web).52- Database: `getDb()` singleton; Drizzle `casing: 'snake_case'`; raw SQL via `sql` template for53  aggregates. Schema changes = new migration via `pnpm db:generate` (one per integration), never54  manual prod mutation. Agents developing in parallel use `drizzle-kit push` on their own DB and do55  **not** commit migration files; the integrator generates the migration.56- Connector = `class extends Connector` with `manifest`, `healthCheck(ctx)`, `sync(ctx)`; register57  in `packages/connectors/src/registry.ts`. Use `ctx.http` (rate-limited), `ctx.upsertSourceRecord`58  (idempotent + raw lake), `ctx.addProvenance`, `ctx.recordUnresolved`, `ctx.cursor` (restart),59  `ctx.shouldStop()` between pages (time budget), validators from `sdk/validate.ts`.60- Reconciliation: `new CancerResolver(db)` → `warm()` → `byCode()/byLabel()/resolve()`; store61  `match_type` on every mapping; enqueue misses with `ctx.recordUnresolved`.62- Tests: vitest; connector tests run against **fixtures** (sanitized JSON in `fixtures/`), never63  live APIs; cover normal / empty / pagination / rate-limit / server error / malformed record.64- UI: scientific-editorial aesthetic (off-white, charcoal, muted neutrals, dense tables, no65  gradients/cards-everywhere), WCAG-minded, mobile first-class, light + dark themes (tokens only:66  `--color-*` and `--color-series-*` in globals.css; never hard-code hex in components). Server67  components must not receive a prop named `ref` (React reserves it; rendering fails silently in prod). Every number shows source badge,68  unit, population, period and a freshness line. Empty state = "Data not yet available".69- Language: English UI; code comments English; commit messages English.7071## Running locally72```73createdb cancerindex && cp .env.example .env74pnpm install && pnpm db:migrate && pnpm db:seed && pnpm cix sources:sync75pnpm cix run oncotree --mode dry_run   # smoke; then without --mode76pnpm cix counters && pnpm cix rank77pnpm dev:api & pnpm dev:web78```7980## Deployment81**Since 2026-09-14: OVH server BHS128b** (`ssh BHS128b`, Ubuntu 24.04), outside `mld` — code `~/apps/cancerindex`, Postgres 17 + pgvector82native (db/role `cancerindex`), PM2 under systemd (`~/apps/pm2.cancerindex.config.cjs`, generated from the mld manifest kept in83`~/apps/.manifests/`). Web :8250, API :8251, PM2 processes `cancerindex-web|api|worker|backup`, domain www.cancerindex.io served by the84MacLustr Tunnel (BHS64 Caddy → wg1 10.67.0.62:8250). Release procedure: `deploy/README.md` (rsync + pnpm install + migrate + build + pm2 restart).85Former home: MacLustr node M4M64b via `mld` (manifest `~/dispatch/apps/cancerindex.json`), kept as a cold copy.86Source of truth for the code: spbgit `cancerindex.git`. Daily maintenance (UTC, worker): counters8706:00 → intelligence 06:15 → rankings 06:30; `pnpm cix intel` recomputes the derived layer by hand.88