spb/cancerindex

1# CLAUDE.md — CancerIndex.io (repository guide)23The full product specification (364 sections) lives in `docs/SPEC-original.md`. This file is the4condensed, operational version every contributor (human or agent) must follow.56## Mission7Build **the global index of cancer**: a provenance-first, continuously updated, transparently8sourced oncology ontology connecting epidemiology, genomics, biomarkers, therapies, trials,9regulatory evidence and literature — for every recognized malignant disease entity, not a list of1030 common cancers. Not a physician, not a diagnostic tool, no treatment recommendations.1112## Non-negotiables (from the spec)131. **Provenance first** — no scientific number without a `provenance` row; derived values carry a14   `formula_version` and their inputs (§2, §250-253). Raw payloads go to the data lake (§26).152. **Layers stay separable**: RAW → NORMALIZED → CANONICAL → DERIVED → RANKED → AI (§2).163. **Never fake data**: no mock/hardcoded statistics in UI; show "Data not yet available" (§281-283).174. **Identifiers are first-class**: keep every upstream ID in `*_codes` / dedicated columns (§347);18   public IDs are `CI-<NS>-00000001` minted via `mintId` (§6), never DB integers.195. **Reconciliation before ingestion**: IDs → curated aliases → normalized strings; LLM only as a20   *candidate* generator; unknown labels go to `unresolved_labels`, never dropped (§69, §222, §340).216. **Time-aware observations**: never overwrite year X with year Y; store both (§71).227. **Context on every edge**: cancer context, direction, evidence level, provenance (§29, §244).238. **Licensing gate**: no connector goes live before manifest `licenseStatus` is reviewed; IARC/24   GLOBOCAN stays `review`, SEER `awaiting_credentials` until keys/terms are settled (§10.4, §142).259. **Verify docs before coding a connector** — official docs, smoke test with curl, fixture, tests,26   prove on 10/100/1000 records before a full sync (§224-227). Record `documentationVerifiedAt`.2710. **Rate limits, idempotency, restartability, anomaly guard** (never mass-delete on a shrunken28    response) (§90-92, §171, §229).2911. **Rankings need scope + year + source + formula version + "Why this rank?"** (§33, §183).30    Composite scores are Phase 2+ (§353).3112. **Scientific safety labels**: observed / published / curated / regulatory / guideline /32    computed / AI-generated — never merged (§3). Population survival ≠ individual prognosis (§325).3334## Repository layout35```36apps/web        Next.js 16 (webpack build, --webpack), Tailwind v4, server components, /api/v1 proxied to apps/api37apps/api        Fastify /v1 public API (JSON envelope { data, sources, dataRelease })38workers/        pg-boss scheduler: connector schedules, counters, rankings39packages/shared     ids, provenance types, normalization, logger40packages/database   Drizzle schema (snake_case casing), migrations, seed (metrics + geographies only)41packages/ontology   qualifier rules, CancerResolver (alias/code reconciliation), TOP_LEVEL_CANCERS42packages/connectors SDK (manifest, HttpClient, RawLake, RunContext, validators) + connectors/<id>/{manifest.ts,index.ts,fixtures/,*.test.ts}43packages/ranking    counters (entity_counters), ranking engine (snapshots + lineage), trace44scripts/ci.ts       operator CLI: `pnpm cix connectors|run <id>|run-all|health|sources:sync|counters|rank|stats|trace`45docs/               architecture, data model, methodology, source policy, ADRs, connector docs46data/raw            data lake (gitignored) — {source}/{date}/{entity}/{runId}-{part}.jsonl.gz47```4849## Conventions50- TypeScript strict, ESM, NodeNext imports with `.js` suffix inside packages; web app uses51  `moduleResolution: bundler` + webpack `extensionAlias` (never import `@cancerindex/connectors` in web).52- Database: `getDb()` singleton; Drizzle `casing: 'snake_case'`; raw SQL via `sql` template for53  aggregates. Schema changes = new migration via `pnpm db:generate` (one per integration), never54  manual prod mutation. Agents developing in parallel use `drizzle-kit push` on their own DB and do55  **not** commit migration files; the integrator generates the migration.56- Connector = `class extends Connector` with `manifest`, `healthCheck(ctx)`, `sync(ctx)`; register57  in `packages/connectors/src/registry.ts`. Use `ctx.http` (rate-limited), `ctx.upsertSourceRecord`58  (idempotent + raw lake), `ctx.addProvenance`, `ctx.recordUnresolved`, `ctx.cursor` (restart),59  `ctx.shouldStop()` between pages (time budget), validators from `sdk/validate.ts`.60- Reconciliation: `new CancerResolver(db)` → `warm()` → `byCode()/byLabel()/resolve()`; store61  `match_type` on every mapping; enqueue misses with `ctx.recordUnresolved`.62- Tests: vitest; connector tests run against **fixtures** (sanitized JSON in `fixtures/`), never63  live APIs; cover normal / empty / pagination / rate-limit / server error / malformed record.64- UI: scientific-editorial aesthetic (off-white, charcoal, muted neutrals, dense tables, no65  gradients/cards-everywhere), WCAG-minded, mobile first-class. Every number shows source badge,66  unit, population, period and a freshness line. Empty state = "Data not yet available".67- Language: English UI; code comments English; commit messages English.6869## Running locally70```71createdb cancerindex && cp .env.example .env72pnpm install && pnpm db:migrate && pnpm db:seed && pnpm cix sources:sync73pnpm cix run oncotree --mode dry_run   # smoke; then without --mode74pnpm cix counters && pnpm cix rank75pnpm dev:api & pnpm dev:web76```7778## Deployment79MacLustr cluster via `mld` (gateway M1M32): manifest `~/dispatch/apps/cancerindex.json`,80`mld stage <dir> cancerindex` then `mld deploy cancerindex`. Web :8250, API :8251, PM2 processes81`cancerindex-web|api|worker|ngrok`, domain www.cancerindex.io (ngrok reserved). Postgres 17 +82pgvector on the node, database `cancerindex`. Source of truth for the code: spbgit `cancerindex.git`.83