# CLAUDE.md — CancerIndex.io (repository guide)

The full product specification (364 sections) lives in `docs/SPEC-original.md`. This file is the
condensed, operational version every contributor (human or agent) must follow.

## Mission
Build **the global index of cancer**: a provenance-first, continuously updated, transparently
sourced oncology ontology connecting epidemiology, genomics, biomarkers, therapies, trials,
regulatory evidence and literature — for every recognized malignant disease entity, not a list of
30 common cancers. Not a physician, not a diagnostic tool, no treatment recommendations.

## Non-negotiables (from the spec)
1. **Provenance first** — no scientific number without a `provenance` row; derived values carry a
   `formula_version` and their inputs (§2, §250-253). Raw payloads go to the data lake (§26).
2. **Layers stay separable**: RAW → NORMALIZED → CANONICAL → DERIVED → RANKED → AI (§2).
3. **Never fake data**: no mock/hardcoded statistics in UI; show "Data not yet available" (§281-283).
4. **Identifiers are first-class**: keep every upstream ID in `*_codes` / dedicated columns (§347);
   public IDs are `CI-<NS>-00000001` minted via `mintId` (§6), never DB integers.
5. **Reconciliation before ingestion**: IDs → curated aliases → normalized strings; LLM only as a
   *candidate* generator; unknown labels go to `unresolved_labels`, never dropped (§69, §222, §340).
6. **Time-aware observations**: never overwrite year X with year Y; store both (§71).
7. **Context on every edge**: cancer context, direction, evidence level, provenance (§29, §244).
8. **Licensing gate**: no connector goes live before manifest `licenseStatus` is reviewed; IARC/
   GLOBOCAN stays `review`, SEER `awaiting_credentials` until keys/terms are settled (§10.4, §142).
9. **Verify docs before coding a connector** — official docs, smoke test with curl, fixture, tests,
   prove on 10/100/1000 records before a full sync (§224-227). Record `documentationVerifiedAt`.
10. **Rate limits, idempotency, restartability, anomaly guard** (never mass-delete on a shrunken
    response) (§90-92, §171, §229).
11. **Rankings need scope + year + source + formula version + "Why this rank?"** (§33, §183).
    Composite scores are Phase 2+ (§353).
12. **Scientific safety labels**: observed / published / curated / regulatory / guideline /
    computed / AI-generated — never merged (§3). Population survival ≠ individual prognosis (§325).

## Repository layout
```
apps/web        Next.js 16 (webpack build, --webpack), Tailwind v4, server components, /api/v1 proxied to apps/api; pages: cancers, explore (data explorer), trials (+intelligence, map, terminated), drugs, genes, approvals, pipeline, rankings, research-gap, graph, pulse, data-updates, countries, compare, taxonomy, sources, methodology
apps/api        Fastify /v1 public API (JSON envelope { data, sources, dataRelease }); routes/*.ts incl. epidemiology, intelligence, trial-sites, research-gap, approvals (+pipeline), graph
workers/        pg-boss scheduler: connector schedules, counters, rankings
packages/shared     ids, provenance types, normalization, logger
packages/database   Drizzle schema (snake_case casing), migrations, seed (metrics + geographies only)
packages/ontology   qualifier rules, CancerResolver (alias/code reconciliation), TOP_LEVEL_CANCERS
packages/connectors SDK (manifest, HttpClient, RawLake, RunContext, validators) + connectors/<id>/{manifest.ts,index.ts,fixtures/,*.test.ts}
packages/ranking    counters (entity_counters), intelligence (trial_intelligence, trial sites, drug_pipeline, research_gap_components), ranking engine (snapshots + lineage), trace
scripts/ci.ts       operator CLI: `pnpm cix connectors|run <id>|run-all|health|sources:sync|counters|intel|reconcile-drugs|rank|stats|trace|doctor|alerts`
docs/               architecture, data model, methodology, source policy, ADRs, connector docs
data/raw            data lake (gitignored) — {source}/{date}/{entity}/{runId}-{part}.jsonl.gz
```

## Conventions
- TypeScript strict, ESM, NodeNext imports with `.js` suffix inside packages; web app uses
  `moduleResolution: bundler` + webpack `extensionAlias` (never import `@cancerindex/connectors` in web).
- Database: `getDb()` singleton; Drizzle `casing: 'snake_case'`; raw SQL via `sql` template for
  aggregates. Schema changes = new migration via `pnpm db:generate` (one per integration), never
  manual prod mutation. Agents developing in parallel use `drizzle-kit push` on their own DB and do
  **not** commit migration files; the integrator generates the migration.
- Connector = `class extends Connector` with `manifest`, `healthCheck(ctx)`, `sync(ctx)`; register
  in `packages/connectors/src/registry.ts`. Use `ctx.http` (rate-limited), `ctx.upsertSourceRecord`
  (idempotent + raw lake), `ctx.addProvenance`, `ctx.recordUnresolved`, `ctx.cursor` (restart),
  `ctx.shouldStop()` between pages (time budget), validators from `sdk/validate.ts`.
- Reconciliation: `new CancerResolver(db)` → `warm()` → `byCode()/byLabel()/resolve()`; store
  `match_type` on every mapping; enqueue misses with `ctx.recordUnresolved`.
- Tests: vitest; connector tests run against **fixtures** (sanitized JSON in `fixtures/`), never
  live APIs; cover normal / empty / pagination / rate-limit / server error / malformed record.
- UI: scientific-editorial aesthetic (off-white, charcoal, muted neutrals, dense tables, no
  gradients/cards-everywhere), WCAG-minded, mobile first-class, light + dark themes (tokens only:
  `--color-*` and `--color-series-*` in globals.css; never hard-code hex in components). Server
  components must not receive a prop named `ref` (React reserves it; rendering fails silently in prod). Every number shows source badge,
  unit, population, period and a freshness line. Empty state = "Data not yet available".
- Language: English UI; code comments English; commit messages English.

## Running locally
```
createdb cancerindex && cp .env.example .env
pnpm install && pnpm db:migrate && pnpm db:seed && pnpm cix sources:sync
pnpm cix run oncotree --mode dry_run   # smoke; then without --mode
pnpm cix counters && pnpm cix rank
pnpm dev:api & pnpm dev:web
```

## Deployment
**Since 2026-09-14: OVH server BHS128b** (`ssh BHS128b`, Ubuntu 24.04), outside `mld` — code `~/apps/cancerindex`, Postgres 17 + pgvector
native (db/role `cancerindex`), PM2 under systemd (`~/apps/pm2.cancerindex.config.cjs`, generated from the mld manifest kept in
`~/apps/.manifests/`). Web :8250, API :8251, PM2 processes `cancerindex-web|api|worker|backup`, domain www.cancerindex.io served by the
MacLustr Tunnel (BHS64 Caddy → wg1 10.67.0.62:8250). Release procedure: `deploy/README.md` (rsync + pnpm install + migrate + build + pm2 restart).
Former home: MacLustr node M4M64b via `mld` (manifest `~/dispatch/apps/cancerindex.json`), kept as a cold copy.
Source of truth for the code: spbgit `cancerindex.git`. Daily maintenance (UTC, worker): counters
06:00 → intelligence 06:15 → rankings 06:30; `pnpm cix intel` recomputes the derived layer by hand.