spb/cancerindex

1# CancerIndex — Architecture23CancerIndex.io is a provenance-first oncology knowledge platform: a canonical cancer ontology4connected to epidemiology, genomics, biomarkers, therapies, clinical trials, regulatory status and5literature, with transparent, reproducible rankings. This document describes how the system is6layered, which packages implement each layer, how data flows, and how the platform is deployed.7The condensed rules every contributor follows are in `CLAUDE.md`; the full specification is8`docs/SPEC-original.md`.910## 1. Layers1112The platform is organised as six separable layers (CLAUDE.md §2). Each layer only reads from the13layer below it and writes to its own tables; nothing is ever overwritten across layers.1415| Layer | What lives here | Tables (examples) | Written by |16|---|---|---|---|17| **RAW** | Exact upstream payloads, gzip JSON Lines in the data lake, plus a hash per record | `source_records`, files under `data/raw/{source}/{date}/{entity}/{runId}-{part}.jsonl.gz` | connectors (`ctx.upsertSourceRecord`) |18| **NORMALIZED** | Source-native structures kept intact but typed and validated (CIViC evidence items, ClinVar interpretations, trial conditions before mapping) | `civic_evidence_items`, `variant_clinical_significance`, `trial_conditions`, `trial_interventions`, `connector_field_stats` | connectors |19| **CANONICAL** | One row per real-world concept with stable `CI-*` identifiers, aliases, cross-reference codes, multi-dimensional hierarchy | `cancers`, `cancer_aliases`, `cancer_codes`, `cancer_hierarchy`, `genes`, `variants`, `drugs`, `clinical_trials`, `publications`, `geographies`, `anatomical_sites` | connectors through `CancerResolver` reconciliation |20| **DERIVED** | Numbers computed from canonical data, always with provenance and a formula version | `epidemiology_observations`, `survival_observations`, `cancer_gene_frequencies`, `literature_counts`, `entity_counters`, `knowledge_edges`, `trial_pulse` | connectors (observations) and `@cancerindex/ranking` (`refreshCounters`) |21| **RANKED** | Ranking snapshots per metric × scope × formula version with per-row lineage | `metric_definitions`, `ranking_snapshots`, `rankings` | `@cancerindex/ranking` (`computeAllRankings`) |22| **AI** | Cached, database-grounded syntheses with model and prompt version (not implemented in Phase 1, see `docs/AI.md`) | `ai_answers`, `entity_embeddings` | — |2324Cross-cutting tables: `sources` (registry + license status), `ingest_runs`, `connector_cursors`,25`provenance`, `unresolved_labels`, `change_events`, `audit_log`, `entity_merges`, `api_keys`.2627Two invariants hold across every layer:28291. **Provenance first.** No scientific number exists without a `provenance` row (source, dataset,30   version, retrieval time, URL, evidence type, license). Derived values also carry31   `formula_version` and their inputs. `traceValue()` in `@cancerindex/ranking` walks a ranked value32   back to the observation, the provenance row and the raw record.332. **Never fake data.** Missing data is absent, never zero. The API returns `counters: null` when34   counters have not been computed, and a ranking without a snapshot returns `status: "not_available"`.3536## 2. Repository and packages3738```39cancerindex/40├── apps/41│   ├── web/                 Next.js 16 (webpack), Tailwind v4, server components; proxies /api/v1/* → API42│   └── api/                 Fastify 5 public API (/v1), OpenAPI from zod route schemas, rate limiting, admin43├── workers/                 pg-boss scheduler (connector cron, counters, rankings, health probes)44├── packages/45│   ├── shared/              ids (CI-*), provenance types, normalization, pino logger, env helpers46│   ├── database/            Drizzle schema (snake_case), migrations, seed (metrics + geographies)47│   ├── ontology/            qualifier rules, CancerResolver, TOP_LEVEL_CANCERS, NCIt roots48│   ├── connectors/          SDK (manifest, HttpClient, RawLake, RunContext, validators) + connectors/<id>/49│   └── ranking/             counters, ranking engine (snapshots + lineage), trace50├── scripts/ci.ts            operator CLI (`pnpm cix …`)51├── deploy/                  mld manifest, first-run bootstrap, deployment notes52├── docs/                    this folder53└── data/raw                 data lake (gitignored)54```5556Dependency direction (no cycles): `shared` ← `database` ← `ontology` ← `connectors` ← `ranking`;57`apps/api` depends on `database`, `shared`, `ranking`, `ontology`; `workers` additionally depends58on `connectors`; `apps/web` never imports `connectors` (CLAUDE.md conventions).5960### Connector SDK6162A connector is a class extending `Connector` with a validated `manifest` (zod, see63`packages/connectors/src/sdk/manifest.ts`), a `healthCheck(ctx)` and a `sync(ctx)`. The `RunContext`64provides a rate-limited `HttpClient` (token bucket, bounded concurrency, exponential backoff,65`Retry-After`), a `RawLake` writer, idempotent `upsertSourceRecord` (sha256 payload hash),66`addProvenance`, `recordUnresolved` (curation queue), `recordChange`, a restartable `cursor`, and a67time budget checked with `shouldStop()`. Schema drift is detected by recording observed field68names/types per entity (`connector_field_stats`). `runConnector()` handles bookkeeping in69`ingest_runs` / `connector_cursors`, the credentials gate (`awaiting_credentials`) and the license70gate (`licenseStatus: blocked` never ingests).7172### Reconciliation7374Labels are mapped to canonical cancers by `CancerResolver`: identifiers first (NCIt, DOID, OncoTree,75UMLS, MeSH…), then curated aliases, then normalized strings. Every mapping stores a `match_type`76(`EXACT_IDENTIFIER`, `CURATED_EXACT`, `ONTOLOGY_EXACT`, `CURATED_BROADER`, `CURATED_NARROWER`,77`ALIAS`, `PROBABILISTIC`, `UNRESOLVED`). Unknown labels are stored in `unresolved_labels` with an78optional trigram suggestion and are resolved by a curator through the admin API, which adds a curated79alias so the next connector run picks it up.8081## 3. Data flow8283```84 upstream source ──HTTP──▶ connector.sync(ctx)85        │                     │  upsertSourceRecord → source_records + data/raw/*.jsonl.gz   (RAW)86        │                     │  validate / typed rows                                         (NORMALIZED)87        │                     │  CancerResolver → canonical ids, match_type; misses → unresolved_labels88        │                     │  addProvenance → provenance; observations / evidence / edges   (CANONICAL, DERIVED)89        │                     └  recordChange → change_events ; ingest_runs finalised90        ▼91 workers: maintenance.counters → refreshCounters() → entity_counters                            (DERIVED)92          maintenance.rank     → computeAllRankings() → ranking_snapshots + rankings            (RANKED)93        ▼94 apps/api (/v1) ── envelope { data, sources, dataRelease, generatedAt } ──▶ apps/web, API consumers95```9697Runs are scheduled by the worker from each manifest's `schedule` (cron, UTC) and can be triggered98by an operator (`pnpm cix run <id>` or `POST /v1/admin/connectors/:id/run`). A `connector.run` job is99`stately` per connector id (never queued twice, never concurrent). When a run creates or updates100records, the worker enqueues `maintenance.counters` with `thenRank: true`, so derived and ranked101layers follow canonical changes. A daily counters (06:00 UTC) and rank (06:30 UTC) pass covers102everything else; an hourly `health.probe` refreshes `connector_cursors.health`.103104## 4. Public API105106`apps/api` is a read-only Fastify 5 service. Route schemas are written in zod (via107`fastify-type-provider-zod`) and the OpenAPI 3.1 document is generated from them (`/v1/openapi.json`,108Swagger UI at `/v1/docs`) — documentation is never maintained by hand (CLAUDE.md §357).109110Every `/v1` response uses the envelope `{ data, sources, dataRelease, generatedAt, …pagination }`.111`sources` lists the distinct upstream sources behind the payload with license and attribution;112`dataRelease` is `CancerIndex <YYYY-MM>` derived from the latest successful ingest.113114Other properties: request correlation id (`x-request-id`, honoured when supplied), pino structured115logs, `@fastify/rate-limit` (anonymous 60/min per IP; API keys looked up by sha256 hash with116per-key limits, `X-RateLimit-*` headers), ETag + gzip/brotli, CORS. Admin endpoints under117`/v1/admin/*` require `x-admin-token` (constant-time compare) and write `audit_log` rows for every118mutation; they never execute ingestion in-process, they enqueue pg-boss jobs.119120See `docs/API.md` for endpoint examples.121122## 5. Web application123124`apps/web` (Next.js 16, server components) renders the public site. It talks to the API through a125`/api/v1/*` proxy to `http://127.0.0.1:${API_PORT}/v1/*`, so the browser only ever sees one origin.126The UI follows the scientific-editorial aesthetic and shows, for every number: source badge, unit,127population, period and freshness; empty states read "Data not yet available".128129## 6. Ports, processes and environment130131| Process | Command | Port | Notes |132|---|---|---|---|133| `cancerindex-web` | `next start -p 8250 -H 0.0.0.0` | 8250 | public entry (ngrok → www.cancerindex.io) |134| `cancerindex-api` | `tsx apps/api/src/server.ts` | 8251 (127.0.0.1) | reached only via the web proxy / localhost |135| `cancerindex-worker` | `tsx workers/main.ts` | — | pg-boss schema `pgboss` in the same database |136| PostgreSQL 17 | database `cancerindex`, extensions `pg_trgm`, `unaccent`, `vector` | 5432 | single node |137138Key environment variables (see `.env.example`): `DATABASE_URL`, `API_HOST`/`API_PORT`, `WEB_PORT`,139`CI_API_URL`, `CI_DATA_DIR` (data lake root), `ADMIN_TOKEN`, `NCBI_TOOL`/`NCBI_EMAIL`/`NCBI_API_KEY`,140`SEER_API_KEY`, `WORKER_CONCURRENCY`, `CI_MAX_RUN_MINUTES`, `LOG_LEVEL`, `CI_SERVICE` (log label).141142## 7. Deployment143144Production runs on the MacLustr cluster through the `mld` orchestrator (gateway M1M32). The145manifest is `deploy/mld-manifest.cancerindex.json` (copied to `M1M32:~/dispatch/apps/cancerindex.json`);146`mld stage <dir> cancerindex` uploads a `git archive` of the repository and `mld deploy cancerindex`147places it on the preferred node (M4M64b: Postgres 17 + pgvector, pnpm, large disk), runs the148`post_sync` hooks (`pnpm install`, `createdb` if missing, migrate, seed, `sources:sync`, web build)149and starts three PM2 processes plus the ngrok tunnel. `deploy/first-run.sh` bootstraps the data in150the recommended connector order. Details in `deploy/README.md`.151152## 7b. Operations: checkpoints, anomaly guard, alerts, backups, doctor153154Reference: `docs/connectors/README.md` (runtime guarantees), `deploy/README.md` (day-2 procedures),155`docs/schema-changes-ops.md` (ops tables and indexes added outside the generated migrations).156157- **Checkpoints (§90).** `RunContext` persists `connector_cursors.cursor` (and158  `ingest_runs.cursor_after`) automatically every `manifest.checkpointEvery` upserted records159  (default 2 000) or 60 s when the cursor changed, on `ctx.saveCursor()`, and on SIGTERM/SIGINT160  (`runConnector` installs handlers for the duration of the run: first signal → cursor saved, run161  `aborted`, connector stops at its next `shouldStop()`; second signal → immediate exit). A killed162  run therefore resumes from the last completed page; payload hashes make the overlap idempotent.163  `RawLake` serialises raw writes through one promise chain (single shared drain, no listener164  pile-up) and can `flush()` gzip blocks before a kill.165- **Anomaly guard (§171).** `ctx.guardCount(entity, total)` compares a declared total with the166  previous successful run's `records_fetched` (`anomalyGuard.minRatioOfPrevious`, default 0.5) and167  refuses to continue: run `failed` + `ingest_runs.anomaly`, critical alert, previous data untouched.168  Wired into OncoTree, NCIt and the ClinicalTrials.gov full crawl; connectors keep their fixed floors169  (HGNC, GDC, USCS, CIViC…). Nothing is ever mass-deleted on a shrunken response.170- **Alerts (§170).** Table `system_alerts` (`kind`, `severity info|warn|critical`, `connector_id`,171  `message`, `detail`, `first_seen_at`, `last_seen_at`, `count`, `status open|acknowledged|resolved`),172  deduplicated on kind + connector + message. Raised by `runConnector` (failure, anomaly, aborted,173  schema drift) and by the worker's hourly health probe (`source_failing`, `source_stale` = no174  success within 2× the cron interval of an active connector); resolved on the next success.175  Surfaces: `pnpm cix alerts`, `pnpm cix doctor`, `GET /v1/admin/alerts`.176- **Backups (§172).** PM2 process `cancerindex-backup` (`cron_restart "20 5 * * *"`,177  `autorestart: false`) runs `deploy/backup.sh`: `pg_dump -Fc` to178  `~/apps/cancerindex/backups/cancerindex-YYYYMMDD-HHMM.dump`, verified with `pg_restore --list`,179  14 daily + 8 weekly retained, `logs/backup.log`, non-zero exit on failure. `deploy/restore.sh180  <dump>` restores into a fresh `cancerindex_restore_<ts>` database (never over production). The raw181  lake (`data/raw`) is mirrored separately with rsync; both are needed for TRACE.182- **Doctor.** `pnpm cix doctor` prints the readiness report (environment, database + extensions +183  pending migrations vs `drizzle.__drizzle_migrations`, table sizes, every connector's status /184  license / health / last success age / last run status-anomaly-drift / cursor summary / stale flag,185  unresolved-label backlog, `data/raw` disk usage and free space, ranking-snapshot freshness, open186  alerts) and exits 1 on a hard failure (unreachable database, missing required extension, pending187  migration, unwritable data dir, missing ops schema, critical alert).188- **Indexes.** Indexes that drizzle cannot express or that were added by the performance review are189  created idempotently in `packages/database/src/migrate.ts` after the SQL migrations190  (`createPerformanceIndexes`): GIN on `civic_evidence_items.gene_ids/variant_ids/therapy_ids`191  (the counters use the containment form `@> ARRAY[id]::text[]`), `trial_conditions(cancer_id,192  trial_id)`, `clinical_trials(overall_status, study_type)` and `(overall_status,193  last_update_posted_date)`, `cancer_aliases(normalized text_pattern_ops)`,194  `ingest_runs(connector_id, status, started_at)`.195196## 8. Phase 1 boundaries197198- Rankings are count-based (trials, literature, curated evidence, genes, cohorts) for all entities199  and burden/lethality/gap metrics only where epidemiology observations exist for the scope.200- No composite score (ADR-006). No AI synthesis (docs/AI.md). No HGVS normalisation / liftover201  (ADR-002 plans Python/DuckDB workers for that).202- GLOBOCAN stays in license review and SEER in `awaiting_credentials`; the platform is honest about203  the resulting gaps ("awaiting license review").204