README: detailed project guide with brand mark, product map, architecture, data model, connectors, derived layer, metrics, API, CLI, setup, operations, gotchas, roadmap · 91a859e · cancerindex

1

−# CancerIndex.io — the global index of cancer

1

+<p align="center">

2

+ <img src="apps/web/public/brand/logo.svg" alt="CancerIndex — the global index of cancer" width="360">

3

+</p>

2

4

3

−CancerIndex is a provenance-first, continuously updated, transparently sourced oncology knowledge

4

−platform. It connects a canonical cancer ontology (NCIt-anchored, OncoTree and registry dimensions)

5

−to epidemiology, genomics, biomarkers, therapies, clinical trials, regulatory evidence and literature

6

−— for every recognised malignant disease entity — and publishes reproducible rankings with their

7

−scope, formula version and lineage down to the raw record.

5

+<h1 align="center">CancerIndex.io</h1>

8

6

9

−It is not a physician, not a diagnostic tool, and gives no treatment recommendations.

7

+<p align="center"><strong>Understand cancer through data.</strong><br>

8

+A provenance-first, continuously updated, transparently sourced index of every recognized cancer entity —

9

+epidemiology, genomics, biomarkers, drugs, regulatory approvals, clinical trials, research and derived

10

+intelligence, cross-linked in one coherent dataset.</p>

10

11

−## Status

12

+<p align="center">

13

+ <a href="https://www.cancerindex.io">www.cancerindex.io</a> ·

14

+ <a href="https://www.cancerindex.io/api/v1/docs">API docs</a> ·

15

+ <a href="https://www.cancerindex.io/methodology">Methodology</a> ·

16

+ <a href="https://www.cancerindex.io/sources">Sources</a> ·

17

+ <a href="https://www.cancerindex.io/pulse">Pulse</a>

18

+</p>

12

19

13

−Phase 1 (2026-09). Foundation, connectors, public API, worker and web app are being built in

14

−parallel. Data currently ingested locally: OncoTree (865 cancer entities), with NCIt EVS, HGNC,

15

−CIViC, GDC, ClinVar, ClinicalTrials.gov, PubMed and CDC WONDER connectors in progress. GLOBOCAN

16

−remains under license review and SEER awaits credentials — the platform says so rather than showing

17

−numbers it cannot source.

20

+---

18

21

19

−## Repository

22

+## Table of contents

23

+

24

+1. [What CancerIndex is](#what-cancerindex-is)

25

+2. [What it refuses to do](#what-it-refuses-to-do)

26

+3. [The index in numbers](#the-index-in-numbers)

27

+4. [Product map](#product-map)

28

+5. [Architecture](#architecture)

29

+6. [Repository layout](#repository-layout)

30

+7. [Data model](#data-model)

31

+8. [Connectors](#connectors)

32

+9. [Derived intelligence layer](#derived-intelligence-layer)

33

+10. [Rankings and metrics](#rankings-and-metrics)

34

+11. [Public API](#public-api)

35

+12. [Operator CLI](#operator-cli)

36

+13. [Local development](#local-development)

37

+14. [Configuration](#configuration)

38

+15. [Quality: tests, QA, accessibility, performance](#quality-tests-qa-accessibility-performance)

39

+16. [Deployment and operations](#deployment-and-operations)

40

+17. [Brand](#brand)

41

+18. [Design language](#design-language)

42

+19. [Known gotchas](#known-gotchas)

43

+20. [Roadmap](#roadmap)

44

+21. [Licensing, attribution and contact](#licensing-attribution-and-contact)

45

+

46

+---

47

+

48

+## What CancerIndex is

49

+

50

+CancerIndex organizes fragmented oncology data into one structured, navigable system. A reader moves

51

+from a **cancer** to its **incidence and mortality**, to the **genes** altered in it, to a **variant**, to the

52

+**drugs** that target it, to the **regulatory approvals** of those drugs in each jurisdiction, to the

53

+**clinical trials** that investigate them and to the **papers** that describe them — every step a link, every

54

+number a sourced record.

55

+

56

+It is built for researchers, students, journalists, policymakers, biotech professionals and informed

57

+members of the public. It feels like a research terminal, not a health-content website: dense tables,

58

+sparse charts, one accent colour, a source badge on every figure.

59

+

60

+Five things make it different from a cancer information site:

61

+

62

+- **A canonical taxonomy first.** 9 510 active disease entities anchored on the NCI Thesaurus, with

63

+ OncoTree, ICD-10, ICD-O, DOID, UMLS, MeSH and MONDO cross-references; 36 mutually exclusive

64

+ top-level site groups (GLOBOCAN / ICD-10 ranges) for burden rankings without double counting.

65

+- **Provenance on every value.** Each observation, edge, approval and count points to a `provenance`

66

+ row (source, dataset, version, URL, retrieval date, population, methodology); raw payloads are kept

67

+ in a gzip JSON lake and can be replayed.

68

+- **Layers that stay separable.** RAW → NORMALIZED → CANONICAL → DERIVED → RANKED. Derived values

69

+ carry a `formula_version` and the inputs that produced them ("Why this rank?").

70

+- **Claim categories that never merge.** Observed · published · curated · regulatory · guideline ·

71

+ computed · AI-generated — labelled everywhere, mixed nowhere.

72

+- **No invented numbers.** Missing data is shown as *Data not yet available*. No estimates, no

73

+ placeholder statistics, no composite "worst cancer" score.

74

+

75

+## What it refuses to do

76

+

77

+- It is **not a physician**: no diagnosis, no individual prognosis, no dosing, no treatment

78

+ recommendation. Population statistics never predict an individual outcome and the UI says so.

79

+- It does not rank cancers by an opaque composite; each ranking is one metric, one scope, one formula.

80

+- It does not infer facts: no edge is created by a language model, no reason for a terminated trial is

81

+ guessed, no indication is derived from an ATC class, no cancer is attached to an approval unless the

82

+ official text names exactly one.

83

+- It does not redistribute data it may not redistribute: every connector manifest is licence-reviewed

84

+ before it goes live (IARC / GLOBOCAN stays under review, SEER awaits credentials).

85

+

86

+## The index in numbers

87

+

88

+Production database on 2026-09-11 (`SELECT count(*)` — live counts, never estimates):

89

+

90

+| Layer | Entity | Count |

91

+|---|---|---:|

92

+| Taxonomy | active malignant cancer entities (of 9 510 active entities) | 5 595 |

93

+| Taxonomy | top-level site groups (ranking scope) | 36 |

94

+| Genomics | genes (HGNC) | 45 170 |

95

+| Genomics | variants (CIViC, ClinVar) | 417 000 |

96

+| Genomics | curated CIViC evidence items | 11 984 |

97

+| Biomarkers | canonical biomarkers (NCIt-verified) | 54 |

98

+| Drugs | canonical drugs (CIViC, ChEMBL, openFDA, Health Canada, EMA) | 814 |

99

+| Regulatory | approval records — US 1 570 · CA 1 534 · EU 459 | 3 563 |

100

+| Trials | ClinicalTrials.gov oncology studies | 126 238 |

101

+| Trials | registrant-entered study locations (1.19 M geocoded) | 1 212 144 |

102

+| Literature | PubMed records linked to entities (plus 19 080 per-cancer count windows) | 4 287 |

103

+| Epidemiology | US observations (CDC WONDER, U.S. Cancer Statistics), 1999–2024 | 10 674 |

104

+| Graph | active knowledge edges with cancer context | 18 822 |

105

+| Derived | trial-intelligence rows · drug-pipeline rows · research-gap components | 2 240 · 5 498 · 2 966 |

106

+| Derived | country × cancer × phase trial-site aggregates | 11 805 |

107

+| Rankings | current snapshots over 25 metric definitions | 875 |

108

+| Provenance | source records · provenance rows | 629 889 · 106 263 |

109

+| Sources | connectors registered (17 active, 1 under licence review, 1 awaiting credentials) | 19 |

110

+| Storage | PostgreSQL 17 | 2.5 GB |

111

+

112

+## Product map

113

+

114

+| Area | Route(s) | What it shows |

115

+|---|---|---|

116

+| Home | `/` | Hero, live ticker (cancers, active trials, recruiting Phase III, approved drugs), US burden, data-explorer chart, trial intelligence, new approvals, rankings preview, fastest-rising incidence, gap ratios, curated evidence; aside: where trials recruit, graph teaser, taxonomy, rare spotlight, sources |

117

+| Cancer profile | `/cancer/[slug]` + tabs `statistics` `survival` `genomics` `evidence` `drugs` `trials` `research` `rankings` `sources` | Header with codes and badges; observations by geography/year/sex with charts; cohort alteration frequencies with denominators; CIViC evidence with native levels; jurisdiction-aware approvals; trials with an intelligence strip; literature; "Why this rank?"; every source behind the page |

118

+| Cancers, taxonomy | `/cancers`, `/taxonomy` | Filterable entity table (level, type, malignant, hematologic, pediatric, rare); tree browser across hierarchy types |

119

+| Data explorer | `/explore`, `/explore/coverage`, `/api/export/epidemiology.csv` | Metric × cancers × geography × sex × age × years → comparable chart groups (metric, unit, geography, source, standard population, age never mixed), observations table, permalink, CSV with attribution, API call; coverage matrix |

120

+| Countries | `/countries`, `/country/[slug]` | Latest-year summary, top cancers per metric, trends, clinical trial activity (sites by cancer and phase), the jurisdiction's approval records, sources |

121

+| Compare | `/compare?ids=` | 2–4 cancers side by side (registry figures, counters, ranks) |

122

+| Trials | `/trials`, `/trial/[nct]`, `/trials/intelligence`, `/trials/terminated`, `/trials/map`, `/api/export/trial-intelligence.csv` | Search and filters; study record with mappings, locations, publications; per-cancer trial metrics (growth, enrollment, sponsor and country concentration, termination share, trials per 1 000 deaths); failure tracking with registrant-stated reasons classified by explicit rules; Equal Earth choropleth of sites |

123

+| Drugs | `/drugs`, `/drug/[slug]`, `/pipeline` | Canonical drugs with brands as aliases; identifiers (ATC, DIN, UNII, ChEMBL…), approvals by jurisdiction, development stage per cancer, evidence, trials; pipeline funnel per cancer |

124

+| Approvals | `/approvals` | Dated feed of FDA, Health Canada and EMA records grouped by month, filters by authority, jurisdiction, cancer, status, year |

125

+| Genes, variants | `/genes`, `/gene/[symbol]`, `/variant/[slug]` | HGNC genes; evidence by cancer; variants with ClinVar interpretations; cohort frequencies; publications |

126

+| Biomarkers | `/biomarkers`, `/biomarker/[slug]` | 54 curated biomarkers with NCIt codes verified against EVS; derived cancers, drugs, approvals (tumour-agnostic from real rows), trials, publications |

127

+| Rankings | `/rankings`, `/rankings/[metric]` | One metric, one scope, one formula version per table; lineage on every row; CSV export |

128

+| Research gap | `/research-gap`, `/api/export/research-gap.csv` | Death share vs trial and publication shares, log₂ gap ratios, per-1 000-deaths intensities, log-log scatter |

129

+| Knowledge graph | `/graph?focus=type:ref` | Contextual neighbourhood (source-native edges vs derived registry links), cancer → gene → variant → drug → approval → trials paths, accessible edge table |

130

+| Pulse, history | `/pulse`, `/year/[year]`, `/data-updates` | What changed (approvals, new recruiting Phase III, registration momentum, ranking moves, publications, dataset refreshes); a year in cancer 1999–now; connector state and ingestion log |

131

+| Sources, method | `/sources`, `/source/[slug]`, `/methodology`, `/methodology/trial-map`, `/trust`, `/about`, `/developers`, `/data` | Licence registry and connector health; every formula and threshold; policies; team and hosting; API guide; redistributable downloads |

132

+| Search | ⌘K / Ctrl+K anywhere, `/search` | Cancer, gene, variant, drug, trial, publication, source — exact › alias › prefix › fuzzy |

133

+| Admin | `/admin/*` (token) | Connectors, runs, unresolved labels, rankings, trace |

134

+

135

+Every page has a light and a dark theme (toggle in the header, `prefers-color-scheme` by default),

136

+a canonical URL, structured metadata and a server-rendered Open Graph / Twitter image.

137

+

138

+## Architecture

139

+

140

+```

141

+ ┌──────────────────────── sources (19 connectors) ────────────────────────┐

142

+ │ NCIt EVS · OncoTree · HGNC · CIViC · ClinVar · GDC · cBioPortal · MeSH │

143

+ │ ClinicalTrials.gov · PubMed · CDC WONDER · CDC USCS · ChEMBL · openFDA │

144

+ │ Health Canada DPD · EMA · (IARC GLOBOCAN: review) · (SEER: credentials) │

145

+ └───────────────┬───────────────────────────────────────────────────────────┘

146

+ │ HTTP / bulk files, rate-limited, restartable cursors

147

+ ▼

148

+ workers/main.ts ──► packages/connectors (SDK: manifest · HttpClient · RawLake · RunContext)

149

+ pg-boss scheduler │ RAW → data/raw/{source}/{date}/{entity}/*.jsonl.gz + source_records

150

+ cron per manifest │ NORMALIZED → validators, schema-drift stats, unresolved_labels

151

+ counters 06:00 UTC │ CANONICAL → cancers · genes · variants · drugs · trials · publications

152

+ intel 06:15 UTC │ observations · approvals · knowledge_edges · provenance

153

+ rank 06:30 UTC ▼

154

+ PostgreSQL 17 (+ pg_trgm, unaccent, pgvector) — Drizzle schema, snake_case

155

+ │

156

+ packages/ranking: counters → intelligence → rankings (DERIVED / RANKED)

157

+ │

158

+ ┌────────────────────┴────────────────────┐

159

+ ▼ ▼

160

+ apps/api Fastify 5 · /v1 · zod · OpenAPI apps/web Next.js 16 (webpack) · server components

161

+ envelope { data, sources, dataRelease } Tailwind v4 tokens · pure SVG charts · dark mode

162

+ rate limits · API keys · x-request-id /api/v1/* proxied to the API · ISR caching

163

+ └────────────────────┬────────────────────┘

164

+ ▼

165

+ MacLustr node M4M64b · PM2 (web :8250, api :8251, worker, backup)

166

+ MacLustr Tunnel (WireGuard + Caddy on BHS64) → https://www.cancerindex.io

167

+```

168

+

169

+**Principles baked into the code**

170

+

171

+- *Reconciliation before ingestion.* Labels map to entities by shared identifier, then curated alias,

172

+ then normalized string; every mapping stores a `match_type` (`EXACT_IDENTIFIER`, `CURATED_EXACT`,

173

+ `ONTOLOGY_EXACT`, `CURATED_BROADER`, `ALIAS`, `PROBABILISTIC`, `UNRESOLVED`). Unknown labels go to

174

+ `unresolved_labels`, never to `/dev/null`.

175

+- *Time-aware observations.* A value for year X never overwrites year Y; multi-year aggregates keep

176

+ both bounds; rates keep their standard population.

177

+- *Idempotent, restartable connectors.* Payload hashes, cursor checkpoints every 2 000 records or 60 s,

178

+ SIGTERM-safe aborts, anomaly guard refusing destructive updates when a source shrinks by more than half.

179

+- *Deterministic derived layers.* Counters, intelligence tables and rankings are full rebuilds in one

180

+ transaction; snapshots are immutable and keep `previous_rank` for change explanation.

181

+

182

+## Repository layout

183

+

184

+```

185

+apps/

186

+ web/ Next.js 16 app (React 19, server components, Tailwind v4, --webpack)

187

+ src/app/ routes (see Product map), opengraph-image.tsx per entity, icon.svg, manifest.ts

188

+ src/components/ ui · charts (SVG) · cancer tabs · graph · explorer · country · home modules · layout

189

+ src/lib/ db · format · queries/* (server-only SQL) · graph-model · explorer-* · og.tsx · site.ts

190

+ src/assets/fonts/ WOFF copies of Newsreader / Inter / IBM Plex Mono for server-side images

191

+ public/brand/ logo.svg · logo-mark.svg · logo-mark-dark.svg · PNG icons

192

+ qa/smoke.mjs read-only smoke suite (44 routes, weight caps, mobile overflow, console errors)

193

+ api/ Fastify 5 public API (/v1), zod schemas, OpenAPI at /v1/docs, admin routes

194

+packages/

195

+ shared/ CI-XXX-00000001 ids, provenance types, normalization, logger, env

196

+ database/ Drizzle schema (12 files, ~50 tables), migrations 0000–0002, seeds (metrics,

197

+ geographies, biomarkers), alerts, ids

198

+ ontology/ qualifier rules, CancerResolver (alias/code reconciliation), TOP_LEVEL_CANCERS

199

+ connectors/ SDK (manifest · HttpClient · RawLake · RunContext · validators · doctor) and

200

+ connectors/<id>/{manifest.ts,index.ts,normalize.ts,fixtures/,*.test.ts}

201

+ ranking/ counters · intelligence (trial-intelligence, trial-sites, drug-pipeline,

202

+ drug-duplicates, research-gap) · engine (snapshots) · trace · country-codes

203

+workers/ pg-boss scheduler and job handlers

204

+scripts/ci.ts operator CLI (`pnpm cix …`)

205

+deploy/ mld manifest, first-run bootstrap, backup / restore scripts

206

+docs/ ARCHITECTURE · DATA-MODEL · METHODOLOGY · API · SECURITY · AI · source-policy ·

207

+ schema-changes-ops · adr/ (6 ADRs) · connectors/ (18 pages) · methodology/ (7 pages)

208

+CLAUDE.md operational rules every contributor (human or agent) follows

209

+```

210

+

211

+## Data model

212

+

213

+Source of truth: `packages/database/src/schema/*.ts` (documented in `docs/DATA-MODEL.md`). Public

214

+identifiers are `CI-<NS>-00000001` minted per namespace (`CAN`, `GENE`, `VAR`, `DRUG`, `TRIAL`, `PUB`,

215

+`BIO`, `STUDY`, `METRIC`, `SOURCE`, `GEO`, `PROV`…) — never database integers.

216

+

217

+| Group | Tables | Notes |

218

+|---|---|---|

219

+| Registry & operations | `sources`, `ingest_runs`, `connector_cursors`, `connector_field_stats`, `source_records`, `provenance`, `unresolved_labels`, `change_events`, `audit_log`, `entity_merges`, `system_alerts`, `api_keys`, `id_sequences` | licence status per source; one row per run with counters and log; raw-lake index with payload hashes; merge queue (proposed, never automatic) |

220

+| Cancer ontology | `cancers`, `cancer_aliases`, `cancer_hierarchy` (ncit · oncotree · anatomical · …), `cancer_codes`, `anatomical_sites`, `cancer_anatomy`, `geographies`, `cohort_definitions` | one row per disease concept, aliases and codes searchable, several trees coexist |

221

+| Genomics | `genes`, `gene_aliases`, `variants`, `variant_aliases`, `variant_clinical_significance`, `biomarkers`, `genomic_cohorts`, `cancer_gene_frequencies`, `entity_embeddings` | coordinates carry their assembly; frequencies carry their denominator |

222

+| Drugs & regulatory | `drugs`, `drug_aliases`, `drug_codes`, `treatment_regimens`, `drug_approvals` | brands are aliases; approvals are per jurisdiction, authority, application/DIN/EMA number, dated, with status and verbatim source status |

223

+| Trials | `clinical_trials`, `trial_conditions`, `trial_interventions`, `trial_locations`, `trial_pulse` | conditions and interventions reconciled with `match_type`; 1.2 M locations |

224

+| Literature | `publications`, `publication_entity_edges`, `literature_counts` | the exact PubMed query is stored with every count |

225

+| Evidence & graph | `knowledge_edges`, `civic_evidence_items`, `risk_factors` | edges carry cancer context, direction, native evidence level, category, provenance ids, status (`active` · `superseded`) |

226

+| Epidemiology | `epidemiology_observations`, `survival_observations` | metric, unit, sex, age group, year(s), CI, standard population, estimate type, site definition |

227

+| Derived | `entity_counters`, `trial_intelligence`, `trial_site_country_counts`, `drug_pipeline`, `research_gap_components`, `metric_definitions`, `ranking_snapshots`, `rankings`, `ai_answers` | every row has a `formula_version` and `inputs` |

228

+

229

+Schema changes are migrations generated with `pnpm db:generate` (one per integration) and applied by

230

+`pnpm db:migrate`, which also creates the extensions and the trigram / GIN performance indexes.

231

+

232

+## Connectors

233

+

234

+Each connector is a class with a manifest (licence, terms review date, documentation verification

235

+date, rate limits, schedule, anomaly guard), a health check and a `sync()`; it is tested against

236

+sanitized fixtures (normal, empty, pagination, rate-limit, server error, malformed) and documented in

237

+`docs/connectors/<id>.md`. `pnpm cix sources:sync` seeds manifests into `sources`; `/sources` shows them.

238

+

239

240

+|---|---|---|---|---|

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

+

261

+Run order for a fresh database: `deploy/first-run.sh` (terminology → genes → evidence/genomics/variants

262

+→ trials → literature → epidemiology), then the regulatory connectors, `pnpm cix reconcile-drugs`,

263

+`pnpm cix counters`, `pnpm cix intel`, `pnpm cix rank`.

264

+

265

+## Derived intelligence layer

266

+

267

+Recomputed daily by the worker (`maintenance.intel`, between counters and rankings) or with

268

+`pnpm cix intel` (≈ 50 s on the production database). Every row stores `formula_version` and `inputs`.

269

+Methods: `docs/methodology/*.md` and `/methodology`.

270

+

271

272

+|---|---|---|---|

273

+| Clinical trial intelligence | `trial_intelligence` | `ci-trial-intel-v1`, stop reasons `ci-stop-reasons-v1` | per cancer (top and all levels, descendants included): total / active / recruiting / Phase I–IV counts, growth of first-posted studies (12 m vs prior 12 m, ≥ 20), enrollment mean/median, sponsor and country HHI, industry and US shares, termination share (≥ 2010, ≥ 30 terminal), stop-reason breakdown by explicit keyword rules, trials per 1 000 deaths / per 100 k cases (US, latest year, deaths ≥ 100) |

274

+| Trial map | `trial_site_country_counts` | `ci-trial-sites-v1` | sites and studies per country × (all / each top-level cancer) × (any / each phase) × (all / recruiting only); ISO 3166-1 alpha-3 mapping with explicit unmapped historical names |

275

+| Drug pipeline | `drug_pipeline`, `entity_merges` | `ci-drug-pipeline-v1` | stage per drug and per drug × top-level cancer (approved › withdrawn › highest registry phase › phase not stated); salt-form duplicates proposed to the merge queue |

276

+| Research Gap Index | `research_gap_components` + 4 ranking metrics | `ci-research-gap-components-v1` | for each burden scope: death share vs active-trial and publication shares over the eligible set (deaths ≥ 100), `log₂` gap ratios, per-1 000-deaths intensities |

277

+| Trial → drug reconciliation | `trial_interventions.drug_id` | (in the ClinicalTrials connector) | exact alias › salt/dose/label-stripped alias › probabilistic head; shared aliases resolved by rule (own generic name › brand › base molecule) or left unresolved — 78 956 rows linked |

278

+

279

+## Rankings and metrics

280

+

281

+25 metric definitions live in `metric_definitions` (seeded from `packages/database/src/seed-data/metrics.ts`)

282

+and are rendered live on `/methodology#metrics`. A ranking snapshot is one metric × one scope

283

+(`geo=USA|sex=all|age=all|year=2024|level=top`) × one formula version; rows keep rank, percentile,

284

+confidence, previous rank and the exact inputs.

285

+

286

+| Category | Metrics |

287

+|---|---|

288

+| Burden | `incidence_count`, `mortality_count`, `as_incidence_rate`, `as_mortality_rate` |

289

+| Lethality | `mortality_incidence_ratio`, `five_year_survival` (awaits survival observations) |

290

+| Clinical research | `active_trials`, `recruiting_trials`, `phase3_trials`, `phase3_recruiting_trials`, `trial_termination_share`, `sponsor_concentration` |

291

+| Research activity & trends | `publications_5y`, `publications_12m`, `publication_growth`, `trial_growth_yoy` |

292

+| Molecular knowledge | `curated_evidence_items`, `associated_genes`, `genomic_cohorts` |

293

+| Unmet need | `trial_gap`, `research_gap` (percentile-based), `trial_gap_ratio`, `research_gap_ratio` (share-based, log₂), `trials_per_1000_deaths`, `publications_per_1000_deaths` |

294

+

295

+Burden, lethality and gap metrics exist only for scopes with licensed observations (currently the

296

+United States, per year and sex, top level). No composite score is published (ADR-006).

297

+

298

+## Public API

299

+

300

+Base URL `https://www.cancerindex.io/api/v1` (proxied to Fastify). Read-only JSON. Every response is an

301

+envelope `{ data, sources, dataRelease, generatedAt, total?, limit?, offset?, hasMore? }` where `sources`

302

+lists the upstream sources, licences and attributions behind the returned data. Swagger UI at `/api/v1/docs`,

303

+OpenAPI 3.1 at `/api/v1/openapi.json`. Rate limits per IP; an optional bearer API key raises them.

304

+

305

+| Endpoints | Purpose |

306

+|---|---|

307

+| `/cancers`, `/cancers/:id`, `/cancers/:id/{statistics,survival,genes,variants,drugs,trials,publications}` | taxonomy, observations, evidence, approvals, trials per cancer (descendants included) |

308

+| `/genes`, `/genes/:symbol`, `/variants/:id`, `/drugs`, `/drugs/:id`, `/biomarkers`, `/biomarkers/:slug` | entity details with derived links |

309

+| `/trials`, `/trials/:nct`, `/trials/intelligence[/:cancer]`, `/trials/terminated`, `/trials/sites` | search, study record, per-cancer intelligence, failures with classified reasons, country/city site aggregates |

310

+| `/epidemiology`, `/epidemiology/coverage`, `/epidemiology/metrics` | time-aware observations with provenance (Data explorer) |

311

+| `/approvals`, `/approvals/recent`, `/pipeline`, `/pipeline/summary` | jurisdiction-aware regulatory records and development stages |

312

+| `/research-gap`, `/research-gap/scopes` | gap components per burden scope |

313

+| `/graph/:type/:id`, `/graph/cancer/:id/paths` | knowledge-graph neighbourhoods and chains |

314

+| `/rankings/metrics`, `/rankings`, `/rankings/:metric/:cancerId/explain` | catalogue, snapshots, "Why this rank?" |

315

+| `/search`, `/sources`, `/sources/:slug`, `/stats`, `/changes`, `/healthz` | search, registry and health, live counts, change events |

316

+| `/admin/*` (`x-admin-token`) | connectors, runs, unresolved labels, jobs, trace, audit |

317

+

318

+Details and examples: `docs/API.md`.

319

+

320

+## Operator CLI

20

321

21

322

```

22

−apps/web Next.js 16 public site (port 8250), proxies /api/v1/* to the API

23

−apps/api Fastify 5 public API /v1 (port 8251), OpenAPI at /v1/docs

24

−workers/ pg-boss scheduler: connector cron, counters, rankings, health probes

25

−packages/ shared · database (Drizzle) · ontology · connectors (SDK + connectors) · ranking

26

−scripts/ci.ts operator CLI (pnpm cix …)

27

−deploy/ MacLustr mld manifest, first-run bootstrap

28

−docs/ architecture, data model, methodology, API, security, AI policy, ADRs, connector docs

323

+pnpm cix connectors list connectors, licence status, health, last success

324

+pnpm cix run <id> [--mode full|incremental|backfill|dry_run] [--max-records N] [--max-minutes M] [--reset-cursor]

325

+pnpm cix run-all [--max-minutes M] every active connector in registry order

326

+pnpm cix health <id> source liveness probe

327

+pnpm cix sources:sync manifests → sources table

328

+pnpm cix reconcile-drugs [--remap] trial interventions → canonical drugs

329

+pnpm cix counters rebuild entity_counters

330

+pnpm cix intel trial intelligence · trial sites · drug pipeline · research gap

331

+pnpm cix rank recompute every ranking snapshot

332

+pnpm cix stats | trace <type> <id> | doctor [--no-disk] | alerts [ack|resolve <id>]

29

333

```

30

334

31

−Start with `CLAUDE.md` (operational rules), then `docs/ARCHITECTURE.md`, `docs/DATA-MODEL.md` and

32

−`docs/METHODOLOGY.md`.

335

+`--mode backfill` replays a connector's raw lake without HTTP (used after a mapping-rule change); `dry_run`

336

+never writes and never moves a cursor.

33

337

34

−## Quickstart

338

+## Local development

35

339

36

−Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with `pg_trgm`, `unaccent` and `vector`.

340

+Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with `pg_trgm`, `unaccent` and `vector`; `rsvg-convert`

341

+and Python Pillow only if you regenerate brand rasters.

37

342

38

343

```bash

39

−createdb cancerindex && cp .env.example .env # set NCBI_EMAIL, ADMIN_TOKEN

344

+createdb cancerindex && cp .env.example .env # set NCBI_EMAIL, ADMIN_TOKEN

40

345

pnpm install

41

346

pnpm db:migrate && pnpm db:seed && pnpm cix sources:sync

42

−pnpm cix run oncotree --mode dry_run # smoke test against the live API (no writes)

43

−pnpm cix run oncotree # first real ingest

44

−pnpm cix counters && pnpm cix rank

45

−pnpm dev:api & # http://127.0.0.1:8251/v1/docs

46

−pnpm worker & # schedules from connector manifests

47

−pnpm dev:web # http://localhost:8250

347

+pnpm cix run oncotree --mode dry_run # smoke against the live API, no writes

348

+bash deploy/first-run.sh # ordered first ingestion (hours; restartable)

349

+pnpm cix run health-canada-dpd && pnpm cix run ema && pnpm cix run openfda

350

+pnpm cix reconcile-drugs && pnpm cix counters && pnpm cix intel && pnpm cix rank

351

+pnpm dev:api & # http://127.0.0.1:8251/v1/docs

352

+pnpm worker & # schedules from connector manifests

353

+pnpm dev:web # http://localhost:8250

48

354

```

49

355

50

−Verify: `curl -s localhost:8251/healthz`, `curl -s "localhost:8251/v1/search?q=glio"`.

356

+A faster path for UI work is to restore a production dump (`bash deploy/restore.sh <dump> --target

357

+cancerindex_prodcopy`, then rename databases) and run `pnpm db:migrate && pnpm db:seed`.

358

+Note that Next.js allows a single `next dev` per app directory (`.next/dev/lock`).

51

359

52

−## CLI

360

+## Configuration

53

361

54

−```

55

−pnpm cix connectors list connectors, license status, health

56

−pnpm cix run <id> [--mode full|incremental|dry_run] [--max-records N] [--max-minutes M] [--reset-cursor]

57

−pnpm cix run-all [--max-minutes M] every active connector in registry order

58

−pnpm cix health <id> source liveness probe

59

−pnpm cix sources:sync manifests → sources table

60

−pnpm cix counters rebuild entity_counters

61

−pnpm cix rank recompute ranking snapshots

62

−pnpm cix stats table counts

63

−pnpm cix trace <table> <id> lineage of a value

64

−

65

66

−pnpm --filter @cancerindex/api create-key -- --label "Lab" --tier research --rpm 600

67

−```

362

+`.env` at the repository root (read by the API, the worker and the web app).

363

+

364

+| Variable | Purpose |

365

+|---|---|

366

+| `DATABASE_URL`, `DB_POOL_MAX` | PostgreSQL connection |

367

+| `WEB_PORT` (8250), `API_PORT` (8251), `API_HOST`, `CI_API_URL` | ports and the API URL the web app proxies to |

368

+| `NEXT_PUBLIC_SITE_URL` | canonical site URL (metadata, sitemaps, share images) |

369

+| `CI_DATA_DIR` | raw data lake root (`data/raw`, `data/cache`) |

370

+| `ADMIN_TOKEN` | `/admin` console and `/v1/admin/*` |

371

+| `NCBI_TOOL`, `NCBI_EMAIL`, `NCBI_API_KEY` | E-utilities (PubMed, ClinVar); the key raises the limit to 10 req/s |

372

+| `SEER_API_KEY` | unlocks the SEER connector |

373

+| `OPENFDA_API_KEY` | raises openFDA from 1 000 to 120 000 requests/day |

374

+| `IARC_TERMS_ACCEPTED_BY` | human gate for GLOBOCAN (stays unset until terms are settled) |

375

+| `WORKER_CONCURRENCY`, `CI_MAX_RUN_MINUTES`, `LOG_LEVEL` | worker tuning |

376

+| `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `OPENAI_BASE_URL` | reserved for the future `/ask` layer (not enabled) |

377

+

378

+## Quality: tests, QA, accessibility, performance

68

379

69

−## Quality gates

380

+- `pnpm typecheck` — strict TypeScript across the 8 workspaces.

381

+- `pnpm test` — 528 vitest tests: connector fixtures (every connector), SDK (lake, run, checkpoints),

382

+ formulas (shares, log ratios, HHI, growth, termination share, stop-reason rules, pipeline stages,

383

+ country codes, map scales, projections, explorer comparability, CSV escaping, graph layout), API

384

+ smoke tests against the local database (skipped when unreachable).

385

+- `pnpm --filter @cancerindex/web qa` / `qa:prod` — read-only smoke suite over 44 routes: HTTP 200,

386

+ expected text, per-route weight caps, no "Data not yet available" where data must exist, no horizontal

387

+ overflow at 390 px, console-error scan (Playwright when available).

388

+- Accessibility: keyboard navigation, focus rings, colour never the only carrier (badges carry text,

389

+ charts have legends and data tables, maps have equivalent tables), `aria` labels on SVG figures.

390

+- Performance: ISR caching (`revalidate` per page, `Cache-Control` on public routes), materialized

391

+ derived tables, GIN / trigram indexes, `sql.param` for array parameters, request budgets on heavy

392

+ pages (graph groups, city layers).

393

+

394

+## Deployment and operations

395

+

396

+Production runs on the MacLustr cluster (node **M4M64b**: Postgres 17 + pgvector, PM2) behind the

397

+MacLustr Tunnel (WireGuard + Caddy on the BHS64 gateway) at https://www.cancerindex.io.

70

398

71

399

```bash

72

−pnpm typecheck # every workspace

73

−pnpm -r test # vitest; connector tests use fixtures, API smoke tests skip without a DB

400

+# from the laptop — gateway M1M32 orchestrates (mld)

401

+rsync -a --exclude '/data/' --exclude node_modules --exclude .next --exclude .git --exclude '.env*' ./ /tmp/cancerindex-deploy/

402

+mld stage /tmp/cancerindex-deploy cancerindex

403

+mld deploy cancerindex --node M4M64b # install, migrate, seed, sources:sync, build, PM2 restart, health checks

74

404

```

75

405

76

−## Deployment

406

+| Process (PM2) | Role |

407

+|---|---|

408

+| `cancerindex-web` (:8250) | `next start` |

409

+| `cancerindex-api` (:8251) | Fastify API |

410

+| `cancerindex-worker` | pg-boss: connector schedules, `maintenance.counters` 06:00 UTC → `maintenance.intel` 06:15 → `maintenance.rank` 06:30, hourly health probes and alerts |

411

+| `cancerindex-backup` | nightly `pg_dump` 05:20, 14 daily + 8 weekly retained (`deploy/backup.sh`, `restore.sh`) |

412

+

413

+Operational visibility: `/data-updates` (connector state, ingestion log), `/admin` (runs, unresolved

414

+labels, alerts), `pnpm cix doctor`, `system_alerts`. Deploy manifest: `deploy/mld-manifest.cancerindex.json`

415

+(note the excludes are anchored — `/coverage/`, `/data/` — so route folders with those names are kept).

416

+

417

+## Brand

418

+

419

+<p>

420

+ <img src="apps/web/public/brand/logo-mark.svg" alt="CancerIndex mark" width="72" align="left" hspace="12">

421

+ The mark reads <strong>CI</strong>: an open ring — a cell with its nucleus, the "C" — and a graduated

422

+ index scale in the accent teal, the "I". It is drawn once in SVG and used everywhere: the header and

423

+ footer (<code>BrandMark</code>, theme-aware through <code>currentColor</code> and the accent token),

424

+ the favicon (<code>app/icon.svg</code>, <code>favicon.ico</code>, <code>apple-icon.png</code>), the web

425

+ manifest and the share images.

426

+</p>

427

+<br clear="all">

428

+

429

+| Asset | Path |

430

+|---|---|

431

+| Logo with wordmark | `apps/web/public/brand/logo.svg` |

432

+| Mark (light / dark backgrounds) | `apps/web/public/brand/logo-mark.svg`, `logo-mark-dark.svg` (+ PNG 512) |

433

+| Favicon and app icons | `apps/web/src/app/icon.svg`, `favicon.ico`, `apple-icon.png`, `public/brand/icon-{192,512}.png` |

434

+| Share images | `apps/web/src/lib/og.tsx` (next/og, 1200 × 630, site fonts); `app/opengraph-image.tsx` (live counts) and `opengraph-image.tsx` under `cancer/[slug]`, `gene/[symbol]`, `drug/[slug]`, `biomarker/[slug]`, `trial/[nct]` |

435

+

436

+Colours: paper `#fafaf7`, ink `#1c1c1a`, accent teal `#0f5f63` (dark theme `#6cc3c6`). Type: Newsreader

437

+(display), Inter (UI), IBM Plex Mono (identifiers).

438

+

439

+## Design language

440

+

441

+Scientific, editorial, institutional. Off-white paper, charcoal ink, one restrained teal accent, thin

442

+rules, tabular numerals, dense tables, sparklines and small multiples rather than decorative charts. All

443

+charts are server-rendered SVG with a legend, units, the dataset year and a source line; every value shows

444

+a claim badge (observed · published · curated · regulatory · computed) and a source badge whose popover

445

+gives dataset, version, retrieval date and URL. Mobile is first-class (tables scroll inside their wrap; no

446

+horizontal overflow at 390 px). Light and dark themes share one token set (`--color-*`, `--color-series-*`).

447

+

448

+## Known gotchas

449

+

450

+- A JS array inside a Drizzle template — `` sql`x = ANY(${arr}::text[])` `` — becomes a `($1,$2,…)` tuple

451

+ and fails; use `sql.param(arr)` or `sql.raw` for constant lists.

452

+- Columns declared with the `updatedAt()` helper are physically `updated_at` even when named `computedAt`.

453

+- `db.execute<T>` needs a `type` alias, not an `interface`, for `T`.

454

+- A prop named `ref` cannot be passed to a server component (React reserves it) — the page fails in

455

+ production with minified error #441; use another name (`entityRef`).

456

+- SVG `<title>` must be a single text node or hydration fails; `array_agg` over empty arrays errors — use

457

+ `unnest` / `string_agg`.

458

+- `opengraph-image.tsx` cannot live inside an optional catch-all segment (`[[...tab]]`); place it one

459

+ level up.

460

+- `ranking_snapshots.scope_key` has no source dimension: two sources for the same geography/year/sex

461

+ overwrite each other's current snapshot, so a preferred source is chosen per scope.

462

+- A shared alias between a concept and its descendant ("breast cancer") must resolve to the **broadest**

463

+ concept when building a dictionary; the narrowest-of-lineage rule is only for sentence-level mentions.

464

+- OncoTree's WAF rejects any User-Agent containing a URL; NCBI throttles at 3 req/s without a key;

465

+ openFDA allows 1 000 requests/day without a key; the EMA xlsx has 39 columns and header row 9.

466

+- mld `sync_excludes` patterns must be anchored (`/data/`, `/coverage/`), otherwise route folders with

467

+ the same name are dropped from the deploy.

77

468

78

−Production runs on the MacLustr cluster through `mld` (gateway M1M32): manifest

79

−`deploy/mld-manifest.cancerindex.json`, `mld stage <dir> cancerindex`, `mld deploy cancerindex`,

80

−then `deploy/first-run.sh` for the ordered first ingestion. PM2 processes `cancerindex-web`,

81

−`cancerindex-api`, `cancerindex-worker`; ngrok `www.cancerindex.io` → 8250. See `deploy/README.md`.

469

+## Roadmap

82

470

83

−## Principles (short form)

471

+- Global burden once IARC / GLOBOCAN terms are settled (human decision) and SEER credentials are set;

472

+ survival observations and the `five_year_survival` metric follow.

473

+- MHRA and TGA regulatory connectors; populating `drug_approvals.biomarker_ids` from label text.

474

+- Risk factors and attributable burden, screening and guideline registries (metadata and links only,

475

+ with change detection), hereditary syndromes, pediatric and rare-cancer views.

476

+- Researcher, institution and funding entities (OpenAlex, NIH RePORTER, CIHR) for the funding-gap view.

477

+- Watchlists, alerts and a grounded, cited `/ask` layer that summarizes indexed sources only.

478

+- Lighter default views for the heaviest pages (`/graph`, biomarker pages with hundreds of approvals).

84

479

85

−1. No scientific number without provenance; derived values carry a formula version and inputs.

86

−2. Layers stay separable: RAW → NORMALIZED → CANONICAL → DERIVED → RANKED → AI.

87

−3. Never fake data: missing is missing ("Data not yet available"), never zero.

88

−4. Identifiers are first-class: every upstream id is kept; public ids are `CI-<NS>-00000001`.

89

−5. Reconciliation before ingestion; unknown labels go to a curation queue, never dropped.

90

−6. Time-aware observations: a new year is a new row.

91

−7. Every edge has context: cancer, direction, evidence level, provenance.

92

−8. Licensing gate on every source; scientific safety labels are never merged.

480

+## Licensing, attribution and contact

93

481

94

−## License and attribution

482

+- **Code**: private repository (spbgit `cancerindex.git`).

483

484

+ attribution rows naming the underlying providers.

485

+- **Source data** remains under the licence of each provider — see `/sources` and the `sources` array of

486

+ every API response (NCIt CC BY 4.0, OncoTree CC BY 4.0, HGNC CC0, CIViC CC0, openFDA CC0, Health Canada

487

+ OGL, EMA with acknowledgement, ClinVar / PubMed / ClinicalTrials.gov / CDC public domain, GDC open tier,

488

+ ChEMBL CC BY-SA).

489

+- **Not medical advice.** CancerIndex provides research and educational information only.

95

490

96

−Code: private (Groupe/SPB). Data: each source keeps its own license and attribution, surfaced in

97

−every API response (`sources`) and on `/sources`. See `docs/source-policy.md`.

491

+Built and maintained by **Simon-Pierre Boucher** — contact **contact@spboucher.ai** — hosted on

492

+**MacLustr** (<https://www.maclustr.io>).

98

493