spb/cancerindex

1# ADR-003 — Raw data lake as gzip JSON Lines on disk23Status: accepted (2026-09-08)45## Context67Provenance-first means every canonical or derived value must be traceable to the exact upstream8payload that produced it (CLAUDE.md §1, §26). Payloads are heterogeneous JSON (and TSV converted9to JSON records), volumes range from hundreds of rows (OncoTree) to millions (ClinicalTrials.gov,10PubMed), and connectors run repeatedly. Storing raw payloads in PostgreSQL would bloat the11canonical database and slow backups; an object store adds an external dependency the single-node12deployment does not otherwise need.1314## Decision1516Raw payloads are written by `RawLake` (`packages/connectors/src/sdk/lake.ts`) to the local17filesystem under `CI_DATA_DIR/raw`, organised as1819```20data/raw/{source}/{YYYY-MM-DD}/{entity}/{runId}-{part}.jsonl.gz21```2223One record per line, gzip-compressed, files rotated by size. `source_records` keeps, per upstream24record, the sha256 `payload_hash` and the `raw_path` of the file that contains it, plus the first and25last run that saw it. Manifests declare `rawRetention: full | summary | none`; `dry_run` mode never26writes to the lake. The lake directory is git-ignored and excluded from deployment syncs.2728## Consequences2930- Idempotency and change detection are cheap (hash compare) and independent of the lake.31- `traceValue()` can point to the exact file for any observation; re-ingestion or re-derivation is32  possible without hitting the upstream API.33- Files are append-only per run; nothing in the lake is ever rewritten. Retention policies (e.g.34  keep the last N runs of high-volume sources) are an operational decision handled by a cleanup job,35  not by connectors.36- JSON Lines gzip is directly readable by DuckDB (`read_json_auto`) and Python, which keeps the door37  open for the analytical workers of ADR-002 and for a later Parquet conversion.38- Backups must include both the PostgreSQL dump and `data/raw`; the deploy notes list the lake as a39  path to back up on the node (it is not synced by `mld`).40