# ADR-003 — Raw data lake as gzip JSON Lines on disk

Status: accepted (2026-09-08)

# Context

Provenance-first means every canonical or derived value must be traceable to the exact upstream payload that produced it (CLAUDE.md §1, §26). Payloads are heterogeneous JSON (and TSV converted to JSON records), volumes range from hundreds of rows (OncoTree) to millions (ClinicalTrials.gov, PubMed), and connectors run repeatedly. Storing raw payloads in PostgreSQL would bloat the canonical database and slow backups; an object store adds an external dependency the single-node deployment does not otherwise need.

# Decision

Raw payloads are written by RawLake (packages/connectors/src/sdk/lake.ts) to the local filesystem under CI_DATA_DIR/raw, organised as

text

data/raw/{source}/{YYYY-MM-DD}/{entity}/{runId}-{part}.jsonl.gz

One record per line, gzip-compressed, files rotated by size. source_records keeps, per upstream record, the sha256 payload_hash and the raw_path of the file that contains it, plus the first and last run that saw it. Manifests declare rawRetention: full | summary | none; dry_run mode never writes to the lake. The lake directory is git-ignored and excluded from deployment syncs.

# Consequences

Idempotency and change detection are cheap (hash compare) and independent of the lake.
traceValue() can point to the exact file for any observation; re-ingestion or re-derivation is possible without hitting the upstream API.
Files are append-only per run; nothing in the lake is ever rewritten. Retention policies (e.g. keep the last N runs of high-volume sources) are an operational decision handled by a cleanup job, not by connectors.
JSON Lines gzip is directly readable by DuckDB (read_json_auto) and Python, which keeps the door open for the analytical workers of ADR-002 and for a later Parquet conversion.
Backups must include both the PostgreSQL dump and data/raw; the deploy notes list the lake as a path to back up on the node (it is not synced by mld).