# ADR-003 — Raw data lake as gzip JSON Lines on disk

Status: accepted (2026-09-08)

## Context

Provenance-first means every canonical or derived value must be traceable to the exact upstream
payload that produced it (CLAUDE.md §1, §26). Payloads are heterogeneous JSON (and TSV converted
to JSON records), volumes range from hundreds of rows (OncoTree) to millions (ClinicalTrials.gov,
PubMed), and connectors run repeatedly. Storing raw payloads in PostgreSQL would bloat the
canonical database and slow backups; an object store adds an external dependency the single-node
deployment does not otherwise need.

## Decision

Raw payloads are written by `RawLake` (`packages/connectors/src/sdk/lake.ts`) to the local
filesystem under `CI_DATA_DIR/raw`, organised as

```
data/raw/{source}/{YYYY-MM-DD}/{entity}/{runId}-{part}.jsonl.gz
```

One record per line, gzip-compressed, files rotated by size. `source_records` keeps, per upstream
record, the sha256 `payload_hash` and the `raw_path` of the file that contains it, plus the first and
last run that saw it. Manifests declare `rawRetention: full | summary | none`; `dry_run` mode never
writes to the lake. The lake directory is git-ignored and excluded from deployment syncs.

## Consequences

- Idempotency and change detection are cheap (hash compare) and independent of the lake.
- `traceValue()` can point to the exact file for any observation; re-ingestion or re-derivation is
  possible without hitting the upstream API.
- Files are append-only per run; nothing in the lake is ever rewritten. Retention policies (e.g.
  keep the last N runs of high-volume sources) are an operational decision handled by a cleanup job,
  not by connectors.
- JSON Lines gzip is directly readable by DuckDB (`read_json_auto`) and Python, which keeps the door
  open for the analytical workers of ADR-002 and for a later Parquet conversion.
- Backups must include both the PostgreSQL dump and `data/raw`; the deploy notes list the lake as a
  path to back up on the node (it is not synced by `mld`).