# API_INTEGRATION.md — how WP39 talks to FinData OS

## Authorization
- Header `Authorization: Bearer <key>`; the key is read from `FINDATA_API_KEY` or the file `FINDATA_API_KEY_FILE`
  (default `~/findata-secrets/api-key-wp39.txt`, mode 600). The client never logs headers, never puts the key in a URL,
  and redacts any string starting with `fd_` in exceptions (`src/wp39/api_client.py::_redact`).
- Dedicated key `wp39_uqo` (scopes `read:market, read:fundamentals, query, export`; **no admin**), created 2026-11-02 with
  the node CLI `findata create-key` because `POST /admin/keys` returned HTTP 500 for API-key callers (bug logged in
  DATA_AUDIT §8).
- Compute runs on **M3U96a** against `http://127.0.0.1:8600/api/v1` (same host as the lake; the public URL
  `https://findata.spboucher.ai/api/v1` is used from the laptop for probes only).

## Client (`src/wp39/api_client.py`)
- `httpx.Client` with 60 s timeout, retry with bounded exponential backoff (5 attempts, 0.5 → 8 s, jitter) on 429/5xx and
  network errors; idempotent GETs only are retried automatically; export creation is retried only when no export id was
  received.
- Every response's `X-Request-ID`, `X-Dataset-Id`, `X-Dataset-Version` are recorded in the provenance log
  (`artifacts/logs/api_calls.jsonl`, one line per call, no payloads).
- Cursor iteration for `/market/bars` and `/coverage/symbols`; export polling (`/exports/{id}`) every 1 s up to 10 min.
- Arrow/Parquet schema validation against `GET /catalog/schema/bars` (schema v1) before any file is accepted.

## Calls used by the pipeline
| Step | Call | Scope | Volume |
|---|---|---|---|
| 00 probe | `/health`, `/catalog/datasets`, `/catalog/schema/bars`, `/openapi.json` | read | 4 |
| 01 universe | `/coverage/symbols?dataset_id=market_bars.stock.1m.raw` (cursor, 5,000/page), `/instruments/{symbol}` | read | ≈ 4 + N |
| 02 PIT panel | `/fundamentals/{entity}?period_type=q&limit=20000` (all facts, local as-of join), `/fundamentals/{entity}/periods` | read:fundamentals | 2 × N entities |
| 03 features | `POST /exports` (1m raw, ≤ 12 y, 1 symbol) ×2 per symbol, `/exports/{id}`, `/download`; SPY from `market_bars.etf.1m.raw` | export | 2 × N + 2 |
| checks | `POST /query` (cross-section counts, spot checks ≤ 100 k rows) | query | few |

Exception (documented): earnings-release rows are extracted once on M3U96a with
`psql -d findata -c "\copy (select ... from earnings_events) to '…'"` (read-only). The extract's SHA-256, row count and
timestamp are in `artifacts/manifests/earnings_events.json`.

## Idempotent partitioning and manifests
- Exports are cached per `(symbol, chunk)` as `$WP39_CACHE/bars/<symbol>__<start>__<end>.parquet` and deleted after the
  feature rows are written; `artifacts/manifests/bars_manifest.jsonl` keeps `symbol, start, end, dataset_id,
  dataset_version, row_count, bytes, sha256, exported_at, request_id`.
- Fundamentals extracts: `$WP39_CACHE/fundamentals/<entity>.parquet` with the same manifest fields.
- Re-running a script skips completed partitions (manifest lookup) and never overwrites a partition with a different
  dataset version without logging a `version_drift` warning.