SPB Git
spboucher.ai

spb/findata

Public

fdq shell helper + Claude Code API usage guide

Simon-Pierre Boucher committed (Nov 2, 2026) parent 8a7c5bd Browse files

2 changed files +115 −0

2 changed files
Unified Split
added docs/CLAUDE-API-USAGE.md +58 −0
@@ -0,0 +1,58 @@
1 +# Using FinData OS from a Claude Code session (or any shell)
2 +
3 +**Key**: `~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt` (scopes: read:market, read:fundamentals, query, export, admin;
4 +created 2026-11-02 for the owner's Claude Code sessions). Never paste it into code, docs or chat; the helper reads it.
5 +**Base URL**: https://findata.spboucher.ai (API prefix `/api/v1`, Swagger at `/api/docs`). Auth header: `Authorization: Bearer <key>`.
6 +
7 +## Fastest path: the `fdq` helper
8 +```bash
9 +FDQ=~/Desktop/Cluster/projets/findata/scripts/fdq # laptop mirror; also at M3U96a:~/apps/findata/scripts/fdq
10 +$FDQ health # status, alerts, lake free space, worker, backup age
11 +$FDQ datasets # 64 datasets: id, status, rows, first/last session
12 +$FDQ bars AAPL 1m 2024-03-01 2024-03-31 # JSON page (<= 10 000 rows/page, 1m <= 31 days per call); raw|split|split_div as 5th arg
13 +$FDQ sql "SELECT session_date, min(low) lo, max(high) hi, sum(volume) v FROM market_bars_stock_1m_raw WHERE symbol='AAPL' AND year=2024 GROUP BY 1 ORDER BY 1"
14 +$FDQ sql "SELECT trade_date, strike, call_put, bid, ask, open_interest, delta FROM options_eod WHERE underlying='SPY' AND trade_date=DATE '2024-03-15' AND expiry=DATE '2024-03-28' ORDER BY strike" 500
15 +$FDQ fund MSFT 2015-01-01T00:00:00Z rev,net_inc,eps_d # point-in-time: only facts available at as_of
16 +$FDQ export "AAPL,MSFT,NVDA" 2024-01-01 2024-12-31 market_bars.stock.1m.raw /tmp/x.parquet # large ranges -> async Parquet
17 +$FDQ jobs ; $FDQ enqueue '{"kind":"incremental","params":{"type":"stock","period":"day"}}' # admin (ingestion queue)
18 +```
19 +
20 +## Python (SDK)
21 +```bash
22 +cd ~/Desktop/Cluster/projets/findata
23 +FINDATA_API_KEY=$(cat ~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt) .venv/bin/python my_script.py
24 +```
25 +```python
26 +import os
27 +from findata_sdk import FinData
28 +fd = FinData("https://findata.spboucher.ai", api_key=os.environ["FINDATA_API_KEY"])
29 +bars = fd.bars("AAPL", "1m", "2024-03-01", "2024-03-31") # pyarrow.Table (all pages, Arrow IPC transport)
30 +df = bars.to_pandas()
31 +tbl = fd.sql("SELECT symbol, count(*) n FROM market_bars_stock_1m_split_div WHERE symbol IN ('AAPL','MSFT') AND year=2020 GROUP BY 1")
32 +facts = fd.fundamentals("AAPL", as_of="2020-01-01T00:00:00Z", metrics=["rev", "net_inc"], period_type="q")
33 +path = fd.export(["AAPL", "MSFT"], "2015-01-01", "2024-12-31", dest="/tmp/aapl_msft_1m.parquet") # ~2 s per 1 M rows
34 +```
35 +
36 +## What is there (SQL view names = dataset id with dots -> underscores)
37 +| View | Content | Partition columns for the fast path |
38 +|---|---|---|
39 +| `market_bars_stock_1m_raw` (+ `_split`, `_split_div`) | US stocks 1-minute, 16 195 symbols incl. `-DELISTED`, 2000 -> today | `symbol`, `year`, `session_date` |
40 +| `market_bars_stock_{5m,30m,1h,1d}_{raw,split,split_div}` | coarser bars (5m raw is derived from 1m; vendor 5m/30m/1h exist only adjusted) | same |
41 +| `market_bars_etf_*` | same structure for 5 161 ETFs | same |
42 +| `market_bars_index_{1m,5m,30m,1h,1d}_raw` | 122 indices (no volume) | same |
43 +| `market_bars_fx_*_raw`, `market_bars_crypto_*_raw` | FX majors (volume = tick count), crypto (UTC) | same |
44 +| `market_bars_futures_{tf}_continuous_{raw,ratio,absolute}` | continuous series per root (ES, NQ, CL...) | `symbol`, `year` |
45 +| `market_bars_futures_contracts_{tf}_raw` | individual contracts, symbol = `ROOT_MYY` (e.g. `ES_H25`) | `symbol`, `year` |
46 +| `options_eod` | EOD chains 2010 -> today, 11 582 underlyings, IV + Greeks | `underlying`, `period` (`2024q1`, `latest`), `trade_date` |
47 +
48 +Bars columns: `instrument_id, symbol_as_reported, asset_class, interval, timestamp_utc (bar OPEN, UTC), session_date, open, high, low, close, volume, currency, trade_count, vwap, adjustment_type, source_id, ingested_at, quality_flags, revision_id`.
49 +Fundamentals live in Postgres (not in SQL views): use `/api/v1/fundamentals/{ticker}` (as_of semantics) — 5 451 entities, 76 metrics (`rev, net_inc, eps_d, assets_tot, cash_op, shares, roe` ...).
50 +
51 +## Performance rules (measured on the full lake)
52 +- Put `symbol='X'` / `symbol IN (...)` (or `underlying='X'`) and ideally `year=` or a date range in every SQL statement -> 30-300 ms. Without them the view is a directory scan of up to 320 k files (30-60 s, may hit the 30 s timeout).
53 +- Cross-sections: use `market_bars_stock_1d_raw WHERE session_date = DATE '...'` (about 1.3 s) or a `session_date =` predicate on 1m (about 3 s).
54 +- REST bars endpoint: 1m <= 31 days per call, 5m <= 186 days, 1d <= 10 years; pages of <= 10 000 rows with `next_cursor`; larger ranges -> `POST /exports` (Parquet, dataset version pinned).
55 +- Rate limit 600 req/min per key. Responses carry `X-Request-ID` and `X-Dataset-Version`.
56 +
57 +## Operations (admin scope)
58 +`fdq jobs`, `fdq enqueue '{...}'` (kinds: incremental, backfill, derive, compaction, fundamentals, options, backup, quality, gc), `fdq get /api/v1/admin/storage`, `fdq get /api/v1/quality`. Full operator notes: `docs/OPERATIONS.md`. License: FirstRate Data private use — never republish raw bars/tables.
added scripts/fdq +57 −0
@@ -0,0 +1,57 @@
1 +#!/bin/bash
2 +# fdq — quick FinData OS client for shell sessions (Claude Code or human).
3 +# Reads the API key from ~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt (or $FINDATA_API_KEY).
4 +# fdq health # status + alerts
5 +# fdq datasets # catalog (id, status, rows)
6 +# fdq bars AAPL 1m 2024-03-01 2024-03-31 [raw|split|split_div] # JSON page (<= 10 000 rows); use export for more
7 +# fdq sql "SELECT ... FROM market_bars_stock_1m_raw WHERE symbol='AAPL' AND year=2024 ..." [max_rows]
8 +# fdq fund AAPL [as_of_iso] [metrics,comma] # point-in-time fundamentals
9 +# fdq export "AAPL,MSFT" 2024-01-01 2024-12-31 [dataset_id] [out.parquet] # async Parquet export -> file
10 +# fdq jobs | fdq enqueue '{"kind":"incremental","params":{"type":"stock","period":"day"}}' # admin
11 +# fdq get /api/v1/... | fdq post /api/v1/... '{json}' # raw
12 +set -euo pipefail
13 +BASE="${FINDATA_URL:-https://findata.spboucher.ai}"
14 +KEY="${FINDATA_API_KEY:-$(cat "$HOME/Desktop/Cluster/secrets/findata/api-key-claude-code.txt")}"
15 +export BASE KEY
16 +j() { python3 -c 'import sys,json; d=json.load(sys.stdin); print(json.dumps(d, indent=1, ensure_ascii=False))'; }
17 +cmd="${1:-help}"; shift || true
18 +case "$cmd" in
19 + health) curl -s -H "Authorization: Bearer $KEY" "$BASE/api/v1/health" | j ;;
20 + datasets) curl -s -H "Authorization: Bearer $KEY" "$BASE/api/v1/catalog/datasets" | python3 -c 'import sys,json
21 +for d in json.load(sys.stdin)["data"]:
22 + print(f"{d[\"id\"]:<44} {d[\"status\"]:<10} rows={(d.get(\"row_count\") or 0):>14,} {d.get(\"first_session\") or \"\"} -> {d.get(\"last_session\") or \"\"}")' ;;
23 + bars) curl -s -H "Authorization: Bearer $KEY" --get "$BASE/api/v1/market/bars" --data-urlencode "symbol=$1" --data-urlencode "interval=$2" --data-urlencode "start=$3" --data-urlencode "end=$4" --data-urlencode "adjustment=${5:-raw}" --data-urlencode "limit=${LIMIT:-10000}" | j ;;
24 + sql) SQL="$1" MAXROWS="${2:-1000}" python3 -c '
25 +import os, sys, json, urllib.request
26 +req = urllib.request.Request(os.environ["BASE"] + "/api/v1/query", data=json.dumps({"sql": os.environ["SQL"], "max_rows": int(os.environ["MAXROWS"])}).encode(),
27 + headers={"Authorization": "Bearer " + os.environ["KEY"], "Content-Type": "application/json"})
28 +try:
29 + with urllib.request.urlopen(req, timeout=180) as r: d = json.load(r)
30 +except urllib.error.HTTPError as e:
31 + sys.exit(e.read().decode()[:800])
32 +print(json.dumps({"meta": d["meta"], "columns": [c["name"] for c in d["columns"]]}, indent=1))
33 +for row in d["rows"]: print(json.dumps(row, ensure_ascii=False))' ;;
34 + fund) curl -s -H "Authorization: Bearer $KEY" --get "$BASE/api/v1/fundamentals/$1" ${2:+--data-urlencode "as_of=$2"} ${3:+--data-urlencode "metrics=$3"} | j ;;
35 + export) SYMS="$1" START="$2" END="$3" DS="${4:-market_bars.stock.1m.raw}" OUT="${5:-findata-export.parquet}" python3 -c '
36 +import os, sys, json, time, urllib.request
37 +base, key = os.environ["BASE"], os.environ["KEY"]
38 +hdr = {"Authorization": "Bearer " + key, "Content-Type": "application/json"}
39 +body = {"dataset_id": os.environ["DS"], "symbols": os.environ["SYMS"].split(","), "start": os.environ["START"], "end": os.environ["END"]}
40 +r = json.load(urllib.request.urlopen(urllib.request.Request(base + "/api/v1/exports", data=json.dumps(body).encode(), headers=hdr)))
41 +eid = r["id"]
42 +while True:
43 + st = json.load(urllib.request.urlopen(urllib.request.Request(base + "/api/v1/exports/" + eid, headers=hdr)))
44 + if st["status"] == "ready": break
45 + if st["status"] in ("failed", "expired"): sys.exit("export " + st["status"] + ": " + str(st.get("error")))
46 + time.sleep(2)
47 +with urllib.request.urlopen(urllib.request.Request(base + "/api/v1/exports/" + eid + "/download", headers=hdr)) as resp, open(os.environ["OUT"], "wb") as f:
48 + f.write(resp.read())
49 +print(os.environ["OUT"] + ": " + str(st["row_count"]) + " rows, " + str(st["bytes"]) + " bytes, dataset v" + str(st["snapshot"].get("version")))' ;;
50 + jobs) curl -s -H "Authorization: Bearer $KEY" "$BASE/api/v1/admin/jobs?limit=${LIMIT:-30}" | python3 -c 'import sys,json
51 +for j in json.load(sys.stdin)["data"]:
52 + print(f"{j[\"id\"][:8]} {j[\"kind\"]:<12} {str(j[\"dataset_id\"]):<36} {j[\"status\"]:<9} {str(j[\"created_at\"])[:16]} {str(j.get(\"last_error\") or \"\")[:60]}")' ;;
53 + enqueue) curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$1" "$BASE/api/v1/admin/jobs" | j ;;
54 + get) curl -s -H "Authorization: Bearer $KEY" "$BASE$1" | j ;;
55 + post) curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "${2:-{}}" "$BASE$1" | j ;;
56 + *) sed -n '2,12p' "$0" ;;
57 +esac
58
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere