SPB Git
spboucher.ai

spb/findata

Public

fdq shell helper + Claude Code API usage guide

Simon-Pierre Boucher committed (Nov 2, 2026) parent 8a7c5bd Browse files

2 changed files +115 −0

2 changed files
Unified Split
added docs/CLAUDE-API-USAGE.md +58 −0
@@ -0,0 +1,58 @@
1+# Using FinData OS from a Claude Code session (or any shell)
2+
3+**Key**: `~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt` (scopes: read:market, read:fundamentals, query, export, admin;
4+created 2026-11-02 for the owner's Claude Code sessions). Never paste it into code, docs or chat; the helper reads it.
5+**Base URL**: https://findata.spboucher.ai (API prefix `/api/v1`, Swagger at `/api/docs`). Auth header: `Authorization: Bearer <key>`.
6+
7+## Fastest path: the `fdq` helper
8+```bash
9+FDQ=~/Desktop/Cluster/projets/findata/scripts/fdq # laptop mirror; also at M3U96a:~/apps/findata/scripts/fdq
10+$FDQ health # status, alerts, lake free space, worker, backup age
11+$FDQ datasets # 64 datasets: id, status, rows, first/last session
12+$FDQ bars AAPL 1m 2024-03-01 2024-03-31 # JSON page (<= 10 000 rows/page, 1m <= 31 days per call); raw|split|split_div as 5th arg
13+$FDQ sql "SELECT session_date, min(low) lo, max(high) hi, sum(volume) v FROM market_bars_stock_1m_raw WHERE symbol='AAPL' AND year=2024 GROUP BY 1 ORDER BY 1"
14+$FDQ sql "SELECT trade_date, strike, call_put, bid, ask, open_interest, delta FROM options_eod WHERE underlying='SPY' AND trade_date=DATE '2024-03-15' AND expiry=DATE '2024-03-28' ORDER BY strike" 500
15+$FDQ fund MSFT 2015-01-01T00:00:00Z rev,net_inc,eps_d # point-in-time: only facts available at as_of
16+$FDQ export "AAPL,MSFT,NVDA" 2024-01-01 2024-12-31 market_bars.stock.1m.raw /tmp/x.parquet # large ranges -> async Parquet
17+$FDQ jobs ; $FDQ enqueue '{"kind":"incremental","params":{"type":"stock","period":"day"}}' # admin (ingestion queue)
18+```
19+
20+## Python (SDK)
21+```bash
22+cd ~/Desktop/Cluster/projets/findata
23+FINDATA_API_KEY=$(cat ~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt) .venv/bin/python my_script.py
24+```
25+```python
26+import os
27+from findata_sdk import FinData
28+fd = FinData("https://findata.spboucher.ai", api_key=os.environ["FINDATA_API_KEY"])
29+bars = fd.bars("AAPL", "1m", "2024-03-01", "2024-03-31") # pyarrow.Table (all pages, Arrow IPC transport)
30+df = bars.to_pandas()
31+tbl = fd.sql("SELECT symbol, count(*) n FROM market_bars_stock_1m_split_div WHERE symbol IN ('AAPL','MSFT') AND year=2020 GROUP BY 1")
32+facts = fd.fundamentals("AAPL", as_of="2020-01-01T00:00:00Z", metrics=["rev", "net_inc"], period_type="q")
33+path = fd.export(["AAPL", "MSFT"], "2015-01-01", "2024-12-31", dest="/tmp/aapl_msft_1m.parquet") # ~2 s per 1 M rows
34+```
35+
36+## What is there (SQL view names = dataset id with dots -> underscores)
37+| View | Content | Partition columns for the fast path |
38+|---|---|---|
39+| `market_bars_stock_1m_raw` (+ `_split`, `_split_div`) | US stocks 1-minute, 16 195 symbols incl. `-DELISTED`, 2000 -> today | `symbol`, `year`, `session_date` |
40+| `market_bars_stock_{5m,30m,1h,1d}_{raw,split,split_div}` | coarser bars (5m raw is derived from 1m; vendor 5m/30m/1h exist only adjusted) | same |
41+| `market_bars_etf_*` | same structure for 5 161 ETFs | same |
42+| `market_bars_index_{1m,5m,30m,1h,1d}_raw` | 122 indices (no volume) | same |
43+| `market_bars_fx_*_raw`, `market_bars_crypto_*_raw` | FX majors (volume = tick count), crypto (UTC) | same |
44+| `market_bars_futures_{tf}_continuous_{raw,ratio,absolute}` | continuous series per root (ES, NQ, CL...) | `symbol`, `year` |
45+| `market_bars_futures_contracts_{tf}_raw` | individual contracts, symbol = `ROOT_MYY` (e.g. `ES_H25`) | `symbol`, `year` |
46+| `options_eod` | EOD chains 2010 -> today, 11 582 underlyings, IV + Greeks | `underlying`, `period` (`2024q1`, `latest`), `trade_date` |
47+
48+Bars columns: `instrument_id, symbol_as_reported, asset_class, interval, timestamp_utc (bar OPEN, UTC), session_date, open, high, low, close, volume, currency, trade_count, vwap, adjustment_type, source_id, ingested_at, quality_flags, revision_id`.
49+Fundamentals live in Postgres (not in SQL views): use `/api/v1/fundamentals/{ticker}` (as_of semantics) — 5 451 entities, 76 metrics (`rev, net_inc, eps_d, assets_tot, cash_op, shares, roe` ...).
50+
51+## Performance rules (measured on the full lake)
52+- Put `symbol='X'` / `symbol IN (...)` (or `underlying='X'`) and ideally `year=` or a date range in every SQL statement -> 30-300 ms. Without them the view is a directory scan of up to 320 k files (30-60 s, may hit the 30 s timeout).
53+- Cross-sections: use `market_bars_stock_1d_raw WHERE session_date = DATE '...'` (about 1.3 s) or a `session_date =` predicate on 1m (about 3 s).
54+- REST bars endpoint: 1m <= 31 days per call, 5m <= 186 days, 1d <= 10 years; pages of <= 10 000 rows with `next_cursor`; larger ranges -> `POST /exports` (Parquet, dataset version pinned).
55+- Rate limit 600 req/min per key. Responses carry `X-Request-ID` and `X-Dataset-Version`.
56+
57+## Operations (admin scope)
58+`fdq jobs`, `fdq enqueue '{...}'` (kinds: incremental, backfill, derive, compaction, fundamentals, options, backup, quality, gc), `fdq get /api/v1/admin/storage`, `fdq get /api/v1/quality`. Full operator notes: `docs/OPERATIONS.md`. License: FirstRate Data private use — never republish raw bars/tables.
added scripts/fdq +57 −0
@@ -0,0 +1,57 @@
1+#!/bin/bash
2+# fdq — quick FinData OS client for shell sessions (Claude Code or human).
3+# Reads the API key from ~/Desktop/Cluster/secrets/findata/api-key-claude-code.txt (or $FINDATA_API_KEY).
4+# fdq health # status + alerts
5+# fdq datasets # catalog (id, status, rows)
6+# fdq bars AAPL 1m 2024-03-01 2024-03-31 [raw|split|split_div] # JSON page (<= 10 000 rows); use export for more
7+# fdq sql "SELECT ... FROM market_bars_stock_1m_raw WHERE symbol='AAPL' AND year=2024 ..." [max_rows]
8+# fdq fund AAPL [as_of_iso] [metrics,comma] # point-in-time fundamentals
9+# fdq export "AAPL,MSFT" 2024-01-01 2024-12-31 [dataset_id] [out.parquet] # async Parquet export -> file
10+# fdq jobs | fdq enqueue '{"kind":"incremental","params":{"type":"stock","period":"day"}}' # admin
11+# fdq get /api/v1/... | fdq post /api/v1/... '{json}' # raw
12+set -euo pipefail
13+BASE="${FINDATA_URL:-https://findata.spboucher.ai}"
14+KEY="${FINDATA_API_KEY:-$(cat "$HOME/Desktop/Cluster/secrets/findata/api-key-claude-code.txt")}"
15+export BASE KEY
16+j() { python3 -c 'import sys,json; d=json.load(sys.stdin); print(json.dumps(d, indent=1, ensure_ascii=False))'; }
17+cmd="${1:-help}"; shift || true
18+case "$cmd" in
19+ health) curl -s -H "Authorization: Bearer $KEY" "$BASE/api/v1/health" | j ;;
20+ datasets) curl -s -H "Authorization: Bearer $KEY" "$BASE/api/v1/catalog/datasets" | python3 -c 'import sys,json
21+for d in json.load(sys.stdin)["data"]:
22+ print(f"{d[\"id\"]:<44} {d[\"status\"]:<10} rows={(d.get(\"row_count\") or 0):>14,} {d.get(\"first_session\") or \"\"} -> {d.get(\"last_session\") or \"\"}")' ;;
23+ bars) curl -s -H "Authorization: Bearer $KEY" --get "$BASE/api/v1/market/bars" --data-urlencode "symbol=$1" --data-urlencode "interval=$2" --data-urlencode "start=$3" --data-urlencode "end=$4" --data-urlencode "adjustment=${5:-raw}" --data-urlencode "limit=${LIMIT:-10000}" | j ;;
24+ sql) SQL="$1" MAXROWS="${2:-1000}" python3 -c '
25+import os, sys, json, urllib.request
26+req = urllib.request.Request(os.environ["BASE"] + "/api/v1/query", data=json.dumps({"sql": os.environ["SQL"], "max_rows": int(os.environ["MAXROWS"])}).encode(),
27+ headers={"Authorization": "Bearer " + os.environ["KEY"], "Content-Type": "application/json"})
28+try:
29+ with urllib.request.urlopen(req, timeout=180) as r: d = json.load(r)
30+except urllib.error.HTTPError as e:
31+ sys.exit(e.read().decode()[:800])
32+print(json.dumps({"meta": d["meta"], "columns": [c["name"] for c in d["columns"]]}, indent=1))
33+for row in d["rows"]: print(json.dumps(row, ensure_ascii=False))' ;;
34+ fund) curl -s -H "Authorization: Bearer $KEY" --get "$BASE/api/v1/fundamentals/$1" ${2:+--data-urlencode "as_of=$2"} ${3:+--data-urlencode "metrics=$3"} | j ;;
35+ export) SYMS="$1" START="$2" END="$3" DS="${4:-market_bars.stock.1m.raw}" OUT="${5:-findata-export.parquet}" python3 -c '
36+import os, sys, json, time, urllib.request
37+base, key = os.environ["BASE"], os.environ["KEY"]
38+hdr = {"Authorization": "Bearer " + key, "Content-Type": "application/json"}
39+body = {"dataset_id": os.environ["DS"], "symbols": os.environ["SYMS"].split(","), "start": os.environ["START"], "end": os.environ["END"]}
40+r = json.load(urllib.request.urlopen(urllib.request.Request(base + "/api/v1/exports", data=json.dumps(body).encode(), headers=hdr)))
41+eid = r["id"]
42+while True:
43+ st = json.load(urllib.request.urlopen(urllib.request.Request(base + "/api/v1/exports/" + eid, headers=hdr)))
44+ if st["status"] == "ready": break
45+ if st["status"] in ("failed", "expired"): sys.exit("export " + st["status"] + ": " + str(st.get("error")))
46+ time.sleep(2)
47+with urllib.request.urlopen(urllib.request.Request(base + "/api/v1/exports/" + eid + "/download", headers=hdr)) as resp, open(os.environ["OUT"], "wb") as f:
48+ f.write(resp.read())
49+print(os.environ["OUT"] + ": " + str(st["row_count"]) + " rows, " + str(st["bytes"]) + " bytes, dataset v" + str(st["snapshot"].get("version")))' ;;
50+ jobs) curl -s -H "Authorization: Bearer $KEY" "$BASE/api/v1/admin/jobs?limit=${LIMIT:-30}" | python3 -c 'import sys,json
51+for j in json.load(sys.stdin)["data"]:
52+ print(f"{j[\"id\"][:8]} {j[\"kind\"]:<12} {str(j[\"dataset_id\"]):<36} {j[\"status\"]:<9} {str(j[\"created_at\"])[:16]} {str(j.get(\"last_error\") or \"\")[:60]}")' ;;
53+ enqueue) curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$1" "$BASE/api/v1/admin/jobs" | j ;;
54+ get) curl -s -H "Authorization: Bearer $KEY" "$BASE$1" | j ;;
55+ post) curl -s -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "${2:-{}}" "$BASE$1" | j ;;
56+ *) sed -n '2,12p' "$0" ;;
57+esac
58
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere