SPB Git
spboucher.ai

spb/hfmarketdata

Public

Open high-frequency market data platform — FirstRate full-history downloader, DuckDB/Parquet lake, open REST API and React docs platform (www.hfmarketdata.io)

HF Market Data — FirstRate full-history downloader, DuckDB/Parquet lake, open API and docs platform

- frd_downloader.py: 745-job resumable plan (options 2010→today, stock/etf A-Z,
  futures continuous+contracts, crypto/index/fx, all timeframes & adjustments),
  CSV→Parquet(zstd) via DuckDB, raw archives deleted on the fly
- hfmarketdata/api: open FastAPI+DuckDB API (bars, multi-ticker, point-in-time
  snapshot, options chains/history, live status), JSON+CSV
- hfmarketdata/web: React+Vite documentation platform (editorial light theme)

Author: Simon-Pierre Boucher <contact@spboucher.ai>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
simon-pierre boucher committed (Aug 10, 2026) Browse files

14 changed files +4,352 −0

14 changed files
Load all 14 files Unified Split
added .gitignore +24 −0
@@ -0,0 +1,24 @@
1+# --- heavy data: never in the repo ---
2+*.zip
3+*.parquet
4+*.csv
5+*.duckdb
6+parquet/
7+zips/
8+data/
9+state/
10+logs/
11+
12+# credentials / vendor docs containing the customer id
13+api.txt
14+
15+# environments & builds
16+venv/
17+__pycache__/
18+*.pyc
19+node_modules/
20+hfmarketdata/web/dist/
21+hfmarketdata/venv/
22+
23+# OS / editor
24+.DS_Store
added README.md +114 −0
@@ -0,0 +1,114 @@
1+# HF Market Data
2+
3+**Open high-frequency market data platform** — a full-history downloader, a DuckDB-over-Parquet data lake, an open REST API and a documentation platform, live at **[www.hfmarketdata.io](https://www.hfmarketdata.io)**.
4+
5+[![Live](https://img.shields.io/badge/live-www.hfmarketdata.io-2e5ce6)](https://www.hfmarketdata.io)
6+[![API](https://img.shields.io/badge/API-open%20%C2%B7%20no%20key-2e5ce6)](https://www.hfmarketdata.io/docs)
7+[![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?logo=python&logoColor=white)](#)
8+[![FastAPI](https://img.shields.io/badge/FastAPI-0.115%2B-009688?logo=fastapi&logoColor=white)](#)
9+[![DuckDB](https://img.shields.io/badge/DuckDB-1.x-FFF000?logo=duckdb&logoColor=black)](#)
10+[![React](https://img.shields.io/badge/React-18-61DAFB?logo=react&logoColor=black)](#)
11+[![Vite](https://img.shields.io/badge/Vite-6-646CFF?logo=vite&logoColor=white)](#)
12+[![Parquet](https://img.shields.io/badge/storage-Parquet%20%C2%B7%20zstd-50ABF1)](#)
13+
14+[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-1c2430)](mailto:contact@spboucher.ai)
15+[![Contact](https://img.shields.io/badge/contact-contact%40spboucher.ai-1c2430)](mailto:contact@spboucher.ai)
16+
17+---
18+
19+## Dataset metrics
20+
21+Live snapshot of the lake while ingestion is still running (2026-08-10):
22+
23+| Metric | Value |
24+|---|---:|
25+| Rows converted to Parquet | **1,785,837,666+** |
26+| Parquet files (one per instrument/segment) | **211,140+** |
27+| Lake size (zstd Parquet) | **34 GB+** (raw archives deleted on the fly) |
28+| Options quarters ingested | **30+ / 66** (2010_q1 → today) |
29+| Options underlyings | **5,800+** US equities & indices |
30+| Bar timeframes | 1min · 5min · 30min · 1hour · 1day |
31+| Asset classes | stocks · ETFs · futures (continuous + contracts) · crypto · indices · FX · options |
32+| Download plan | 745 jobs, fully resumable, 0 conversion errors |
33+
34+Query the live inventory anytime: [`GET /v1/status`](https://www.hfmarketdata.io/v1/status).
35+
36+## Architecture
37+
38+```
39+FirstRate Data API ──▶ frd_downloader.py ──▶ Parquet lake (zstd, one file per instrument)
40+ (zip archives) · 745-job plan /parquet/{type}/{timeframe}/{adjustment}/{TICKER}.parquet
41+ · resume manifest /parquet/options/{year}_{quarter}/…
42+ · CSV→Parquet via
43+ DuckDB, raw deleted
44+ │ queried in place (predicate pushdown)
45+ ▼
46+ hfmarketdata/api/main.py (FastAPI + DuckDB)
47+ │ :8090
48+ hfmarketdata/web/ (React + Vite docs platform)
49+ │
50+ https://www.hfmarketdata.io (ngrok)
51+```
52+
53+## Repository layout
54+
55+```
56+frd_downloader.py Full-history downloader & Parquet converter (single file)
57+requirements.txt Downloader dependencies (requests, duckdb)
58+hfmarketdata/
59+ api/main.py Open REST API — FastAPI + DuckDB over the Parquet lake
60+ requirements.txt API dependencies
61+ web/ Documentation platform (React 18 + Vite, English)
62+```
63+
64+Heavy data (Parquet lake, zip archives, venvs, builds) is **not** in this repository — see `.gitignore`.
65+
66+## The downloader
67+
68+Downloads **everything** FirstRate Data publishes and stores it query-ready:
69+
70+- **Options** — every quarterly archive since 2010 + the current partial quarter (16 fields per row: quotes, bid/ask IV, open interest, volume, delta, gamma, vega, theta, rho)
71+- **Stocks & ETFs** — full history, tickers A–Z, all 5 timeframes × `adj_split` / `adj_splitdiv` / `UNADJUSTED`
72+- **Futures** — continuous series (`contin_UNadj` / `contin_adj_ratio` / `contin_adj_absolute`) + individual contracts
73+- **Crypto, indices, FX** — full history, all timeframes
74+- **Metadata** — ticker listings, splits, dividends, update logs, continuous-series audit
75+
76+```bash
77+export FRD_USERID=<your FirstRate customer id>
78+python3 -m venv venv && venv/bin/pip install -r requirements.txt
79+venv/bin/python frd_downloader.py --data-root /Volumes/ssd/firstratedata
80+```
81+
82+Fully resumable (`state/manifest.json`), disk-space guard, retries with backoff, and raw zips/CSVs are deleted as soon as each segment is converted. Notable quirks handled: FirstRate files mix LF/CRLF line endings (`strict_mode=false`), daily files use `yyyyMMdd` dates, the API 302-redirects to a CDN.
83+
84+## The API
85+
86+Open, keyless, JSON + CSV. Highlights:
87+
88+| Endpoint | What it does |
89+|---|---|
90+| `GET /v1/status` | Live inventory of the lake |
91+| `GET /v1/{asset}/tickers` | Instrument discovery per type/timeframe/adjustment |
92+| `GET /v1/bars/{asset}/{ticker}` | OHLCV bars, precise `start`/`end` windows, up to 2M rows in CSV |
93+| `GET /v1/bars/{asset}?tickers=A,B,C` | Multi-ticker bars in one call (per-ticker limit) |
94+| `GET /v1/snapshot/{asset}?tickers=…&at=…` | Whole watchlist at one precise moment |
95+| `GET /v1/options/chain/{ticker}` | Full EOD chain — quotes, IV, Greeks |
96+| `GET /v1/options/history/{ticker}` | One contract's daily life story |
97+| `GET /docs` | OpenAPI / Swagger |
98+
99+```bash
100+# 4 hours of 1-minute bars across a watchlist
101+curl "https://www.hfmarketdata.io/v1/bars/stock?tickers=AAPL,MSFT,NVDA&timeframe=1min&start=2024-06-03%2009:30:00&end=2024-06-03%2013:30:00"
102+```
103+
104+## The platform
105+
106+React 18 + Vite documentation site (English) served by the API process — refined editorial light theme (Fraunces / Inter / JetBrains Mono), full endpoint reference with parameter tables, request snippets (curl / Python / JavaScript), example + **live** responses, response-field dictionaries and a live dataset status page.
107+
108+```bash
109+cd hfmarketdata/web && npm install && npm run build # → dist/, served at /
110+```
111+
112+## Author
113+
114+**Simon-Pierre Boucher** — [contact@spboucher.ai](mailto:contact@spboucher.ai)
added frd_downloader.py +627 −0
@@ -0,0 +1,627 @@
1+#!/usr/bin/env python3
2+# -*- coding: utf-8 -*-
3+"""
4+================================================================================
5+ FirstRate Data — Full Historical Downloader & Parquet Converter
6+================================================================================
7+
8+Downloads the COMPLETE historical datasets from the FirstRate Data API and
9+stores them as query-optimized Parquet files (zstd compressed), ready to be
10+served by an ultra-fast query layer (DuckDB / pyarrow).
11+
12+Coverage
13+--------
14+ * options : full archives per year/quarter (2010 -> today)
15+ + 'month' update for the current partial quarter
16+ * stock, etf : period=full, ticker_range A-Z,
17+ timeframes 1min/5min/30min/1hour/1day,
18+ adjustments adj_split / adj_splitdiv / UNADJUSTED
19+ (UNADJUSTED only exists for 1min & 1day)
20+ * futures (continuous) : timeframes x contin_UNadj / contin_adj_ratio /
21+ contin_adj_absolute
22+ * futures (individual) : contract_files=archive + update, all timeframes
23+ * crypto, index, fx : period=full, all timeframes
24+ * metadata : ticker listings, splits, dividends, update logs,
25+ futures continuous-series audit file
26+ * delisted stocks : optional (--include-delisted), archives 1-4 +
27+ 2026 update, all timeframe/adjustment combos
28+
29+Storage layout (under --data-root)
30+----------------------------------
31+ parquet/{type}/{timeframe}/{adjustment}/{TICKER}.parquet (bar data)
32+ parquet/options/{year}_{quarter}/{file}.parquet (options)
33+ meta/… (csv metadata)
34+ zips/… (only w/ --keep-zips)
35+ state/manifest.json (resume state)
36+ logs/frd_downloader.log
37+
38+Every Parquet file carries a `ticker` column, so the whole tree can be
39+queried directly, e.g.:
40+
41+ SELECT * FROM read_parquet('parquet/stock/1day/adj_splitdiv/*.parquet')
42+ WHERE ticker = 'AAPL' AND datetime >= '2020-01-01';
43+
44+Usage
45+-----
46+ python3 frd_downloader.py --data-root /Volumes/ssd/firstratedata
47+ python3 frd_downloader.py --list-jobs # preview the job plan
48+ python3 frd_downloader.py --types options stock # restrict asset types
49+ python3 frd_downloader.py --include-delisted # add delisted stocks
50+
51+The run is fully resumable: completed jobs are recorded in the manifest and
52+skipped on re-run.
53+
54+Author : Simon-Pierre Boucher
55+Contact : contact@spboucher.ai
56+Created : 2026-08-09
57+================================================================================
58+"""
59+
60+from __future__ import annotations
61+
62+import argparse
63+import json
64+import logging
65+import os
66+import shutil
67+import sys
68+import tempfile
69+import threading
70+import time
71+import zipfile
72+from concurrent.futures import ThreadPoolExecutor, as_completed
73+from dataclasses import dataclass, field
74+from datetime import date
75+from pathlib import Path
76+from typing import Iterator
77+from urllib.parse import urlencode
78+
79+import duckdb
80+import requests
81+
82+__author__ = "Simon-Pierre Boucher"
83+__contact__ = "contact@spboucher.ai"
84+__version__ = "1.0.0"
85+
86+# ------------------------------------------------------------------------------
87+# Configuration
88+# ------------------------------------------------------------------------------
89+
90+BASE_URL = "https://firstratedata.com/api"
91+# FirstRate Data customer id — never hardcoded: export FRD_USERID=<your id>
92+USERID = os.environ.get("FRD_USERID", "")
93+
94+TIMEFRAMES = ["1day", "1hour", "30min", "5min", "1min"] # small -> large
95+STOCK_ETF_ADJUSTMENTS = ["adj_split", "adj_splitdiv", "UNADJUSTED"]
96+UNADJUSTED_TIMEFRAMES = {"1min", "1day"} # per API doc
97+FUTURES_ADJUSTMENTS = ["contin_UNadj", "contin_adj_ratio", "contin_adj_absolute"]
98+TICKER_RANGES = [chr(c) for c in range(ord("A"), ord("Z") + 1)]
99+OPTIONS_FIRST_YEAR = 2010
100+
101+DOWNLOAD_TIMEOUT = 7200 # seconds per request (archives can be huge)
102+CHUNK_SIZE = 4 * 1024 * 1024 # 4 MiB streaming chunks
103+MAX_RETRIES = 5
104+RETRY_BACKOFF = 60 # seconds, multiplied by attempt number
105+MIN_FREE_GB = 30 # abort downloads below this free-space floor
106+
107+log = logging.getLogger("frd")
108+
109+# ------------------------------------------------------------------------------
110+# Job model
111+# ------------------------------------------------------------------------------
112+
113+
114+@dataclass
115+class Job:
116+ """One downloadable unit: an API request plus its destination."""
117+
118+ key: str # unique id, used in the manifest
119+ endpoint: str # API endpoint path (e.g. 'data_file')
120+ params: dict # query parameters (userid added at request time)
121+ dest_subdir: str # where converted output lands, under data root
122+ kind: str = "bars" # 'bars' (zip of csv -> parquet) or 'meta' (raw text)
123+ priority: int = 50 # lower runs first
124+
125+ @property
126+ def url(self) -> str:
127+ q = dict(self.params)
128+ q["userid"] = USERID
129+ return f"{BASE_URL}/{self.endpoint}?{urlencode(q)}"
130+
131+ @property
132+ def display_url(self) -> str:
133+ return f"{BASE_URL}/{self.endpoint}?{urlencode(self.params)}&userid=***"
134+
135+
136+def _current_quarter(today: date) -> tuple[int, int]:
137+ return today.year, (today.month - 1) // 3 + 1
138+
139+
140+def build_jobs(types: list[str], include_delisted: bool) -> list[Job]:
141+ """Build the full download plan, ordered so that the small/most-useful
142+ datasets (daily bars, metadata) land first and the huge ones last."""
143+ jobs: list[Job] = []
144+ today = date.today()
145+
146+ # -- metadata (tiny, always useful, run first) ------------------------------
147+ for t in ["stock", "etf", "futures", "crypto", "index", "fx"]:
148+ if t in types:
149+ jobs.append(Job(
150+ key=f"ticker_listing|{t}", endpoint="ticker_listing",
151+ params={"type": t}, dest_subdir=f"meta/{t}",
152+ kind="meta", priority=0))
153+ for t in ["stock", "etf"]:
154+ if t in types:
155+ for mft in ["splits", "dividends"]:
156+ jobs.append(Job(
157+ key=f"meta|{t}|{mft}", endpoint="meta_file",
158+ params={"type": t, "metafile_type": mft},
159+ dest_subdir=f"meta/{t}", kind="meta", priority=1))
160+ if "stock" in types:
161+ for lt in ["delisted", "added", "changed"]:
162+ jobs.append(Job(
163+ key=f"update_log|stock|{lt}", endpoint="update_log",
164+ params={"type": "stock", "log_type": lt},
165+ dest_subdir="meta/stock", kind="meta", priority=1))
166+ if "futures" in types:
167+ jobs.append(Job(
168+ key="meta|futures|contin_audit", endpoint="meta_file",
169+ params={"type": "futures", "metafile_type": "contin_audit"},
170+ dest_subdir="meta/futures", kind="meta", priority=1))
171+
172+ # -- stock / etf : full archives, per letter --------------------------------
173+ for t in ["stock", "etf"]:
174+ if t not in types:
175+ continue
176+ for tf_i, tf in enumerate(TIMEFRAMES):
177+ for adj in STOCK_ETF_ADJUSTMENTS:
178+ if adj == "UNADJUSTED" and tf not in UNADJUSTED_TIMEFRAMES:
179+ continue
180+ for letter in TICKER_RANGES:
181+ jobs.append(Job(
182+ key=f"{t}|full|{tf}|{adj}|{letter}",
183+ endpoint="data_file",
184+ params={"type": t, "period": "full",
185+ "ticker_range": letter,
186+ "timeframe": tf, "adjustment": adj},
187+ dest_subdir=f"parquet/{t}/{tf}/{adj}",
188+ priority=10 + tf_i * 4))
189+
190+ # -- futures : continuous series --------------------------------------------
191+ if "futures" in types:
192+ for tf_i, tf in enumerate(TIMEFRAMES):
193+ for adj in FUTURES_ADJUSTMENTS:
194+ jobs.append(Job(
195+ key=f"futures|full|{tf}|{adj}",
196+ endpoint="data_file",
197+ params={"type": "futures", "period": "full",
198+ "timeframe": tf, "adjustment": adj},
199+ dest_subdir=f"parquet/futures/{tf}/{adj}",
200+ priority=10 + tf_i * 4))
201+ # individual contracts (pre-2026 archive + current-year update)
202+ for tf_i, tf in enumerate(TIMEFRAMES):
203+ for cf in ["archive", "update"]:
204+ jobs.append(Job(
205+ key=f"futures_contract|{cf}|{tf}",
206+ endpoint="futures_contract",
207+ params={"contract_files": cf, "timeframe": tf},
208+ dest_subdir=f"parquet/futures_contracts/{tf}/{cf}",
209+ priority=12 + tf_i * 4))
210+
211+ # -- crypto / index / fx : full archives -------------------------------------
212+ for t in ["crypto", "index", "fx"]:
213+ if t not in types:
214+ continue
215+ for tf_i, tf in enumerate(TIMEFRAMES):
216+ jobs.append(Job(
217+ key=f"{t}|full|{tf}",
218+ endpoint="data_file",
219+ params={"type": t, "period": "full", "timeframe": tf},
220+ dest_subdir=f"parquet/{t}/{tf}/none",
221+ priority=10 + tf_i * 4))
222+
223+ # -- options : every quarter since 2010 + current partial quarter ------------
224+ # Priority 5-6: options are the primary dataset, they download right after
225+ # the metadata and before all the bar archives.
226+ if "options" in types:
227+ cur_year, cur_q = _current_quarter(today)
228+ for year in range(OPTIONS_FIRST_YEAR, cur_year + 1):
229+ for q in range(1, 5):
230+ if year == cur_year and q >= cur_q:
231+ continue # current/future quarters have no full archive yet
232+ jobs.append(Job(
233+ key=f"options|{year}|q{q}",
234+ endpoint="data_file",
235+ params={"type": "options", "year": str(year),
236+ "quarter": f"q{q}"},
237+ dest_subdir=f"parquet/options/{year}_q{q}",
238+ priority=5))
239+ jobs.append(Job(
240+ key="options|update|month",
241+ endpoint="data_file",
242+ params={"type": "options", "period": "month"},
243+ dest_subdir=f"parquet/options/{cur_year}_q{cur_q}_partial",
244+ priority=6))
245+
246+ # -- delisted stocks (optional, survivorship-bias-free research) -------------
247+ if include_delisted and "stock" in types:
248+ for tf_i, tf in enumerate(TIMEFRAMES):
249+ for adj in STOCK_ETF_ADJUSTMENTS:
250+ if adj == "UNADJUSTED" and tf != "1min":
251+ continue # per API doc: UNADJUSTED delisted = 1min only
252+ for arc in ["1", "2", "3", "4"]:
253+ jobs.append(Job(
254+ key=f"delisted|archive{arc}|{tf}|{adj}",
255+ endpoint="delisted_data_file",
256+ params={"archive_number": arc,
257+ "timeframe": tf, "adjustment": adj},
258+ dest_subdir=f"parquet/stock_delisted/{tf}/{adj}",
259+ priority=30 + tf_i * 4))
260+ jobs.append(Job(
261+ key=f"delisted|update_year|{tf}|{adj}",
262+ endpoint="delisted_data_file",
263+ params={"update": "year", "timeframe": tf,
264+ "adjustment": adj},
265+ dest_subdir=f"parquet/stock_delisted/{tf}/{adj}",
266+ priority=30 + tf_i * 4))
267+
268+ jobs.sort(key=lambda j: (j.priority, j.key))
269+ return jobs
270+
271+
272+# ------------------------------------------------------------------------------
273+# Manifest (resume state)
274+# ------------------------------------------------------------------------------
275+
276+
277+class Manifest:
278+ """Thread-safe JSON manifest tracking completed/failed jobs."""
279+
280+ def __init__(self, path: Path):
281+ self.path = path
282+ self._lock = threading.Lock()
283+ self._data: dict = {"jobs": {}, "version": __version__}
284+ if path.exists():
285+ try:
286+ self._data = json.loads(path.read_text())
287+ except (json.JSONDecodeError, OSError):
288+ log.warning("Manifest unreadable, starting fresh: %s", path)
289+
290+ def is_done(self, key: str) -> bool:
291+ return self._data["jobs"].get(key, {}).get("status") == "done"
292+
293+ def mark(self, key: str, status: str, **info) -> None:
294+ with self._lock:
295+ entry = self._data["jobs"].setdefault(key, {})
296+ entry.update(status=status, updated=time.strftime("%Y-%m-%d %H:%M:%S"), **info)
297+ tmp = self.path.with_suffix(".tmp")
298+ tmp.write_text(json.dumps(self._data, indent=1))
299+ tmp.replace(self.path)
300+
301+ def summary(self) -> dict:
302+ counts: dict[str, int] = {}
303+ for e in self._data["jobs"].values():
304+ counts[e.get("status", "?")] = counts.get(e.get("status", "?"), 0) + 1
305+ return counts
306+
307+
308+# ------------------------------------------------------------------------------
309+# CSV -> Parquet conversion
310+# ------------------------------------------------------------------------------
311+
312+# Column layouts by column count (FirstRate bar files have no header row).
313+# The first field is read as VARCHAR because FirstRate mixes date formats
314+# ('2005-01-03 17:00:00' intraday vs '20100104' daily) — it is normalized to a
315+# proper TIMESTAMP at conversion time (see _DT_PARSE).
316+BAR_SCHEMAS = {
317+ 5: {"dt_raw": "VARCHAR", "open": "DOUBLE", "high": "DOUBLE",
318+ "low": "DOUBLE", "close": "DOUBLE"},
319+ 6: {"dt_raw": "VARCHAR", "open": "DOUBLE", "high": "DOUBLE",
320+ "low": "DOUBLE", "close": "DOUBLE", "volume": "DOUBLE"},
321+ 7: {"dt_raw": "VARCHAR", "open": "DOUBLE", "high": "DOUBLE",
322+ "low": "DOUBLE", "close": "DOUBLE", "volume": "DOUBLE",
323+ "open_interest": "DOUBLE"},
324+}
325+
326+_DT_PARSE = ("COALESCE("
327+ "try_strptime(dt_raw, '%Y-%m-%d %H:%M:%S'), "
328+ "try_strptime(dt_raw, '%Y%m%d'), "
329+ "try_strptime(dt_raw, '%Y-%m-%d'), "
330+ "try_strptime(dt_raw, '%m/%d/%Y %H:%M:%S'), "
331+ "try_strptime(dt_raw, '%m/%d/%Y'))")
332+
333+# Options chain layout, per https://firstratedata.com/_readme/options.txt :
334+# Trade Date, Strike, Expiry Date, Call/Put, Last Trade Price, Bid Price,
335+# Ask Price, Bid IV, Ask IV, Open Interest, Volume, Delta, Gamma, Vega,
336+# Theta, Rho
337+OPTIONS_SCHEMA = {
338+ "trade_date": "DATE", "strike": "DOUBLE", "expiry": "DATE",
339+ "call_put": "VARCHAR", "last_price": "DOUBLE", "bid": "DOUBLE",
340+ "ask": "DOUBLE", "bid_iv": "DOUBLE", "ask_iv": "DOUBLE",
341+ "open_interest": "DOUBLE", "volume": "DOUBLE", "delta": "DOUBLE",
342+ "gamma": "DOUBLE", "vega": "DOUBLE", "theta": "DOUBLE", "rho": "DOUBLE",
343+}
344+
345+
346+def _sniff_column_count(csv_path: Path) -> int:
347+ with open(csv_path, "r", errors="replace") as f:
348+ for line in f:
349+ line = line.strip()
350+ if line:
351+ return line.count(",") + 1
352+ return 0
353+
354+
355+def _ticker_from_filename(path: Path) -> str:
356+ """'AAPL_1min.txt' -> 'AAPL' ; 'ES_contin_adj.txt' -> 'ES'."""
357+ return path.stem.split("_")[0].upper()
358+
359+
360+def csv_to_parquet(con: duckdb.DuckDBPyConnection, csv_path: Path,
361+ out_path: Path, asset_type: str) -> int:
362+ """Convert one extracted csv/txt file to zstd Parquet. Returns row count."""
363+ ncols = _sniff_column_count(csv_path)
364+ if ncols == 0:
365+ return 0
366+ out_path.parent.mkdir(parents=True, exist_ok=True)
367+ ticker = _ticker_from_filename(csv_path)
368+ src = str(csv_path).replace("'", "''")
369+ dst = str(out_path).replace("'", "''")
370+
371+ # strict_mode=false: FirstRate files mix LF and CRLF line endings (daily
372+ # rows are appended with CRLF onto LF history), which the strict CSV
373+ # sniffer rejects outright.
374+ common = "strict_mode=false, ignore_errors=true"
375+ if asset_type == "options" and ncols == len(OPTIONS_SCHEMA):
376+ cols = json.dumps(OPTIONS_SCHEMA).replace('"', "'")
377+ select = (f"SELECT '{ticker}' AS ticker, * FROM read_csv('{src}', "
378+ f"header=false, columns={cols}, dateformat='%Y-%m-%d', "
379+ f"{common})")
380+ elif asset_type != "options" and (schema := BAR_SCHEMAS.get(ncols)):
381+ cols = json.dumps(schema).replace('"', "'")
382+ value_cols = ", ".join(c for c in schema if c != "dt_raw")
383+ select = (f"SELECT '{ticker}' AS ticker, {_DT_PARSE} AS datetime, "
384+ f"{value_cols} FROM read_csv('{src}', "
385+ f"header=false, columns={cols}, {common})")
386+ else:
387+ # Unknown layout (options files etc.): let DuckDB auto-detect types,
388+ # header presence and column names — still fully queryable.
389+ select = (f"SELECT '{ticker}' AS ticker, * FROM read_csv_auto('{src}', "
390+ f"{common})")
391+
392+ con.execute(f"COPY ({select}) TO '{dst}' "
393+ f"(FORMAT PARQUET, COMPRESSION ZSTD, ROW_GROUP_SIZE 1000000)")
394+ rows = con.execute(f"SELECT count(*) FROM read_parquet('{dst}')").fetchone()[0]
395+ if rows == 0 and csv_path.stat().st_size > 0:
396+ log.warning("0 rows converted from non-empty file %s (ncols=%d)",
397+ csv_path.name, ncols)
398+ return rows
399+
400+
401+def extract_and_convert(zip_path: Path, dest_dir: Path, asset_type: str) -> tuple[int, int]:
402+ """Extract a downloaded zip (recursively if it nests zips) and convert every
403+ csv/txt member to Parquet under dest_dir. Returns (files, rows)."""
404+ files = rows = 0
405+ con = duckdb.connect()
406+ con.execute("SET threads TO 4")
407+ try:
408+ with tempfile.TemporaryDirectory(dir=zip_path.parent) as tmp:
409+ tmp_dir = Path(tmp)
410+ _extract_recursive(zip_path, tmp_dir)
411+ members = sorted(p for p in tmp_dir.rglob("*")
412+ if p.is_file() and p.suffix.lower() in (".txt", ".csv"))
413+ for member in members:
414+ out = dest_dir / (member.stem + ".parquet")
415+ try:
416+ n = csv_to_parquet(con, member, out, asset_type)
417+ if n > 0:
418+ files += 1
419+ rows += n
420+ else:
421+ out.unlink(missing_ok=True)
422+ except duckdb.Error as e:
423+ log.error("Convert failed %s: %s", member.name, e)
424+ member.unlink() # free space as we go
425+ finally:
426+ con.close()
427+ return files, rows
428+
429+
430+def _extract_recursive(zip_path: Path, dest: Path, depth: int = 0) -> None:
431+ if depth > 3:
432+ return
433+ with zipfile.ZipFile(zip_path) as zf:
434+ zf.extractall(dest)
435+ for nested in list(dest.rglob("*.zip")):
436+ sub = nested.with_suffix("")
437+ sub.mkdir(exist_ok=True)
438+ _extract_recursive(nested, sub, depth + 1)
439+ nested.unlink()
440+
441+
442+# ------------------------------------------------------------------------------
443+# Downloader
444+# ------------------------------------------------------------------------------
445+
446+
447+@dataclass
448+class Runner:
449+ data_root: Path
450+ manifest: Manifest
451+ keep_zips: bool = False
452+ min_free_gb: int = MIN_FREE_GB
453+ session: requests.Session = field(default_factory=requests.Session)
454+
455+ def _free_gb(self) -> float:
456+ return shutil.disk_usage(self.data_root).free / 1e9
457+
458+ def _download(self, job: Job, dest: Path) -> Path | None:
459+ """Stream the response to dest (via .part). Returns path or None."""
460+ part = dest.with_suffix(dest.suffix + ".part")
461+ for attempt in range(1, MAX_RETRIES + 1):
462+ if self._free_gb() < self.min_free_gb:
463+ raise RuntimeError(
464+ f"Only {self._free_gb():.1f} GB free (< {self.min_free_gb} GB floor)")
465+ try:
466+ with self.session.get(job.url, stream=True,
467+ timeout=DOWNLOAD_TIMEOUT) as r:
468+ if r.status_code in (429, 500, 502, 503, 504):
469+ raise requests.HTTPError(f"HTTP {r.status_code}")
470+ if r.status_code == 404:
471+ log.warning("[%s] 404 — no data for this combination", job.key)
472+ return None
473+ r.raise_for_status()
474+ dest.parent.mkdir(parents=True, exist_ok=True)
475+ size = 0
476+ with open(part, "wb") as f:
477+ for chunk in r.iter_content(CHUNK_SIZE):
478+ f.write(chunk)
479+ size += len(chunk)
480+ part.replace(dest)
481+ log.info("[%s] downloaded %.1f MB", job.key, size / 1e6)
482+ return dest
483+ except (requests.RequestException, OSError) as e:
484+ part.unlink(missing_ok=True)
485+ if attempt == MAX_RETRIES:
486+ raise
487+ wait = RETRY_BACKOFF * attempt
488+ log.warning("[%s] attempt %d/%d failed (%s), retrying in %ds",
489+ job.key, attempt, MAX_RETRIES, e, wait)
490+ time.sleep(wait)
491+ return None
492+
493+ def run_job(self, job: Job) -> None:
494+ if self.manifest.is_done(job.key):
495+ log.info("[%s] already done, skipping", job.key)
496+ return
497+ log.info("[%s] GET %s", job.key, job.display_url)
498+ t0 = time.time()
499+ try:
500+ if job.kind == "meta":
501+ self._run_meta(job)
502+ else:
503+ self._run_bars(job)
504+ except Exception as e:
505+ log.error("[%s] FAILED: %s", job.key, e)
506+ self.manifest.mark(job.key, "failed", error=str(e))
507+ return
508+ log.info("[%s] done in %.0fs", job.key, time.time() - t0)
509+
510+ def _run_meta(self, job: Job) -> None:
511+ out_dir = self.data_root / job.dest_subdir
512+ out_dir.mkdir(parents=True, exist_ok=True)
513+ fname = "_".join(str(v) for v in job.params.values()) + ".csv"
514+ out = out_dir / fname
515+ r = self.session.get(job.url, timeout=600)
516+ r.raise_for_status()
517+ out.write_bytes(r.content)
518+ self.manifest.mark(job.key, "done", bytes=len(r.content), file=str(out))
519+
520+ def _run_bars(self, job: Job) -> None:
521+ zip_dir = self.data_root / "zips"
522+ zip_dir.mkdir(parents=True, exist_ok=True)
523+ zip_path = zip_dir / (job.key.replace("|", "_") + ".zip")
524+
525+ got = self._download(job, zip_path)
526+ if got is None:
527+ self.manifest.mark(job.key, "empty")
528+ return
529+ if not zipfile.is_zipfile(zip_path):
530+ head = zip_path.read_bytes()[:200]
531+ zip_path.unlink()
532+ raise RuntimeError(f"Response is not a zip (starts with {head[:60]!r})")
533+
534+ asset_type = job.key.split("|")[0]
535+ dest_dir = self.data_root / job.dest_subdir
536+ files, rows = extract_and_convert(zip_path, dest_dir, asset_type)
537+ size = zip_path.stat().st_size
538+ if not self.keep_zips:
539+ zip_path.unlink()
540+ self.manifest.mark(job.key, "done", bytes=size, files=files, rows=rows)
541+ log.info("[%s] converted %d files / %s rows -> %s",
542+ job.key, files, f"{rows:,}", dest_dir)
543+
544+
545+# ------------------------------------------------------------------------------
546+# CLI
547+# ------------------------------------------------------------------------------
548+
549+
550+def setup_logging(data_root: Path) -> None:
551+ log_dir = data_root / "logs"
552+ log_dir.mkdir(parents=True, exist_ok=True)
553+ fmt = logging.Formatter("%(asctime)s %(levelname)-7s %(message)s",
554+ "%Y-%m-%d %H:%M:%S")
555+ for handler in (logging.StreamHandler(sys.stdout),
556+ logging.FileHandler(log_dir / "frd_downloader.log")):
557+ handler.setFormatter(fmt)
558+ log.addHandler(handler)
559+ log.setLevel(logging.INFO)
560+
561+
562+def main() -> int:
563+ ap = argparse.ArgumentParser(
564+ description="FirstRate Data full-history downloader "
565+ f"(v{__version__} — {__author__} <{__contact__}>)")
566+ ap.add_argument("--data-root", type=Path,
567+ default=Path("/Volumes/ssd/firstratedata"),
568+ help="Root directory for all data (default: %(default)s)")
569+ ap.add_argument("--types", nargs="+",
570+ default=["options", "stock", "etf", "futures",
571+ "crypto", "index", "fx"],
572+ choices=["options", "stock", "etf", "futures",
573+ "crypto", "index", "fx"],
574+ help="Asset types to download (default: all)")
575+ ap.add_argument("--workers", type=int, default=2,
576+ help="Parallel downloads (default: %(default)s — be polite)")
577+ ap.add_argument("--keep-zips", action="store_true",
578+ help="Keep raw zip archives after conversion")
579+ ap.add_argument("--include-delisted", action="store_true",
580+ help="Also download the delisted-stocks archives")
581+ ap.add_argument("--min-free-gb", type=int, default=MIN_FREE_GB,
582+ help="Stop downloading below this free space (default: %(default)s)")
583+ ap.add_argument("--list-jobs", action="store_true",
584+ help="Print the job plan and exit")
585+ args = ap.parse_args()
586+
587+ if not USERID:
588+ ap.error("FRD_USERID environment variable is not set "
589+ "(your FirstRate Data customer id)")
590+
591+ jobs = build_jobs(args.types, args.include_delisted)
592+
593+ if args.list_jobs:
594+ for j in jobs:
595+ print(f"{j.priority:3d} {j.key:45s} {j.display_url}")
596+ print(f"\nTotal: {len(jobs)} jobs")
597+ return 0
598+
599+ args.data_root.mkdir(parents=True, exist_ok=True)
600+ setup_logging(args.data_root)
601+ log.info("FirstRate Data downloader v%s — %s <%s>",
602+ __version__, __author__, __contact__)
603+ log.info("Data root: %s | types: %s | %d jobs | %d workers",
604+ args.data_root, ",".join(args.types), len(jobs), args.workers)
605+
606+ manifest = Manifest(args.data_root / "state" / "manifest.json")
607+ (args.data_root / "state").mkdir(parents=True, exist_ok=True)
608+ runner = Runner(args.data_root, manifest,
609+ keep_zips=args.keep_zips, min_free_gb=args.min_free_gb)
610+
611+ pending = [j for j in jobs if not manifest.is_done(j.key)]
612+ log.info("%d jobs pending (%d already complete)",
613+ len(pending), len(jobs) - len(pending))
614+
615+ with ThreadPoolExecutor(max_workers=args.workers) as pool:
616+ futures = {pool.submit(runner.run_job, j): j for j in pending}
617+ for fut in as_completed(futures):
618+ fut.result() # exceptions are handled inside run_job
619+
620+ counts = manifest.summary()
621+ log.info("Run complete. Manifest: %s", counts)
622+ failed = counts.get("failed", 0)
623+ return 1 if failed else 0
624+
625+
626+if __name__ == "__main__":
627+ sys.exit(main())
added hfmarketdata/api/main.py +476 −0
@@ -0,0 +1,476 @@
1+#!/usr/bin/env python3
2+# -*- coding: utf-8 -*-
3+"""
4+================================================================================
5+ HF Market Data — Open High-Frequency Market Data API
6+ https://www.hfmarketdata.io
7+================================================================================
8+
9+Ultra-fast open REST API serving historical market data (stocks, ETFs,
10+futures, crypto, indices, FX — 1-minute to daily bars — plus full end-of-day
11+options chains with quotes, implied volatilities and Greeks).
12+
13+The data layer is a partitioned Parquet lake (zstd) queried in-place with
14+DuckDB — no database server, predicate pushdown straight into the row groups.
15+
16+Layout expected under DATA_ROOT (built by frd_downloader.py):
17+ parquet/{type}/{timeframe}/{adjustment}/{TICKER}_*.parquet
18+ parquet/options/{year}_{quarter}/{TICKER}_*_option_chain.parquet
19+
20+Author : Simon-Pierre Boucher
21+Contact : contact@spboucher.ai
22+================================================================================
23+"""
24+
25+from __future__ import annotations
26+
27+import os
28+import threading
29+import time
30+from pathlib import Path
31+
32+import duckdb
33+from fastapi import FastAPI, HTTPException, Query
34+from fastapi.middleware.cors import CORSMiddleware
35+from fastapi.responses import PlainTextResponse
36+from fastapi.staticfiles import StaticFiles
37+
38+__author__ = "Simon-Pierre Boucher"
39+__contact__ = "contact@spboucher.ai"
40+__version__ = "1.0.0"
41+
42+DATA_ROOT = Path(os.environ.get("HFMD_DATA_ROOT", "/Volumes/ssd/firstratedata"))
43+PARQUET = DATA_ROOT / "parquet"
44+WEB_DIST = Path(os.environ.get("HFMD_WEB_DIST",
45+ Path(__file__).resolve().parent.parent / "web" / "dist"))
46+
47+BAR_TYPES = ["stock", "etf", "futures", "futures_contracts", "crypto", "index", "fx"]
48+TIMEFRAMES = ["1min", "5min", "30min", "1hour", "1day"]
49+DEFAULT_ADJUSTMENT = {
50+ "stock": "adj_splitdiv", "etf": "adj_splitdiv",
51+ "futures": "contin_UNadj", "futures_contracts": "archive",
52+ "crypto": "none", "index": "none", "fx": "none",
53+}
54+MAX_LIMIT_JSON = 50_000
55+MAX_LIMIT_CSV = 2_000_000
56+
57+app = FastAPI(
58+ title="HF Market Data API",
59+ description=(
60+ "Open high-frequency market data API — intraday & daily bars for "
61+ "stocks, ETFs, futures, crypto, indices and FX, plus full historical "
62+ "options chains (quotes, implied volatility, Greeks) since 2010. "
63+ f"Built by {__author__} ({__contact__})."
64+ ),
65+ version=__version__,
66+ contact={"name": __author__, "email": __contact__,
67+ "url": "https://www.hfmarketdata.io"},
68+)
69+app.add_middleware(
70+ CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"],
71+)
72+
73+# ------------------------------------------------------------------------------
74+# DuckDB — one connection per thread, read-only usage
75+# ------------------------------------------------------------------------------
76+
77+_tls = threading.local()
78+
79+
80+def db() -> duckdb.DuckDBPyConnection:
81+ if not hasattr(_tls, "con"):
82+ _tls.con = duckdb.connect()
83+ _tls.con.execute("SET threads TO 4")
84+ _tls.con.execute("SET enable_object_cache=true")
85+ return _tls.con
86+
87+
88+# ------------------------------------------------------------------------------
89+# File resolution with a small TTL cache (directory scans are the slow part)
90+# ------------------------------------------------------------------------------
91+
92+_cache: dict[str, tuple[float, object]] = {}
93+_cache_lock = threading.Lock()
94+CACHE_TTL = 300 # seconds
95+
96+
97+def cached(key: str, builder):
98+ now = time.time()
99+ with _cache_lock:
100+ hit = _cache.get(key)
101+ if hit and now - hit[0] < CACHE_TTL:
102+ return hit[1]
103+ value = builder()
104+ with _cache_lock:
105+ _cache[key] = (now, value)
106+ return value
107+
108+
109+def bar_dir(asset: str, timeframe: str, adjustment: str) -> Path:
110+ return PARQUET / asset / timeframe / adjustment
111+
112+
113+def list_adjustments(asset: str, timeframe: str) -> list[str]:
114+ base = PARQUET / asset / timeframe
115+ if not base.is_dir():
116+ return []
117+ return sorted(p.name for p in base.iterdir() if p.is_dir())
118+
119+
120+def ticker_index(asset: str, timeframe: str, adjustment: str) -> dict[str, str]:
121+ """ticker -> parquet file path, cached."""
122+ d = bar_dir(asset, timeframe, adjustment)
123+
124+ def build():
125+ idx: dict[str, str] = {}
126+ if d.is_dir():
127+ for p in d.iterdir():
128+ if p.suffix == ".parquet":
129+ idx[p.stem.split("_")[0].upper()] = str(p)
130+ return idx
131+
132+ return cached(f"idx|{d}", build)
133+
134+
135+def options_quarters() -> list[str]:
136+ def build():
137+ base = PARQUET / "options"
138+ if not base.is_dir():
139+ return []
140+ return sorted(p.name for p in base.iterdir() if p.is_dir())
141+ return cached("opt_quarters", build)
142+
143+
144+def options_files(ticker: str) -> list[str]:
145+ """All quarterly chain files for a ticker, cheap direct-path checks."""
146+ t = ticker.upper()
147+
148+ def build():
149+ out = []
150+ for q in options_quarters():
151+ d = PARQUET / "options" / q
152+ for p in d.glob(f"{t}_*.parquet"):
153+ if p.stem.split("_")[0].upper() == t:
154+ out.append(str(p))
155+ return out
156+
157+ return cached(f"optfiles|{t}", build)
158+
159+
160+def _validate(asset: str, timeframe: str, adjustment: str | None) -> str:
161+ if asset not in BAR_TYPES:
162+ raise HTTPException(404, f"Unknown asset type '{asset}'. "
163+ f"Available: {', '.join(BAR_TYPES)}")
164+ if timeframe not in TIMEFRAMES:
165+ raise HTTPException(400, f"Unknown timeframe '{timeframe}'. "
166+ f"Available: {', '.join(TIMEFRAMES)}")
167+ adj = adjustment or DEFAULT_ADJUSTMENT[asset]
168+ available = list_adjustments(asset, timeframe)
169+ if available and adj not in available:
170+ raise HTTPException(400, f"Adjustment '{adj}' not available for "
171+ f"{asset}/{timeframe}. Available: {', '.join(available)}")
172+ return adj
173+
174+
175+def _rows_to_response(rel, limit: int, fmt: str, order_col: str):
176+ if fmt == "csv":
177+ limit = min(limit, MAX_LIMIT_CSV)
178+ rel = rel.limit(limit)
179+ df = rel.df()
180+ return PlainTextResponse(df.to_csv(index=False), media_type="text/csv")
181+ limit = min(limit, MAX_LIMIT_JSON)
182+ rel = rel.limit(limit)
183+ df = rel.df()
184+ for col in df.columns:
185+ if str(df[col].dtype).startswith("datetime"):
186+ df[col] = df[col].astype(str)
187+ return {"count": len(df), "data": df.to_dict(orient="records")}
188+
189+
190+# ------------------------------------------------------------------------------
191+# Meta endpoints
192+# ------------------------------------------------------------------------------
193+
194+
195+@app.get("/health", tags=["meta"])
196+def health():
197+ return {"status": "ok", "service": "hfmarketdata-api", "version": __version__,
198+ "data_root_present": PARQUET.is_dir()}
199+
200+
201+@app.get("/v1/status", tags=["meta"])
202+def status():
203+ """Dataset inventory: every asset type, timeframe and adjustment with
204+ the number of instruments currently available."""
205+ def build():
206+ out = {}
207+ for asset in BAR_TYPES:
208+ base = PARQUET / asset
209+ if not base.is_dir():
210+ continue
211+ tfs = {}
212+ for tf in sorted(p.name for p in base.iterdir() if p.is_dir()):
213+ adjs = {}
214+ for adj in list_adjustments(asset, tf):
215+ n = sum(1 for p in (base / tf / adj).iterdir()
216+ if p.suffix == ".parquet")
217+ adjs[adj] = n
218+ tfs[tf] = adjs
219+ out[asset] = tfs
220+ quarters = options_quarters()
221+ if quarters:
222+ out["options"] = {
223+ "quarters": quarters,
224+ "files_latest_quarter":
225+ sum(1 for _ in (PARQUET / "options" / quarters[-1]).glob("*.parquet")),
226+ }
227+ return out
228+ return {"datasets": cached("status", build)}
229+
230+
231+@app.get("/v1/{asset}/tickers", tags=["bars"])
232+def tickers(asset: str,
233+ timeframe: str = Query("1day"),
234+ adjustment: str | None = Query(None),
235+ search: str | None = Query(None, description="Substring filter"),
236+ limit: int = Query(10_000, ge=1)):
237+ """List available tickers for an asset type / timeframe / adjustment.
238+ For `options`, this delegates to the options underlyings listing."""
239+ if asset == "options":
240+ return opt_tickers(quarter=None, search=search, limit=limit)
241+ adj = _validate(asset, timeframe, adjustment)
242+ idx = ticker_index(asset, timeframe, adj)
243+ names = sorted(idx)
244+ if search:
245+ s = search.upper()
246+ names = [t for t in names if s in t]
247+ return {"asset": asset, "timeframe": timeframe, "adjustment": adj,
248+ "count": len(names[:limit]), "tickers": names[:limit]}
249+
250+
251+# ------------------------------------------------------------------------------
252+# Bars
253+# ------------------------------------------------------------------------------
254+
255+
256+@app.get("/v1/bars/{asset}/{ticker}", tags=["bars"])
257+def bars(asset: str, ticker: str,
258+ timeframe: str = Query("1day", description="1min|5min|30min|1hour|1day"),
259+ adjustment: str | None = Query(None, description="Default depends on asset type"),
260+ start: str | None = Query(None, description="ISO date/datetime lower bound"),
261+ end: str | None = Query(None, description="ISO date/datetime upper bound"),
262+ order: str = Query("asc", pattern="^(asc|desc)$"),
263+ limit: int = Query(5_000, ge=1),
264+ format: str = Query("json", pattern="^(json|csv)$")):
265+ """OHLCV bars for one instrument. Daily futures bars also carry open
266+ interest. `format=csv` allows bulk extraction (up to 2M rows)."""
267+ adj = _validate(asset, timeframe, adjustment)
268+ idx = ticker_index(asset, timeframe, adj)
269+ path = idx.get(ticker.upper())
270+ if not path:
271+ raise HTTPException(404, f"Ticker '{ticker.upper()}' not found in "
272+ f"{asset}/{timeframe}/{adj}")
273+ con = db()
274+ conds, params = [], []
275+ if start:
276+ conds.append("datetime >= ?"); params.append(start)
277+ if end:
278+ conds.append("datetime <= ?"); params.append(end)
279+ where = ("WHERE " + " AND ".join(conds)) if conds else ""
280+ rel = con.sql(
281+ f"SELECT * FROM read_parquet(?) {where} ORDER BY datetime {order.upper()}",
282+ params=[path, *params])
283+ return _rows_to_response(rel, limit, format, "datetime")
284+
285+
286+@app.get("/v1/bars/{asset}", tags=["bars"])
287+def bars_multi(asset: str,
288+ tickers: str = Query(..., description="Comma-separated list, e.g. AAPL,MSFT,TSLA (max 50)"),
289+ timeframe: str = Query("1day"),
290+ adjustment: str | None = Query(None),
291+ start: str | None = Query(None, description="ISO date/datetime lower bound"),
292+ end: str | None = Query(None, description="ISO date/datetime upper bound"),
293+ order: str = Query("asc", pattern="^(asc|desc)$"),
294+ limit: int = Query(5_000, ge=1, description="Max rows PER TICKER"),
295+ format: str = Query("json", pattern="^(json|csv)$")):
296+ """Bars for SEVERAL instruments in one call. Combine with `start`/`end`
297+ to slice any precise window — e.g. 4 hours of 1-minute bars across a
298+ whole watchlist. The row limit applies per ticker."""
299+ adj = _validate(asset, timeframe, adjustment)
300+ idx = ticker_index(asset, timeframe, adj)
301+ names = [t.strip().upper() for t in tickers.split(",") if t.strip()][:50]
302+ if not names:
303+ raise HTTPException(400, "No tickers given")
304+ missing = [t for t in names if t not in idx]
305+ if missing:
306+ raise HTTPException(404, f"Not found in {asset}/{timeframe}/{adj}: "
307+ f"{', '.join(missing)}")
308+ paths = [idx[t] for t in names]
309+ con = db()
310+ conds, params = [], []
311+ if start:
312+ conds.append("datetime >= ?"); params.append(start)
313+ if end:
314+ conds.append("datetime <= ?"); params.append(end)
315+ where = ("WHERE " + " AND ".join(conds)) if conds else ""
316+ per_ticker = min(limit, MAX_LIMIT_CSV if format == "csv" else MAX_LIMIT_JSON)
317+ rel = con.sql(
318+ f"SELECT * FROM read_parquet(?) {where} "
319+ f"QUALIFY row_number() OVER (PARTITION BY ticker ORDER BY datetime "
320+ f"{order.upper()}) <= ? ORDER BY ticker, datetime {order.upper()}",
321+ params=[paths, *params, per_ticker])
322+ return _rows_to_response(rel, per_ticker * len(names), format, "datetime")
323+
324+
325+@app.get("/v1/snapshot/{asset}", tags=["bars"])
326+def snapshot(asset: str,
327+ tickers: str = Query(..., description="Comma-separated list, e.g. AAPL,MSFT,TSLA (max 50)"),
328+ at: str = Query(..., description="Precise moment, e.g. 2024-06-03 10:35:00"),
329+ timeframe: str = Query("1min"),
330+ adjustment: str | None = Query(None),
331+ format: str = Query("json", pattern="^(json|csv)$")):
332+ """Cross-sectional snapshot: for each requested instrument, the last bar
333+ at or before the given moment — the state of a whole watchlist at one
334+ precise point in time."""
335+ adj = _validate(asset, timeframe, adjustment)
336+ idx = ticker_index(asset, timeframe, adj)
337+ names = [t.strip().upper() for t in tickers.split(",") if t.strip()][:50]
338+ if not names:
339+ raise HTTPException(400, "No tickers given")
340+ missing = [t for t in names if t not in idx]
341+ if missing:
342+ raise HTTPException(404, f"Not found in {asset}/{timeframe}/{adj}: "
343+ f"{', '.join(missing)}")
344+ paths = [idx[t] for t in names]
345+ con = db()
346+ rel = con.sql(
347+ "SELECT * FROM read_parquet(?) WHERE datetime <= ? "
348+ "QUALIFY row_number() OVER (PARTITION BY ticker ORDER BY datetime DESC) = 1 "
349+ "ORDER BY ticker",
350+ params=[paths, at])
351+ return _rows_to_response(rel, len(names), format, "datetime")
352+
353+
354+# ------------------------------------------------------------------------------
355+# Options
356+# ------------------------------------------------------------------------------
357+
358+
359+@app.get("/v1/options/quarters", tags=["options"])
360+def opt_quarters():
361+ """Available options archive quarters (e.g. 2010_q1 … current)."""
362+ return {"quarters": options_quarters()}
363+
364+
365+@app.get("/v1/options/tickers", tags=["options"])
366+def opt_tickers(quarter: str | None = Query(None, description="e.g. 2024_q4; default latest"),
367+ search: str | None = None,
368+ limit: int = Query(10_000, ge=1)):
369+ """Underlyings available in a given options quarter."""
370+ quarters = options_quarters()
371+ if not quarters:
372+ raise HTTPException(503, "Options dataset not yet available")
373+ q = quarter or quarters[-1]
374+ if q not in quarters:
375+ raise HTTPException(404, f"Unknown quarter '{q}'")
376+
377+ def build():
378+ return sorted({p.stem.split("_")[0].upper()
379+ for p in (PARQUET / "options" / q).glob("*.parquet")})
380+ names = cached(f"opt_tickers|{q}", build)
381+ if search:
382+ s = search.upper()
383+ names = [t for t in names if s in t]
384+ return {"quarter": q, "count": len(names[:limit]), "tickers": names[:limit]}
385+
386+
387+@app.get("/v1/options/chain/{ticker}", tags=["options"])
388+def opt_chain(ticker: str,
389+ trade_date: str | None = Query(None, description="yyyy-mm-dd; default latest available"),
390+ expiry: str | None = Query(None, description="Filter on expiry date"),
391+ call_put: str | None = Query(None, pattern="^(c|p)$"),
392+ strike_min: float | None = None,
393+ strike_max: float | None = None,
394+ min_volume: float | None = None,
395+ limit: int = Query(20_000, ge=1),
396+ format: str = Query("json", pattern="^(json|csv)$")):
397+ """Full end-of-day option chain for an underlying: quotes, bid/ask implied
398+ volatility, open interest, volume and Greeks (delta, gamma, vega, theta, rho)."""
399+ files = options_files(ticker)
400+ if not files:
401+ raise HTTPException(404, f"No options data for '{ticker.upper()}'")
402+ con = db()
403+ if trade_date is None:
404+ trade_date = con.execute(
405+ "SELECT max(trade_date) FROM read_parquet(?)", [files[-1]]
406+ ).fetchone()[0]
407+ files_q = [files[-1]]
408+ else:
409+ files_q = files
410+ conds, params = ["trade_date = ?"], [str(trade_date)]
411+ if expiry:
412+ conds.append("expiry = ?"); params.append(expiry)
413+ if call_put:
414+ conds.append("call_put = ?"); params.append(call_put)
415+ if strike_min is not None:
416+ conds.append("strike >= ?"); params.append(strike_min)
417+ if strike_max is not None:
418+ conds.append("strike <= ?"); params.append(strike_max)
419+ if min_volume is not None:
420+ conds.append("volume >= ?"); params.append(min_volume)
421+ rel = con.sql(
422+ f"SELECT * FROM read_parquet(?) WHERE {' AND '.join(conds)} "
423+ f"ORDER BY expiry, strike, call_put",
424+ params=[files_q, *params])
425+ return _rows_to_response(rel, limit, format, "expiry")
426+
427+
428+@app.get("/v1/options/expirations/{ticker}", tags=["options"])
429+def opt_expirations(ticker: str,
430+ trade_date: str | None = Query(None, description="yyyy-mm-dd")):
431+ """Available expiry dates (optionally as of one trade date)."""
432+ files = options_files(ticker)
433+ if not files:
434+ raise HTTPException(404, f"No options data for '{ticker.upper()}'")
435+ con = db()
436+ if trade_date:
437+ rows = con.execute(
438+ "SELECT DISTINCT expiry FROM read_parquet(?) WHERE trade_date = ? ORDER BY expiry",
439+ [files, trade_date]).fetchall()
440+ else:
441+ rows = con.execute(
442+ "SELECT DISTINCT expiry FROM read_parquet(?) ORDER BY expiry",
443+ [files[-1]]).fetchall()
444+ return {"ticker": ticker.upper(), "expirations": [str(r[0]) for r in rows]}
445+
446+
447+@app.get("/v1/options/history/{ticker}", tags=["options"])
448+def opt_history(ticker: str,
449+ strike: float = Query(...),
450+ expiry: str = Query(..., description="yyyy-mm-dd"),
451+ call_put: str = Query(..., pattern="^(c|p)$"),
452+ limit: int = Query(5_000, ge=1),
453+ format: str = Query("json", pattern="^(json|csv)$")):
454+ """Daily time series for one specific contract across its whole life."""
455+ files = options_files(ticker)
456+ if not files:
457+ raise HTTPException(404, f"No options data for '{ticker.upper()}'")
458+ con = db()
459+ rel = con.sql(
460+ "SELECT * FROM read_parquet(?) WHERE strike = ? AND expiry = ? AND call_put = ? "
461+ "ORDER BY trade_date",
462+ params=[files, strike, expiry, call_put])
463+ return _rows_to_response(rel, limit, format, "trade_date")
464+
465+
466+# ------------------------------------------------------------------------------
467+# Static React platform (mounted last so it doesn't shadow the API)
468+# ------------------------------------------------------------------------------
469+
470+if WEB_DIST.is_dir():
471+ app.mount("/", StaticFiles(directory=str(WEB_DIST), html=True), name="web")
472+
473+
474+if __name__ == "__main__":
475+ import uvicorn
476+ uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("HFMD_PORT", 8090)))
added hfmarketdata/requirements.txt +4 −0
@@ -0,0 +1,4 @@
1+fastapi>=0.115
2+uvicorn>=0.30
3+duckdb>=1.0
4+pandas>=2.0
added hfmarketdata/web/index.html +16 −0
@@ -0,0 +1,16 @@
1+<!doctype html>
2+<html lang="en">
3+ <head>
4+ <meta charset="UTF-8" />
5+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
6+ <meta name="description" content="HF Market Data — free open API for high-frequency historical market data: stocks, ETFs, futures, crypto, indices, FX (1-minute to daily) and full options chains with Greeks since 2010." />
7+ <title>HF Market Data — Open High-Frequency Market Data API</title>
8+ <link rel="preconnect" href="https://fonts.googleapis.com" />
9+ <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
10+ <link href="https://fonts.googleapis.com/css2?family=Fraunces:ital,opsz,wght@0,9..144,400;0,9..144,500;0,9..144,600;1,9..144,400;1,9..144,500&family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;600&display=swap" rel="stylesheet" />
11+ </head>
12+ <body>
13+ <div id="root"></div>
14+ <script type="module" src="/src/main.jsx"></script>
15+ </body>
16+</html>
added hfmarketdata/web/package-lock.json +1857 −0
Large diff (1,857 lines) is not loaded by default to keep the page fast.Load diff
added hfmarketdata/web/package.json +21 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added hfmarketdata/web/src/App.jsx +403 −0
Large diff (403 lines) is not loaded by default to keep the page fast.Load diff
added hfmarketdata/web/src/main.jsx +10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added hfmarketdata/web/src/spec.js +329 −0
Large diff (329 lines) is not loaded by default to keep the page fast.Load diff
added hfmarketdata/web/src/styles.css +457 −0
Large diff (457 lines) is not loaded by default to keep the page fast.Load diff
added hfmarketdata/web/vite.config.js +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added requirements.txt +2 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere