SPB Git

spb/hfmarketdata Public

Open high-frequency market data platform — FirstRate full-history downloader, DuckDB/Parquet lake, open REST API and React docs platform (www.hfmarketdata.io)

Python 46.7% JavaScript 37.4% CSS 14.9% HTML 0.9%
ZIP
NameLast commitUpdated
hfmarketdata fix(api): NaN/±inf dans les Greeks sérialisés en null (500 sur... 6 h ago
.gitignore HF Market Data — FirstRate full-history downloader, DuckDB/Parquet... 20 h ago
frd_downloader.py Handle FX intraday split date+time columns; 7zz fallback for Deflate64 8 h ago
README.md README: final dataset metrics — 26.5B rows, 558,814 files, 350 GB,... 8 h ago
requirements.txt HF Market Data — FirstRate full-history downloader, DuckDB/Parquet... 20 h ago
README.md

# HF Market Data

Open high-frequency market data platform — a full-history downloader, a DuckDB-over-Parquet data lake, an open REST API and a documentation platform, live at www.hfmarketdata.io.

Live API Python FastAPI DuckDB React Vite Parquet

Author Contact

# Dataset metrics

Full ingestion completed 2026-08-10 — 745/745 jobs, zero data loss:

Metric Value
Rows in the lake 26,464,775,098
Parquet files (one per instrument/segment) 558,814
Lake size (zstd Parquet) 350 GB (raw archives deleted on the fly)
Options quarters 66 / 66 (2010_q1 → today)
Options underlyings 5,800+ US equities & indices
Stocks · ETFs at 1-minute 7,670 · 5,161 instruments
Bar timeframes 1min · 5min · 30min · 1hour · 1day
Asset classes stocks · ETFs · futures (continuous + contracts) · crypto · indices · FX · options

Query the live inventory anytime: GET /v1/status.

# Architecture

text
FirstRate Data API ──▶ frd_downloader.py ──▶ Parquet lake (zstd, one file per instrument)
   (zip archives)        · 745-job plan          /parquet/{type}/{timeframe}/{adjustment}/{TICKER}.parquet
                         · resume manifest       /parquet/options/{year}_{quarter}/…
                         · CSV→Parquet via
                           DuckDB, raw deleted
                                                        │ queried in place (predicate pushdown)
                                                        ▼
                                        hfmarketdata/api/main.py  (FastAPI + DuckDB)
                                                        │  :8090
                                        hfmarketdata/web/          (React + Vite docs platform)
                                                        │
                                          https://www.hfmarketdata.io  (ngrok)

# Repository layout

text
frd_downloader.py        Full-history downloader & Parquet converter (single file)
requirements.txt         Downloader dependencies (requests, duckdb)
hfmarketdata/
  api/main.py            Open REST API — FastAPI + DuckDB over the Parquet lake
  requirements.txt       API dependencies
  web/                   Documentation platform (React 18 + Vite, English)

Heavy data (Parquet lake, zip archives, venvs, builds) is not in this repository — see .gitignore.

# The downloader

Downloads everything FirstRate Data publishes and stores it query-ready:

  • Options — every quarterly archive since 2010 + the current partial quarter (16 fields per row: quotes, bid/ask IV, open interest, volume, delta, gamma, vega, theta, rho)
  • Stocks & ETFs — full history, tickers A–Z, all 5 timeframes × adj_split / adj_splitdiv / UNADJUSTED
  • Futures — continuous series (contin_UNadj / contin_adj_ratio / contin_adj_absolute) + individual contracts
  • Crypto, indices, FX — full history, all timeframes
  • Metadata — ticker listings, splits, dividends, update logs, continuous-series audit
bash
export FRD_USERID=<your FirstRate customer id>
python3 -m venv venv && venv/bin/pip install -r requirements.txt
venv/bin/python frd_downloader.py --data-root /Volumes/ssd/firstratedata

Fully resumable (state/manifest.json), disk-space guard, retries with backoff, and raw zips/CSVs are deleted as soon as each segment is converted. Notable quirks handled: FirstRate files mix LF/CRLF line endings (strict_mode=false), daily files use yyyyMMdd dates, the API 302-redirects to a CDN.

# The API

Open, keyless, JSON + CSV. Highlights:

Endpoint What it does
GET /v1/status Live inventory of the lake
GET /v1/{asset}/tickers Instrument discovery per type/timeframe/adjustment
GET /v1/bars/{asset}/{ticker} OHLCV bars, precise start/end windows, up to 2M rows in CSV
GET /v1/bars/{asset}?tickers=A,B,C Multi-ticker bars in one call (per-ticker limit)
GET /v1/snapshot/{asset}?tickers=…&at=… Whole watchlist at one precise moment
GET /v1/options/chain/{ticker} Full EOD chain — quotes, IV, Greeks
GET /v1/options/history/{ticker} One contract's daily life story
GET /docs OpenAPI / Swagger
bash
# 4 hours of 1-minute bars across a watchlist
curl "https://www.hfmarketdata.io/v1/bars/stock?tickers=AAPL,MSFT,NVDA&timeframe=1min&start=2024-06-03%2009:30:00&end=2024-06-03%2013:30:00"

# The platform

React 18 + Vite documentation site (English) served by the API process — refined editorial light theme (Fraunces / Inter / JetBrains Mono), full endpoint reference with parameter tables, request snippets (curl / Python / JavaScript), example + live responses, response-field dictionaries and a live dataset status page.

bash
cd hfmarketdata/web && npm install && npm run build   # → dist/, served at /

# Author

Simon-Pierre Boucher — contact@spboucher.ai