spb/hfmarketdata Public

1# HF Market Data23**Open high-frequency market data platform** — a full-history downloader, a DuckDB-over-Parquet data lake, an open REST API and a documentation platform, live at **[www.hfmarketdata.io](https://www.hfmarketdata.io)**.45[![Live](https://img.shields.io/badge/live-www.hfmarketdata.io-2e5ce6)](https://www.hfmarketdata.io)6[![API](https://img.shields.io/badge/API-open%20%C2%B7%20no%20key-2e5ce6)](https://www.hfmarketdata.io/docs)7[![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?logo=python&logoColor=white)](#)8[![FastAPI](https://img.shields.io/badge/FastAPI-0.115%2B-009688?logo=fastapi&logoColor=white)](#)9[![DuckDB](https://img.shields.io/badge/DuckDB-1.x-FFF000?logo=duckdb&logoColor=black)](#)10[![React](https://img.shields.io/badge/React-18-61DAFB?logo=react&logoColor=black)](#)11[![Vite](https://img.shields.io/badge/Vite-6-646CFF?logo=vite&logoColor=white)](#)12[![Parquet](https://img.shields.io/badge/storage-Parquet%20%C2%B7%20zstd-50ABF1)](#)1314[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-1c2430)](mailto:contact@spboucher.ai)15[![Contact](https://img.shields.io/badge/contact-contact%40spboucher.ai-1c2430)](mailto:contact@spboucher.ai)1617---1819## Dataset metrics2021Full ingestion completed 2026-08-10 — **745/745 jobs, zero data loss**:2223| Metric | Value |24|---|---:|25| Rows in the lake | **26,464,775,098** |26| Parquet files (one per instrument/segment) | **558,814** |27| Lake size (zstd Parquet) | **350 GB** (raw archives deleted on the fly) |28| Options quarters | **66 / 66** (2010_q1 → today) |29| Options underlyings | **5,800+** US equities & indices |30| Stocks · ETFs at 1-minute | 7,670 · 5,161 instruments |31| Bar timeframes | 1min · 5min · 30min · 1hour · 1day |32| Asset classes | stocks · ETFs · futures (continuous + contracts) · crypto · indices · FX · options |3334Query the live inventory anytime: [`GET /v1/status`](https://www.hfmarketdata.io/v1/status).3536## Architecture3738```39FirstRate Data API ──▶ frd_downloader.py ──▶ Parquet lake (zstd, one file per instrument)40   (zip archives)        · 745-job plan          /parquet/{type}/{timeframe}/{adjustment}/{TICKER}.parquet41                         · resume manifest       /parquet/options/{year}_{quarter}/…42                         · CSV→Parquet via43                           DuckDB, raw deleted44                                                        │ queried in place (predicate pushdown)45                                                        ▼46                                        hfmarketdata/api/main.py  (FastAPI + DuckDB)47                                                        │  :809048                                        hfmarketdata/web/          (React + Vite docs platform)49                                                        │50                                          https://www.hfmarketdata.io  (ngrok)51```5253## Repository layout5455```56frd_downloader.py        Full-history downloader & Parquet converter (single file)57requirements.txt         Downloader dependencies (requests, duckdb)58hfmarketdata/59  api/main.py            Open REST API — FastAPI + DuckDB over the Parquet lake60  requirements.txt       API dependencies61  web/                   Documentation platform (React 18 + Vite, English)62```6364Heavy data (Parquet lake, zip archives, venvs, builds) is **not** in this repository — see `.gitignore`.6566## The downloader6768Downloads **everything** FirstRate Data publishes and stores it query-ready:6970- **Options** — every quarterly archive since 2010 + the current partial quarter (16 fields per row: quotes, bid/ask IV, open interest, volume, delta, gamma, vega, theta, rho)71- **Stocks & ETFs** — full history, tickers A–Z, all 5 timeframes × `adj_split` / `adj_splitdiv` / `UNADJUSTED`72- **Futures** — continuous series (`contin_UNadj` / `contin_adj_ratio` / `contin_adj_absolute`) + individual contracts73- **Crypto, indices, FX** — full history, all timeframes74- **Metadata** — ticker listings, splits, dividends, update logs, continuous-series audit7576```bash77export FRD_USERID=<your FirstRate customer id>78python3 -m venv venv && venv/bin/pip install -r requirements.txt79venv/bin/python frd_downloader.py --data-root /Volumes/ssd/firstratedata80```8182Fully resumable (`state/manifest.json`), disk-space guard, retries with backoff, and raw zips/CSVs are deleted as soon as each segment is converted. Notable quirks handled: FirstRate files mix LF/CRLF line endings (`strict_mode=false`), daily files use `yyyyMMdd` dates, the API 302-redirects to a CDN.8384## The API8586Open, keyless, JSON + CSV. Highlights:8788| Endpoint | What it does |89|---|---|90| `GET /v1/status` | Live inventory of the lake |91| `GET /v1/{asset}/tickers` | Instrument discovery per type/timeframe/adjustment |92| `GET /v1/bars/{asset}/{ticker}` | OHLCV bars, precise `start`/`end` windows, up to 2M rows in CSV |93| `GET /v1/bars/{asset}?tickers=A,B,C` | Multi-ticker bars in one call (per-ticker limit) |94| `GET /v1/snapshot/{asset}?tickers=…&at=…` | Whole watchlist at one precise moment |95| `GET /v1/options/chain/{ticker}` | Full EOD chain — quotes, IV, Greeks |96| `GET /v1/options/history/{ticker}` | One contract's daily life story |97| `GET /docs` | OpenAPI / Swagger |9899```bash100# 4 hours of 1-minute bars across a watchlist101curl "https://www.hfmarketdata.io/v1/bars/stock?tickers=AAPL,MSFT,NVDA&timeframe=1min&start=2024-06-03%2009:30:00&end=2024-06-03%2013:30:00"102```103104## The platform105106React 18 + Vite documentation site (English) served by the API process — refined editorial light theme (Fraunces / Inter / JetBrains Mono), full endpoint reference with parameter tables, request snippets (curl / Python / JavaScript), example + **live** responses, response-field dictionaries and a live dataset status page.107108```bash109cd hfmarketdata/web && npm install && npm run build   # → dist/, served at /110```111112## Author113114**Simon-Pierre Boucher** — [contact@spboucher.ai](mailto:contact@spboucher.ai)115