# HF Market Data

**Open high-frequency market data platform** — a full-history downloader, a DuckDB-over-Parquet data lake, an open REST API and a documentation platform, live at **[www.hfmarketdata.io](https://www.hfmarketdata.io)**.

[![Live](https://img.shields.io/badge/live-www.hfmarketdata.io-2e5ce6)](https://www.hfmarketdata.io)
[![API](https://img.shields.io/badge/API-open%20%C2%B7%20no%20key-2e5ce6)](https://www.hfmarketdata.io/docs)
[![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?logo=python&logoColor=white)](#)
[![FastAPI](https://img.shields.io/badge/FastAPI-0.115%2B-009688?logo=fastapi&logoColor=white)](#)
[![DuckDB](https://img.shields.io/badge/DuckDB-1.x-FFF000?logo=duckdb&logoColor=black)](#)
[![React](https://img.shields.io/badge/React-18-61DAFB?logo=react&logoColor=black)](#)
[![Vite](https://img.shields.io/badge/Vite-6-646CFF?logo=vite&logoColor=white)](#)
[![Parquet](https://img.shields.io/badge/storage-Parquet%20%C2%B7%20zstd-50ABF1)](#)

[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-1c2430)](mailto:contact@spboucher.ai)
[![Contact](https://img.shields.io/badge/contact-contact%40spboucher.ai-1c2430)](mailto:contact@spboucher.ai)

---

## Dataset metrics

Full ingestion completed 2026-08-10 — **745/745 jobs, zero data loss**:

| Metric | Value |
|---|---:|
| Rows in the lake | **26,464,775,098** |
| Parquet files (one per instrument/segment) | **558,814** |
| Lake size (zstd Parquet) | **350 GB** (raw archives deleted on the fly) |
| Options quarters | **66 / 66** (2010_q1 → today) |
| Options underlyings | **5,800+** US equities & indices |
| Stocks · ETFs at 1-minute | 7,670 · 5,161 instruments |
| Bar timeframes | 1min · 5min · 30min · 1hour · 1day |
| Asset classes | stocks · ETFs · futures (continuous + contracts) · crypto · indices · FX · options |

Query the live inventory anytime: [`GET /v1/status`](https://www.hfmarketdata.io/v1/status).

## Architecture

```
FirstRate Data API ──▶ frd_downloader.py ──▶ Parquet lake (zstd, one file per instrument)
   (zip archives)        · 745-job plan          /parquet/{type}/{timeframe}/{adjustment}/{TICKER}.parquet
                         · resume manifest       /parquet/options/{year}_{quarter}/…
                         · CSV→Parquet via
                           DuckDB, raw deleted
                                                        │ queried in place (predicate pushdown)
                                                        ▼
                                        hfmarketdata/api/main.py  (FastAPI + DuckDB)
                                                        │  :8090
                                        hfmarketdata/web/          (React + Vite docs platform)
                                                        │
                                          https://www.hfmarketdata.io  (ngrok)
```

## Repository layout

```
frd_downloader.py        Full-history downloader & Parquet converter (single file)
requirements.txt         Downloader dependencies (requests, duckdb)
hfmarketdata/
  api/main.py            Open REST API — FastAPI + DuckDB over the Parquet lake
  requirements.txt       API dependencies
  web/                   Documentation platform (React 18 + Vite, English)
```

Heavy data (Parquet lake, zip archives, venvs, builds) is **not** in this repository — see `.gitignore`.

## The downloader

Downloads **everything** FirstRate Data publishes and stores it query-ready:

- **Options** — every quarterly archive since 2010 + the current partial quarter (16 fields per row: quotes, bid/ask IV, open interest, volume, delta, gamma, vega, theta, rho)
- **Stocks & ETFs** — full history, tickers A–Z, all 5 timeframes × `adj_split` / `adj_splitdiv` / `UNADJUSTED`
- **Futures** — continuous series (`contin_UNadj` / `contin_adj_ratio` / `contin_adj_absolute`) + individual contracts
- **Crypto, indices, FX** — full history, all timeframes
- **Metadata** — ticker listings, splits, dividends, update logs, continuous-series audit

```bash
export FRD_USERID=<your FirstRate customer id>
python3 -m venv venv && venv/bin/pip install -r requirements.txt
venv/bin/python frd_downloader.py --data-root /Volumes/ssd/firstratedata
```

Fully resumable (`state/manifest.json`), disk-space guard, retries with backoff, and raw zips/CSVs are deleted as soon as each segment is converted. Notable quirks handled: FirstRate files mix LF/CRLF line endings (`strict_mode=false`), daily files use `yyyyMMdd` dates, the API 302-redirects to a CDN.

## The API

Open, keyless, JSON + CSV. Highlights:

| Endpoint | What it does |
|---|---|
| `GET /v1/status` | Live inventory of the lake |
| `GET /v1/{asset}/tickers` | Instrument discovery per type/timeframe/adjustment |
| `GET /v1/bars/{asset}/{ticker}` | OHLCV bars, precise `start`/`end` windows, up to 2M rows in CSV |
| `GET /v1/bars/{asset}?tickers=A,B,C` | Multi-ticker bars in one call (per-ticker limit) |
| `GET /v1/snapshot/{asset}?tickers=…&at=…` | Whole watchlist at one precise moment |
| `GET /v1/options/chain/{ticker}` | Full EOD chain — quotes, IV, Greeks |
| `GET /v1/options/history/{ticker}` | One contract's daily life story |
| `GET /docs` | OpenAPI / Swagger |

```bash
# 4 hours of 1-minute bars across a watchlist
curl "https://www.hfmarketdata.io/v1/bars/stock?tickers=AAPL,MSFT,NVDA&timeframe=1min&start=2024-06-03%2009:30:00&end=2024-06-03%2013:30:00"
```

## The platform

React 18 + Vite documentation site (English) served by the API process — refined editorial light theme (Fraunces / Inter / JetBrains Mono), full endpoint reference with parameter tables, request snippets (curl / Python / JavaScript), example + **live** responses, response-field dictionaries and a live dataset status page.

```bash
cd hfmarketdata/web && npm install && npm run build   # → dist/, served at /
```

## Author

**Simon-Pierre Boucher** — [contact@spboucher.ai](mailto:contact@spboucher.ai)