Name	Last commit	Updated
docs	ratelimit: principal `site` — les requêtes de données du site...	18 days ago
hfmarketdata	web: charts — catalogue moteur enregistré au chargement du module...	18 days ago
mcp	mcp: formulation « plateforme 100 % gratuite » (MCP, skills, page...	20 days ago
scripts	fundamentals: script d'attente pour lancer migrate_coverage après le...	18 days ago
skills	mcp: formulation « plateforme 100 % gratuite » (MCP, skills, page...	20 days ago
tests	ratelimit: principal `site` — les requêtes de données du site...	18 days ago
.gitignore	web: pas de tarification — page Limits (/limits, /pricing redirige),...	20 days ago
CLAUDE.md	docs: M3U96b copy deleted 2026-09-14, BHS128 holds the only lake copy	11 days ago
frd_downloader.py	frd: row groups de 65 536 lignes pour l'intraday + script de...	18 days ago
pytest.ini	futures: lac synthétique enrichi (cycle de vie volume/OI, ETH, racine...	20 days ago
README.md	chore(retrait cluster): synchro copie du nœud M4M64a + README « URL...	20 days ago
requirements-dev.txt	accounts: codes d'erreur comptes/quotas (EMAIL_TAKEN, INVALID_TOKEN,...	20 days ago
requirements.txt	HF Market Data — FirstRate full-history downloader, DuckDB/Parquet...	1 mo ago

⚠️ URL live hors fonction pour le moment. L'application a été retirée du cluster MacLustr le 2026-09-04 (processus arrêtés, copie du nœud supprimée). Ce dépôt spbgit est désormais la seule source de vérité du projet. Ancienne URL : https://www.hfmarketdata.io

# HF Market Data

Open high-frequency market data platform — a full-history downloader, a DuckDB-over-Parquet data lake, an open REST API and a documentation platform, live at www.hfmarketdata.io.

# Dataset metrics

Full ingestion completed 2026-08-10 — 745/745 jobs, zero data loss:

Metric	Value
Rows in the lake	26,464,775,098
Parquet files (one per instrument/segment)	558,814
Lake size (zstd Parquet)	350 GB (raw archives deleted on the fly)
Options quarters	66 / 66 (2010_q1 → today)
Options underlyings	5,800+ US equities & indices
Stocks · ETFs at 1-minute	7,670 · 5,161 instruments
Bar timeframes	1min · 5min · 30min · 1hour · 1day
Asset classes	stocks · ETFs · futures (continuous + contracts) · crypto · indices · FX · options

Query the live inventory anytime: GET /v1/status.

# Architecture

text

FirstRate Data API ──▶ frd_downloader.py ──▶ Parquet lake (zstd, one file per instrument)
   (zip archives)        · 745-job plan          /parquet/{type}/{timeframe}/{adjustment}/{TICKER}.parquet
                         · resume manifest       /parquet/options/{year}_{quarter}/…
                         · CSV→Parquet via
                           DuckDB, raw deleted
                                                        │ queried in place (predicate pushdown)
                                                        ▼
                                        hfmarketdata/api/main.py  (FastAPI + DuckDB)
                                                        │  :8090
                                        hfmarketdata/web/          (React + Vite docs platform)
                                                        │
                                          https://www.hfmarketdata.io  (ngrok)

# Repository layout

text

frd_downloader.py        Full-history downloader & Parquet converter (single file)
requirements.txt         Downloader dependencies (requests, duckdb)
hfmarketdata/
  api/main.py            Open REST API — FastAPI + DuckDB over the Parquet lake
  requirements.txt       API dependencies
  web/                   Documentation platform (React 18 + Vite, English)

Heavy data (Parquet lake, zip archives, venvs, builds) is not in this repository — see .gitignore.

# The downloader

Downloads everything FirstRate Data publishes and stores it query-ready:

Options — every quarterly archive since 2010 + the current partial quarter (16 fields per row: quotes, bid/ask IV, open interest, volume, delta, gamma, vega, theta, rho)
Stocks & ETFs — full history, tickers A–Z, all 5 timeframes × adj_split / adj_splitdiv / UNADJUSTED
Futures — continuous series (contin_UNadj / contin_adj_ratio / contin_adj_absolute) + individual contracts
Crypto, indices, FX — full history, all timeframes
Metadata — ticker listings, splits, dividends, update logs, continuous-series audit

bash

export FRD_USERID=<your FirstRate customer id>
python3 -m venv venv && venv/bin/pip install -r requirements.txt
venv/bin/python frd_downloader.py --data-root /Volumes/ssd/firstratedata

Fully resumable (state/manifest.json), disk-space guard, retries with backoff, and raw zips/CSVs are deleted as soon as each segment is converted. Notable quirks handled: FirstRate files mix LF/CRLF line endings (strict_mode=false), daily files use yyyyMMdd dates, the API 302-redirects to a CDN.

# The API

Open, keyless, JSON + CSV. Highlights:

Endpoint	What it does
`GET /v1/status`	Live inventory of the lake
`GET /v1/{asset}/tickers`	Instrument discovery per type/timeframe/adjustment
`GET /v1/bars/{asset}/{ticker}`	OHLCV bars, precise `start`/`end` windows, up to 2M rows in CSV
`GET /v1/bars/{asset}?tickers=A,B,C`	Multi-ticker bars in one call (per-ticker limit)
`GET /v1/snapshot/{asset}?tickers=…&at=…`	Whole watchlist at one precise moment
`GET /v1/options/chain/{ticker}`	Full EOD chain — quotes, IV, Greeks
`GET /v1/options/history/{ticker}`	One contract's daily life story
`GET /docs`	OpenAPI / Swagger

bash

# 4 hours of 1-minute bars across a watchlist
curl "https://www.hfmarketdata.io/v1/bars/stock?tickers=AAPL,MSFT,NVDA&timeframe=1min&start=2024-06-03%2009:30:00&end=2024-06-03%2013:30:00"

# The platform

React 18 + Vite documentation site (English) served by the API process — refined editorial light theme (Fraunces / Inter / JetBrains Mono), full endpoint reference with parameter tables, request snippets (curl / Python / JavaScript), example + live responses, response-field dictionaries and a live dataset status page.

bash

cd hfmarketdata/web && npm install && npm run build   # → dist/, served at /

# Author

Simon-Pierre Boucher — contact@spboucher.ai