HF Market Data — FirstRate full-history downloader, DuckDB/Parquet lake, open API and docs platform · dd1eb9a · hfmarketdata

HF Market Data — FirstRate full-history downloader, DuckDB/Parquet lake, open API and docs platform

- frd_downloader.py: 745-job resumable plan (options 2010→today, stock/etf A-Z,
  futures continuous+contracts, crypto/index/fx, all timeframes & adjustments),
  CSV→Parquet(zstd) via DuckDB, raw archives deleted on the fly
- hfmarketdata/api: open FastAPI+DuckDB API (bars, multi-ticker, point-in-time
  snapshot, options chains/history, live status), JSON+CSV
- hfmarketdata/web: React+Vite documentation platform (editorial light theme)

Author: Simon-Pierre Boucher <contact@spboucher.ai>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

simon-pierre boucher committed 3 mo ago (Aug 10, 2026) Browse files

14 changed files +4,352 −0

added .gitignore +24 −0

		`@@ -0,0 +1,24 @@`
	1	+# --- heavy data: never in the repo ---
	2	+*.zip
	3	+*.parquet
	4	+*.csv
	5	+*.duckdb
	6	+parquet/
	7	+zips/
	8	+data/
	9	+state/
	10	+logs/
	11	+
	12	+# credentials / vendor docs containing the customer id
	13	+api.txt
	14	+
	15	+# environments & builds
	16	+venv/
	17	+__pycache__/
	18	+*.pyc
	19	+node_modules/
	20	+hfmarketdata/web/dist/
	21	+hfmarketdata/venv/
	22	+
	23	+# OS / editor
	24	+.DS_Store

added README.md +114 −0

		`@@ -0,0 +1,114 @@`
	1	+# HF Market Data
	2	+
	3	+Open high-frequency market data platform — a full-history downloader, a DuckDB-over-Parquet data lake, an open REST API and a documentation platform, live at [www.hfmarketdata.io](https://www.hfmarketdata.io).
	4	+
	5	+[![Live](https://img.shields.io/badge/live-www.hfmarketdata.io-2e5ce6)](https://www.hfmarketdata.io)
	6	+[![API](https://img.shields.io/badge/API-open%20%C2%B7%20no%20key-2e5ce6)](https://www.hfmarketdata.io/docs)
	7	+[![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?logo=python&logoColor=white)](#)
	8	+[![FastAPI](https://img.shields.io/badge/FastAPI-0.115%2B-009688?logo=fastapi&logoColor=white)](#)
	9	+[![DuckDB](https://img.shields.io/badge/DuckDB-1.x-FFF000?logo=duckdb&logoColor=black)](#)
	10	+[![React](https://img.shields.io/badge/React-18-61DAFB?logo=react&logoColor=black)](#)
	11	+[![Vite](https://img.shields.io/badge/Vite-6-646CFF?logo=vite&logoColor=white)](#)
	12	+[![Parquet](https://img.shields.io/badge/storage-Parquet%20%C2%B7%20zstd-50ABF1)](#)
	13	+
	14	+[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-1c2430)](mailto:contact@spboucher.ai)
	15	+[![Contact](https://img.shields.io/badge/contact-contact%40spboucher.ai-1c2430)](mailto:contact@spboucher.ai)
	16	+
	17	+---
	18	+
	19	+## Dataset metrics
	20	+
	21	+Live snapshot of the lake while ingestion is still running (2026-08-10):
	22	+
	23	+\| Metric \| Value \|
	24	+\|---\|---:\|
	25	+\| Rows converted to Parquet \| 1,785,837,666+ \|
	26	+\| Parquet files (one per instrument/segment) \| 211,140+ \|
	27	+\| Lake size (zstd Parquet) \| 34 GB+ (raw archives deleted on the fly) \|
	28	+\| Options quarters ingested \| 30+ / 66 (2010_q1 → today) \|
	29	+\| Options underlyings \| 5,800+ US equities & indices \|
	30	+\| Bar timeframes \| 1min · 5min · 30min · 1hour · 1day \|
	31	+\| Asset classes \| stocks · ETFs · futures (continuous + contracts) · crypto · indices · FX · options \|
	32	+\| Download plan \| 745 jobs, fully resumable, 0 conversion errors \|
	33	+
	34	+Query the live inventory anytime: [`GET /v1/status`](https://www.hfmarketdata.io/v1/status).
	35	+
	36	+## Architecture
	37	+
	38	+```
	39	+FirstRate Data API ──▶ frd_downloader.py ──▶ Parquet lake (zstd, one file per instrument)
	40	+ (zip archives) · 745-job plan /parquet/{type}/{timeframe}/{adjustment}/{TICKER}.parquet
	41	+ · resume manifest /parquet/options/{year}_{quarter}/…
	42	+ · CSV→Parquet via
	43	+ DuckDB, raw deleted
	44	+ │ queried in place (predicate pushdown)
	45	+ ▼
	46	+ hfmarketdata/api/main.py (FastAPI + DuckDB)
	47	+ │ :8090
	48	+ hfmarketdata/web/ (React + Vite docs platform)
	49	+ │
	50	+ https://www.hfmarketdata.io (ngrok)
	51	+```
	52	+
	53	+## Repository layout
	54	+
	55	+```
	56	+frd_downloader.py Full-history downloader & Parquet converter (single file)
	57	+requirements.txt Downloader dependencies (requests, duckdb)
	58	+hfmarketdata/
	59	+ api/main.py Open REST API — FastAPI + DuckDB over the Parquet lake
	60	+ requirements.txt API dependencies
	61	+ web/ Documentation platform (React 18 + Vite, English)
	62	+```
	63	+
	64	+Heavy data (Parquet lake, zip archives, venvs, builds) is not in this repository — see `.gitignore`.
	65	+
	66	+## The downloader
	67	+
	68	+Downloads everything FirstRate Data publishes and stores it query-ready:
	69	+
	70	+- Options — every quarterly archive since 2010 + the current partial quarter (16 fields per row: quotes, bid/ask IV, open interest, volume, delta, gamma, vega, theta, rho)
	71	+- Stocks & ETFs — full history, tickers A–Z, all 5 timeframes × `adj_split` / `adj_splitdiv` / `UNADJUSTED`
	72	+- Futures — continuous series (`contin_UNadj` / `contin_adj_ratio` / `contin_adj_absolute`) + individual contracts
	73	+- Crypto, indices, FX — full history, all timeframes
	74	+- Metadata — ticker listings, splits, dividends, update logs, continuous-series audit
	75	+
	76	+```bash
	77	+export FRD_USERID=<your FirstRate customer id>
	78	+python3 -m venv venv && venv/bin/pip install -r requirements.txt
	79	+venv/bin/python frd_downloader.py --data-root /Volumes/ssd/firstratedata
	80	+```
	81	+
	82	+Fully resumable (`state/manifest.json`), disk-space guard, retries with backoff, and raw zips/CSVs are deleted as soon as each segment is converted. Notable quirks handled: FirstRate files mix LF/CRLF line endings (`strict_mode=false`), daily files use `yyyyMMdd` dates, the API 302-redirects to a CDN.
	83	+
	84	+## The API
	85	+
	86	+Open, keyless, JSON + CSV. Highlights:
	87	+
	88	+\| Endpoint \| What it does \|
	89	+\|---\|---\|
	90	+\| `GET /v1/status` \| Live inventory of the lake \|
	91	+\| `GET /v1/{asset}/tickers` \| Instrument discovery per type/timeframe/adjustment \|
	92	+\| `GET /v1/bars/{asset}/{ticker}` \| OHLCV bars, precise `start`/`end` windows, up to 2M rows in CSV \|
	93	+\| `GET /v1/bars/{asset}?tickers=A,B,C` \| Multi-ticker bars in one call (per-ticker limit) \|
	94	+\| `GET /v1/snapshot/{asset}?tickers=…&at=…` \| Whole watchlist at one precise moment \|
	95	+\| `GET /v1/options/chain/{ticker}` \| Full EOD chain — quotes, IV, Greeks \|
	96	+\| `GET /v1/options/history/{ticker}` \| One contract's daily life story \|
	97	+\| `GET /docs` \| OpenAPI / Swagger \|
	98	+
	99	+```bash
	100	+# 4 hours of 1-minute bars across a watchlist
	101	+curl "https://www.hfmarketdata.io/v1/bars/stock?tickers=AAPL,MSFT,NVDA&timeframe=1min&start=2024-06-03%2009:30:00&end=2024-06-03%2013:30:00"
	102	+```
	103	+
	104	+## The platform
	105	+
	106	+React 18 + Vite documentation site (English) served by the API process — refined editorial light theme (Fraunces / Inter / JetBrains Mono), full endpoint reference with parameter tables, request snippets (curl / Python / JavaScript), example + live responses, response-field dictionaries and a live dataset status page.
	107	+
	108	+```bash
	109	+cd hfmarketdata/web && npm install && npm run build # → dist/, served at /
	110	+```
	111	+
	112	+## Author
	113	+
	114	+Simon-Pierre Boucher — [contact@spboucher.ai](mailto:contact@spboucher.ai)

added frd_downloader.py +627 −0

		`@@ -0,0 +1,627 @@`
	1	+#!/usr/bin/env python3
	2	+# -- coding: utf-8 --
	3	+"""
	4	+================================================================================
	5	+ FirstRate Data — Full Historical Downloader & Parquet Converter
	6	+================================================================================
	7	+
	8	+Downloads the COMPLETE historical datasets from the FirstRate Data API and
	9	+stores them as query-optimized Parquet files (zstd compressed), ready to be
	10	+served by an ultra-fast query layer (DuckDB / pyarrow).
	11	+
	12	+Coverage
	13	+--------
	14	+ * options : full archives per year/quarter (2010 -> today)
	15	+ + 'month' update for the current partial quarter
	16	+ * stock, etf : period=full, ticker_range A-Z,
	17	+ timeframes 1min/5min/30min/1hour/1day,
	18	+ adjustments adj_split / adj_splitdiv / UNADJUSTED
	19	+ (UNADJUSTED only exists for 1min & 1day)
	20	+ * futures (continuous) : timeframes x contin_UNadj / contin_adj_ratio /
	21	+ contin_adj_absolute
	22	+ * futures (individual) : contract_files=archive + update, all timeframes
	23	+ * crypto, index, fx : period=full, all timeframes
	24	+ * metadata : ticker listings, splits, dividends, update logs,
	25	+ futures continuous-series audit file
	26	+ * delisted stocks : optional (--include-delisted), archives 1-4 +
	27	+ 2026 update, all timeframe/adjustment combos
	28	+
	29	+Storage layout (under --data-root)
	30	+----------------------------------
	31	+ parquet/{type}/{timeframe}/{adjustment}/{TICKER}.parquet (bar data)
	32	+ parquet/options/{year}_{quarter}/{file}.parquet (options)
	33	+ meta/… (csv metadata)
	34	+ zips/… (only w/ --keep-zips)
	35	+ state/manifest.json (resume state)
	36	+ logs/frd_downloader.log
	37	+
	38	+Every Parquet file carries a `ticker` column, so the whole tree can be
	39	+queried directly, e.g.:
	40	+
	41	+ SELECT * FROM read_parquet('parquet/stock/1day/adj_splitdiv/*.parquet')
	42	+ WHERE ticker = 'AAPL' AND datetime >= '2020-01-01';
	43	+
	44	+Usage
	45	+-----
	46	+ python3 frd_downloader.py --data-root /Volumes/ssd/firstratedata
	47	+ python3 frd_downloader.py --list-jobs # preview the job plan
	48	+ python3 frd_downloader.py --types options stock # restrict asset types
	49	+ python3 frd_downloader.py --include-delisted # add delisted stocks
	50	+
	51	+The run is fully resumable: completed jobs are recorded in the manifest and
	52	+skipped on re-run.
	53	+
	54	+Author : Simon-Pierre Boucher
	55	+Contact : contact@spboucher.ai
	56	+Created : 2026-08-09
	57	+================================================================================
	58	+"""
	59	+
	60	+from __future__ import annotations
	61	+
	62	+import argparse
	63	+import json
	64	+import logging
	65	+import os
	66	+import shutil
	67	+import sys
	68	+import tempfile
	69	+import threading
	70	+import time
	71	+import zipfile
	72	+from concurrent.futures import ThreadPoolExecutor, as_completed
	73	+from dataclasses import dataclass, field
	74	+from datetime import date
	75	+from pathlib import Path
	76	+from typing import Iterator
	77	+from urllib.parse import urlencode
	78	+
	79	+import duckdb
	80	+import requests
	81	+
	82	+__author__ = "Simon-Pierre Boucher"
	83	+__contact__ = "contact@spboucher.ai"
	84	+__version__ = "1.0.0"
	85	+
	86	+# ------------------------------------------------------------------------------
	87	+# Configuration
	88	+# ------------------------------------------------------------------------------
	89	+
	90	+BASE_URL = "https://firstratedata.com/api"
	91	+# FirstRate Data customer id — never hardcoded: export FRD_USERID=<your id>
	92	+USERID = os.environ.get("FRD_USERID", "")
	93	+
	94	+TIMEFRAMES = ["1day", "1hour", "30min", "5min", "1min"] # small -> large
	95	+STOCK_ETF_ADJUSTMENTS = ["adj_split", "adj_splitdiv", "UNADJUSTED"]
	96	+UNADJUSTED_TIMEFRAMES = {"1min", "1day"} # per API doc
	97	+FUTURES_ADJUSTMENTS = ["contin_UNadj", "contin_adj_ratio", "contin_adj_absolute"]
	98	+TICKER_RANGES = [chr(c) for c in range(ord("A"), ord("Z") + 1)]
	99	+OPTIONS_FIRST_YEAR = 2010
	100	+
	101	+DOWNLOAD_TIMEOUT = 7200 # seconds per request (archives can be huge)
	102	+CHUNK_SIZE = 4 * 1024 * 1024 # 4 MiB streaming chunks
	103	+MAX_RETRIES = 5
	104	+RETRY_BACKOFF = 60 # seconds, multiplied by attempt number
	105	+MIN_FREE_GB = 30 # abort downloads below this free-space floor
	106	+
	107	+log = logging.getLogger("frd")
	108	+
	109	+# ------------------------------------------------------------------------------
	110	+# Job model
	111	+# ------------------------------------------------------------------------------
	112	+
	113	+
	114	+@dataclass
	115	+class Job:
	116	+ """One downloadable unit: an API request plus its destination."""
	117	+
	118	+ key: str # unique id, used in the manifest
	119	+ endpoint: str # API endpoint path (e.g. 'data_file')
	120	+ params: dict # query parameters (userid added at request time)
	121	+ dest_subdir: str # where converted output lands, under data root
	122	+ kind: str = "bars" # 'bars' (zip of csv -> parquet) or 'meta' (raw text)
	123	+ priority: int = 50 # lower runs first
	124	+
	125	+ @property
	126	+ def url(self) -> str:
	127	+ q = dict(self.params)
	128	+ q["userid"] = USERID
	129	+ return f"{BASE_URL}/{self.endpoint}?{urlencode(q)}"
	130	+
	131	+ @property
	132	+ def display_url(self) -> str:
	133	+ return f"{BASE_URL}/{self.endpoint}?{urlencode(self.params)}&userid=***"
	134	+
	135	+
	136	+def _current_quarter(today: date) -> tuple[int, int]:
	137	+ return today.year, (today.month - 1) // 3 + 1
	138	+
	139	+
	140	+def build_jobs(types: list[str], include_delisted: bool) -> list[Job]:
	141	+ """Build the full download plan, ordered so that the small/most-useful
	142	+ datasets (daily bars, metadata) land first and the huge ones last."""
	143	+ jobs: list[Job] = []
	144	+ today = date.today()
	145	+
	146	+ # -- metadata (tiny, always useful, run first) ------------------------------
	147	+ for t in ["stock", "etf", "futures", "crypto", "index", "fx"]:
	148	+ if t in types:
	149	+ jobs.append(Job(
	150	+ key=f"ticker_listing\|{t}", endpoint="ticker_listing",
	151	+ params={"type": t}, dest_subdir=f"meta/{t}",
	152	+ kind="meta", priority=0))
	153	+ for t in ["stock", "etf"]:
	154	+ if t in types:
	155	+ for mft in ["splits", "dividends"]:
	156	+ jobs.append(Job(
	157	+ key=f"meta\|{t}\|{mft}", endpoint="meta_file",
	158	+ params={"type": t, "metafile_type": mft},
	159	+ dest_subdir=f"meta/{t}", kind="meta", priority=1))
	160	+ if "stock" in types:
	161	+ for lt in ["delisted", "added", "changed"]:
	162	+ jobs.append(Job(
	163	+ key=f"update_log\|stock\|{lt}", endpoint="update_log",
	164	+ params={"type": "stock", "log_type": lt},
	165	+ dest_subdir="meta/stock", kind="meta", priority=1))
	166	+ if "futures" in types:
	167	+ jobs.append(Job(
	168	+ key="meta\|futures\|contin_audit", endpoint="meta_file",
	169	+ params={"type": "futures", "metafile_type": "contin_audit"},
	170	+ dest_subdir="meta/futures", kind="meta", priority=1))
	171	+
	172	+ # -- stock / etf : full archives, per letter --------------------------------
	173	+ for t in ["stock", "etf"]:
	174	+ if t not in types:
	175	+ continue
	176	+ for tf_i, tf in enumerate(TIMEFRAMES):
	177	+ for adj in STOCK_ETF_ADJUSTMENTS:
	178	+ if adj == "UNADJUSTED" and tf not in UNADJUSTED_TIMEFRAMES:
	179	+ continue
	180	+ for letter in TICKER_RANGES:
	181	+ jobs.append(Job(
	182	+ key=f"{t}\|full\|{tf}\|{adj}\|{letter}",
	183	+ endpoint="data_file",
	184	+ params={"type": t, "period": "full",
	185	+ "ticker_range": letter,
	186	+ "timeframe": tf, "adjustment": adj},
	187	+ dest_subdir=f"parquet/{t}/{tf}/{adj}",
	188	+ priority=10 + tf_i * 4))
	189	+
	190	+ # -- futures : continuous series --------------------------------------------
	191	+ if "futures" in types:
	192	+ for tf_i, tf in enumerate(TIMEFRAMES):
	193	+ for adj in FUTURES_ADJUSTMENTS:
	194	+ jobs.append(Job(
	195	+ key=f"futures\|full\|{tf}\|{adj}",
	196	+ endpoint="data_file",
	197	+ params={"type": "futures", "period": "full",
	198	+ "timeframe": tf, "adjustment": adj},
	199	+ dest_subdir=f"parquet/futures/{tf}/{adj}",
	200	+ priority=10 + tf_i * 4))
	201	+ # individual contracts (pre-2026 archive + current-year update)
	202	+ for tf_i, tf in enumerate(TIMEFRAMES):
	203	+ for cf in ["archive", "update"]:
	204	+ jobs.append(Job(
	205	+ key=f"futures_contract\|{cf}\|{tf}",
	206	+ endpoint="futures_contract",
	207	+ params={"contract_files": cf, "timeframe": tf},
	208	+ dest_subdir=f"parquet/futures_contracts/{tf}/{cf}",
	209	+ priority=12 + tf_i * 4))
	210	+
	211	+ # -- crypto / index / fx : full archives -------------------------------------
	212	+ for t in ["crypto", "index", "fx"]:
	213	+ if t not in types:
	214	+ continue
	215	+ for tf_i, tf in enumerate(TIMEFRAMES):
	216	+ jobs.append(Job(
	217	+ key=f"{t}\|full\|{tf}",
	218	+ endpoint="data_file",
	219	+ params={"type": t, "period": "full", "timeframe": tf},
	220	+ dest_subdir=f"parquet/{t}/{tf}/none",
	221	+ priority=10 + tf_i * 4))
	222	+
	223	+ # -- options : every quarter since 2010 + current partial quarter ------------
	224	+ # Priority 5-6: options are the primary dataset, they download right after
	225	+ # the metadata and before all the bar archives.
	226	+ if "options" in types:
	227	+ cur_year, cur_q = _current_quarter(today)
	228	+ for year in range(OPTIONS_FIRST_YEAR, cur_year + 1):
	229	+ for q in range(1, 5):
	230	+ if year == cur_year and q >= cur_q:
	231	+ continue # current/future quarters have no full archive yet
	232	+ jobs.append(Job(
	233	+ key=f"options\|{year}\|q{q}",
	234	+ endpoint="data_file",
	235	+ params={"type": "options", "year": str(year),
	236	+ "quarter": f"q{q}"},
	237	+ dest_subdir=f"parquet/options/{year}_q{q}",
	238	+ priority=5))
	239	+ jobs.append(Job(
	240	+ key="options\|update\|month",
	241	+ endpoint="data_file",
	242	+ params={"type": "options", "period": "month"},
	243	+ dest_subdir=f"parquet/options/{cur_year}_q{cur_q}_partial",
	244	+ priority=6))
	245	+
	246	+ # -- delisted stocks (optional, survivorship-bias-free research) -------------
	247	+ if include_delisted and "stock" in types:
	248	+ for tf_i, tf in enumerate(TIMEFRAMES):
	249	+ for adj in STOCK_ETF_ADJUSTMENTS:
	250	+ if adj == "UNADJUSTED" and tf != "1min":
	251	+ continue # per API doc: UNADJUSTED delisted = 1min only
	252	+ for arc in ["1", "2", "3", "4"]:
	253	+ jobs.append(Job(
	254	+ key=f"delisted\|archive{arc}\|{tf}\|{adj}",
	255	+ endpoint="delisted_data_file",
	256	+ params={"archive_number": arc,
	257	+ "timeframe": tf, "adjustment": adj},
	258	+ dest_subdir=f"parquet/stock_delisted/{tf}/{adj}",
	259	+ priority=30 + tf_i * 4))
	260	+ jobs.append(Job(
	261	+ key=f"delisted\|update_year\|{tf}\|{adj}",
	262	+ endpoint="delisted_data_file",
	263	+ params={"update": "year", "timeframe": tf,
	264	+ "adjustment": adj},
	265	+ dest_subdir=f"parquet/stock_delisted/{tf}/{adj}",
	266	+ priority=30 + tf_i * 4))
	267	+
	268	+ jobs.sort(key=lambda j: (j.priority, j.key))
	269	+ return jobs
	270	+
	271	+
	272	+# ------------------------------------------------------------------------------
	273	+# Manifest (resume state)
	274	+# ------------------------------------------------------------------------------
	275	+
	276	+
	277	+class Manifest:
	278	+ """Thread-safe JSON manifest tracking completed/failed jobs."""
	279	+
	280	+ def __init__(self, path: Path):
	281	+ self.path = path
	282	+ self._lock = threading.Lock()
	283	+ self._data: dict = {"jobs": {}, "version": __version__}
	284	+ if path.exists():
	285	+ try:
	286	+ self._data = json.loads(path.read_text())
	287	+ except (json.JSONDecodeError, OSError):
	288	+ log.warning("Manifest unreadable, starting fresh: %s", path)
	289	+
	290	+ def is_done(self, key: str) -> bool:
	291	+ return self._data["jobs"].get(key, {}).get("status") == "done"
	292	+
	293	+ def mark(self, key: str, status: str, **info) -> None:
	294	+ with self._lock:
	295	+ entry = self._data["jobs"].setdefault(key, {})
	296	+ entry.update(status=status, updated=time.strftime("%Y-%m-%d %H:%M:%S"), **info)
	297	+ tmp = self.path.with_suffix(".tmp")
	298	+ tmp.write_text(json.dumps(self._data, indent=1))
	299	+ tmp.replace(self.path)
	300	+
	301	+ def summary(self) -> dict:
	302	+ counts: dict[str, int] = {}
	303	+ for e in self._data["jobs"].values():
	304	+ counts[e.get("status", "?")] = counts.get(e.get("status", "?"), 0) + 1
	305	+ return counts
	306	+
	307	+
	308	+# ------------------------------------------------------------------------------
	309	+# CSV -> Parquet conversion
	310	+# ------------------------------------------------------------------------------
	311	+
	312	+# Column layouts by column count (FirstRate bar files have no header row).
	313	+# The first field is read as VARCHAR because FirstRate mixes date formats
	314	+# ('2005-01-03 17:00:00' intraday vs '20100104' daily) — it is normalized to a
	315	+# proper TIMESTAMP at conversion time (see _DT_PARSE).
	316	+BAR_SCHEMAS = {
	317	+ 5: {"dt_raw": "VARCHAR", "open": "DOUBLE", "high": "DOUBLE",
	318	+ "low": "DOUBLE", "close": "DOUBLE"},
	319	+ 6: {"dt_raw": "VARCHAR", "open": "DOUBLE", "high": "DOUBLE",
	320	+ "low": "DOUBLE", "close": "DOUBLE", "volume": "DOUBLE"},
	321	+ 7: {"dt_raw": "VARCHAR", "open": "DOUBLE", "high": "DOUBLE",
	322	+ "low": "DOUBLE", "close": "DOUBLE", "volume": "DOUBLE",
	323	+ "open_interest": "DOUBLE"},
	324	+}
	325	+
	326	+_DT_PARSE = ("COALESCE("
	327	+ "try_strptime(dt_raw, '%Y-%m-%d %H:%M:%S'), "
	328	+ "try_strptime(dt_raw, '%Y%m%d'), "
	329	+ "try_strptime(dt_raw, '%Y-%m-%d'), "
	330	+ "try_strptime(dt_raw, '%m/%d/%Y %H:%M:%S'), "
	331	+ "try_strptime(dt_raw, '%m/%d/%Y'))")
	332	+
	333	+# Options chain layout, per https://firstratedata.com/_readme/options.txt :
	334	+# Trade Date, Strike, Expiry Date, Call/Put, Last Trade Price, Bid Price,
	335	+# Ask Price, Bid IV, Ask IV, Open Interest, Volume, Delta, Gamma, Vega,
	336	+# Theta, Rho
	337	+OPTIONS_SCHEMA = {
	338	+ "trade_date": "DATE", "strike": "DOUBLE", "expiry": "DATE",
	339	+ "call_put": "VARCHAR", "last_price": "DOUBLE", "bid": "DOUBLE",
	340	+ "ask": "DOUBLE", "bid_iv": "DOUBLE", "ask_iv": "DOUBLE",
	341	+ "open_interest": "DOUBLE", "volume": "DOUBLE", "delta": "DOUBLE",
	342	+ "gamma": "DOUBLE", "vega": "DOUBLE", "theta": "DOUBLE", "rho": "DOUBLE",
	343	+}
	344	+
	345	+
	346	+def _sniff_column_count(csv_path: Path) -> int:
	347	+ with open(csv_path, "r", errors="replace") as f:
	348	+ for line in f:
	349	+ line = line.strip()
	350	+ if line:
	351	+ return line.count(",") + 1
	352	+ return 0
	353	+
	354	+
	355	+def _ticker_from_filename(path: Path) -> str:
	356	+ """'AAPL_1min.txt' -> 'AAPL' ; 'ES_contin_adj.txt' -> 'ES'."""
	357	+ return path.stem.split("_")[0].upper()
	358	+
	359	+
	360	+def csv_to_parquet(con: duckdb.DuckDBPyConnection, csv_path: Path,
	361	+ out_path: Path, asset_type: str) -> int:
	362	+ """Convert one extracted csv/txt file to zstd Parquet. Returns row count."""
	363	+ ncols = _sniff_column_count(csv_path)
	364	+ if ncols == 0:
	365	+ return 0
	366	+ out_path.parent.mkdir(parents=True, exist_ok=True)
	367	+ ticker = _ticker_from_filename(csv_path)
	368	+ src = str(csv_path).replace("'", "''")
	369	+ dst = str(out_path).replace("'", "''")
	370	+
	371	+ # strict_mode=false: FirstRate files mix LF and CRLF line endings (daily
	372	+ # rows are appended with CRLF onto LF history), which the strict CSV
	373	+ # sniffer rejects outright.
	374	+ common = "strict_mode=false, ignore_errors=true"
	375	+ if asset_type == "options" and ncols == len(OPTIONS_SCHEMA):
	376	+ cols = json.dumps(OPTIONS_SCHEMA).replace('"', "'")
	377	+ select = (f"SELECT '{ticker}' AS ticker, * FROM read_csv('{src}', "
	378	+ f"header=false, columns={cols}, dateformat='%Y-%m-%d', "
	379	+ f"{common})")
	380	+ elif asset_type != "options" and (schema := BAR_SCHEMAS.get(ncols)):
	381	+ cols = json.dumps(schema).replace('"', "'")
	382	+ value_cols = ", ".join(c for c in schema if c != "dt_raw")
	383	+ select = (f"SELECT '{ticker}' AS ticker, {_DT_PARSE} AS datetime, "
	384	+ f"{value_cols} FROM read_csv('{src}', "
	385	+ f"header=false, columns={cols}, {common})")
	386	+ else:
	387	+ # Unknown layout (options files etc.): let DuckDB auto-detect types,
	388	+ # header presence and column names — still fully queryable.
	389	+ select = (f"SELECT '{ticker}' AS ticker, * FROM read_csv_auto('{src}', "
	390	+ f"{common})")
	391	+
	392	+ con.execute(f"COPY ({select}) TO '{dst}' "
	393	+ f"(FORMAT PARQUET, COMPRESSION ZSTD, ROW_GROUP_SIZE 1000000)")
	394	+ rows = con.execute(f"SELECT count(*) FROM read_parquet('{dst}')").fetchone()[0]
	395	+ if rows == 0 and csv_path.stat().st_size > 0:
	396	+ log.warning("0 rows converted from non-empty file %s (ncols=%d)",
	397	+ csv_path.name, ncols)
	398	+ return rows
	399	+
	400	+
	401	+def extract_and_convert(zip_path: Path, dest_dir: Path, asset_type: str) -> tuple[int, int]:
	402	+ """Extract a downloaded zip (recursively if it nests zips) and convert every
	403	+ csv/txt member to Parquet under dest_dir. Returns (files, rows)."""
	404	+ files = rows = 0
	405	+ con = duckdb.connect()
	406	+ con.execute("SET threads TO 4")
	407	+ try:
	408	+ with tempfile.TemporaryDirectory(dir=zip_path.parent) as tmp:
	409	+ tmp_dir = Path(tmp)
	410	+ _extract_recursive(zip_path, tmp_dir)
	411	+ members = sorted(p for p in tmp_dir.rglob("*")
	412	+ if p.is_file() and p.suffix.lower() in (".txt", ".csv"))
	413	+ for member in members:
	414	+ out = dest_dir / (member.stem + ".parquet")
	415	+ try:
	416	+ n = csv_to_parquet(con, member, out, asset_type)
	417	+ if n > 0:
	418	+ files += 1
	419	+ rows += n
	420	+ else:
	421	+ out.unlink(missing_ok=True)
	422	+ except duckdb.Error as e:
	423	+ log.error("Convert failed %s: %s", member.name, e)
	424	+ member.unlink() # free space as we go
	425	+ finally:
	426	+ con.close()
	427	+ return files, rows
	428	+
	429	+
	430	+def _extract_recursive(zip_path: Path, dest: Path, depth: int = 0) -> None:
	431	+ if depth > 3:
	432	+ return
	433	+ with zipfile.ZipFile(zip_path) as zf:
	434	+ zf.extractall(dest)
	435	+ for nested in list(dest.rglob("*.zip")):
	436	+ sub = nested.with_suffix("")
	437	+ sub.mkdir(exist_ok=True)
	438	+ _extract_recursive(nested, sub, depth + 1)
	439	+ nested.unlink()
	440	+
	441	+
	442	+# ------------------------------------------------------------------------------
	443	+# Downloader
	444	+# ------------------------------------------------------------------------------
	445	+
	446	+
	447	+@dataclass
	448	+class Runner:
	449	+ data_root: Path
	450	+ manifest: Manifest
	451	+ keep_zips: bool = False
	452	+ min_free_gb: int = MIN_FREE_GB
	453	+ session: requests.Session = field(default_factory=requests.Session)
	454	+
	455	+ def _free_gb(self) -> float:
	456	+ return shutil.disk_usage(self.data_root).free / 1e9
	457	+
	458	+ def _download(self, job: Job, dest: Path) -> Path \| None:
	459	+ """Stream the response to dest (via .part). Returns path or None."""
	460	+ part = dest.with_suffix(dest.suffix + ".part")
	461	+ for attempt in range(1, MAX_RETRIES + 1):
	462	+ if self._free_gb() < self.min_free_gb:
	463	+ raise RuntimeError(
	464	+ f"Only {self._free_gb():.1f} GB free (< {self.min_free_gb} GB floor)")
	465	+ try:
	466	+ with self.session.get(job.url, stream=True,
	467	+ timeout=DOWNLOAD_TIMEOUT) as r:
	468	+ if r.status_code in (429, 500, 502, 503, 504):
	469	+ raise requests.HTTPError(f"HTTP {r.status_code}")
	470	+ if r.status_code == 404:
	471	+ log.warning("[%s] 404 — no data for this combination", job.key)
	472	+ return None
	473	+ r.raise_for_status()
	474	+ dest.parent.mkdir(parents=True, exist_ok=True)
	475	+ size = 0
	476	+ with open(part, "wb") as f:
	477	+ for chunk in r.iter_content(CHUNK_SIZE):
	478	+ f.write(chunk)
	479	+ size += len(chunk)
	480	+ part.replace(dest)
	481	+ log.info("[%s] downloaded %.1f MB", job.key, size / 1e6)
	482	+ return dest
	483	+ except (requests.RequestException, OSError) as e:
	484	+ part.unlink(missing_ok=True)
	485	+ if attempt == MAX_RETRIES:
	486	+ raise
	487	+ wait = RETRY_BACKOFF * attempt
	488	+ log.warning("[%s] attempt %d/%d failed (%s), retrying in %ds",
	489	+ job.key, attempt, MAX_RETRIES, e, wait)
	490	+ time.sleep(wait)
	491	+ return None
	492	+
	493	+ def run_job(self, job: Job) -> None:
	494	+ if self.manifest.is_done(job.key):
	495	+ log.info("[%s] already done, skipping", job.key)
	496	+ return
	497	+ log.info("[%s] GET %s", job.key, job.display_url)
	498	+ t0 = time.time()
	499	+ try:
	500	+ if job.kind == "meta":
	501	+ self._run_meta(job)
	502	+ else:
	503	+ self._run_bars(job)
	504	+ except Exception as e:
	505	+ log.error("[%s] FAILED: %s", job.key, e)
	506	+ self.manifest.mark(job.key, "failed", error=str(e))
	507	+ return
	508	+ log.info("[%s] done in %.0fs", job.key, time.time() - t0)
	509	+
	510	+ def _run_meta(self, job: Job) -> None:
	511	+ out_dir = self.data_root / job.dest_subdir
	512	+ out_dir.mkdir(parents=True, exist_ok=True)
	513	+ fname = "_".join(str(v) for v in job.params.values()) + ".csv"
	514	+ out = out_dir / fname
	515	+ r = self.session.get(job.url, timeout=600)
	516	+ r.raise_for_status()
	517	+ out.write_bytes(r.content)
	518	+ self.manifest.mark(job.key, "done", bytes=len(r.content), file=str(out))
	519	+
	520	+ def _run_bars(self, job: Job) -> None:
	521	+ zip_dir = self.data_root / "zips"
	522	+ zip_dir.mkdir(parents=True, exist_ok=True)
	523	+ zip_path = zip_dir / (job.key.replace("\|", "_") + ".zip")
	524	+
	525	+ got = self._download(job, zip_path)
	526	+ if got is None:
	527	+ self.manifest.mark(job.key, "empty")
	528	+ return
	529	+ if not zipfile.is_zipfile(zip_path):
	530	+ head = zip_path.read_bytes()[:200]
	531	+ zip_path.unlink()
	532	+ raise RuntimeError(f"Response is not a zip (starts with {head[:60]!r})")
	533	+
	534	+ asset_type = job.key.split("\|")[0]
	535	+ dest_dir = self.data_root / job.dest_subdir
	536	+ files, rows = extract_and_convert(zip_path, dest_dir, asset_type)
	537	+ size = zip_path.stat().st_size
	538	+ if not self.keep_zips:
	539	+ zip_path.unlink()
	540	+ self.manifest.mark(job.key, "done", bytes=size, files=files, rows=rows)
	541	+ log.info("[%s] converted %d files / %s rows -> %s",
	542	+ job.key, files, f"{rows:,}", dest_dir)
	543	+
	544	+
	545	+# ------------------------------------------------------------------------------
	546	+# CLI
	547	+# ------------------------------------------------------------------------------
	548	+
	549	+
	550	+def setup_logging(data_root: Path) -> None:
	551	+ log_dir = data_root / "logs"
	552	+ log_dir.mkdir(parents=True, exist_ok=True)
	553	+ fmt = logging.Formatter("%(asctime)s %(levelname)-7s %(message)s",
	554	+ "%Y-%m-%d %H:%M:%S")
	555	+ for handler in (logging.StreamHandler(sys.stdout),
	556	+ logging.FileHandler(log_dir / "frd_downloader.log")):
	557	+ handler.setFormatter(fmt)
	558	+ log.addHandler(handler)
	559	+ log.setLevel(logging.INFO)
	560	+
	561	+
	562	+def main() -> int:
	563	+ ap = argparse.ArgumentParser(
	564	+ description="FirstRate Data full-history downloader "
	565	+ f"(v{__version__} — {__author__} <{__contact__}>)")
	566	+ ap.add_argument("--data-root", type=Path,
	567	+ default=Path("/Volumes/ssd/firstratedata"),
	568	+ help="Root directory for all data (default: %(default)s)")
	569	+ ap.add_argument("--types", nargs="+",
	570	+ default=["options", "stock", "etf", "futures",
	571	+ "crypto", "index", "fx"],
	572	+ choices=["options", "stock", "etf", "futures",
	573	+ "crypto", "index", "fx"],
	574	+ help="Asset types to download (default: all)")
	575	+ ap.add_argument("--workers", type=int, default=2,
	576	+ help="Parallel downloads (default: %(default)s — be polite)")
	577	+ ap.add_argument("--keep-zips", action="store_true",
	578	+ help="Keep raw zip archives after conversion")
	579	+ ap.add_argument("--include-delisted", action="store_true",
	580	+ help="Also download the delisted-stocks archives")
	581	+ ap.add_argument("--min-free-gb", type=int, default=MIN_FREE_GB,
	582	+ help="Stop downloading below this free space (default: %(default)s)")
	583	+ ap.add_argument("--list-jobs", action="store_true",
	584	+ help="Print the job plan and exit")
	585	+ args = ap.parse_args()
	586	+
	587	+ if not USERID:
	588	+ ap.error("FRD_USERID environment variable is not set "
	589	+ "(your FirstRate Data customer id)")
	590	+
	591	+ jobs = build_jobs(args.types, args.include_delisted)
	592	+
	593	+ if args.list_jobs:
	594	+ for j in jobs:
	595	+ print(f"{j.priority:3d} {j.key:45s} {j.display_url}")
	596	+ print(f"\nTotal: {len(jobs)} jobs")
	597	+ return 0
	598	+
	599	+ args.data_root.mkdir(parents=True, exist_ok=True)
	600	+ setup_logging(args.data_root)
	601	+ log.info("FirstRate Data downloader v%s — %s <%s>",
	602	+ __version__, __author__, __contact__)
	603	+ log.info("Data root: %s \| types: %s \| %d jobs \| %d workers",
	604	+ args.data_root, ",".join(args.types), len(jobs), args.workers)
	605	+
	606	+ manifest = Manifest(args.data_root / "state" / "manifest.json")
	607	+ (args.data_root / "state").mkdir(parents=True, exist_ok=True)
	608	+ runner = Runner(args.data_root, manifest,
	609	+ keep_zips=args.keep_zips, min_free_gb=args.min_free_gb)
	610	+
	611	+ pending = [j for j in jobs if not manifest.is_done(j.key)]
	612	+ log.info("%d jobs pending (%d already complete)",
	613	+ len(pending), len(jobs) - len(pending))
	614	+
	615	+ with ThreadPoolExecutor(max_workers=args.workers) as pool:
	616	+ futures = {pool.submit(runner.run_job, j): j for j in pending}
	617	+ for fut in as_completed(futures):
	618	+ fut.result() # exceptions are handled inside run_job
	619	+
	620	+ counts = manifest.summary()
	621	+ log.info("Run complete. Manifest: %s", counts)
	622	+ failed = counts.get("failed", 0)
	623	+ return 1 if failed else 0
	624	+
	625	+
	626	+if __name__ == "__main__":
	627	+ sys.exit(main())

added hfmarketdata/api/main.py +476 −0

		`@@ -0,0 +1,476 @@`
	1	+#!/usr/bin/env python3
	2	+# -- coding: utf-8 --
	3	+"""
	4	+================================================================================
	5	+ HF Market Data — Open High-Frequency Market Data API
	6	+ https://www.hfmarketdata.io
	7	+================================================================================
	8	+
	9	+Ultra-fast open REST API serving historical market data (stocks, ETFs,
	10	+futures, crypto, indices, FX — 1-minute to daily bars — plus full end-of-day
	11	+options chains with quotes, implied volatilities and Greeks).
	12	+
	13	+The data layer is a partitioned Parquet lake (zstd) queried in-place with
	14	+DuckDB — no database server, predicate pushdown straight into the row groups.
	15	+
	16	+Layout expected under DATA_ROOT (built by frd_downloader.py):
	17	+ parquet/{type}/{timeframe}/{adjustment}/{TICKER}_*.parquet
	18	+ parquet/options/{year}_{quarter}/{TICKER}_*_option_chain.parquet
	19	+
	20	+Author : Simon-Pierre Boucher
	21	+Contact : contact@spboucher.ai
	22	+================================================================================
	23	+"""
	24	+
	25	+from __future__ import annotations
	26	+
	27	+import os
	28	+import threading
	29	+import time
	30	+from pathlib import Path
	31	+
	32	+import duckdb
	33	+from fastapi import FastAPI, HTTPException, Query
	34	+from fastapi.middleware.cors import CORSMiddleware
	35	+from fastapi.responses import PlainTextResponse
	36	+from fastapi.staticfiles import StaticFiles
	37	+
	38	+__author__ = "Simon-Pierre Boucher"
	39	+__contact__ = "contact@spboucher.ai"
	40	+__version__ = "1.0.0"
	41	+
	42	+DATA_ROOT = Path(os.environ.get("HFMD_DATA_ROOT", "/Volumes/ssd/firstratedata"))
	43	+PARQUET = DATA_ROOT / "parquet"
	44	+WEB_DIST = Path(os.environ.get("HFMD_WEB_DIST",
	45	+ Path(__file__).resolve().parent.parent / "web" / "dist"))
	46	+
	47	+BAR_TYPES = ["stock", "etf", "futures", "futures_contracts", "crypto", "index", "fx"]
	48	+TIMEFRAMES = ["1min", "5min", "30min", "1hour", "1day"]
	49	+DEFAULT_ADJUSTMENT = {
	50	+ "stock": "adj_splitdiv", "etf": "adj_splitdiv",
	51	+ "futures": "contin_UNadj", "futures_contracts": "archive",
	52	+ "crypto": "none", "index": "none", "fx": "none",
	53	+}
	54	+MAX_LIMIT_JSON = 50_000
	55	+MAX_LIMIT_CSV = 2_000_000
	56	+
	57	+app = FastAPI(
	58	+ title="HF Market Data API",
	59	+ description=(
	60	+ "Open high-frequency market data API — intraday & daily bars for "
	61	+ "stocks, ETFs, futures, crypto, indices and FX, plus full historical "
	62	+ "options chains (quotes, implied volatility, Greeks) since 2010. "
	63	+ f"Built by {__author__} ({__contact__})."
	64	+ ),
	65	+ version=__version__,
	66	+ contact={"name": __author__, "email": __contact__,
	67	+ "url": "https://www.hfmarketdata.io"},
	68	+)
	69	+app.add_middleware(
	70	+ CORSMiddleware, allow_origins=[""], allow_methods=[""], allow_headers=["*"],
	71	+)
	72	+
	73	+# ------------------------------------------------------------------------------
	74	+# DuckDB — one connection per thread, read-only usage
	75	+# ------------------------------------------------------------------------------
	76	+
	77	+_tls = threading.local()
	78	+
	79	+
	80	+def db() -> duckdb.DuckDBPyConnection:
	81	+ if not hasattr(_tls, "con"):
	82	+ _tls.con = duckdb.connect()
	83	+ _tls.con.execute("SET threads TO 4")
	84	+ _tls.con.execute("SET enable_object_cache=true")
	85	+ return _tls.con
	86	+
	87	+
	88	+# ------------------------------------------------------------------------------
	89	+# File resolution with a small TTL cache (directory scans are the slow part)
	90	+# ------------------------------------------------------------------------------
	91	+
	92	+_cache: dict[str, tuple[float, object]] = {}
	93	+_cache_lock = threading.Lock()
	94	+CACHE_TTL = 300 # seconds
	95	+
	96	+
	97	+def cached(key: str, builder):
	98	+ now = time.time()
	99	+ with _cache_lock:
	100	+ hit = _cache.get(key)
	101	+ if hit and now - hit[0] < CACHE_TTL:
	102	+ return hit[1]
	103	+ value = builder()
	104	+ with _cache_lock:
	105	+ _cache[key] = (now, value)
	106	+ return value
	107	+
	108	+
	109	+def bar_dir(asset: str, timeframe: str, adjustment: str) -> Path:
	110	+ return PARQUET / asset / timeframe / adjustment
	111	+
	112	+
	113	+def list_adjustments(asset: str, timeframe: str) -> list[str]:
	114	+ base = PARQUET / asset / timeframe
	115	+ if not base.is_dir():
	116	+ return []
	117	+ return sorted(p.name for p in base.iterdir() if p.is_dir())
	118	+
	119	+
	120	+def ticker_index(asset: str, timeframe: str, adjustment: str) -> dict[str, str]:
	121	+ """ticker -> parquet file path, cached."""
	122	+ d = bar_dir(asset, timeframe, adjustment)
	123	+
	124	+ def build():
	125	+ idx: dict[str, str] = {}
	126	+ if d.is_dir():
	127	+ for p in d.iterdir():
	128	+ if p.suffix == ".parquet":
	129	+ idx[p.stem.split("_")[0].upper()] = str(p)
	130	+ return idx
	131	+
	132	+ return cached(f"idx\|{d}", build)
	133	+
	134	+
	135	+def options_quarters() -> list[str]:
	136	+ def build():
	137	+ base = PARQUET / "options"
	138	+ if not base.is_dir():
	139	+ return []
	140	+ return sorted(p.name for p in base.iterdir() if p.is_dir())
	141	+ return cached("opt_quarters", build)
	142	+
	143	+
	144	+def options_files(ticker: str) -> list[str]:
	145	+ """All quarterly chain files for a ticker, cheap direct-path checks."""
	146	+ t = ticker.upper()
	147	+
	148	+ def build():
	149	+ out = []
	150	+ for q in options_quarters():
	151	+ d = PARQUET / "options" / q
	152	+ for p in d.glob(f"{t}_*.parquet"):
	153	+ if p.stem.split("_")[0].upper() == t:
	154	+ out.append(str(p))
	155	+ return out
	156	+
	157	+ return cached(f"optfiles\|{t}", build)
	158	+
	159	+
	160	+def _validate(asset: str, timeframe: str, adjustment: str \| None) -> str:
	161	+ if asset not in BAR_TYPES:
	162	+ raise HTTPException(404, f"Unknown asset type '{asset}'. "
	163	+ f"Available: {', '.join(BAR_TYPES)}")
	164	+ if timeframe not in TIMEFRAMES:
	165	+ raise HTTPException(400, f"Unknown timeframe '{timeframe}'. "
	166	+ f"Available: {', '.join(TIMEFRAMES)}")
	167	+ adj = adjustment or DEFAULT_ADJUSTMENT[asset]
	168	+ available = list_adjustments(asset, timeframe)
	169	+ if available and adj not in available:
	170	+ raise HTTPException(400, f"Adjustment '{adj}' not available for "
	171	+ f"{asset}/{timeframe}. Available: {', '.join(available)}")
	172	+ return adj
	173	+
	174	+
	175	+def _rows_to_response(rel, limit: int, fmt: str, order_col: str):
	176	+ if fmt == "csv":
	177	+ limit = min(limit, MAX_LIMIT_CSV)
	178	+ rel = rel.limit(limit)
	179	+ df = rel.df()
	180	+ return PlainTextResponse(df.to_csv(index=False), media_type="text/csv")
	181	+ limit = min(limit, MAX_LIMIT_JSON)
	182	+ rel = rel.limit(limit)
	183	+ df = rel.df()
	184	+ for col in df.columns:
	185	+ if str(df[col].dtype).startswith("datetime"):
	186	+ df[col] = df[col].astype(str)
	187	+ return {"count": len(df), "data": df.to_dict(orient="records")}
	188	+
	189	+
	190	+# ------------------------------------------------------------------------------
	191	+# Meta endpoints
	192	+# ------------------------------------------------------------------------------
	193	+
	194	+
	195	+@app.get("/health", tags=["meta"])
	196	+def health():
	197	+ return {"status": "ok", "service": "hfmarketdata-api", "version": __version__,
	198	+ "data_root_present": PARQUET.is_dir()}
	199	+
	200	+
	201	+@app.get("/v1/status", tags=["meta"])
	202	+def status():
	203	+ """Dataset inventory: every asset type, timeframe and adjustment with
	204	+ the number of instruments currently available."""
	205	+ def build():
	206	+ out = {}
	207	+ for asset in BAR_TYPES:
	208	+ base = PARQUET / asset
	209	+ if not base.is_dir():
	210	+ continue
	211	+ tfs = {}
	212	+ for tf in sorted(p.name for p in base.iterdir() if p.is_dir()):
	213	+ adjs = {}
	214	+ for adj in list_adjustments(asset, tf):
	215	+ n = sum(1 for p in (base / tf / adj).iterdir()
	216	+ if p.suffix == ".parquet")
	217	+ adjs[adj] = n
	218	+ tfs[tf] = adjs
	219	+ out[asset] = tfs
	220	+ quarters = options_quarters()
	221	+ if quarters:
	222	+ out["options"] = {
	223	+ "quarters": quarters,
	224	+ "files_latest_quarter":
	225	+ sum(1 for _ in (PARQUET / "options" / quarters[-1]).glob("*.parquet")),
	226	+ }
	227	+ return out
	228	+ return {"datasets": cached("status", build)}
	229	+
	230	+
	231	+@app.get("/v1/{asset}/tickers", tags=["bars"])
	232	+def tickers(asset: str,
	233	+ timeframe: str = Query("1day"),
	234	+ adjustment: str \| None = Query(None),
	235	+ search: str \| None = Query(None, description="Substring filter"),
	236	+ limit: int = Query(10_000, ge=1)):
	237	+ """List available tickers for an asset type / timeframe / adjustment.
	238	+ For `options`, this delegates to the options underlyings listing."""
	239	+ if asset == "options":
	240	+ return opt_tickers(quarter=None, search=search, limit=limit)
	241	+ adj = _validate(asset, timeframe, adjustment)
	242	+ idx = ticker_index(asset, timeframe, adj)
	243	+ names = sorted(idx)
	244	+ if search:
	245	+ s = search.upper()
	246	+ names = [t for t in names if s in t]
	247	+ return {"asset": asset, "timeframe": timeframe, "adjustment": adj,
	248	+ "count": len(names[:limit]), "tickers": names[:limit]}
	249	+
	250	+
	251	+# ------------------------------------------------------------------------------
	252	+# Bars
	253	+# ------------------------------------------------------------------------------
	254	+
	255	+
	256	+@app.get("/v1/bars/{asset}/{ticker}", tags=["bars"])
	257	+def bars(asset: str, ticker: str,
	258	+ timeframe: str = Query("1day", description="1min\|5min\|30min\|1hour\|1day"),
	259	+ adjustment: str \| None = Query(None, description="Default depends on asset type"),
	260	+ start: str \| None = Query(None, description="ISO date/datetime lower bound"),
	261	+ end: str \| None = Query(None, description="ISO date/datetime upper bound"),
	262	+ order: str = Query("asc", pattern="^(asc\|desc)$"),
	263	+ limit: int = Query(5_000, ge=1),
	264	+ format: str = Query("json", pattern="^(json\|csv)$")):
	265	+ """OHLCV bars for one instrument. Daily futures bars also carry open
	266	+ interest. `format=csv` allows bulk extraction (up to 2M rows)."""
	267	+ adj = _validate(asset, timeframe, adjustment)
	268	+ idx = ticker_index(asset, timeframe, adj)
	269	+ path = idx.get(ticker.upper())
	270	+ if not path:
	271	+ raise HTTPException(404, f"Ticker '{ticker.upper()}' not found in "
	272	+ f"{asset}/{timeframe}/{adj}")
	273	+ con = db()
	274	+ conds, params = [], []
	275	+ if start:
	276	+ conds.append("datetime >= ?"); params.append(start)
	277	+ if end:
	278	+ conds.append("datetime <= ?"); params.append(end)
	279	+ where = ("WHERE " + " AND ".join(conds)) if conds else ""
	280	+ rel = con.sql(
	281	+ f"SELECT * FROM read_parquet(?) {where} ORDER BY datetime {order.upper()}",
	282	+ params=[path, *params])
	283	+ return _rows_to_response(rel, limit, format, "datetime")
	284	+
	285	+
	286	+@app.get("/v1/bars/{asset}", tags=["bars"])
	287	+def bars_multi(asset: str,
	288	+ tickers: str = Query(..., description="Comma-separated list, e.g. AAPL,MSFT,TSLA (max 50)"),
	289	+ timeframe: str = Query("1day"),
	290	+ adjustment: str \| None = Query(None),
	291	+ start: str \| None = Query(None, description="ISO date/datetime lower bound"),
	292	+ end: str \| None = Query(None, description="ISO date/datetime upper bound"),
	293	+ order: str = Query("asc", pattern="^(asc\|desc)$"),
	294	+ limit: int = Query(5_000, ge=1, description="Max rows PER TICKER"),
	295	+ format: str = Query("json", pattern="^(json\|csv)$")):
	296	+ """Bars for SEVERAL instruments in one call. Combine with `start`/`end`
	297	+ to slice any precise window — e.g. 4 hours of 1-minute bars across a
	298	+ whole watchlist. The row limit applies per ticker."""
	299	+ adj = _validate(asset, timeframe, adjustment)
	300	+ idx = ticker_index(asset, timeframe, adj)
	301	+ names = [t.strip().upper() for t in tickers.split(",") if t.strip()][:50]
	302	+ if not names:
	303	+ raise HTTPException(400, "No tickers given")
	304	+ missing = [t for t in names if t not in idx]
	305	+ if missing:
	306	+ raise HTTPException(404, f"Not found in {asset}/{timeframe}/{adj}: "
	307	+ f"{', '.join(missing)}")
	308	+ paths = [idx[t] for t in names]
	309	+ con = db()
	310	+ conds, params = [], []
	311	+ if start:
	312	+ conds.append("datetime >= ?"); params.append(start)
	313	+ if end:
	314	+ conds.append("datetime <= ?"); params.append(end)
	315	+ where = ("WHERE " + " AND ".join(conds)) if conds else ""
	316	+ per_ticker = min(limit, MAX_LIMIT_CSV if format == "csv" else MAX_LIMIT_JSON)
	317	+ rel = con.sql(
	318	+ f"SELECT * FROM read_parquet(?) {where} "
	319	+ f"QUALIFY row_number() OVER (PARTITION BY ticker ORDER BY datetime "
	320	+ f"{order.upper()}) <= ? ORDER BY ticker, datetime {order.upper()}",
	321	+ params=[paths, *params, per_ticker])
	322	+ return _rows_to_response(rel, per_ticker * len(names), format, "datetime")
	323	+
	324	+
	325	+@app.get("/v1/snapshot/{asset}", tags=["bars"])
	326	+def snapshot(asset: str,
	327	+ tickers: str = Query(..., description="Comma-separated list, e.g. AAPL,MSFT,TSLA (max 50)"),
	328	+ at: str = Query(..., description="Precise moment, e.g. 2024-06-03 10:35:00"),
	329	+ timeframe: str = Query("1min"),
	330	+ adjustment: str \| None = Query(None),
	331	+ format: str = Query("json", pattern="^(json\|csv)$")):
	332	+ """Cross-sectional snapshot: for each requested instrument, the last bar
	333	+ at or before the given moment — the state of a whole watchlist at one
	334	+ precise point in time."""
	335	+ adj = _validate(asset, timeframe, adjustment)
	336	+ idx = ticker_index(asset, timeframe, adj)
	337	+ names = [t.strip().upper() for t in tickers.split(",") if t.strip()][:50]
	338	+ if not names:
	339	+ raise HTTPException(400, "No tickers given")
	340	+ missing = [t for t in names if t not in idx]
	341	+ if missing:
	342	+ raise HTTPException(404, f"Not found in {asset}/{timeframe}/{adj}: "
	343	+ f"{', '.join(missing)}")
	344	+ paths = [idx[t] for t in names]
	345	+ con = db()
	346	+ rel = con.sql(
	347	+ "SELECT * FROM read_parquet(?) WHERE datetime <= ? "
	348	+ "QUALIFY row_number() OVER (PARTITION BY ticker ORDER BY datetime DESC) = 1 "
	349	+ "ORDER BY ticker",
	350	+ params=[paths, at])
	351	+ return _rows_to_response(rel, len(names), format, "datetime")
	352	+
	353	+
	354	+# ------------------------------------------------------------------------------
	355	+# Options
	356	+# ------------------------------------------------------------------------------
	357	+
	358	+
	359	+@app.get("/v1/options/quarters", tags=["options"])
	360	+def opt_quarters():
	361	+ """Available options archive quarters (e.g. 2010_q1 … current)."""
	362	+ return {"quarters": options_quarters()}
	363	+
	364	+
	365	+@app.get("/v1/options/tickers", tags=["options"])
	366	+def opt_tickers(quarter: str \| None = Query(None, description="e.g. 2024_q4; default latest"),
	367	+ search: str \| None = None,
	368	+ limit: int = Query(10_000, ge=1)):
	369	+ """Underlyings available in a given options quarter."""
	370	+ quarters = options_quarters()
	371	+ if not quarters:
	372	+ raise HTTPException(503, "Options dataset not yet available")
	373	+ q = quarter or quarters[-1]
	374	+ if q not in quarters:
	375	+ raise HTTPException(404, f"Unknown quarter '{q}'")
	376	+
	377	+ def build():
	378	+ return sorted({p.stem.split("_")[0].upper()
	379	+ for p in (PARQUET / "options" / q).glob("*.parquet")})
	380	+ names = cached(f"opt_tickers\|{q}", build)
	381	+ if search:
	382	+ s = search.upper()
	383	+ names = [t for t in names if s in t]
	384	+ return {"quarter": q, "count": len(names[:limit]), "tickers": names[:limit]}
	385	+
	386	+
	387	+@app.get("/v1/options/chain/{ticker}", tags=["options"])
	388	+def opt_chain(ticker: str,
	389	+ trade_date: str \| None = Query(None, description="yyyy-mm-dd; default latest available"),
	390	+ expiry: str \| None = Query(None, description="Filter on expiry date"),
	391	+ call_put: str \| None = Query(None, pattern="^(c\|p)$"),
	392	+ strike_min: float \| None = None,
	393	+ strike_max: float \| None = None,
	394	+ min_volume: float \| None = None,
	395	+ limit: int = Query(20_000, ge=1),
	396	+ format: str = Query("json", pattern="^(json\|csv)$")):
	397	+ """Full end-of-day option chain for an underlying: quotes, bid/ask implied
	398	+ volatility, open interest, volume and Greeks (delta, gamma, vega, theta, rho)."""
	399	+ files = options_files(ticker)
	400	+ if not files:
	401	+ raise HTTPException(404, f"No options data for '{ticker.upper()}'")
	402	+ con = db()
	403	+ if trade_date is None:
	404	+ trade_date = con.execute(
	405	+ "SELECT max(trade_date) FROM read_parquet(?)", [files[-1]]
	406	+ ).fetchone()[0]
	407	+ files_q = [files[-1]]
	408	+ else:
	409	+ files_q = files
	410	+ conds, params = ["trade_date = ?"], [str(trade_date)]
	411	+ if expiry:
	412	+ conds.append("expiry = ?"); params.append(expiry)
	413	+ if call_put:
	414	+ conds.append("call_put = ?"); params.append(call_put)
	415	+ if strike_min is not None:
	416	+ conds.append("strike >= ?"); params.append(strike_min)
	417	+ if strike_max is not None:
	418	+ conds.append("strike <= ?"); params.append(strike_max)
	419	+ if min_volume is not None:
	420	+ conds.append("volume >= ?"); params.append(min_volume)
	421	+ rel = con.sql(
	422	+ f"SELECT * FROM read_parquet(?) WHERE {' AND '.join(conds)} "
	423	+ f"ORDER BY expiry, strike, call_put",
	424	+ params=[files_q, *params])
	425	+ return _rows_to_response(rel, limit, format, "expiry")
	426	+
	427	+
	428	+@app.get("/v1/options/expirations/{ticker}", tags=["options"])
	429	+def opt_expirations(ticker: str,
	430	+ trade_date: str \| None = Query(None, description="yyyy-mm-dd")):
	431	+ """Available expiry dates (optionally as of one trade date)."""
	432	+ files = options_files(ticker)
	433	+ if not files:
	434	+ raise HTTPException(404, f"No options data for '{ticker.upper()}'")
	435	+ con = db()
	436	+ if trade_date:
	437	+ rows = con.execute(
	438	+ "SELECT DISTINCT expiry FROM read_parquet(?) WHERE trade_date = ? ORDER BY expiry",
	439	+ [files, trade_date]).fetchall()
	440	+ else:
	441	+ rows = con.execute(
	442	+ "SELECT DISTINCT expiry FROM read_parquet(?) ORDER BY expiry",
	443	+ [files[-1]]).fetchall()
	444	+ return {"ticker": ticker.upper(), "expirations": [str(r[0]) for r in rows]}
	445	+
	446	+
	447	+@app.get("/v1/options/history/{ticker}", tags=["options"])
	448	+def opt_history(ticker: str,
	449	+ strike: float = Query(...),
	450	+ expiry: str = Query(..., description="yyyy-mm-dd"),
	451	+ call_put: str = Query(..., pattern="^(c\|p)$"),
	452	+ limit: int = Query(5_000, ge=1),
	453	+ format: str = Query("json", pattern="^(json\|csv)$")):
	454	+ """Daily time series for one specific contract across its whole life."""
	455	+ files = options_files(ticker)
	456	+ if not files:
	457	+ raise HTTPException(404, f"No options data for '{ticker.upper()}'")
	458	+ con = db()
	459	+ rel = con.sql(
	460	+ "SELECT * FROM read_parquet(?) WHERE strike = ? AND expiry = ? AND call_put = ? "
	461	+ "ORDER BY trade_date",
	462	+ params=[files, strike, expiry, call_put])
	463	+ return _rows_to_response(rel, limit, format, "trade_date")
	464	+
	465	+
	466	+# ------------------------------------------------------------------------------
	467	+# Static React platform (mounted last so it doesn't shadow the API)
	468	+# ------------------------------------------------------------------------------
	469	+
	470	+if WEB_DIST.is_dir():
	471	+ app.mount("/", StaticFiles(directory=str(WEB_DIST), html=True), name="web")
	472	+
	473	+
	474	+if __name__ == "__main__":
	475	+ import uvicorn
	476	+ uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("HFMD_PORT", 8090)))

added hfmarketdata/requirements.txt +4 −0

		`@@ -0,0 +1,4 @@`
	1	+fastapi>=0.115
	2	+uvicorn>=0.30
	3	+duckdb>=1.0
	4	+pandas>=2.0

added hfmarketdata/web/index.html +16 −0

		`@@ -0,0 +1,16 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+ <head>
	4	+ <meta charset="UTF-8" />
	5	+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
	6	+ <meta name="description" content="HF Market Data — free open API for high-frequency historical market data: stocks, ETFs, futures, crypto, indices, FX (1-minute to daily) and full options chains with Greeks since 2010." />
	7	+ <title>HF Market Data — Open High-Frequency Market Data API</title>
	8	+ <link rel="preconnect" href="https://fonts.googleapis.com" />
	9	+ <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
	10	+ <link href="https://fonts.googleapis.com/css2?family=Fraunces:ital,opsz,wght@0,9..144,400;0,9..144,500;0,9..144,600;1,9..144,400;1,9..144,500&family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;600&display=swap" rel="stylesheet" />
	11	+ </head>
	12	+ <body>
	13	+ <div id="root"></div>
	14	+ <script type="module" src="/src/main.jsx"></script>
	15	+ </body>
	16	+</html>

added hfmarketdata/web/package-lock.json +1857 −0

Large diff (1,857 lines) is not loaded by default to keep the page fast.Load diff

added hfmarketdata/web/package.json +21 −0

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

added hfmarketdata/web/src/App.jsx +403 −0

Large diff (403 lines) is not loaded by default to keep the page fast.Load diff

added hfmarketdata/web/src/main.jsx +10 −0

Large diff (10 lines) is not loaded by default to keep the page fast.Load diff

added hfmarketdata/web/src/spec.js +329 −0

Large diff (329 lines) is not loaded by default to keep the page fast.Load diff

added hfmarketdata/web/src/styles.css +457 −0

Large diff (457 lines) is not loaded by default to keep the page fast.Load diff

added hfmarketdata/web/vite.config.js +12 −0

Large diff (12 lines) is not loaded by default to keep the page fast.Load diff

added requirements.txt +2 −0

Large diff (3 lines) is not loaded by default to keep the page fast.Load diff

spb/hfmarketdata

HF Market Data — FirstRate full-history downloader, DuckDB/Parquet lake, open API and docs platform

Theme