SPB Git forge

spb/hfmarketdata

Public

Open high-frequency market data platform — FirstRate full-history downloader, DuckDB/Parquet lake, open REST API and React docs platform (www.hfmarketdata.io)

127commits 1branches 0releases
24.7 MBsize
maindefault branch
11 days agolast push
JavaScript 53.7% Python 38.3% CSS 4.6% TypeScript 3.1%

# name: hfmd-continuous-futures description: Build and compare continuous futures series from HF Market Data — server-side v2 continuous endpoint (roll=volume/open_interest/calendar, adjust=none/back_adjusted/ratio, depth) versus the vendor continuous series, or stitched locally from individual contracts with explicit roll dates. Use when the user asks for a continuous futures price, questions how rolls/adjustments were done, or needs a series suitable for backtesting.

# hfmd-continuous-futures

A futures "price history" is a construction: someone chose when to jump from one contract to the next (roll) and what to do with the price gap at the jump (adjust). This skill makes those choices explicit, reproducible and comparable.

# Vocabulary (say this to the user)

  • Roll rule — volume: switch when the next contract's volume exceeds the front's (most common, tracks where liquidity is); open_interest: same with OI (smoother, lags a bit); calendar: N business days before expiry/first notice (deterministic, what many CTAs do).
  • Adjustment — none: raw prices, discontinuous at rolls (fine for levels, wrong for returns); back_adjusted (additive): shift all earlier history by the roll gap so the series is continuous in points (good for P&L in ticks, can go negative on long histories); ratio (multiplicative): scale earlier history by the gap ratio, continuous in percent (best for returns/backtests, levels are not real prices).
  • Depth — 1 = front month, 2 = second month … (depth 2 avoids expiry noise for spread work).
  • Vendor series (v1 /v1/bars/futures/{ROOT}?adjustment=contin_UNadj|contin_adj_ratio|contin_adj_absolute) are FirstRate Data's own construction — a fixed rule you cannot change. The v2 endpoint lets you pick.

# When to use

  • "Get me continuous ES since 2015", "why does the crude series jump in April?", "back-adjust NG for a backtest", "compare volume-roll vs calendar-roll on CL"

# Steps

  1. Confirm root (/v1/futures/roots or search_symbols(asset=futures)), timeframe, range, and the intended use (levels → none; P&L in points → back_adjusted; returns → ratio).
  2. Server-side build + comparison: python3 scripts/continuous_compare.py --root CL --start 2018-01-01 --rolls volume,calendar --adjusts back_adjusted,ratio --vendor contin_adj_ratio --plot cl.png
    • prints each series' roll dates (from meta.roll_dates), the number of rolls per year, the mean absolute gap at rolls, and the return correlation / tracking difference between the variants and the vendor series.
  3. If the v2 endpoint is not available on the server yet (404 NOT_FOUND), stitch locally to show the methodology: python3 scripts/stitch_local.py --root CL --start 2023-01-01 --roll volume --adjust ratio --out cl_local.csv (lists contracts via /v1/futures/{root}/contracts, pulls each contract's daily bars, rolls on volume crossover, adjusts; prints the roll table). This costs one request per contract — needs an API key for long histories.
  4. Report: chosen rule + why, roll dates table, adjusted vs raw last price (to remind that adjusted levels ≠ tradable prices), and any gap in coverage (/v1/futures/contract/{symbol}/coverage).

# Examples

bash
# ES front month, volume roll, ratio-adjusted, daily since 2015 → CSV for a backtest
python3 scripts/continuous_compare.py --root ES --start 2015-01-01 --rolls volume --adjusts ratio --out es_cont.csv

# Second-month natural gas vs front month (seasonal spread work)
python3 scripts/continuous_compare.py --root NG --depth 2 --rolls open_interest --adjusts none --start 2020-01-01

# How different are the vendor's absolute-adjusted and the v2 back_adjusted series?
python3 scripts/continuous_compare.py --root CL --rolls volume --adjusts back_adjusted --vendor contin_adj_absolute --start 2019-01-01

# Gotchas

  • Never compute returns on none/contin_UNadj: the roll gap (often 0.5-3 %) is not a market move.
  • Back-adjusted (additive) series on long histories can go negative (CL 2000-2020 does) — use ratio for percent returns.
  • Roll dates depend on the timeframe used to measure volume (daily is standard); intraday continuous series inherit the daily roll calendar.
  • Coverage: contracts have data since 2010 (archive ≤ 2025 and update ≥ 2025 files are already merged by the API); a missing contract is reported as a gap, not filled.
  • FirstRate roots may differ from exchange codes (E6 is euro FX 6E; the API accepts both).