# Futures v2 — individual contracts, chains, continuous series, term structure

Design notes for chantier 1 (hfmarketdata/api/futures/). The web team turns this into the "Futures individual contracts" guide; the reference documentation is generated from /openapi.json.

# 1. Endpoints

Common: {"data": [...], "meta": {"count", "next_cursor", ...}}, X-Row-Count, errors 400 INVALID_CONTRACT_SYMBOL, 404 CONTRACT_NOT_FOUND, 404 ROOT_NOT_FOUND, 400 INVALID_PARAMETER, 400 ROW_LIMIT_EXCEEDED (tier cap via request.state.max_rows). Limits: default 5 000 rows, max 200 000 (JSON), 2 000 000 (CSV/Parquet). continuous has request_cost = 2.

# 2. Symbols and root aliases (symbols.py)

Accepted: ESZ25, ESZ2025, ES_Z25, ES-Z25, lower-case. Roots are 1–4 chars incl. digits (C, 6E, M2K, FDAX, SR3). Month codes F G H J K M N Q U V X Z. Two-digit years 00–79 → 20xx, 80–99 → 19xx.

The canonical root is the lake (FirstRate) code because the files are named that way. CME codes are aliases:

J7 and E7 in the lake are the E-mini yen / euro (distinct products, not aliases of 6J/6E). Unknown lake roots (no reference entry, not in meta/futures/futures.csv): CPO FBTM FID FNMY FOAM JB JG NK RU ST TWN ZK → source="derived", expiry_rule="data", name null.

Responses always show the canonical form (E6Z25); /roots lists aliases per root.

# 3. Reference table (specs.py)

~120 roots with exchange specs (contract size + unit, tick size/value, settlement, month cycle, expiry rule, first-notice rule, business-day calendar us|eurex, RTH window in US/Eastern). Fields not known with certainty are null — never invented. Roots present in the lake but absent from the table get a derived placeholder.

RTH windows (Eastern): equity index 09:30–16:00 (default), CL/NG/HO/RB 09:00–14:30, GC/SI/HG 08:20–13:30, treasuries & FX & DX 08:20–15:00, grains 09:30–14:20, livestock 09:30–14:05, Eurex 03:00–11:30, ICE softs product-specific (cotton is an overnight window 21:00–14:20, handled as wrap-around).

# 4. Expiry rules (expiry.py, calendar in calendar_us.py)

US calendar (CME/NYSE): New Year (Sun→Mon, Sat→none), MLK, Presidents, Good Friday, Memorial, Juneteenth (≥ 2022, observed), Independence (observed), Labor, Thanksgiving, Christmas (observed). Eurex: 1 Jan, Good Friday, Easter Monday, 1 May, 24–26 Dec, 31 Dec. Weather / state-funeral NYSE closures are not included (CME futures kept trading).

First notice: treasuries / grains / metals → last business day of the preceding month; CL → business day after the last trade; KC → 7 business days before the first business day of the contract month; cash-settled → null. Every contract row records expiration_source (rule | data) and last_trading_date (null when data).

Known lake quirks: FirstRate sometimes keeps 1–2 post-expiry rows (HEZ24 last bar 12-17 vs rule 12-13, BZZ24 11-01 vs 10-31, GFX24 11-22 vs 11-21, ZQZ24 2025-01-02 vs 12-31). The rule wins for expiration_date; last_data_date is reported separately in /coverage.

# 5. Metadata tables (models.py) and backfill (backfill.py, scripts/backfill_contracts.py)

futures_roots, futures_contracts, futures_contract_gaps as in UPGRADE-PLAN §1.1–1.3, plus documented extras: roots.aliases/contract_size_unit/first_notice_rule/calendar/rth_start/rth_end, contracts.expiration_source/bars_1day, and contracts.timeframes is a JSON object {tf: {first, last, rows}} (needed by /coverage).

Backfill = one DuckDB aggregate per {tf}/{archive|update} directory (read_parquet('dir/*.parquet', filename=true), min/max/count on datetime only), one scan of all 1day files (archive ∪ update, dedup on datetime, update wins) for 20-session volume, last OI and session lists (gaps > 3 business days), then SQLite upserts. Status: expired when last_data_date < today − 7 and the contract month/expiration has passed, else active. --roots ES,CL and --since YYYY-MM-DD (mtime filter) give incremental runs; run_backfill() is the library entry point.

Production (M3U96b):

ssh M3U96b
cd ~/hfmarketdata && HFMD_DATA_ROOT=~/firstratedata venv/bin/python scripts/backfill_contracts.py -v
# nightly refresh (PM2 cron or launchd): --since $(date -v-2d +%F)

# 6. Sessions and time zones (service.py)

Lake timestamps are naive US/Eastern. Intraday output is localised with zoneinfo("America/New_York") (DST-aware) and emitted in UTC (2024-12-19T14:30:00Z); daily bars stay dates. from/to: YYYY-MM-DD = Eastern trading dates (inclusive), ISO datetimes with Z/offset are converted, naive datetimes are UTC. session=rth keeps bars whose start time is in the root's RTH window; eth the complement; ignored for 1d. Cursor = last bar datetime.

# 7. Continuous series (rolls.py)

Contracts of the root are ordered by expiration_date; the front holds contract k from the previous roll until roll_k, the first session on which contract k+1 is the front:

• calendar — held through the expiration date; roll on the next session.
• first_notice — roll on first_notice_date (held through the session before), else calendar.
• volume (default) / open_interest — roll after the 2nd consecutive session where the next contract's metric exceeds the front's, evaluated on daily data in the last 30 sessions of the front (far-dated noise cannot trigger a roll a year early); never later than the calendar roll.

depth=2|3 uses the same windows with the 2nd/3rd contract of the sequence. Intraday intervals apply the daily schedule by Eastern calendar date of the bar (evening-session bars of day D belong to D — documented simplification). Contracts already expired when the previous roll happens are skipped.

# Adjustments — worked example

Three contracts, closes: A = 100 (flat), B = 105, C = 110. Calendar rolls A→B on 2024-03-18 and B→C on 2024-06-24.

• gap_k = close_new − close_old on the last session of the old segment where both traded (nearest common session within 5 sessions): gap₁ = 105 − 100 = 5 (session 2024-03-15), gap₂ = 110 − 105 = 5.
• back_adjusted (additive): segment offsets are the cumulative gaps of later rolls: C: 0, B: +5, A: +10. A bars become 110, B bars 110, C unchanged → no jump at the rolls; the latest contract is always unadjusted.
• ratio_adjusted (multiplicative): factors 105/100 and 110/105; A × 1.05 × 1.0476 = 110, B × 1.0476 = 110.
• Volume and open interest are never adjusted. If no common session exists, gap: null, adjusted: false and that roll contributes nothing (no invented value).

meta.roll_dates lists {date, from_symbol, to_symbol, gap, ratio, gap_session, adjusted} for rolls inside the returned window; meta.segments, meta.rolls_total and meta.unadjusted_symbol complete the picture; each row carries its source symbol.

# 8. Term structure

For every live contract on as_of: settle = last daily close ≤ as_of (≤ 7 days old, else null), spread_vs_front = settle − settle_front, slope_annualized = (settle / settle_front − 1) × 365 / (dte − dte_front). meta.structure = contango if the 2nd contract settles above the front, backwardation below, flat equal.

# 9. Tests

tests/test_futures_symbols.py (parsing, aliases, bad input), tests/test_futures_expiry.py (calendar + 39 real dates), tests/test_futures_rolls.py (schedules, depth, gaps, adjustment math, gap detection), tests/test_futures_api.py (backfill → 7 endpoints → formats → pagination → errors → OpenAPI), tests/test_futures_perf.py (10 000 bars < 300 ms, -m "not slow" to skip). Fixture lake: tests/fixtures/make_fixtures.py (ES/CL/NG/E6 contracts with realistic volume/OI life-cycles and ETH bars).

# 10. Known limitations

• Expiry rules use one US (CME) calendar and one Eurex calendar; ICE Europe/UK, Euronext, HKEX bank holidays are not modelled (rare 1-day differences possible on London cocoa C, Brent B, CNH). Products without an implemented rule (Nikkei variants excepted) fall back to data.
• First-notice dates are only computed for treasuries, grains, metals, CL and KC.
• Intraday continuous series attribute evening-session bars to the calendar date of the bar (not the CME trading date).
• Volume/OI rolls are evaluated inside the last 30 sessions of the front; products whose liquidity migrates earlier (e.g. some ags around FND) should use first_notice.
• contract_size/tick values are null for ~25 minor roots (select-sector minis, small Eurex indices, Euribor…).