# Futures v2 — individual contracts, chains, continuous series, term structure

Design notes for chantier 1 (`hfmarketdata/api/futures/`). The web team turns this into the
"Futures individual contracts" guide; the reference documentation is generated from `/openapi.json`.

## 1. Endpoints

| Endpoint | Purpose |
|---|---|
| `GET /v1/futures/roots` | 142 roots with reference specs (name, exchange, asset class, size, tick, settlement, expiry rule, month cycle, RTH window, aliases, coverage). |
| `GET /v1/futures/{root}/contracts` | Contracts of a root: expiration (+ source), last trading / first notice, real data range, status, 20-session volume, last OI, intervals, files. |
| `GET /v1/futures/contract/{symbol}/bars` | OHLCV of one contract, `interval=1m|5m|30m|1h|1d`, `session=rth|eth|all`, `from`/`to`, cursor pagination, `format=json|csv|parquet`. |
| `GET /v1/futures/contract/{symbol}/coverage` | What the lake really holds (per interval), gaps, explicit reasons for nulls. |
| `GET /v1/futures/{root}/chain?as_of=` | Live contracts on a date, ordered by expiration, with last close/volume/OI. |
| `GET /v1/futures/{root}/continuous?roll=&adjust=&depth=` | Continuous series stitched from individual contracts with a reproducible roll schedule (`meta.roll_dates`). |
| `GET /v1/futures/{root}/term-structure?as_of=` | Forward curve: settle per contract, spread and annualised slope vs front, contango/backwardation flag. |

Common: `{"data": [...], "meta": {"count", "next_cursor", ...}}`, `X-Row-Count`, errors
`400 INVALID_CONTRACT_SYMBOL`, `404 CONTRACT_NOT_FOUND`, `404 ROOT_NOT_FOUND`, `400 INVALID_PARAMETER`,
`400 ROW_LIMIT_EXCEEDED` (tier cap via `request.state.max_rows`). Limits: default 5 000 rows, max 200 000 (JSON),
2 000 000 (CSV/Parquet). `continuous` has `request_cost = 2`.

## 2. Symbols and root aliases (`symbols.py`)

Accepted: `ESZ25`, `ESZ2025`, `ES_Z25`, `ES-Z25`, lower-case. Roots are 1–4 chars incl. digits (`C`, `6E`, `M2K`,
`FDAX`, `SR3`). Month codes `F G H J K M N Q U V X Z`. Two-digit years 00–79 → 20xx, 80–99 → 19xx.

The **canonical root is the lake (FirstRate) code** because the files are named that way. CME codes are aliases:

| CME | lake | product | | CME | lake | product |
|---|---|---|---|---|---|---|
| 6E | E6 | Euro FX | | 6N | N6 | New Zealand dollar |
| 6J | J1 | Japanese yen (full) | | 6M | MP | Mexican peso |
| 6B | B6 | British pound | | 6L | BR | Brazilian real |
| 6A | A6 | Australian dollar | | 6Z | T6 | South African rand |
| 6C | AD | Canadian dollar | | ZB | US | 30-year T-bond |
| 6S | E1 | Swiss franc | | EMD | EW | E-mini S&P MidCap 400 |
| | | | | BRN | B | ICE Brent |

`J7` and `E7` in the lake are the **E-mini** yen / euro (distinct products, not aliases of 6J/6E).
Unknown lake roots (no reference entry, not in `meta/futures/futures.csv`): `CPO FBTM FID FNMY FOAM JB JG NK RU ST TWN ZK`
→ `source="derived"`, `expiry_rule="data"`, name null.

Responses always show the canonical form (`E6Z25`); `/roots` lists `aliases` per root.

## 3. Reference table (`specs.py`)

~120 roots with exchange specs (contract size + unit, tick size/value, settlement, month cycle, expiry rule,
first-notice rule, business-day calendar `us|eurex`, RTH window in US/Eastern). Fields not known with certainty are
`null` — never invented. Roots present in the lake but absent from the table get a derived placeholder.

RTH windows (Eastern): equity index 09:30–16:00 (default), CL/NG/HO/RB 09:00–14:30, GC/SI/HG 08:20–13:30,
treasuries & FX & DX 08:20–15:00, grains 09:30–14:20, livestock 09:30–14:05, Eurex 03:00–11:30, ICE softs product-specific
(cotton is an overnight window 21:00–14:20, handled as wrap-around).

## 4. Expiry rules (`expiry.py`, calendar in `calendar_us.py`)

US calendar (CME/NYSE): New Year (Sun→Mon, Sat→none), MLK, Presidents, Good Friday, Memorial, Juneteenth (≥ 2022,
observed), Independence (observed), Labor, Thanksgiving, Christmas (observed). Eurex: 1 Jan, Good Friday, Easter Monday,
1 May, 24–26 Dec, 31 Dec. Weather / state-funeral NYSE closures are not included (CME futures kept trading).

| rule | roots | definition | verified |
|---|---|---|---|
| `third_friday` | ES NQ YM RTY MES MNQ M2K EW ESG XA* MFS MME | 3rd Friday of the contract month (preceding business day if holiday) | ESZ24 → 2024-12-20 |
| `third_friday_eurex` | FDAX FESX FDXM FDXS FXXP FCE FTI FTUK … | same, Eurex calendar | FDAXZ24 → 2024-12-20 |
| `second_friday_minus_1` | NKD NIY | business day before the 2nd Friday | NKDZ24 → 2024-12-12 |
| `cl_rule` | CL | 3 business days before the 25th of the preceding month (25th → prior business day if not one) | CLZ24 → 2024-11-20, CLF25 → 2024-12-19 |
| `cl_minus_1` | MCL | one business day before CL | MCLZ24 → 2024-11-19 |
| `ng_rule` | NG HH | 3 business days before the 1st of the contract month | NGZ24 → 2024-11-26 |
| `ng_minus_1` | QG | one business day before NG | — |
| `prior_month_last_business_day` | HO RB SB BR | last business day of the preceding month | HOZ24 → 2024-11-29, SBH25 → 2025-02-28 |
| `bz_rule` | BZ B | last business day of the 2nd preceding month | BZZ24 → 2024-10-31 |
| `metals_rule` | GC SI HG PL PA MGC SIL ALI | 3rd last business day of the contract month | GCZ24 → 2024-12-27, PLF25 → 2025-01-29 |
| `treasury_rule` | ZN US(ZB) UB TN | 7th business day before the last business day | ZNZ24 → 2024-12-19 |
| `last_business_day` | ZT ZF ZQ SR1 LE HRC | last business day of the contract month | ZTZ24 → 2024-12-31, LEZ24 → 2024-12-31 |
| `grains_rule` | ZC ZS ZW KE ZM ZL ZO ZR XC RS | business day before the 15th | ZCZ24 → 2024-12-13 |
| `fx_rule` | E6 J1 B6 A6 AD E1 N6 MP T6 E7 J7 NOK SEK RP RY PJY CNH DX | 2nd business day before the 3rd Wednesday | E6Z24 → 2024-12-16 |
| `vx_rule` | VX VXM FVSA | Wednesday 30 days before the 3rd Friday of the following month (holiday-adjusted) | VXZ24 → 2024-12-18, VXH25 → 2025-03-18 |
| `last_friday` | BTC MBT MET | last Friday of the contract month | BTCZ24 → 2024-12-27 |
| `he_rule` | HE PRK | 10th business day of the contract month | HEZ24 → 2024-12-13 |
| `gf_rule` | GF | last Thursday (Nov: Thursday before Thanksgiving) | GFX24 → 2024-11-21 |
| `kc_rule` / `cc_rule` / `ct_rule` / `oj_rule` | KC / CC C / CT / OJ | 8 / 11 / 16 / 14 business days before the last business day | KCZ24 12-18, CCZ24 12-13, CTZ24 12-06, OJF25 2025-01-10 |
| `bund_rule` | FGBL FGBM FGBS FGBX FOAT FBTP FBTS FBON | 2 exchange days before the 10th (delivery day) | FGBLZ24 → 2024-12-06 |
| `sr3_rule` | SR3 | business day before the 3rd Wednesday of the 3rd following month | SR3Z24 → 2025-03-18 |
| `data` | everything else | `expiration_date = last_data_date`, `expiration_source="data"` | |

First notice: treasuries / grains / metals → last business day of the preceding month; CL → business day after the
last trade; KC → 7 business days before the first business day of the contract month; cash-settled → null.
Every contract row records `expiration_source` (`rule` | `data`) and `last_trading_date` (null when `data`).

Known lake quirks: FirstRate sometimes keeps 1–2 post-expiry rows (HEZ24 last bar 12-17 vs rule 12-13, BZZ24 11-01 vs
10-31, GFX24 11-22 vs 11-21, ZQZ24 2025-01-02 vs 12-31). The rule wins for `expiration_date`; `last_data_date` is
reported separately in `/coverage`.

## 5. Metadata tables (`models.py`) and backfill (`backfill.py`, `scripts/backfill_contracts.py`)

`futures_roots`, `futures_contracts`, `futures_contract_gaps` as in UPGRADE-PLAN §1.1–1.3, plus documented extras:
`roots.aliases/contract_size_unit/first_notice_rule/calendar/rth_start/rth_end`, `contracts.expiration_source/bars_1day`,
and `contracts.timeframes` is a JSON object `{tf: {first, last, rows}}` (needed by `/coverage`).

Backfill = one DuckDB aggregate per `{tf}/{archive|update}` directory (`read_parquet('dir/*.parquet', filename=true)`,
min/max/count on `datetime` only), one scan of all `1day` files (archive ∪ update, dedup on `datetime`, update wins)
for 20-session volume, last OI and session lists (gaps > 3 business days), then SQLite upserts. Status: `expired` when
`last_data_date < today − 7` **and** the contract month/expiration has passed, else `active`. `--roots ES,CL` and
`--since YYYY-MM-DD` (mtime filter) give incremental runs; `run_backfill()` is the library entry point.

Production (M3U96b):

```bash
ssh M3U96b
cd ~/hfmarketdata && HFMD_DATA_ROOT=~/firstratedata venv/bin/python scripts/backfill_contracts.py -v
# nightly refresh (PM2 cron or launchd): --since $(date -v-2d +%F)
```

## 6. Sessions and time zones (`service.py`)

Lake timestamps are naive US/Eastern. Intraday output is localised with `zoneinfo("America/New_York")` (DST-aware)
and emitted in UTC (`2024-12-19T14:30:00Z`); daily bars stay dates. `from`/`to`: `YYYY-MM-DD` = Eastern trading
dates (inclusive), ISO datetimes with `Z`/offset are converted, naive datetimes are UTC. `session=rth` keeps bars whose
start time is in the root's RTH window; `eth` the complement; ignored for `1d`. Cursor = last bar datetime.

## 7. Continuous series (`rolls.py`)

Contracts of the root are ordered by `expiration_date`; the front holds contract *k* from the previous roll until
`roll_k`, the first session on which contract *k+1* is the front:

* `calendar` — held through the expiration date; roll on the next session.
* `first_notice` — roll on `first_notice_date` (held through the session before), else calendar.
* `volume` (default) / `open_interest` — roll after the 2nd consecutive session where the next contract's metric
  exceeds the front's, evaluated on daily data in the **last 30 sessions** of the front (far-dated noise cannot
  trigger a roll a year early); never later than the calendar roll.

`depth=2|3` uses the same windows with the 2nd/3rd contract of the sequence. Intraday intervals apply the daily
schedule by Eastern calendar date of the bar (evening-session bars of day *D* belong to *D* — documented simplification).
Contracts already expired when the previous roll happens are skipped.

### Adjustments — worked example

Three contracts, closes: A = 100 (flat), B = 105, C = 110. Calendar rolls A→B on 2024-03-18 and B→C on 2024-06-24.

* gap_k = close_new − close_old on the **last session of the old segment** where both traded (nearest common session
  within 5 sessions): gap₁ = 105 − 100 = 5 (session 2024-03-15), gap₂ = 110 − 105 = 5.
* `back_adjusted` (additive): segment offsets are the cumulative gaps of *later* rolls: C: 0, B: +5, A: +10.
  A bars become 110, B bars 110, C unchanged → no jump at the rolls; the latest contract is always unadjusted.
* `ratio_adjusted` (multiplicative): factors 105/100 and 110/105; A × 1.05 × 1.0476 = 110, B × 1.0476 = 110.
* Volume and open interest are never adjusted. If no common session exists, `gap: null`, `adjusted: false` and that
  roll contributes nothing (no invented value).

`meta.roll_dates` lists `{date, from_symbol, to_symbol, gap, ratio, gap_session, adjusted}` for rolls inside the
returned window; `meta.segments`, `meta.rolls_total` and `meta.unadjusted_symbol` complete the picture; each row carries
its source `symbol`.

## 8. Term structure

For every live contract on `as_of`: `settle` = last daily close ≤ `as_of` (≤ 7 days old, else null),
`spread_vs_front = settle − settle_front`, `slope_annualized = (settle / settle_front − 1) × 365 / (dte − dte_front)`.
`meta.structure` = `contango` if the 2nd contract settles above the front, `backwardation` below, `flat` equal.

## 9. Tests

`tests/test_futures_symbols.py` (parsing, aliases, bad input), `tests/test_futures_expiry.py` (calendar + 39 real dates),
`tests/test_futures_rolls.py` (schedules, depth, gaps, adjustment math, gap detection), `tests/test_futures_api.py`
(backfill → 7 endpoints → formats → pagination → errors → OpenAPI), `tests/test_futures_perf.py` (10 000 bars < 300 ms,
`-m "not slow"` to skip). Fixture lake: `tests/fixtures/make_fixtures.py` (ES/CL/NG/E6 contracts with realistic
volume/OI life-cycles and ETH bars).

## 10. Known limitations

* Expiry rules use one US (CME) calendar and one Eurex calendar; ICE Europe/UK, Euronext, HKEX bank holidays are not
  modelled (rare 1-day differences possible on London cocoa `C`, Brent `B`, `CNH`). Products without an implemented
  rule (Nikkei variants excepted) fall back to `data`.
* First-notice dates are only computed for treasuries, grains, metals, CL and KC.
* Intraday continuous series attribute evening-session bars to the calendar date of the bar (not the CME trading date).
* Volume/OI rolls are evaluated inside the last 30 sessions of the front; products whose liquidity migrates earlier
  (e.g. some ags around FND) should use `first_notice`.
* `contract_size`/`tick` values are null for ~25 minor roots (select-sector minis, small Eurex indices, Euribor…).