SPB Git forge

spb/hfmarketdata

Public

Open high-frequency market data platform — FirstRate full-history downloader, DuckDB/Parquet lake, open REST API and React docs platform (www.hfmarketdata.io)

127commits 1branches 0releases
24.7 MBsize
maindefault branch
11 days agolast push
JavaScript 53.7% Python 38.3% CSS 4.6% TypeScript 3.1%

docs: notes de conception du module futures (alias, règles d'échéance, rolls, ajustements, limites)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed 20 days ago (Sep 4, 2026) parent daadb87

1 changed file +176 −0

added docs/futures.md +176 −0
@@ -0,0 +1,176 @@
1 +# Futures v2 — individual contracts, chains, continuous series, term structure
2 +
3 +Design notes for chantier 1 (`hfmarketdata/api/futures/`). The web team turns this into the
4 +"Futures individual contracts" guide; the reference documentation is generated from `/openapi.json`.
5 +
6 +## 1. Endpoints
7 +
8 +| Endpoint | Purpose |
9 +|---|---|
10 +| `GET /v1/futures/roots` | 142 roots with reference specs (name, exchange, asset class, size, tick, settlement, expiry rule, month cycle, RTH window, aliases, coverage). |
11 +| `GET /v1/futures/{root}/contracts` | Contracts of a root: expiration (+ source), last trading / first notice, real data range, status, 20-session volume, last OI, intervals, files. |
12 +| `GET /v1/futures/contract/{symbol}/bars` | OHLCV of one contract, `interval=1m|5m|30m|1h|1d`, `session=rth|eth|all`, `from`/`to`, cursor pagination, `format=json|csv|parquet`. |
13 +| `GET /v1/futures/contract/{symbol}/coverage` | What the lake really holds (per interval), gaps, explicit reasons for nulls. |
14 +| `GET /v1/futures/{root}/chain?as_of=` | Live contracts on a date, ordered by expiration, with last close/volume/OI. |
15 +| `GET /v1/futures/{root}/continuous?roll=&adjust=&depth=` | Continuous series stitched from individual contracts with a reproducible roll schedule (`meta.roll_dates`). |
16 +| `GET /v1/futures/{root}/term-structure?as_of=` | Forward curve: settle per contract, spread and annualised slope vs front, contango/backwardation flag. |
17 +
18 +Common: `{"data": [...], "meta": {"count", "next_cursor", ...}}`, `X-Row-Count`, errors
19 +`400 INVALID_CONTRACT_SYMBOL`, `404 CONTRACT_NOT_FOUND`, `404 ROOT_NOT_FOUND`, `400 INVALID_PARAMETER`,
20 +`400 ROW_LIMIT_EXCEEDED` (tier cap via `request.state.max_rows`). Limits: default 5 000 rows, max 200 000 (JSON),
21 +2 000 000 (CSV/Parquet). `continuous` has `request_cost = 2`.
22 +
23 +## 2. Symbols and root aliases (`symbols.py`)
24 +
25 +Accepted: `ESZ25`, `ESZ2025`, `ES_Z25`, `ES-Z25`, lower-case. Roots are 1–4 chars incl. digits (`C`, `6E`, `M2K`,
26 +`FDAX`, `SR3`). Month codes `F G H J K M N Q U V X Z`. Two-digit years 00–79 → 20xx, 80–99 → 19xx.
27 +
28 +The **canonical root is the lake (FirstRate) code** because the files are named that way. CME codes are aliases:
29 +
30 +| CME | lake | product | | CME | lake | product |
31 +|---|---|---|---|---|---|---|
32 +| 6E | E6 | Euro FX | | 6N | N6 | New Zealand dollar |
33 +| 6J | J1 | Japanese yen (full) | | 6M | MP | Mexican peso |
34 +| 6B | B6 | British pound | | 6L | BR | Brazilian real |
35 +| 6A | A6 | Australian dollar | | 6Z | T6 | South African rand |
36 +| 6C | AD | Canadian dollar | | ZB | US | 30-year T-bond |
37 +| 6S | E1 | Swiss franc | | EMD | EW | E-mini S&P MidCap 400 |
38 +| | | | | BRN | B | ICE Brent |
39 +
40 +`J7` and `E7` in the lake are the **E-mini** yen / euro (distinct products, not aliases of 6J/6E).
41 +Unknown lake roots (no reference entry, not in `meta/futures/futures.csv`): `CPO FBTM FID FNMY FOAM JB JG NK RU ST TWN ZK`
42 +→ `source="derived"`, `expiry_rule="data"`, name null.
43 +
44 +Responses always show the canonical form (`E6Z25`); `/roots` lists `aliases` per root.
45 +
46 +## 3. Reference table (`specs.py`)
47 +
48 +~120 roots with exchange specs (contract size + unit, tick size/value, settlement, month cycle, expiry rule,
49 +first-notice rule, business-day calendar `us|eurex`, RTH window in US/Eastern). Fields not known with certainty are
50 +`null` — never invented. Roots present in the lake but absent from the table get a derived placeholder.
51 +
52 +RTH windows (Eastern): equity index 09:30–16:00 (default), CL/NG/HO/RB 09:00–14:30, GC/SI/HG 08:20–13:30,
53 +treasuries & FX & DX 08:20–15:00, grains 09:30–14:20, livestock 09:30–14:05, Eurex 03:00–11:30, ICE softs product-specific
54 +(cotton is an overnight window 21:00–14:20, handled as wrap-around).
55 +
56 +## 4. Expiry rules (`expiry.py`, calendar in `calendar_us.py`)
57 +
58 +US calendar (CME/NYSE): New Year (Sun→Mon, Sat→none), MLK, Presidents, Good Friday, Memorial, Juneteenth (≥ 2022,
59 +observed), Independence (observed), Labor, Thanksgiving, Christmas (observed). Eurex: 1 Jan, Good Friday, Easter Monday,
60 +1 May, 24–26 Dec, 31 Dec. Weather / state-funeral NYSE closures are not included (CME futures kept trading).
61 +
62 +| rule | roots | definition | verified |
63 +|---|---|---|---|
64 +| `third_friday` | ES NQ YM RTY MES MNQ M2K EW ESG XA* MFS MME | 3rd Friday of the contract month (preceding business day if holiday) | ESZ24 → 2024-12-20 |
65 +| `third_friday_eurex` | FDAX FESX FDXM FDXS FXXP FCE FTI FTUK … | same, Eurex calendar | FDAXZ24 → 2024-12-20 |
66 +| `second_friday_minus_1` | NKD NIY | business day before the 2nd Friday | NKDZ24 → 2024-12-12 |
67 +| `cl_rule` | CL | 3 business days before the 25th of the preceding month (25th → prior business day if not one) | CLZ24 → 2024-11-20, CLF25 → 2024-12-19 |
68 +| `cl_minus_1` | MCL | one business day before CL | MCLZ24 → 2024-11-19 |
69 +| `ng_rule` | NG HH | 3 business days before the 1st of the contract month | NGZ24 → 2024-11-26 |
70 +| `ng_minus_1` | QG | one business day before NG | — |
71 +| `prior_month_last_business_day` | HO RB SB BR | last business day of the preceding month | HOZ24 → 2024-11-29, SBH25 → 2025-02-28 |
72 +| `bz_rule` | BZ B | last business day of the 2nd preceding month | BZZ24 → 2024-10-31 |
73 +| `metals_rule` | GC SI HG PL PA MGC SIL ALI | 3rd last business day of the contract month | GCZ24 → 2024-12-27, PLF25 → 2025-01-29 |
74 +| `treasury_rule` | ZN US(ZB) UB TN | 7th business day before the last business day | ZNZ24 → 2024-12-19 |
75 +| `last_business_day` | ZT ZF ZQ SR1 LE HRC | last business day of the contract month | ZTZ24 → 2024-12-31, LEZ24 → 2024-12-31 |
76 +| `grains_rule` | ZC ZS ZW KE ZM ZL ZO ZR XC RS | business day before the 15th | ZCZ24 → 2024-12-13 |
77 +| `fx_rule` | E6 J1 B6 A6 AD E1 N6 MP T6 E7 J7 NOK SEK RP RY PJY CNH DX | 2nd business day before the 3rd Wednesday | E6Z24 → 2024-12-16 |
78 +| `vx_rule` | VX VXM FVSA | Wednesday 30 days before the 3rd Friday of the following month (holiday-adjusted) | VXZ24 → 2024-12-18, VXH25 → 2025-03-18 |
79 +| `last_friday` | BTC MBT MET | last Friday of the contract month | BTCZ24 → 2024-12-27 |
80 +| `he_rule` | HE PRK | 10th business day of the contract month | HEZ24 → 2024-12-13 |
81 +| `gf_rule` | GF | last Thursday (Nov: Thursday before Thanksgiving) | GFX24 → 2024-11-21 |
82 +| `kc_rule` / `cc_rule` / `ct_rule` / `oj_rule` | KC / CC C / CT / OJ | 8 / 11 / 16 / 14 business days before the last business day | KCZ24 12-18, CCZ24 12-13, CTZ24 12-06, OJF25 2025-01-10 |
83 +| `bund_rule` | FGBL FGBM FGBS FGBX FOAT FBTP FBTS FBON | 2 exchange days before the 10th (delivery day) | FGBLZ24 → 2024-12-06 |
84 +| `sr3_rule` | SR3 | business day before the 3rd Wednesday of the 3rd following month | SR3Z24 → 2025-03-18 |
85 +| `data` | everything else | `expiration_date = last_data_date`, `expiration_source="data"` | |
86 +
87 +First notice: treasuries / grains / metals → last business day of the preceding month; CL → business day after the
88 +last trade; KC → 7 business days before the first business day of the contract month; cash-settled → null.
89 +Every contract row records `expiration_source` (`rule` | `data`) and `last_trading_date` (null when `data`).
90 +
91 +Known lake quirks: FirstRate sometimes keeps 1–2 post-expiry rows (HEZ24 last bar 12-17 vs rule 12-13, BZZ24 11-01 vs
92 +10-31, GFX24 11-22 vs 11-21, ZQZ24 2025-01-02 vs 12-31). The rule wins for `expiration_date`; `last_data_date` is
93 +reported separately in `/coverage`.
94 +
95 +## 5. Metadata tables (`models.py`) and backfill (`backfill.py`, `scripts/backfill_contracts.py`)
96 +
97 +`futures_roots`, `futures_contracts`, `futures_contract_gaps` as in UPGRADE-PLAN §1.1–1.3, plus documented extras:
98 +`roots.aliases/contract_size_unit/first_notice_rule/calendar/rth_start/rth_end`, `contracts.expiration_source/bars_1day`,
99 +and `contracts.timeframes` is a JSON object `{tf: {first, last, rows}}` (needed by `/coverage`).
100 +
101 +Backfill = one DuckDB aggregate per `{tf}/{archive|update}` directory (`read_parquet('dir/*.parquet', filename=true)`,
102 +min/max/count on `datetime` only), one scan of all `1day` files (archive ∪ update, dedup on `datetime`, update wins)
103 +for 20-session volume, last OI and session lists (gaps > 3 business days), then SQLite upserts. Status: `expired` when
104 +`last_data_date < today − 7` **and** the contract month/expiration has passed, else `active`. `--roots ES,CL` and
105 +`--since YYYY-MM-DD` (mtime filter) give incremental runs; `run_backfill()` is the library entry point.
106 +
107 +Production (M3U96b):
108 +
109 +```bash
110 +ssh M3U96b
111 +cd ~/hfmarketdata && HFMD_DATA_ROOT=~/firstratedata venv/bin/python scripts/backfill_contracts.py -v
112 +# nightly refresh (PM2 cron or launchd): --since $(date -v-2d +%F)
113 +```
114 +
115 +## 6. Sessions and time zones (`service.py`)
116 +
117 +Lake timestamps are naive US/Eastern. Intraday output is localised with `zoneinfo("America/New_York")` (DST-aware)
118 +and emitted in UTC (`2024-12-19T14:30:00Z`); daily bars stay dates. `from`/`to`: `YYYY-MM-DD` = Eastern trading
119 +dates (inclusive), ISO datetimes with `Z`/offset are converted, naive datetimes are UTC. `session=rth` keeps bars whose
120 +start time is in the root's RTH window; `eth` the complement; ignored for `1d`. Cursor = last bar datetime.
121 +
122 +## 7. Continuous series (`rolls.py`)
123 +
124 +Contracts of the root are ordered by `expiration_date`; the front holds contract *k* from the previous roll until
125 +`roll_k`, the first session on which contract *k+1* is the front:
126 +
127 +* `calendar` — held through the expiration date; roll on the next session.
128 +* `first_notice` — roll on `first_notice_date` (held through the session before), else calendar.
129 +* `volume` (default) / `open_interest` — roll after the 2nd consecutive session where the next contract's metric
130 + exceeds the front's, evaluated on daily data in the **last 30 sessions** of the front (far-dated noise cannot
131 + trigger a roll a year early); never later than the calendar roll.
132 +
133 +`depth=2|3` uses the same windows with the 2nd/3rd contract of the sequence. Intraday intervals apply the daily
134 +schedule by Eastern calendar date of the bar (evening-session bars of day *D* belong to *D* — documented simplification).
135 +Contracts already expired when the previous roll happens are skipped.
136 +
137 +### Adjustments — worked example
138 +
139 +Three contracts, closes: A = 100 (flat), B = 105, C = 110. Calendar rolls A→B on 2024-03-18 and B→C on 2024-06-24.
140 +
141 +* gap_k = close_new − close_old on the **last session of the old segment** where both traded (nearest common session
142 + within 5 sessions): gap₁ = 105 − 100 = 5 (session 2024-03-15), gap₂ = 110 − 105 = 5.
143 +* `back_adjusted` (additive): segment offsets are the cumulative gaps of *later* rolls: C: 0, B: +5, A: +10.
144 + A bars become 110, B bars 110, C unchanged → no jump at the rolls; the latest contract is always unadjusted.
145 +* `ratio_adjusted` (multiplicative): factors 105/100 and 110/105; A × 1.05 × 1.0476 = 110, B × 1.0476 = 110.
146 +* Volume and open interest are never adjusted. If no common session exists, `gap: null`, `adjusted: false` and that
147 + roll contributes nothing (no invented value).
148 +
149 +`meta.roll_dates` lists `{date, from_symbol, to_symbol, gap, ratio, gap_session, adjusted}` for rolls inside the
150 +returned window; `meta.segments`, `meta.rolls_total` and `meta.unadjusted_symbol` complete the picture; each row carries
151 +its source `symbol`.
152 +
153 +## 8. Term structure
154 +
155 +For every live contract on `as_of`: `settle` = last daily close ≤ `as_of` (≤ 7 days old, else null),
156 +`spread_vs_front = settle − settle_front`, `slope_annualized = (settle / settle_front − 1) × 365 / (dte − dte_front)`.
157 +`meta.structure` = `contango` if the 2nd contract settles above the front, `backwardation` below, `flat` equal.
158 +
159 +## 9. Tests
160 +
161 +`tests/test_futures_symbols.py` (parsing, aliases, bad input), `tests/test_futures_expiry.py` (calendar + 39 real dates),
162 +`tests/test_futures_rolls.py` (schedules, depth, gaps, adjustment math, gap detection), `tests/test_futures_api.py`
163 +(backfill → 7 endpoints → formats → pagination → errors → OpenAPI), `tests/test_futures_perf.py` (10 000 bars < 300 ms,
164 +`-m "not slow"` to skip). Fixture lake: `tests/fixtures/make_fixtures.py` (ES/CL/NG/E6 contracts with realistic
165 +volume/OI life-cycles and ETH bars).
166 +
167 +## 10. Known limitations
168 +
169 +* Expiry rules use one US (CME) calendar and one Eurex calendar; ICE Europe/UK, Euronext, HKEX bank holidays are not
170 + modelled (rare 1-day differences possible on London cocoa `C`, Brent `B`, `CNH`). Products without an implemented
171 + rule (Nikkei variants excepted) fall back to `data`.
172 +* First-notice dates are only computed for treasuries, grains, metals, CL and KC.
173 +* Intraday continuous series attribute evening-session bars to the calendar date of the bar (not the CME trading date).
174 +* Volume/OI rolls are evaluated inside the last 30 sessions of the front; products whose liquidity migrates earlier
175 + (e.g. some ags around FND) should use `first_notice`.
176 +* `contract_size`/`tick` values are null for ~25 minor roots (select-sector minis, small Eurex indices, Euribor…).
177