docs: notes de conception du module futures (alias, règles d'échéance, rolls, ajustements, limites)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 changed file +176 −0
added
docs/futures.md
+176 −0
@@ -0,0 +1,176 @@ | ||
| 1 | +# Futures v2 — individual contracts, chains, continuous series, term structure | |
| 2 | + | |
| 3 | +Design notes for chantier 1 (`hfmarketdata/api/futures/`). The web team turns this into the | |
| 4 | +"Futures individual contracts" guide; the reference documentation is generated from `/openapi.json`. | |
| 5 | + | |
| 6 | +## 1. Endpoints | |
| 7 | + | |
| 8 | +| Endpoint | Purpose | | |
| 9 | +|---|---| | |
| 10 | +| `GET /v1/futures/roots` | 142 roots with reference specs (name, exchange, asset class, size, tick, settlement, expiry rule, month cycle, RTH window, aliases, coverage). | | |
| 11 | +| `GET /v1/futures/{root}/contracts` | Contracts of a root: expiration (+ source), last trading / first notice, real data range, status, 20-session volume, last OI, intervals, files. | | |
| 12 | +| `GET /v1/futures/contract/{symbol}/bars` | OHLCV of one contract, `interval=1m|5m|30m|1h|1d`, `session=rth|eth|all`, `from`/`to`, cursor pagination, `format=json|csv|parquet`. | | |
| 13 | +| `GET /v1/futures/contract/{symbol}/coverage` | What the lake really holds (per interval), gaps, explicit reasons for nulls. | | |
| 14 | +| `GET /v1/futures/{root}/chain?as_of=` | Live contracts on a date, ordered by expiration, with last close/volume/OI. | | |
| 15 | +| `GET /v1/futures/{root}/continuous?roll=&adjust=&depth=` | Continuous series stitched from individual contracts with a reproducible roll schedule (`meta.roll_dates`). | | |
| 16 | +| `GET /v1/futures/{root}/term-structure?as_of=` | Forward curve: settle per contract, spread and annualised slope vs front, contango/backwardation flag. | | |
| 17 | + | |
| 18 | +Common: `{"data": [...], "meta": {"count", "next_cursor", ...}}`, `X-Row-Count`, errors | |
| 19 | +`400 INVALID_CONTRACT_SYMBOL`, `404 CONTRACT_NOT_FOUND`, `404 ROOT_NOT_FOUND`, `400 INVALID_PARAMETER`, | |
| 20 | +`400 ROW_LIMIT_EXCEEDED` (tier cap via `request.state.max_rows`). Limits: default 5 000 rows, max 200 000 (JSON), | |
| 21 | +2 000 000 (CSV/Parquet). `continuous` has `request_cost = 2`. | |
| 22 | + | |
| 23 | +## 2. Symbols and root aliases (`symbols.py`) | |
| 24 | + | |
| 25 | +Accepted: `ESZ25`, `ESZ2025`, `ES_Z25`, `ES-Z25`, lower-case. Roots are 1–4 chars incl. digits (`C`, `6E`, `M2K`, | |
| 26 | +`FDAX`, `SR3`). Month codes `F G H J K M N Q U V X Z`. Two-digit years 00–79 → 20xx, 80–99 → 19xx. | |
| 27 | + | |
| 28 | +The **canonical root is the lake (FirstRate) code** because the files are named that way. CME codes are aliases: | |
| 29 | + | |
| 30 | +| CME | lake | product | | CME | lake | product | | |
| 31 | +|---|---|---|---|---|---|---| | |
| 32 | +| 6E | E6 | Euro FX | | 6N | N6 | New Zealand dollar | | |
| 33 | +| 6J | J1 | Japanese yen (full) | | 6M | MP | Mexican peso | | |
| 34 | +| 6B | B6 | British pound | | 6L | BR | Brazilian real | | |
| 35 | +| 6A | A6 | Australian dollar | | 6Z | T6 | South African rand | | |
| 36 | +| 6C | AD | Canadian dollar | | ZB | US | 30-year T-bond | | |
| 37 | +| 6S | E1 | Swiss franc | | EMD | EW | E-mini S&P MidCap 400 | | |
| 38 | +| | | | | BRN | B | ICE Brent | | |
| 39 | + | |
| 40 | +`J7` and `E7` in the lake are the **E-mini** yen / euro (distinct products, not aliases of 6J/6E). | |
| 41 | +Unknown lake roots (no reference entry, not in `meta/futures/futures.csv`): `CPO FBTM FID FNMY FOAM JB JG NK RU ST TWN ZK` | |
| 42 | +→ `source="derived"`, `expiry_rule="data"`, name null. | |
| 43 | + | |
| 44 | +Responses always show the canonical form (`E6Z25`); `/roots` lists `aliases` per root. | |
| 45 | + | |
| 46 | +## 3. Reference table (`specs.py`) | |
| 47 | + | |
| 48 | +~120 roots with exchange specs (contract size + unit, tick size/value, settlement, month cycle, expiry rule, | |
| 49 | +first-notice rule, business-day calendar `us|eurex`, RTH window in US/Eastern). Fields not known with certainty are | |
| 50 | +`null` — never invented. Roots present in the lake but absent from the table get a derived placeholder. | |
| 51 | + | |
| 52 | +RTH windows (Eastern): equity index 09:30–16:00 (default), CL/NG/HO/RB 09:00–14:30, GC/SI/HG 08:20–13:30, | |
| 53 | +treasuries & FX & DX 08:20–15:00, grains 09:30–14:20, livestock 09:30–14:05, Eurex 03:00–11:30, ICE softs product-specific | |
| 54 | +(cotton is an overnight window 21:00–14:20, handled as wrap-around). | |
| 55 | + | |
| 56 | +## 4. Expiry rules (`expiry.py`, calendar in `calendar_us.py`) | |
| 57 | + | |
| 58 | +US calendar (CME/NYSE): New Year (Sun→Mon, Sat→none), MLK, Presidents, Good Friday, Memorial, Juneteenth (≥ 2022, | |
| 59 | +observed), Independence (observed), Labor, Thanksgiving, Christmas (observed). Eurex: 1 Jan, Good Friday, Easter Monday, | |
| 60 | +1 May, 24–26 Dec, 31 Dec. Weather / state-funeral NYSE closures are not included (CME futures kept trading). | |
| 61 | + | |
| 62 | +| rule | roots | definition | verified | | |
| 63 | +|---|---|---|---| | |
| 64 | +| `third_friday` | ES NQ YM RTY MES MNQ M2K EW ESG XA* MFS MME | 3rd Friday of the contract month (preceding business day if holiday) | ESZ24 → 2024-12-20 | | |
| 65 | +| `third_friday_eurex` | FDAX FESX FDXM FDXS FXXP FCE FTI FTUK … | same, Eurex calendar | FDAXZ24 → 2024-12-20 | | |
| 66 | +| `second_friday_minus_1` | NKD NIY | business day before the 2nd Friday | NKDZ24 → 2024-12-12 | | |
| 67 | +| `cl_rule` | CL | 3 business days before the 25th of the preceding month (25th → prior business day if not one) | CLZ24 → 2024-11-20, CLF25 → 2024-12-19 | | |
| 68 | +| `cl_minus_1` | MCL | one business day before CL | MCLZ24 → 2024-11-19 | | |
| 69 | +| `ng_rule` | NG HH | 3 business days before the 1st of the contract month | NGZ24 → 2024-11-26 | | |
| 70 | +| `ng_minus_1` | QG | one business day before NG | — | | |
| 71 | +| `prior_month_last_business_day` | HO RB SB BR | last business day of the preceding month | HOZ24 → 2024-11-29, SBH25 → 2025-02-28 | | |
| 72 | +| `bz_rule` | BZ B | last business day of the 2nd preceding month | BZZ24 → 2024-10-31 | | |
| 73 | +| `metals_rule` | GC SI HG PL PA MGC SIL ALI | 3rd last business day of the contract month | GCZ24 → 2024-12-27, PLF25 → 2025-01-29 | | |
| 74 | +| `treasury_rule` | ZN US(ZB) UB TN | 7th business day before the last business day | ZNZ24 → 2024-12-19 | | |
| 75 | +| `last_business_day` | ZT ZF ZQ SR1 LE HRC | last business day of the contract month | ZTZ24 → 2024-12-31, LEZ24 → 2024-12-31 | | |
| 76 | +| `grains_rule` | ZC ZS ZW KE ZM ZL ZO ZR XC RS | business day before the 15th | ZCZ24 → 2024-12-13 | | |
| 77 | +| `fx_rule` | E6 J1 B6 A6 AD E1 N6 MP T6 E7 J7 NOK SEK RP RY PJY CNH DX | 2nd business day before the 3rd Wednesday | E6Z24 → 2024-12-16 | | |
| 78 | +| `vx_rule` | VX VXM FVSA | Wednesday 30 days before the 3rd Friday of the following month (holiday-adjusted) | VXZ24 → 2024-12-18, VXH25 → 2025-03-18 | | |
| 79 | +| `last_friday` | BTC MBT MET | last Friday of the contract month | BTCZ24 → 2024-12-27 | | |
| 80 | +| `he_rule` | HE PRK | 10th business day of the contract month | HEZ24 → 2024-12-13 | | |
| 81 | +| `gf_rule` | GF | last Thursday (Nov: Thursday before Thanksgiving) | GFX24 → 2024-11-21 | | |
| 82 | +| `kc_rule` / `cc_rule` / `ct_rule` / `oj_rule` | KC / CC C / CT / OJ | 8 / 11 / 16 / 14 business days before the last business day | KCZ24 12-18, CCZ24 12-13, CTZ24 12-06, OJF25 2025-01-10 | | |
| 83 | +| `bund_rule` | FGBL FGBM FGBS FGBX FOAT FBTP FBTS FBON | 2 exchange days before the 10th (delivery day) | FGBLZ24 → 2024-12-06 | | |
| 84 | +| `sr3_rule` | SR3 | business day before the 3rd Wednesday of the 3rd following month | SR3Z24 → 2025-03-18 | | |
| 85 | +| `data` | everything else | `expiration_date = last_data_date`, `expiration_source="data"` | | | |
| 86 | + | |
| 87 | +First notice: treasuries / grains / metals → last business day of the preceding month; CL → business day after the | |
| 88 | +last trade; KC → 7 business days before the first business day of the contract month; cash-settled → null. | |
| 89 | +Every contract row records `expiration_source` (`rule` | `data`) and `last_trading_date` (null when `data`). | |
| 90 | + | |
| 91 | +Known lake quirks: FirstRate sometimes keeps 1–2 post-expiry rows (HEZ24 last bar 12-17 vs rule 12-13, BZZ24 11-01 vs | |
| 92 | +10-31, GFX24 11-22 vs 11-21, ZQZ24 2025-01-02 vs 12-31). The rule wins for `expiration_date`; `last_data_date` is | |
| 93 | +reported separately in `/coverage`. | |
| 94 | + | |
| 95 | +## 5. Metadata tables (`models.py`) and backfill (`backfill.py`, `scripts/backfill_contracts.py`) | |
| 96 | + | |
| 97 | +`futures_roots`, `futures_contracts`, `futures_contract_gaps` as in UPGRADE-PLAN §1.1–1.3, plus documented extras: | |
| 98 | +`roots.aliases/contract_size_unit/first_notice_rule/calendar/rth_start/rth_end`, `contracts.expiration_source/bars_1day`, | |
| 99 | +and `contracts.timeframes` is a JSON object `{tf: {first, last, rows}}` (needed by `/coverage`). | |
| 100 | + | |
| 101 | +Backfill = one DuckDB aggregate per `{tf}/{archive|update}` directory (`read_parquet('dir/*.parquet', filename=true)`, | |
| 102 | +min/max/count on `datetime` only), one scan of all `1day` files (archive ∪ update, dedup on `datetime`, update wins) | |
| 103 | +for 20-session volume, last OI and session lists (gaps > 3 business days), then SQLite upserts. Status: `expired` when | |
| 104 | +`last_data_date < today − 7` **and** the contract month/expiration has passed, else `active`. `--roots ES,CL` and | |
| 105 | +`--since YYYY-MM-DD` (mtime filter) give incremental runs; `run_backfill()` is the library entry point. | |
| 106 | + | |
| 107 | +Production (M3U96b): | |
| 108 | + | |
| 109 | +```bash | |
| 110 | +ssh M3U96b | |
| 111 | +cd ~/hfmarketdata && HFMD_DATA_ROOT=~/firstratedata venv/bin/python scripts/backfill_contracts.py -v | |
| 112 | +# nightly refresh (PM2 cron or launchd): --since $(date -v-2d +%F) | |
| 113 | +``` | |
| 114 | + | |
| 115 | +## 6. Sessions and time zones (`service.py`) | |
| 116 | + | |
| 117 | +Lake timestamps are naive US/Eastern. Intraday output is localised with `zoneinfo("America/New_York")` (DST-aware) | |
| 118 | +and emitted in UTC (`2024-12-19T14:30:00Z`); daily bars stay dates. `from`/`to`: `YYYY-MM-DD` = Eastern trading | |
| 119 | +dates (inclusive), ISO datetimes with `Z`/offset are converted, naive datetimes are UTC. `session=rth` keeps bars whose | |
| 120 | +start time is in the root's RTH window; `eth` the complement; ignored for `1d`. Cursor = last bar datetime. | |
| 121 | + | |
| 122 | +## 7. Continuous series (`rolls.py`) | |
| 123 | + | |
| 124 | +Contracts of the root are ordered by `expiration_date`; the front holds contract *k* from the previous roll until | |
| 125 | +`roll_k`, the first session on which contract *k+1* is the front: | |
| 126 | + | |
| 127 | +* `calendar` — held through the expiration date; roll on the next session. | |
| 128 | +* `first_notice` — roll on `first_notice_date` (held through the session before), else calendar. | |
| 129 | +* `volume` (default) / `open_interest` — roll after the 2nd consecutive session where the next contract's metric | |
| 130 | + exceeds the front's, evaluated on daily data in the **last 30 sessions** of the front (far-dated noise cannot | |
| 131 | + trigger a roll a year early); never later than the calendar roll. | |
| 132 | + | |
| 133 | +`depth=2|3` uses the same windows with the 2nd/3rd contract of the sequence. Intraday intervals apply the daily | |
| 134 | +schedule by Eastern calendar date of the bar (evening-session bars of day *D* belong to *D* — documented simplification). | |
| 135 | +Contracts already expired when the previous roll happens are skipped. | |
| 136 | + | |
| 137 | +### Adjustments — worked example | |
| 138 | + | |
| 139 | +Three contracts, closes: A = 100 (flat), B = 105, C = 110. Calendar rolls A→B on 2024-03-18 and B→C on 2024-06-24. | |
| 140 | + | |
| 141 | +* gap_k = close_new − close_old on the **last session of the old segment** where both traded (nearest common session | |
| 142 | + within 5 sessions): gap₁ = 105 − 100 = 5 (session 2024-03-15), gap₂ = 110 − 105 = 5. | |
| 143 | +* `back_adjusted` (additive): segment offsets are the cumulative gaps of *later* rolls: C: 0, B: +5, A: +10. | |
| 144 | + A bars become 110, B bars 110, C unchanged → no jump at the rolls; the latest contract is always unadjusted. | |
| 145 | +* `ratio_adjusted` (multiplicative): factors 105/100 and 110/105; A × 1.05 × 1.0476 = 110, B × 1.0476 = 110. | |
| 146 | +* Volume and open interest are never adjusted. If no common session exists, `gap: null`, `adjusted: false` and that | |
| 147 | + roll contributes nothing (no invented value). | |
| 148 | + | |
| 149 | +`meta.roll_dates` lists `{date, from_symbol, to_symbol, gap, ratio, gap_session, adjusted}` for rolls inside the | |
| 150 | +returned window; `meta.segments`, `meta.rolls_total` and `meta.unadjusted_symbol` complete the picture; each row carries | |
| 151 | +its source `symbol`. | |
| 152 | + | |
| 153 | +## 8. Term structure | |
| 154 | + | |
| 155 | +For every live contract on `as_of`: `settle` = last daily close ≤ `as_of` (≤ 7 days old, else null), | |
| 156 | +`spread_vs_front = settle − settle_front`, `slope_annualized = (settle / settle_front − 1) × 365 / (dte − dte_front)`. | |
| 157 | +`meta.structure` = `contango` if the 2nd contract settles above the front, `backwardation` below, `flat` equal. | |
| 158 | + | |
| 159 | +## 9. Tests | |
| 160 | + | |
| 161 | +`tests/test_futures_symbols.py` (parsing, aliases, bad input), `tests/test_futures_expiry.py` (calendar + 39 real dates), | |
| 162 | +`tests/test_futures_rolls.py` (schedules, depth, gaps, adjustment math, gap detection), `tests/test_futures_api.py` | |
| 163 | +(backfill → 7 endpoints → formats → pagination → errors → OpenAPI), `tests/test_futures_perf.py` (10 000 bars < 300 ms, | |
| 164 | +`-m "not slow"` to skip). Fixture lake: `tests/fixtures/make_fixtures.py` (ES/CL/NG/E6 contracts with realistic | |
| 165 | +volume/OI life-cycles and ETH bars). | |
| 166 | + | |
| 167 | +## 10. Known limitations | |
| 168 | + | |
| 169 | +* Expiry rules use one US (CME) calendar and one Eurex calendar; ICE Europe/UK, Euronext, HKEX bank holidays are not | |
| 170 | + modelled (rare 1-day differences possible on London cocoa `C`, Brent `B`, `CNH`). Products without an implemented | |
| 171 | + rule (Nikkei variants excepted) fall back to `data`. | |
| 172 | +* First-notice dates are only computed for treasuries, grains, metals, CL and KC. | |
| 173 | +* Intraday continuous series attribute evening-session bars to the calendar date of the bar (not the CME trading date). | |
| 174 | +* Volume/OI rolls are evaluated inside the last 30 sessions of the front; products whose liquidity migrates earlier | |
| 175 | + (e.g. some ags around FND) should use `first_notice`. | |
| 176 | +* `contract_size`/`tick` values are null for ~25 minor roots (select-sector minis, small Eurex indices, Euribor…). | |
| 177 | ||