skills: pack de 5 skills Claude Code (analyse, backtest walk-forward, futures continus, screener fondamentaux, structure par terme) + zip téléchargeable
- chaque skill : SKILL.md (frontmatter, quand l'utiliser, étapes, exemples, pièges) + scripts Python requests/pandas - helper commun skills/_shared/hfmd.py (enveloppes v1/v2, retry 429, pagination) copié dans chaque skill par le build - skills/scripts/build_skills.py : synchro + zip déterministe → hfmarketdata/web/public/downloads/hfmarketdata-skills.zip - scripts v1 validés en live (SPY 1 405 barres ; walk-forward 14 blocs) ; scripts v2 échouent proprement (404 = module pas encore déployé) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
21 changed files +2,236 −0
added
hfmarketdata/web/public/downloads/hfmarketdata-skills.zip
+0 −0
Binary file not shown.
added
skills/README.md
+60 −0
@@ -0,0 +1,60 @@ | ||
| 1 | +# HF Market Data — skills pack | |
| 2 | + | |
| 3 | +Five Claude Code / Claude skills that turn <https://www.hfmarketdata.io> into a data-analysis workbench. | |
| 4 | +Each skill is a folder with a `SKILL.md` (when to use it, step-by-step, examples) and small Python scripts | |
| 5 | +(`requests` + `pandas`, `matplotlib` optional). No other dependency. | |
| 6 | + | |
| 7 | +| Skill | Use it when you want to… | | |
| 8 | +|---|---| | |
| 9 | +| `hfmd-data-analysis` | pull bars for any symbol into pandas and get returns, volatility, drawdowns, seasonality, a chart | | |
| 10 | +| `hfmd-quick-backtest` | test a moving-average crossover (or any signal) with an honest walk-forward, costs, and a report | | |
| 11 | +| `hfmd-continuous-futures` | build or compare continuous futures series — roll rules, back-adjustment, vs the vendor series | | |
| 12 | +| `hfmd-fundamentals-screen` | screen US stocks on ratios, enrich with statements, keep point-in-time discipline | | |
| 13 | +| `hfmd-term-structure` | draw a futures curve, quantify contango/backwardation and roll yield, compare two dates | | |
| 14 | + | |
| 15 | +## Install | |
| 16 | + | |
| 17 | +```bash | |
| 18 | +# download | |
| 19 | +curl -LO https://www.hfmarketdata.io/downloads/hfmarketdata-skills.zip | |
| 20 | + | |
| 21 | +# for every project (personal skills) | |
| 22 | +unzip -o hfmarketdata-skills.zip -d ~/.claude/skills/ | |
| 23 | + | |
| 24 | +# or for one project only (shared with the team through git) | |
| 25 | +unzip -o hfmarketdata-skills.zip -d .claude/skills/ | |
| 26 | + | |
| 27 | +pip install requests pandas matplotlib # matplotlib only for the charts | |
| 28 | +export HFMD_API_KEY=hfmd_live_… # optional: free account = 120 req/min (keyless = 30 req/h) | |
| 29 | +``` | |
| 30 | + | |
| 31 | +Claude Code lists them under `/skills`; you can also just ask — "backtest a 20/100 MA crossover on SPY since 2015" triggers `hfmd-quick-backtest`. | |
| 32 | + | |
| 33 | +## Layout | |
| 34 | + | |
| 35 | +``` | |
| 36 | +skills/ | |
| 37 | + _shared/hfmd.py single source of the API helper (copied into each skill by the build) | |
| 38 | + hfmd-data-analysis/ SKILL.md · scripts/fetch_bars.py · scripts/analyze.py · scripts/hfmd.py | |
| 39 | + hfmd-quick-backtest/ SKILL.md · scripts/ma_crossover.py · scripts/hfmd.py | |
| 40 | + hfmd-continuous-futures/ SKILL.md · scripts/continuous_compare.py · scripts/stitch_local.py · scripts/hfmd.py | |
| 41 | + hfmd-fundamentals-screen/ SKILL.md · scripts/screen.py · scripts/hfmd.py | |
| 42 | + hfmd-term-structure/ SKILL.md · scripts/term_structure.py · scripts/hfmd.py | |
| 43 | + scripts/build_skills.py syncs _shared/hfmd.py → each skill, zips → web/public/downloads/hfmarketdata-skills.zip | |
| 44 | +``` | |
| 45 | + | |
| 46 | +## Build the zip | |
| 47 | + | |
| 48 | +```bash | |
| 49 | +python3 skills/scripts/build_skills.py # → hfmarketdata/web/public/downloads/hfmarketdata-skills.zip | |
| 50 | +python3 skills/scripts/build_skills.py --check # fail if a copied hfmd.py drifted from _shared/ | |
| 51 | +``` | |
| 52 | + | |
| 53 | +The zip is committed so the website can serve it without a build step. | |
| 54 | + | |
| 55 | +## Conventions the scripts follow | |
| 56 | + | |
| 57 | +- Never invent data: gaps stay `NaN`, and every script prints the row count and the date range it actually received. | |
| 58 | +- Intraday timestamps are naive US/Eastern (as delivered by the API); daily bars are dates. | |
| 59 | +- Respect the quota: keyless is 30 requests/hour — scripts paginate by date only when needed and honour `Retry-After` on 429. | |
| 60 | +- Fundamentals are point-in-time: use `filed_at` (not `period_end`) to decide what was knowable on a given day. | |
added
skills/_shared/hfmd.py
+156 −0
@@ -0,0 +1,156 @@ | ||
| 1 | +"""Tiny HF Market Data client shared by the hfmd-* skills (requests + pandas only). | |
| 2 | + | |
| 3 | +Copied verbatim into every skill's `scripts/` folder by `skills/scripts/build_skills.py` | |
| 4 | +so each skill stays self-contained. Do not edit the copies — edit `skills/_shared/hfmd.py`. | |
| 5 | + | |
| 6 | +Environment: HFMD_API_KEY (optional, free account = 120 req/min; keyless = 30 req/h), | |
| 7 | + HFMD_BASE_URL (default https://www.hfmarketdata.io). | |
| 8 | +""" | |
| 9 | +from __future__ import annotations | |
| 10 | + | |
| 11 | +import os | |
| 12 | +import sys | |
| 13 | +import time | |
| 14 | +from typing import Any | |
| 15 | + | |
| 16 | +import pandas as pd | |
| 17 | +import requests | |
| 18 | + | |
| 19 | +BASE_URL = os.environ.get("HFMD_BASE_URL", "https://www.hfmarketdata.io").rstrip("/") | |
| 20 | +API_KEY = os.environ.get("HFMD_API_KEY", "").strip() | |
| 21 | +UA = "hfmd-skills/1.0 (+https://www.hfmarketdata.io/integrations/skills)" | |
| 22 | + | |
| 23 | +TIMEFRAMES = {"1m": "1min", "5m": "5min", "30m": "30min", "1h": "1hour", "1d": "1day", | |
| 24 | + "1min": "1min", "5min": "5min", "30min": "30min", "1hour": "1hour", "1day": "1day"} | |
| 25 | +V2_INTERVAL = {"1min": "1m", "5min": "5m", "30min": "30m", "1hour": "1h", "1day": "1d"} | |
| 26 | + | |
| 27 | + | |
| 28 | +class HfmdError(RuntimeError): | |
| 29 | + def __init__(self, status: int, code: str, message: str, url: str): | |
| 30 | + super().__init__(f"HF Market Data error {status} {code}: {message} ({url})") | |
| 31 | + self.status, self.code, self.url = status, code, url | |
| 32 | + | |
| 33 | + | |
| 34 | +_session = requests.Session() | |
| 35 | +_session.headers.update({"Accept": "application/json", "User-Agent": UA}) | |
| 36 | +if API_KEY: | |
| 37 | + _session.headers["Authorization"] = f"Bearer {API_KEY}" | |
| 38 | + | |
| 39 | + | |
| 40 | +def log(msg: str) -> None: | |
| 41 | + print(msg, file=sys.stderr) | |
| 42 | + | |
| 43 | + | |
| 44 | +def get(path: str, params: dict[str, Any] | None = None, *, retries: int = 3) -> tuple[Any, dict[str, str]]: | |
| 45 | + """GET a JSON endpoint. Returns (body, headers). Retries on 429 honouring Retry-After.""" | |
| 46 | + url = f"{BASE_URL}{path}" | |
| 47 | + p = {k: (",".join(v) if isinstance(v, (list, tuple)) else v) for k, v in (params or {}).items() if v not in (None, "")} | |
| 48 | + p.setdefault("format", "json") | |
| 49 | + for attempt in range(retries + 1): | |
| 50 | + r = _session.get(url, params=p, timeout=120) | |
| 51 | + if r.status_code == 429 and attempt < retries: | |
| 52 | + wait = float(r.headers.get("Retry-After", "5")) | |
| 53 | + log(f"429 rate limited — sleeping {wait:.0f}s (set HFMD_API_KEY for 120 req/min)") | |
| 54 | + time.sleep(min(wait, 120)) | |
| 55 | + continue | |
| 56 | + if r.status_code >= 400: | |
| 57 | + try: | |
| 58 | + body = r.json() | |
| 59 | + except ValueError: | |
| 60 | + body = {} | |
| 61 | + err = body.get("error") or {} | |
| 62 | + raise HfmdError(r.status_code, err.get("code", "HTTP_ERROR"), err.get("message") or body.get("detail") or r.text[:200], r.url) | |
| 63 | + rem = r.headers.get("X-RateLimit-Remaining-Requests") | |
| 64 | + if rem is not None: | |
| 65 | + log(f"rate limit: {rem}/{r.headers.get('X-RateLimit-Limit-Requests')} requests left") | |
| 66 | + return r.json(), dict(r.headers) | |
| 67 | + raise HfmdError(429, "RATE_LIMIT_EXCEEDED", "gave up after retries", url) | |
| 68 | + | |
| 69 | + | |
| 70 | +def rows_of(body: Any) -> list[dict]: | |
| 71 | + """Rows from either envelope: v1 {count,data} or v2 {data,meta}.""" | |
| 72 | + if isinstance(body, list): | |
| 73 | + return body | |
| 74 | + if isinstance(body, dict): | |
| 75 | + data = body.get("data") | |
| 76 | + if isinstance(data, list): | |
| 77 | + return data | |
| 78 | + return [] | |
| 79 | + | |
| 80 | + | |
| 81 | +def meta_of(body: Any) -> dict: | |
| 82 | + if isinstance(body, dict): | |
| 83 | + m = dict(body.get("meta") or {}) | |
| 84 | + if "count" in body and "count" not in m: | |
| 85 | + m["count"] = body["count"] | |
| 86 | + return m | |
| 87 | + return {} | |
| 88 | + | |
| 89 | + | |
| 90 | +def to_df(body: Any, time_col: str | None = "datetime") -> pd.DataFrame: | |
| 91 | + df = pd.DataFrame(rows_of(body)) | |
| 92 | + if time_col and time_col in df.columns: | |
| 93 | + df[time_col] = pd.to_datetime(df[time_col]) | |
| 94 | + df = df.sort_values(time_col).reset_index(drop=True) | |
| 95 | + return df | |
| 96 | + | |
| 97 | + | |
| 98 | +def bars(asset: str, ticker: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, | |
| 99 | + adjustment: str | None = None, limit: int = 50_000) -> pd.DataFrame: | |
| 100 | + """v1 bars (stock/etf/crypto/index/fx, or futures = vendor continuous). Paginates by date when needed.""" | |
| 101 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 102 | + frames: list[pd.DataFrame] = [] | |
| 103 | + cursor_start = start | |
| 104 | + while True: | |
| 105 | + body, _ = get(f"/v1/bars/{asset}/{ticker}", {"timeframe": tf, "start": cursor_start, "end": end, "adjustment": adjustment, "limit": limit, "order": "asc"}) | |
| 106 | + df = to_df(body) | |
| 107 | + if df.empty: | |
| 108 | + break | |
| 109 | + frames.append(df) | |
| 110 | + if len(df) < limit: | |
| 111 | + break | |
| 112 | + last = df["datetime"].iloc[-1] | |
| 113 | + cursor_start = (last + pd.Timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M") if tf != "1day" else (last + pd.Timedelta(days=1)).strftime("%Y-%m-%d") | |
| 114 | + out = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(columns=["ticker", "datetime", "open", "high", "low", "close", "volume"]) | |
| 115 | + return out.drop_duplicates("datetime").reset_index(drop=True) | |
| 116 | + | |
| 117 | + | |
| 118 | +def contract_bars(symbol: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, limit: int = 50_000) -> pd.DataFrame: | |
| 119 | + """v2 individual futures contract bars (ESZ25…).""" | |
| 120 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 121 | + body, _ = get(f"/v1/futures/contract/{symbol}/bars", {"interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit}) | |
| 122 | + return to_df(body) | |
| 123 | + | |
| 124 | + | |
| 125 | +def continuous(root: str, roll: str = "volume", adjust: str = "back_adjusted", depth: int = 1, timeframe: str = "1day", | |
| 126 | + start: str | None = None, end: str | None = None, limit: int = 50_000) -> tuple[pd.DataFrame, dict]: | |
| 127 | + """v2 server-built continuous series. Returns (df, meta) — meta['roll_dates'] lists the rolls.""" | |
| 128 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 129 | + body, _ = get(f"/v1/futures/{root}/continuous", {"roll": roll, "adjust": adjust, "depth": depth, "interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit}) | |
| 130 | + return to_df(body), meta_of(body) | |
| 131 | + | |
| 132 | + | |
| 133 | +def contracts(root: str, status: str | None = None) -> pd.DataFrame: | |
| 134 | + body, _ = get(f"/v1/futures/{root}/contracts", {"status": status, "limit": 1000}) | |
| 135 | + return to_df(body, time_col=None) | |
| 136 | + | |
| 137 | + | |
| 138 | +def term_structure(root: str, as_of: str | None = None) -> tuple[pd.DataFrame, dict]: | |
| 139 | + body, _ = get(f"/v1/futures/{root}/term-structure", {"as_of": as_of}) | |
| 140 | + return to_df(body, time_col=None), meta_of(body) | |
| 141 | + | |
| 142 | + | |
| 143 | +def screener(filters: str, sort: str | None = None, as_of: str | None = None, limit: int = 100) -> tuple[pd.DataFrame, dict]: | |
| 144 | + body, _ = get("/v1/fundamentals/screener", {"filters": filters, "sort": sort, "as_of": as_of, "limit": limit}) | |
| 145 | + return to_df(body, time_col=None), meta_of(body) | |
| 146 | + | |
| 147 | + | |
| 148 | +def ratios(ticker: str) -> dict: | |
| 149 | + body, _ = get(f"/v1/fundamentals/{ticker}/ratios") | |
| 150 | + d = body.get("data", body) if isinstance(body, dict) else body | |
| 151 | + return d if isinstance(d, dict) else (d[0] if d else {}) | |
| 152 | + | |
| 153 | + | |
| 154 | +def statements(ticker: str, statement: str = "income", period: str = "quarterly", limit: int = 12) -> pd.DataFrame: | |
| 155 | + body, _ = get(f"/v1/fundamentals/{ticker}/statements", {"statement": statement, "period": period, "limit": limit}) | |
| 156 | + return to_df(body, time_col=None) | |
added
skills/hfmd-continuous-futures/SKILL.md
+53 −0
@@ -0,0 +1,53 @@ | ||
| 1 | +--- | |
| 2 | +name: hfmd-continuous-futures | |
| 3 | +description: Build and compare continuous futures series from HF Market Data — server-side v2 continuous endpoint (roll=volume/open_interest/calendar, adjust=none/back_adjusted/ratio, depth) versus the vendor continuous series, or stitched locally from individual contracts with explicit roll dates. Use when the user asks for a continuous futures price, questions how rolls/adjustments were done, or needs a series suitable for backtesting. | |
| 4 | +--- | |
| 5 | + | |
| 6 | +# hfmd-continuous-futures | |
| 7 | + | |
| 8 | +A futures "price history" is a construction: someone chose when to jump from one contract to the next | |
| 9 | +(**roll**) and what to do with the price gap at the jump (**adjust**). This skill makes those choices | |
| 10 | +explicit, reproducible and comparable. | |
| 11 | + | |
| 12 | +## Vocabulary (say this to the user) | |
| 13 | + | |
| 14 | +- **Roll rule** — `volume`: switch when the next contract's volume exceeds the front's (most common, tracks where liquidity is); `open_interest`: same with OI (smoother, lags a bit); `calendar`: N business days before expiry/first notice (deterministic, what many CTAs do). | |
| 15 | +- **Adjustment** — `none`: raw prices, discontinuous at rolls (fine for *levels*, wrong for *returns*); `back_adjusted` (additive): shift all earlier history by the roll gap so the series is continuous in **points** (good for P&L in ticks, can go negative on long histories); `ratio` (multiplicative): scale earlier history by the gap ratio, continuous in **percent** (best for returns/backtests, levels are not real prices). | |
| 16 | +- **Depth** — 1 = front month, 2 = second month … (depth 2 avoids expiry noise for spread work). | |
| 17 | +- Vendor series (v1 `/v1/bars/futures/{ROOT}?adjustment=contin_UNadj|contin_adj_ratio|contin_adj_absolute`) are FirstRate Data's own construction — a fixed rule you cannot change. The v2 endpoint lets you pick. | |
| 18 | + | |
| 19 | +## When to use | |
| 20 | + | |
| 21 | +- "Get me continuous ES since 2015", "why does the crude series jump in April?", "back-adjust NG for a backtest", "compare volume-roll vs calendar-roll on CL" | |
| 22 | + | |
| 23 | +## Steps | |
| 24 | + | |
| 25 | +1. Confirm root (`/v1/futures/roots` or `search_symbols(asset=futures)`), timeframe, range, and the intended *use* (levels → `none`; P&L in points → `back_adjusted`; returns → `ratio`). | |
| 26 | +2. Server-side build + comparison: | |
| 27 | + `python3 scripts/continuous_compare.py --root CL --start 2018-01-01 --rolls volume,calendar --adjusts back_adjusted,ratio --vendor contin_adj_ratio --plot cl.png` | |
| 28 | + - prints each series' roll dates (from `meta.roll_dates`), the number of rolls per year, the mean absolute gap at rolls, and the return correlation / tracking difference between the variants and the vendor series. | |
| 29 | +3. If the v2 endpoint is not available on the server yet (404 `NOT_FOUND`), stitch locally to show the methodology: | |
| 30 | + `python3 scripts/stitch_local.py --root CL --start 2023-01-01 --roll volume --adjust ratio --out cl_local.csv` | |
| 31 | + (lists contracts via `/v1/futures/{root}/contracts`, pulls each contract's daily bars, rolls on volume crossover, adjusts; prints the roll table). This costs one request per contract — needs an API key for long histories. | |
| 32 | +4. Report: chosen rule + why, roll dates table, adjusted vs raw last price (to remind that adjusted levels ≠ tradable prices), and any gap in coverage (`/v1/futures/contract/{symbol}/coverage`). | |
| 33 | + | |
| 34 | +## Examples | |
| 35 | + | |
| 36 | +```bash | |
| 37 | +# ES front month, volume roll, ratio-adjusted, daily since 2015 → CSV for a backtest | |
| 38 | +python3 scripts/continuous_compare.py --root ES --start 2015-01-01 --rolls volume --adjusts ratio --out es_cont.csv | |
| 39 | + | |
| 40 | +# Second-month natural gas vs front month (seasonal spread work) | |
| 41 | +python3 scripts/continuous_compare.py --root NG --depth 2 --rolls open_interest --adjusts none --start 2020-01-01 | |
| 42 | + | |
| 43 | +# How different are the vendor's absolute-adjusted and the v2 back_adjusted series? | |
| 44 | +python3 scripts/continuous_compare.py --root CL --rolls volume --adjusts back_adjusted --vendor contin_adj_absolute --start 2019-01-01 | |
| 45 | +``` | |
| 46 | + | |
| 47 | +## Gotchas | |
| 48 | + | |
| 49 | +- Never compute returns on `none`/`contin_UNadj`: the roll gap (often 0.5-3 %) is not a market move. | |
| 50 | +- Back-adjusted (additive) series on long histories can go **negative** (CL 2000-2020 does) — use `ratio` for percent returns. | |
| 51 | +- Roll dates depend on the timeframe used to measure volume (daily is standard); intraday continuous series inherit the daily roll calendar. | |
| 52 | +- Coverage: contracts have data since 2010 (`archive` ≤ 2025 and `update` ≥ 2025 files are already merged by the API); a missing contract is reported as a gap, not filled. | |
| 53 | +- FirstRate roots may differ from exchange codes (`E6` is euro FX `6E`; the API accepts both). | |
added
skills/hfmd-continuous-futures/scripts/continuous_compare.py
+118 −0
@@ -0,0 +1,118 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +"""Build server-side continuous futures series (v2) with several roll/adjust choices and compare them | |
| 3 | +to each other and to the vendor continuous series (v1). | |
| 4 | + | |
| 5 | + continuous_compare.py --root CL --start 2018-01-01 --rolls volume,calendar --adjusts back_adjusted,ratio \ | |
| 6 | + [--depth 1] [--timeframe 1day] [--vendor contin_adj_ratio] [--out cl.csv] [--plot cl.png] | |
| 7 | +""" | |
| 8 | +from __future__ import annotations | |
| 9 | + | |
| 10 | +import argparse | |
| 11 | +import sys | |
| 12 | +from pathlib import Path | |
| 13 | + | |
| 14 | +import numpy as np | |
| 15 | +import pandas as pd | |
| 16 | + | |
| 17 | +sys.path.insert(0, str(Path(__file__).parent)) | |
| 18 | +import hfmd # noqa: E402 | |
| 19 | + | |
| 20 | + | |
| 21 | +def describe_rolls(name: str, df: pd.DataFrame, meta: dict) -> None: | |
| 22 | + rolls = meta.get("roll_dates") or [] | |
| 23 | + if df.empty: | |
| 24 | + print(f"\n[{name}] no data") | |
| 25 | + return | |
| 26 | + years = max((df["datetime"].iloc[-1] - df["datetime"].iloc[0]).days / 365.25, 1e-9) | |
| 27 | + roll_txt = f"{len(rolls)} rolls ({len(rolls) / years:.1f}/yr)" if rolls or not name.startswith("vendor") else "roll dates not exposed (vendor construction)" | |
| 28 | + print(f"\n[{name}] {len(df):,} bars {df['datetime'].iloc[0].date()} → {df['datetime'].iloc[-1].date()} · {roll_txt}") | |
| 29 | + if rolls: | |
| 30 | + shown = rolls if len(rolls) <= 12 else rolls[:6] + ["…"] + rolls[-6:] | |
| 31 | + print(" roll dates:", ", ".join(str(r) if isinstance(r, str) else str(r.get("date", r)) for r in shown)) | |
| 32 | + if "close" in df and len(df) > 1: | |
| 33 | + print(f" last close {df['close'].iloc[-1]:.4f} · first close {df['close'].iloc[0]:.4f}" | |
| 34 | + + (" (adjusted levels are NOT tradable prices)" if "adjust=none" not in name else "")) | |
| 35 | + | |
| 36 | + | |
| 37 | +def main() -> int: | |
| 38 | + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) | |
| 39 | + ap.add_argument("--root", required=True) | |
| 40 | + ap.add_argument("--start", default="2015-01-01") | |
| 41 | + ap.add_argument("--end") | |
| 42 | + ap.add_argument("--timeframe", default="1day") | |
| 43 | + ap.add_argument("--depth", type=int, default=1) | |
| 44 | + ap.add_argument("--rolls", default="volume", help="comma list: volume, open_interest, calendar") | |
| 45 | + ap.add_argument("--adjusts", default="back_adjusted", help="comma list: none, back_adjusted, ratio") | |
| 46 | + ap.add_argument("--vendor", help="also fetch the v1 vendor continuous with this adjustment: contin_UNadj, contin_adj_ratio, contin_adj_absolute") | |
| 47 | + ap.add_argument("--out", help="CSV with all series' closes aligned on datetime") | |
| 48 | + ap.add_argument("--plot") | |
| 49 | + a = ap.parse_args() | |
| 50 | + root = a.root.upper() | |
| 51 | + | |
| 52 | + series: dict[str, pd.Series] = {} | |
| 53 | + for roll in a.rolls.split(","): | |
| 54 | + for adj in a.adjusts.split(","): | |
| 55 | + name = f"v2 roll={roll} adjust={adj} depth={a.depth}" | |
| 56 | + try: | |
| 57 | + df, meta = hfmd.continuous(root, roll=roll, adjust=adj, depth=a.depth, timeframe=a.timeframe, start=a.start, end=a.end) | |
| 58 | + except hfmd.HfmdError as e: | |
| 59 | + print(f"\n[{name}] {e}", file=sys.stderr) | |
| 60 | + if e.status == 404: | |
| 61 | + print(" → the v2 continuous endpoint is not deployed on this server yet; use scripts/stitch_local.py", file=sys.stderr) | |
| 62 | + continue | |
| 63 | + describe_rolls(name, df, meta) | |
| 64 | + if not df.empty: | |
| 65 | + series[name] = df.set_index("datetime")["close"].astype(float) | |
| 66 | + | |
| 67 | + if a.vendor: | |
| 68 | + name = f"vendor {a.vendor}" | |
| 69 | + try: | |
| 70 | + vdf = hfmd.bars("futures", root, a.timeframe, a.start, a.end, a.vendor) | |
| 71 | + describe_rolls(name, vdf, {}) | |
| 72 | + if not vdf.empty: | |
| 73 | + series[name] = vdf.set_index("datetime")["close"].astype(float) | |
| 74 | + except hfmd.HfmdError as e: | |
| 75 | + print(f"\n[{name}] {e}", file=sys.stderr) | |
| 76 | + | |
| 77 | + if not series: | |
| 78 | + print("\nno series retrieved", file=sys.stderr) | |
| 79 | + return 1 | |
| 80 | + | |
| 81 | + wide = pd.concat(series, axis=1).sort_index() | |
| 82 | + rets = wide.pct_change() | |
| 83 | + print("\nreturn correlation between series (daily pct changes; 'none'/UNadj variants include roll gaps):") | |
| 84 | + with pd.option_context("display.width", 200): | |
| 85 | + print(rets.corr().round(4).to_string()) | |
| 86 | + if len(series) > 1: | |
| 87 | + names = list(series) | |
| 88 | + base = names[-1] | |
| 89 | + print(f"\nannualised tracking difference of log-returns vs '{base}':") | |
| 90 | + for n in names[:-1]: | |
| 91 | + d = (np.log1p(rets[n]) - np.log1p(rets[base])).dropna() | |
| 92 | + print(f" {n:<45} mean={d.mean() * 252:+.4%}/yr std={d.std() * np.sqrt(252):.4%} n={len(d)}") | |
| 93 | + | |
| 94 | + if a.out: | |
| 95 | + wide.to_csv(a.out) | |
| 96 | + print(f"\nwrote {a.out} ({len(wide):,} rows × {wide.shape[1]} series)") | |
| 97 | + if a.plot: | |
| 98 | + try: | |
| 99 | + import matplotlib | |
| 100 | + matplotlib.use("Agg") | |
| 101 | + import matplotlib.pyplot as plt | |
| 102 | + except ImportError: | |
| 103 | + print("matplotlib not installed", file=sys.stderr) | |
| 104 | + return 0 | |
| 105 | + fig, ax = plt.subplots(figsize=(11, 5)) | |
| 106 | + for n, s in series.items(): | |
| 107 | + ax.plot(s.index, s / s.dropna().iloc[0] * 100, lw=1.0, label=n) | |
| 108 | + ax.set_title(f"{root} continuous series rebased to 100 (differences = roll/adjust methodology)") | |
| 109 | + ax.grid(alpha=0.25) | |
| 110 | + ax.legend(fontsize=8) | |
| 111 | + fig.tight_layout() | |
| 112 | + fig.savefig(a.plot, dpi=130) | |
| 113 | + print(f"wrote {a.plot}") | |
| 114 | + return 0 | |
| 115 | + | |
| 116 | + | |
| 117 | +if __name__ == "__main__": | |
| 118 | + sys.exit(main()) | |
added
skills/hfmd-continuous-futures/scripts/hfmd.py
+156 −0
@@ -0,0 +1,156 @@ | ||
| 1 | +"""Tiny HF Market Data client shared by the hfmd-* skills (requests + pandas only). | |
| 2 | + | |
| 3 | +Copied verbatim into every skill's `scripts/` folder by `skills/scripts/build_skills.py` | |
| 4 | +so each skill stays self-contained. Do not edit the copies — edit `skills/_shared/hfmd.py`. | |
| 5 | + | |
| 6 | +Environment: HFMD_API_KEY (optional, free account = 120 req/min; keyless = 30 req/h), | |
| 7 | + HFMD_BASE_URL (default https://www.hfmarketdata.io). | |
| 8 | +""" | |
| 9 | +from __future__ import annotations | |
| 10 | + | |
| 11 | +import os | |
| 12 | +import sys | |
| 13 | +import time | |
| 14 | +from typing import Any | |
| 15 | + | |
| 16 | +import pandas as pd | |
| 17 | +import requests | |
| 18 | + | |
| 19 | +BASE_URL = os.environ.get("HFMD_BASE_URL", "https://www.hfmarketdata.io").rstrip("/") | |
| 20 | +API_KEY = os.environ.get("HFMD_API_KEY", "").strip() | |
| 21 | +UA = "hfmd-skills/1.0 (+https://www.hfmarketdata.io/integrations/skills)" | |
| 22 | + | |
| 23 | +TIMEFRAMES = {"1m": "1min", "5m": "5min", "30m": "30min", "1h": "1hour", "1d": "1day", | |
| 24 | + "1min": "1min", "5min": "5min", "30min": "30min", "1hour": "1hour", "1day": "1day"} | |
| 25 | +V2_INTERVAL = {"1min": "1m", "5min": "5m", "30min": "30m", "1hour": "1h", "1day": "1d"} | |
| 26 | + | |
| 27 | + | |
| 28 | +class HfmdError(RuntimeError): | |
| 29 | + def __init__(self, status: int, code: str, message: str, url: str): | |
| 30 | + super().__init__(f"HF Market Data error {status} {code}: {message} ({url})") | |
| 31 | + self.status, self.code, self.url = status, code, url | |
| 32 | + | |
| 33 | + | |
| 34 | +_session = requests.Session() | |
| 35 | +_session.headers.update({"Accept": "application/json", "User-Agent": UA}) | |
| 36 | +if API_KEY: | |
| 37 | + _session.headers["Authorization"] = f"Bearer {API_KEY}" | |
| 38 | + | |
| 39 | + | |
| 40 | +def log(msg: str) -> None: | |
| 41 | + print(msg, file=sys.stderr) | |
| 42 | + | |
| 43 | + | |
| 44 | +def get(path: str, params: dict[str, Any] | None = None, *, retries: int = 3) -> tuple[Any, dict[str, str]]: | |
| 45 | + """GET a JSON endpoint. Returns (body, headers). Retries on 429 honouring Retry-After.""" | |
| 46 | + url = f"{BASE_URL}{path}" | |
| 47 | + p = {k: (",".join(v) if isinstance(v, (list, tuple)) else v) for k, v in (params or {}).items() if v not in (None, "")} | |
| 48 | + p.setdefault("format", "json") | |
| 49 | + for attempt in range(retries + 1): | |
| 50 | + r = _session.get(url, params=p, timeout=120) | |
| 51 | + if r.status_code == 429 and attempt < retries: | |
| 52 | + wait = float(r.headers.get("Retry-After", "5")) | |
| 53 | + log(f"429 rate limited — sleeping {wait:.0f}s (set HFMD_API_KEY for 120 req/min)") | |
| 54 | + time.sleep(min(wait, 120)) | |
| 55 | + continue | |
| 56 | + if r.status_code >= 400: | |
| 57 | + try: | |
| 58 | + body = r.json() | |
| 59 | + except ValueError: | |
| 60 | + body = {} | |
| 61 | + err = body.get("error") or {} | |
| 62 | + raise HfmdError(r.status_code, err.get("code", "HTTP_ERROR"), err.get("message") or body.get("detail") or r.text[:200], r.url) | |
| 63 | + rem = r.headers.get("X-RateLimit-Remaining-Requests") | |
| 64 | + if rem is not None: | |
| 65 | + log(f"rate limit: {rem}/{r.headers.get('X-RateLimit-Limit-Requests')} requests left") | |
| 66 | + return r.json(), dict(r.headers) | |
| 67 | + raise HfmdError(429, "RATE_LIMIT_EXCEEDED", "gave up after retries", url) | |
| 68 | + | |
| 69 | + | |
| 70 | +def rows_of(body: Any) -> list[dict]: | |
| 71 | + """Rows from either envelope: v1 {count,data} or v2 {data,meta}.""" | |
| 72 | + if isinstance(body, list): | |
| 73 | + return body | |
| 74 | + if isinstance(body, dict): | |
| 75 | + data = body.get("data") | |
| 76 | + if isinstance(data, list): | |
| 77 | + return data | |
| 78 | + return [] | |
| 79 | + | |
| 80 | + | |
| 81 | +def meta_of(body: Any) -> dict: | |
| 82 | + if isinstance(body, dict): | |
| 83 | + m = dict(body.get("meta") or {}) | |
| 84 | + if "count" in body and "count" not in m: | |
| 85 | + m["count"] = body["count"] | |
| 86 | + return m | |
| 87 | + return {} | |
| 88 | + | |
| 89 | + | |
| 90 | +def to_df(body: Any, time_col: str | None = "datetime") -> pd.DataFrame: | |
| 91 | + df = pd.DataFrame(rows_of(body)) | |
| 92 | + if time_col and time_col in df.columns: | |
| 93 | + df[time_col] = pd.to_datetime(df[time_col]) | |
| 94 | + df = df.sort_values(time_col).reset_index(drop=True) | |
| 95 | + return df | |
| 96 | + | |
| 97 | + | |
| 98 | +def bars(asset: str, ticker: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, | |
| 99 | + adjustment: str | None = None, limit: int = 50_000) -> pd.DataFrame: | |
| 100 | + """v1 bars (stock/etf/crypto/index/fx, or futures = vendor continuous). Paginates by date when needed.""" | |
| 101 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 102 | + frames: list[pd.DataFrame] = [] | |
| 103 | + cursor_start = start | |
| 104 | + while True: | |
| 105 | + body, _ = get(f"/v1/bars/{asset}/{ticker}", {"timeframe": tf, "start": cursor_start, "end": end, "adjustment": adjustment, "limit": limit, "order": "asc"}) | |
| 106 | + df = to_df(body) | |
| 107 | + if df.empty: | |
| 108 | + break | |
| 109 | + frames.append(df) | |
| 110 | + if len(df) < limit: | |
| 111 | + break | |
| 112 | + last = df["datetime"].iloc[-1] | |
| 113 | + cursor_start = (last + pd.Timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M") if tf != "1day" else (last + pd.Timedelta(days=1)).strftime("%Y-%m-%d") | |
| 114 | + out = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(columns=["ticker", "datetime", "open", "high", "low", "close", "volume"]) | |
| 115 | + return out.drop_duplicates("datetime").reset_index(drop=True) | |
| 116 | + | |
| 117 | + | |
| 118 | +def contract_bars(symbol: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, limit: int = 50_000) -> pd.DataFrame: | |
| 119 | + """v2 individual futures contract bars (ESZ25…).""" | |
| 120 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 121 | + body, _ = get(f"/v1/futures/contract/{symbol}/bars", {"interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit}) | |
| 122 | + return to_df(body) | |
| 123 | + | |
| 124 | + | |
| 125 | +def continuous(root: str, roll: str = "volume", adjust: str = "back_adjusted", depth: int = 1, timeframe: str = "1day", | |
| 126 | + start: str | None = None, end: str | None = None, limit: int = 50_000) -> tuple[pd.DataFrame, dict]: | |
| 127 | + """v2 server-built continuous series. Returns (df, meta) — meta['roll_dates'] lists the rolls.""" | |
| 128 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 129 | + body, _ = get(f"/v1/futures/{root}/continuous", {"roll": roll, "adjust": adjust, "depth": depth, "interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit}) | |
| 130 | + return to_df(body), meta_of(body) | |
| 131 | + | |
| 132 | + | |
| 133 | +def contracts(root: str, status: str | None = None) -> pd.DataFrame: | |
| 134 | + body, _ = get(f"/v1/futures/{root}/contracts", {"status": status, "limit": 1000}) | |
| 135 | + return to_df(body, time_col=None) | |
| 136 | + | |
| 137 | + | |
| 138 | +def term_structure(root: str, as_of: str | None = None) -> tuple[pd.DataFrame, dict]: | |
| 139 | + body, _ = get(f"/v1/futures/{root}/term-structure", {"as_of": as_of}) | |
| 140 | + return to_df(body, time_col=None), meta_of(body) | |
| 141 | + | |
| 142 | + | |
| 143 | +def screener(filters: str, sort: str | None = None, as_of: str | None = None, limit: int = 100) -> tuple[pd.DataFrame, dict]: | |
| 144 | + body, _ = get("/v1/fundamentals/screener", {"filters": filters, "sort": sort, "as_of": as_of, "limit": limit}) | |
| 145 | + return to_df(body, time_col=None), meta_of(body) | |
| 146 | + | |
| 147 | + | |
| 148 | +def ratios(ticker: str) -> dict: | |
| 149 | + body, _ = get(f"/v1/fundamentals/{ticker}/ratios") | |
| 150 | + d = body.get("data", body) if isinstance(body, dict) else body | |
| 151 | + return d if isinstance(d, dict) else (d[0] if d else {}) | |
| 152 | + | |
| 153 | + | |
| 154 | +def statements(ticker: str, statement: str = "income", period: str = "quarterly", limit: int = 12) -> pd.DataFrame: | |
| 155 | + body, _ = get(f"/v1/fundamentals/{ticker}/statements", {"statement": statement, "period": period, "limit": limit}) | |
| 156 | + return to_df(body, time_col=None) | |
added
skills/hfmd-continuous-futures/scripts/stitch_local.py
+124 −0
@@ -0,0 +1,124 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +"""Stitch a continuous futures series locally from individual contracts (transparent methodology, | |
| 3 | +and a fallback when the v2 /continuous endpoint is unavailable). | |
| 4 | + | |
| 5 | + stitch_local.py --root CL --start 2023-01-01 [--roll volume|open_interest|calendar] [--roll-days 5] | |
| 6 | + [--adjust none|back_adjusted|ratio] [--out cl_local.csv] | |
| 7 | + | |
| 8 | +Costs one request per contract in the range (≈ 12/yr for CL, 4/yr for ES) — set HFMD_API_KEY. | |
| 9 | +""" | |
| 10 | +from __future__ import annotations | |
| 11 | + | |
| 12 | +import argparse | |
| 13 | +import sys | |
| 14 | +from pathlib import Path | |
| 15 | + | |
| 16 | +import numpy as np | |
| 17 | +import pandas as pd | |
| 18 | + | |
| 19 | +sys.path.insert(0, str(Path(__file__).parent)) | |
| 20 | +import hfmd # noqa: E402 | |
| 21 | + | |
| 22 | + | |
| 23 | +def main() -> int: | |
| 24 | + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) | |
| 25 | + ap.add_argument("--root", required=True) | |
| 26 | + ap.add_argument("--start", default="2023-01-01") | |
| 27 | + ap.add_argument("--end") | |
| 28 | + ap.add_argument("--roll", default="volume", choices=["volume", "open_interest", "calendar"]) | |
| 29 | + ap.add_argument("--roll-days", type=int, default=5, help="calendar rule: business days before expiration") | |
| 30 | + ap.add_argument("--adjust", default="ratio", choices=["none", "back_adjusted", "ratio"]) | |
| 31 | + ap.add_argument("--out") | |
| 32 | + a = ap.parse_args() | |
| 33 | + root = a.root.upper() | |
| 34 | + | |
| 35 | + cons = hfmd.contracts(root) | |
| 36 | + if cons.empty: | |
| 37 | + print("no contracts returned", file=sys.stderr) | |
| 38 | + return 1 | |
| 39 | + cons["expiration_date"] = pd.to_datetime(cons["expiration_date"]) | |
| 40 | + cons["last_data_date"] = pd.to_datetime(cons.get("last_data_date")) | |
| 41 | + lo = pd.Timestamp(a.start) - pd.Timedelta(days=120) | |
| 42 | + hi = pd.Timestamp(a.end) if a.end else pd.Timestamp.today() | |
| 43 | + sel = cons[(cons["expiration_date"] >= lo) & (cons["first_data_date"].pipe(pd.to_datetime) <= hi)].sort_values("expiration_date") if "first_data_date" in cons else cons[cons["expiration_date"] >= lo].sort_values("expiration_date") | |
| 44 | + print(f"{root}: {len(sel)} contracts in scope ({sel['symbol'].iloc[0]} … {sel['symbol'].iloc[-1]})") | |
| 45 | + | |
| 46 | + panels: dict[str, pd.DataFrame] = {} | |
| 47 | + for sym in sel["symbol"]: | |
| 48 | + df = hfmd.contract_bars(sym, "1day", a.start and str((pd.Timestamp(a.start) - pd.Timedelta(days=120)).date()), a.end) | |
| 49 | + if df.empty: | |
| 50 | + hfmd.log(f" {sym}: no bars (gap reported, not filled)") | |
| 51 | + continue | |
| 52 | + panels[sym] = df.set_index("datetime") | |
| 53 | + if not panels: | |
| 54 | + return 1 | |
| 55 | + order = [s for s in sel["symbol"] if s in panels] | |
| 56 | + expiry = dict(zip(sel["symbol"], sel["expiration_date"])) | |
| 57 | + | |
| 58 | + # --- choose the active contract on each date --------------------------------------------------- | |
| 59 | + dates = sorted(set().union(*[p.index for p in panels.values()])) | |
| 60 | + dates = [d for d in dates if d >= pd.Timestamp(a.start)] | |
| 61 | + active_idx, active, rolls = 0, order[0], [] | |
| 62 | + chosen: list[tuple[pd.Timestamp, str]] = [] | |
| 63 | + for d in dates: | |
| 64 | + # skip contracts already expired | |
| 65 | + while active_idx + 1 < len(order) and expiry[active] < d: | |
| 66 | + active_idx += 1 | |
| 67 | + active = order[active_idx] | |
| 68 | + rolls.append((d, active, "expired")) | |
| 69 | + if active_idx + 1 < len(order): | |
| 70 | + nxt = order[active_idx + 1] | |
| 71 | + switch = False | |
| 72 | + if a.roll == "calendar": | |
| 73 | + switch = d >= expiry[active] - pd.tseries.offsets.BDay(a.roll_days) | |
| 74 | + elif d in panels[active].index and d in panels[nxt].index: | |
| 75 | + col = "volume" if a.roll == "volume" else "open_interest" | |
| 76 | + if col in panels[nxt].columns and col in panels[active].columns: | |
| 77 | + switch = float(panels[nxt].loc[d, col] or 0) > float(panels[active].loc[d, col] or 0) | |
| 78 | + if switch: | |
| 79 | + active_idx += 1 | |
| 80 | + active = nxt | |
| 81 | + rolls.append((d, active, a.roll)) | |
| 82 | + chosen.append((d, active)) | |
| 83 | + | |
| 84 | + # --- assemble + adjust -------------------------------------------------------------------------- | |
| 85 | + rows = [] | |
| 86 | + for d, sym in chosen: | |
| 87 | + if d in panels[sym].index: | |
| 88 | + r = panels[sym].loc[d] | |
| 89 | + rows.append({"datetime": d, "contract": sym, "open": r["open"], "high": r["high"], "low": r["low"], "close": r["close"], "volume": r.get("volume"), "open_interest": r.get("open_interest")}) | |
| 90 | + else: | |
| 91 | + rows.append({"datetime": d, "contract": sym, "open": np.nan, "high": np.nan, "low": np.nan, "close": np.nan, "volume": np.nan, "open_interest": np.nan}) | |
| 92 | + out = pd.DataFrame(rows).set_index("datetime") | |
| 93 | + | |
| 94 | + if a.adjust != "none": | |
| 95 | + adj = out.copy() | |
| 96 | + # walk backwards: at each roll, compute the gap between the new and old contract on the roll date | |
| 97 | + roll_dates = [d for d, _, _ in rolls if d in out.index] | |
| 98 | + for d in reversed(roll_dates): | |
| 99 | + i = out.index.get_loc(d) | |
| 100 | + new_sym, old_sym = out["contract"].iloc[i], out["contract"].iloc[i - 1] if i > 0 else None | |
| 101 | + if not old_sym or d not in panels[old_sym].index or d not in panels[new_sym].index: | |
| 102 | + continue | |
| 103 | + new_c, old_c = float(panels[new_sym].loc[d, "close"]), float(panels[old_sym].loc[d, "close"]) | |
| 104 | + before = adj.index < d | |
| 105 | + for col in ("open", "high", "low", "close"): | |
| 106 | + if a.adjust == "back_adjusted": | |
| 107 | + adj.loc[before, col] = adj.loc[before, col] + (new_c - old_c) | |
| 108 | + else: | |
| 109 | + adj.loc[before, col] = adj.loc[before, col] * (new_c / old_c if old_c else 1.0) | |
| 110 | + out = adj | |
| 111 | + | |
| 112 | + print(f"\nrolls ({a.roll}{' ' + str(a.roll_days) + 'bd' if a.roll == 'calendar' else ''}, adjust={a.adjust}): {len(rolls)}") | |
| 113 | + for d, sym, why in rolls: | |
| 114 | + print(f" {d.date()} → {sym} ({why})") | |
| 115 | + print(f"\n{len(out):,} rows {out.index[0].date()} → {out.index[-1].date()} · missing closes: {int(out['close'].isna().sum())}") | |
| 116 | + print(f"last close (adjusted={a.adjust}): {out['close'].iloc[-1]:.4f} · contract {out['contract'].iloc[-1]}") | |
| 117 | + if a.out: | |
| 118 | + out.to_csv(a.out) | |
| 119 | + print(f"wrote {a.out}") | |
| 120 | + return 0 | |
| 121 | + | |
| 122 | + | |
| 123 | +if __name__ == "__main__": | |
| 124 | + sys.exit(main()) | |
added
skills/hfmd-data-analysis/SKILL.md
+57 −0
@@ -0,0 +1,57 @@ | ||
| 1 | +--- | |
| 2 | +name: hfmd-data-analysis | |
| 3 | +description: Fetch historical OHLCV bars (stocks, ETFs, futures, crypto, indices, FX; 1-minute to daily) from HF Market Data into pandas and produce summary statistics, return/volatility/drawdown analysis and charts. Use when the user asks to analyse, describe, chart or export price history for a symbol or a basket. | |
| 4 | +--- | |
| 5 | + | |
| 6 | +# hfmd-data-analysis | |
| 7 | + | |
| 8 | +Pull price history from `https://www.hfmarketdata.io` (v1 `/v1/bars/{asset}/{ticker}`) into a DataFrame and | |
| 9 | +answer "what did this instrument do?" questions with numbers, not impressions. | |
| 10 | + | |
| 11 | +## When to use | |
| 12 | + | |
| 13 | +- "Show me AAPL daily since 2020 and summarise it", "how volatile was BTC in 2024?", "export SPY 5-minute bars for last week to CSV" | |
| 14 | +- Any request that needs bars in pandas before something else (feature engineering, correlations, seasonality) | |
| 15 | +- Not for backtests (use `hfmd-quick-backtest`), continuous futures methodology (`hfmd-continuous-futures`) or curves (`hfmd-term-structure`) | |
| 16 | + | |
| 17 | +## Inputs to confirm with the user | |
| 18 | + | |
| 19 | +| Parameter | Values | Default | | |
| 20 | +|---|---|---| | |
| 21 | +| asset | `stock` `etf` `crypto` `index` `fx` `futures` (vendor continuous) | infer from the symbol | | |
| 22 | +| timeframe | `1min` `5min` `30min` `1hour` `1day` | `1day` | | |
| 23 | +| start / end | `YYYY-MM-DD` (intraday: keep ranges short — one day of 1-min bars ≈ 390 rows RTH, 1 440 for crypto) | last 5 years for daily | | |
| 24 | +| adjustment | stock/etf: `UNADJUSTED` `adj_split` `adj_splitdiv` · futures: `contin_UNadj` `contin_adj_ratio` `contin_adj_absolute` | API default (`adj_splitdiv` for stocks) | | |
| 25 | + | |
| 26 | +## Steps | |
| 27 | + | |
| 28 | +1. Resolve the symbol if unsure: `GET /v1/{asset}/tickers?search=AAP` (or ask). | |
| 29 | +2. Fetch: `python3 scripts/fetch_bars.py --asset stock --ticker AAPL --timeframe 1day --start 2020-01-01 --out aapl.csv` | |
| 30 | + - paginates automatically (keyless max 5 000 rows/request) and prints the exact range received — never assume the range you asked for is the range you got. | |
| 31 | +3. Analyse: `python3 scripts/analyze.py aapl.csv --plot aapl.png` | |
| 32 | + - prints: rows, first/last bar, CAGR, annualised volatility, Sharpe (rf = 0), max drawdown (with dates), best/worst bar, skew/kurtosis, gap count, monthly return table; writes a price + drawdown chart if `--plot`. | |
| 33 | +4. Report the numbers with their window and adjustment; flag missing sessions instead of filling them. | |
| 34 | + | |
| 35 | +## Examples | |
| 36 | + | |
| 37 | +```bash | |
| 38 | +# 1. Five years of daily SPY, summary + chart | |
| 39 | +python3 scripts/fetch_bars.py --asset etf --ticker SPY --start 2021-01-01 --out spy.csv && python3 scripts/analyze.py spy.csv --plot spy.png | |
| 40 | + | |
| 41 | +# 2. One week of 5-minute bars for TSLA (intraday: naive US/Eastern timestamps) | |
| 42 | +python3 scripts/fetch_bars.py --asset stock --ticker TSLA --timeframe 5min --start 2025-08-25 --end 2025-08-29 --out tsla_5m.csv | |
| 43 | + | |
| 44 | +# 3. Vendor continuous crude oil, ratio-adjusted, to parquet | |
| 45 | +python3 scripts/fetch_bars.py --asset futures --ticker CL --adjustment contin_adj_ratio --start 2015-01-01 --out cl.parquet | |
| 46 | + | |
| 47 | +# 4. Several tickers → one long DataFrame | |
| 48 | +python3 scripts/fetch_bars.py --asset stock --ticker AAPL MSFT NVDA --start 2024-01-01 --out mega.csv | |
| 49 | +``` | |
| 50 | + | |
| 51 | +## Gotchas | |
| 52 | + | |
| 53 | +- Intraday `datetime` is naive **US/Eastern** (exchange time); daily bars are plain dates. Localise before joining with UTC data. | |
| 54 | +- Stocks default to split+dividend adjusted prices; say so when quoting historical levels. | |
| 55 | +- Keyless quota: 30 requests/hour, 5 000 rows/request. Set `HFMD_API_KEY` (free account, 120 req/min, 50 000 rows/request) before pulling intraday history. | |
| 56 | +- `volume` for FX/indices may be 0/absent — do not compute volume statistics on them. | |
| 57 | +- Error `TICKER_NOT_FOUND` usually means the symbol exists in another asset class (e.g. `SPY` is `etf`, not `stock`). | |
added
skills/hfmd-data-analysis/scripts/analyze.py
+119 −0
@@ -0,0 +1,119 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +"""Summary statistics for a bars file produced by fetch_bars.py (one or several tickers). | |
| 3 | + | |
| 4 | + analyze.py aapl.csv [--plot aapl.png] [--periods-per-year 252] | |
| 5 | + | |
| 6 | +Prints, per ticker: rows, range, CAGR, annualised volatility, Sharpe (rf=0), max drawdown (+ dates), | |
| 7 | +best/worst bar, skew, kurtosis, number of gaps > 3 calendar days, monthly returns table. | |
| 8 | +""" | |
| 9 | +from __future__ import annotations | |
| 10 | + | |
| 11 | +import argparse | |
| 12 | +import sys | |
| 13 | + | |
| 14 | +import numpy as np | |
| 15 | +import pandas as pd | |
| 16 | + | |
| 17 | + | |
| 18 | +def infer_periods_per_year(dt: pd.Series) -> int: | |
| 19 | + if len(dt) < 3: | |
| 20 | + return 252 | |
| 21 | + step = dt.diff().dropna().median() | |
| 22 | + if step >= pd.Timedelta(days=1): | |
| 23 | + return 252 if dt.dt.dayofweek.max() <= 4 else 365 | |
| 24 | + per_day = int(pd.Timedelta(hours=6.5) / step) if step < pd.Timedelta(hours=1) else int(pd.Timedelta(hours=24) / step) | |
| 25 | + return max(per_day, 1) * 252 | |
| 26 | + | |
| 27 | + | |
| 28 | +def stats(df: pd.DataFrame, ppy: int | None) -> dict: | |
| 29 | + df = df.sort_values("datetime").reset_index(drop=True) | |
| 30 | + ppy = ppy or infer_periods_per_year(df["datetime"]) | |
| 31 | + close = df["close"].astype(float) | |
| 32 | + ret = close.pct_change().dropna() | |
| 33 | + years = max((df["datetime"].iloc[-1] - df["datetime"].iloc[0]).days / 365.25, 1e-9) | |
| 34 | + equity = (1 + ret).cumprod() | |
| 35 | + dd = equity / equity.cummax() - 1 | |
| 36 | + trough = dd.idxmin() if len(dd) else None | |
| 37 | + peak = equity.loc[:trough].idxmax() if trough is not None else None | |
| 38 | + gaps = (df["datetime"].diff() > pd.Timedelta(days=3)).sum() | |
| 39 | + out = { | |
| 40 | + "rows": len(df), | |
| 41 | + "first": str(df["datetime"].iloc[0]), | |
| 42 | + "last": str(df["datetime"].iloc[-1]), | |
| 43 | + "periods_per_year": ppy, | |
| 44 | + "total_return": close.iloc[-1] / close.iloc[0] - 1, | |
| 45 | + "cagr": (close.iloc[-1] / close.iloc[0]) ** (1 / years) - 1, | |
| 46 | + "ann_vol": ret.std() * np.sqrt(ppy), | |
| 47 | + "sharpe_rf0": (ret.mean() / ret.std() * np.sqrt(ppy)) if ret.std() > 0 else np.nan, | |
| 48 | + "max_drawdown": dd.min() if len(dd) else np.nan, | |
| 49 | + "dd_peak": str(df["datetime"].iloc[peak]) if peak is not None else None, | |
| 50 | + "dd_trough": str(df["datetime"].iloc[trough]) if trough is not None else None, | |
| 51 | + "best_bar": ret.max(), | |
| 52 | + "worst_bar": ret.min(), | |
| 53 | + "skew": ret.skew(), | |
| 54 | + "kurtosis": ret.kurt(), | |
| 55 | + "gaps_gt_3d": int(gaps), | |
| 56 | + "avg_volume": float(df["volume"].mean()) if "volume" in df else None, | |
| 57 | + } | |
| 58 | + return out | |
| 59 | + | |
| 60 | + | |
| 61 | +def monthly_table(df: pd.DataFrame) -> pd.DataFrame: | |
| 62 | + s = df.set_index("datetime")["close"].astype(float).resample("ME").last().pct_change().dropna() | |
| 63 | + t = s.to_frame("r") | |
| 64 | + t["year"], t["month"] = t.index.year, t.index.month | |
| 65 | + return (t.pivot(index="year", columns="month", values="r") * 100).round(1) | |
| 66 | + | |
| 67 | + | |
| 68 | +def main() -> int: | |
| 69 | + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) | |
| 70 | + ap.add_argument("file") | |
| 71 | + ap.add_argument("--plot", help="write a price + drawdown PNG (needs matplotlib)") | |
| 72 | + ap.add_argument("--periods-per-year", type=int) | |
| 73 | + a = ap.parse_args() | |
| 74 | + | |
| 75 | + df = pd.read_parquet(a.file) if a.file.endswith(".parquet") else pd.read_csv(a.file) | |
| 76 | + df["datetime"] = pd.to_datetime(df["datetime"]) | |
| 77 | + tickers = df["ticker"].unique() if "ticker" in df else ["?"] | |
| 78 | + for t in tickers: | |
| 79 | + sub = df[df["ticker"] == t] if "ticker" in df else df | |
| 80 | + s = stats(sub, a.periods_per_year) | |
| 81 | + print(f"\n=== {t} ===") | |
| 82 | + for k, v in s.items(): | |
| 83 | + if isinstance(v, float): | |
| 84 | + print(f"{k:>18}: {v:,.4f}" if abs(v) < 1000 else f"{k:>18}: {v:,.0f}") | |
| 85 | + else: | |
| 86 | + print(f"{k:>18}: {v}") | |
| 87 | + if s["periods_per_year"] <= 365 and len(sub) > 40: | |
| 88 | + print("\nmonthly returns (%):") | |
| 89 | + print(monthly_table(sub).to_string()) | |
| 90 | + | |
| 91 | + if a.plot: | |
| 92 | + try: | |
| 93 | + import matplotlib | |
| 94 | + matplotlib.use("Agg") | |
| 95 | + import matplotlib.pyplot as plt | |
| 96 | + except ImportError: | |
| 97 | + print("matplotlib not installed: pip install matplotlib", file=sys.stderr) | |
| 98 | + return 0 | |
| 99 | + fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(11, 6.5), sharex=True, gridspec_kw={"height_ratios": [3, 1]}) | |
| 100 | + for t in tickers: | |
| 101 | + sub = (df[df["ticker"] == t] if "ticker" in df else df).sort_values("datetime") | |
| 102 | + close = sub["close"].astype(float) | |
| 103 | + norm = close / close.iloc[0] * 100 if len(tickers) > 1 else close | |
| 104 | + ax1.plot(sub["datetime"], norm, lw=1.1, label=t) | |
| 105 | + eq = (1 + close.pct_change().fillna(0)).cumprod() | |
| 106 | + ax2.fill_between(sub["datetime"], (eq / eq.cummax() - 1) * 100, 0, alpha=0.35) | |
| 107 | + ax1.set_title(f"{', '.join(tickers)} — {'rebased to 100' if len(tickers) > 1 else 'close'}") | |
| 108 | + ax1.grid(alpha=0.25) | |
| 109 | + ax1.legend(loc="upper left") | |
| 110 | + ax2.set_ylabel("drawdown %") | |
| 111 | + ax2.grid(alpha=0.25) | |
| 112 | + fig.tight_layout() | |
| 113 | + fig.savefig(a.plot, dpi=130) | |
| 114 | + print(f"\nwrote {a.plot}") | |
| 115 | + return 0 | |
| 116 | + | |
| 117 | + | |
| 118 | +if __name__ == "__main__": | |
| 119 | + sys.exit(main()) | |
added
skills/hfmd-data-analysis/scripts/fetch_bars.py
+52 −0
@@ -0,0 +1,52 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +"""Fetch OHLCV bars from HF Market Data into CSV or Parquet. | |
| 3 | + | |
| 4 | +Examples: | |
| 5 | + fetch_bars.py --asset stock --ticker AAPL --start 2020-01-01 --out aapl.csv | |
| 6 | + fetch_bars.py --asset stock --ticker AAPL MSFT --timeframe 1hour --start 2025-08-01 --out both.csv | |
| 7 | + fetch_bars.py --asset futures --ticker CL --adjustment contin_adj_ratio --out cl.parquet | |
| 8 | +""" | |
| 9 | +from __future__ import annotations | |
| 10 | + | |
| 11 | +import argparse | |
| 12 | +import sys | |
| 13 | +from pathlib import Path | |
| 14 | + | |
| 15 | +import pandas as pd | |
| 16 | + | |
| 17 | +sys.path.insert(0, str(Path(__file__).parent)) | |
| 18 | +import hfmd # noqa: E402 | |
| 19 | + | |
| 20 | + | |
| 21 | +def main() -> int: | |
| 22 | + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) | |
| 23 | + ap.add_argument("--asset", required=True, choices=["stock", "etf", "crypto", "index", "fx", "futures"]) | |
| 24 | + ap.add_argument("--ticker", required=True, nargs="+", help="one or more symbols") | |
| 25 | + ap.add_argument("--timeframe", default="1day", help="1min 5min 30min 1hour 1day (aliases 1m 5m 30m 1h 1d)") | |
| 26 | + ap.add_argument("--start") | |
| 27 | + ap.add_argument("--end") | |
| 28 | + ap.add_argument("--adjustment", help="stock/etf: UNADJUSTED adj_split adj_splitdiv · futures: contin_UNadj contin_adj_ratio contin_adj_absolute") | |
| 29 | + ap.add_argument("--out", required=True, help=".csv or .parquet") | |
| 30 | + a = ap.parse_args() | |
| 31 | + | |
| 32 | + frames = [] | |
| 33 | + for t in a.ticker: | |
| 34 | + df = hfmd.bars(a.asset, t.upper(), a.timeframe, a.start, a.end, a.adjustment) | |
| 35 | + if df.empty: | |
| 36 | + hfmd.log(f"{t}: no rows returned") | |
| 37 | + continue | |
| 38 | + hfmd.log(f"{t}: {len(df):,} bars from {df['datetime'].iloc[0]} to {df['datetime'].iloc[-1]}") | |
| 39 | + frames.append(df) | |
| 40 | + if not frames: | |
| 41 | + return 1 | |
| 42 | + out = pd.concat(frames, ignore_index=True) | |
| 43 | + if a.out.endswith(".parquet"): | |
| 44 | + out.to_parquet(a.out, index=False) | |
| 45 | + else: | |
| 46 | + out.to_csv(a.out, index=False) | |
| 47 | + print(f"wrote {a.out}: {len(out):,} rows, {out['ticker'].nunique()} ticker(s), columns={list(out.columns)}") | |
| 48 | + return 0 | |
| 49 | + | |
| 50 | + | |
| 51 | +if __name__ == "__main__": | |
| 52 | + sys.exit(main()) | |
added
skills/hfmd-data-analysis/scripts/hfmd.py
+156 −0
@@ -0,0 +1,156 @@ | ||
| 1 | +"""Tiny HF Market Data client shared by the hfmd-* skills (requests + pandas only). | |
| 2 | + | |
| 3 | +Copied verbatim into every skill's `scripts/` folder by `skills/scripts/build_skills.py` | |
| 4 | +so each skill stays self-contained. Do not edit the copies — edit `skills/_shared/hfmd.py`. | |
| 5 | + | |
| 6 | +Environment: HFMD_API_KEY (optional, free account = 120 req/min; keyless = 30 req/h), | |
| 7 | + HFMD_BASE_URL (default https://www.hfmarketdata.io). | |
| 8 | +""" | |
| 9 | +from __future__ import annotations | |
| 10 | + | |
| 11 | +import os | |
| 12 | +import sys | |
| 13 | +import time | |
| 14 | +from typing import Any | |
| 15 | + | |
| 16 | +import pandas as pd | |
| 17 | +import requests | |
| 18 | + | |
| 19 | +BASE_URL = os.environ.get("HFMD_BASE_URL", "https://www.hfmarketdata.io").rstrip("/") | |
| 20 | +API_KEY = os.environ.get("HFMD_API_KEY", "").strip() | |
| 21 | +UA = "hfmd-skills/1.0 (+https://www.hfmarketdata.io/integrations/skills)" | |
| 22 | + | |
| 23 | +TIMEFRAMES = {"1m": "1min", "5m": "5min", "30m": "30min", "1h": "1hour", "1d": "1day", | |
| 24 | + "1min": "1min", "5min": "5min", "30min": "30min", "1hour": "1hour", "1day": "1day"} | |
| 25 | +V2_INTERVAL = {"1min": "1m", "5min": "5m", "30min": "30m", "1hour": "1h", "1day": "1d"} | |
| 26 | + | |
| 27 | + | |
| 28 | +class HfmdError(RuntimeError): | |
| 29 | + def __init__(self, status: int, code: str, message: str, url: str): | |
| 30 | + super().__init__(f"HF Market Data error {status} {code}: {message} ({url})") | |
| 31 | + self.status, self.code, self.url = status, code, url | |
| 32 | + | |
| 33 | + | |
| 34 | +_session = requests.Session() | |
| 35 | +_session.headers.update({"Accept": "application/json", "User-Agent": UA}) | |
| 36 | +if API_KEY: | |
| 37 | + _session.headers["Authorization"] = f"Bearer {API_KEY}" | |
| 38 | + | |
| 39 | + | |
| 40 | +def log(msg: str) -> None: | |
| 41 | + print(msg, file=sys.stderr) | |
| 42 | + | |
| 43 | + | |
| 44 | +def get(path: str, params: dict[str, Any] | None = None, *, retries: int = 3) -> tuple[Any, dict[str, str]]: | |
| 45 | + """GET a JSON endpoint. Returns (body, headers). Retries on 429 honouring Retry-After.""" | |
| 46 | + url = f"{BASE_URL}{path}" | |
| 47 | + p = {k: (",".join(v) if isinstance(v, (list, tuple)) else v) for k, v in (params or {}).items() if v not in (None, "")} | |
| 48 | + p.setdefault("format", "json") | |
| 49 | + for attempt in range(retries + 1): | |
| 50 | + r = _session.get(url, params=p, timeout=120) | |
| 51 | + if r.status_code == 429 and attempt < retries: | |
| 52 | + wait = float(r.headers.get("Retry-After", "5")) | |
| 53 | + log(f"429 rate limited — sleeping {wait:.0f}s (set HFMD_API_KEY for 120 req/min)") | |
| 54 | + time.sleep(min(wait, 120)) | |
| 55 | + continue | |
| 56 | + if r.status_code >= 400: | |
| 57 | + try: | |
| 58 | + body = r.json() | |
| 59 | + except ValueError: | |
| 60 | + body = {} | |
| 61 | + err = body.get("error") or {} | |
| 62 | + raise HfmdError(r.status_code, err.get("code", "HTTP_ERROR"), err.get("message") or body.get("detail") or r.text[:200], r.url) | |
| 63 | + rem = r.headers.get("X-RateLimit-Remaining-Requests") | |
| 64 | + if rem is not None: | |
| 65 | + log(f"rate limit: {rem}/{r.headers.get('X-RateLimit-Limit-Requests')} requests left") | |
| 66 | + return r.json(), dict(r.headers) | |
| 67 | + raise HfmdError(429, "RATE_LIMIT_EXCEEDED", "gave up after retries", url) | |
| 68 | + | |
| 69 | + | |
| 70 | +def rows_of(body: Any) -> list[dict]: | |
| 71 | + """Rows from either envelope: v1 {count,data} or v2 {data,meta}.""" | |
| 72 | + if isinstance(body, list): | |
| 73 | + return body | |
| 74 | + if isinstance(body, dict): | |
| 75 | + data = body.get("data") | |
| 76 | + if isinstance(data, list): | |
| 77 | + return data | |
| 78 | + return [] | |
| 79 | + | |
| 80 | + | |
| 81 | +def meta_of(body: Any) -> dict: | |
| 82 | + if isinstance(body, dict): | |
| 83 | + m = dict(body.get("meta") or {}) | |
| 84 | + if "count" in body and "count" not in m: | |
| 85 | + m["count"] = body["count"] | |
| 86 | + return m | |
| 87 | + return {} | |
| 88 | + | |
| 89 | + | |
| 90 | +def to_df(body: Any, time_col: str | None = "datetime") -> pd.DataFrame: | |
| 91 | + df = pd.DataFrame(rows_of(body)) | |
| 92 | + if time_col and time_col in df.columns: | |
| 93 | + df[time_col] = pd.to_datetime(df[time_col]) | |
| 94 | + df = df.sort_values(time_col).reset_index(drop=True) | |
| 95 | + return df | |
| 96 | + | |
| 97 | + | |
| 98 | +def bars(asset: str, ticker: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, | |
| 99 | + adjustment: str | None = None, limit: int = 50_000) -> pd.DataFrame: | |
| 100 | + """v1 bars (stock/etf/crypto/index/fx, or futures = vendor continuous). Paginates by date when needed.""" | |
| 101 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 102 | + frames: list[pd.DataFrame] = [] | |
| 103 | + cursor_start = start | |
| 104 | + while True: | |
| 105 | + body, _ = get(f"/v1/bars/{asset}/{ticker}", {"timeframe": tf, "start": cursor_start, "end": end, "adjustment": adjustment, "limit": limit, "order": "asc"}) | |
| 106 | + df = to_df(body) | |
| 107 | + if df.empty: | |
| 108 | + break | |
| 109 | + frames.append(df) | |
| 110 | + if len(df) < limit: | |
| 111 | + break | |
| 112 | + last = df["datetime"].iloc[-1] | |
| 113 | + cursor_start = (last + pd.Timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M") if tf != "1day" else (last + pd.Timedelta(days=1)).strftime("%Y-%m-%d") | |
| 114 | + out = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(columns=["ticker", "datetime", "open", "high", "low", "close", "volume"]) | |
| 115 | + return out.drop_duplicates("datetime").reset_index(drop=True) | |
| 116 | + | |
| 117 | + | |
| 118 | +def contract_bars(symbol: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, limit: int = 50_000) -> pd.DataFrame: | |
| 119 | + """v2 individual futures contract bars (ESZ25…).""" | |
| 120 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 121 | + body, _ = get(f"/v1/futures/contract/{symbol}/bars", {"interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit}) | |
| 122 | + return to_df(body) | |
| 123 | + | |
| 124 | + | |
| 125 | +def continuous(root: str, roll: str = "volume", adjust: str = "back_adjusted", depth: int = 1, timeframe: str = "1day", | |
| 126 | + start: str | None = None, end: str | None = None, limit: int = 50_000) -> tuple[pd.DataFrame, dict]: | |
| 127 | + """v2 server-built continuous series. Returns (df, meta) — meta['roll_dates'] lists the rolls.""" | |
| 128 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 129 | + body, _ = get(f"/v1/futures/{root}/continuous", {"roll": roll, "adjust": adjust, "depth": depth, "interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit}) | |
| 130 | + return to_df(body), meta_of(body) | |
| 131 | + | |
| 132 | + | |
| 133 | +def contracts(root: str, status: str | None = None) -> pd.DataFrame: | |
| 134 | + body, _ = get(f"/v1/futures/{root}/contracts", {"status": status, "limit": 1000}) | |
| 135 | + return to_df(body, time_col=None) | |
| 136 | + | |
| 137 | + | |
| 138 | +def term_structure(root: str, as_of: str | None = None) -> tuple[pd.DataFrame, dict]: | |
| 139 | + body, _ = get(f"/v1/futures/{root}/term-structure", {"as_of": as_of}) | |
| 140 | + return to_df(body, time_col=None), meta_of(body) | |
| 141 | + | |
| 142 | + | |
| 143 | +def screener(filters: str, sort: str | None = None, as_of: str | None = None, limit: int = 100) -> tuple[pd.DataFrame, dict]: | |
| 144 | + body, _ = get("/v1/fundamentals/screener", {"filters": filters, "sort": sort, "as_of": as_of, "limit": limit}) | |
| 145 | + return to_df(body, time_col=None), meta_of(body) | |
| 146 | + | |
| 147 | + | |
| 148 | +def ratios(ticker: str) -> dict: | |
| 149 | + body, _ = get(f"/v1/fundamentals/{ticker}/ratios") | |
| 150 | + d = body.get("data", body) if isinstance(body, dict) else body | |
| 151 | + return d if isinstance(d, dict) else (d[0] if d else {}) | |
| 152 | + | |
| 153 | + | |
| 154 | +def statements(ticker: str, statement: str = "income", period: str = "quarterly", limit: int = 12) -> pd.DataFrame: | |
| 155 | + body, _ = get(f"/v1/fundamentals/{ticker}/statements", {"statement": statement, "period": period, "limit": limit}) | |
| 156 | + return to_df(body, time_col=None) | |
added
skills/hfmd-fundamentals-screen/SKILL.md
+57 −0
@@ -0,0 +1,57 @@ | ||
| 1 | +--- | |
| 2 | +name: hfmd-fundamentals-screen | |
| 3 | +description: Screen US stocks on fundamental ratios with HF Market Data (PE, PB, EV/EBITDA, FCF yield, ROE, margins, leverage, growth), enrich the hits with their latest ratios and recent quarterly statements, and document the point-in-time caveats (filing dates, restatements). Use when the user asks to "find stocks with…", "screen for…", or compare companies on fundamentals. | |
| 4 | +--- | |
| 5 | + | |
| 6 | +# hfmd-fundamentals-screen | |
| 7 | + | |
| 8 | +Run a screener → pull ratios and statements for the survivors → produce a table the user can defend, | |
| 9 | +with the date each number became public. | |
| 10 | + | |
| 11 | +## When to use | |
| 12 | + | |
| 13 | +- "Screen US stocks with PE<15 and FCF yield>6% and show their last 3 quarters" | |
| 14 | +- "Cheapest software companies by EV/EBITDA with ROE > 20 %", "which of these tickers has the best balance sheet?" | |
| 15 | +- Historical: "what would this screen have returned on 2020-03-20?" (`--as-of`, point-in-time) | |
| 16 | + | |
| 17 | +## Filter syntax | |
| 18 | + | |
| 19 | +`filters="pe<15,roe>0.15,fcf_yield>0.06,market_cap>1e9"` — comma-separated `metric<op>value`, ops `< <= > >= =`. | |
| 20 | +Ratios are **decimals** (0.15 = 15 %). Known metrics: `pe pb ps ev_ebitda fcf_yield dividend_yield roe roa gross_margin operating_margin net_margin debt_to_equity current_ratio revenue_growth eps_growth market_cap sector`. | |
| 21 | +Sort: `--sort fcf_yield:desc`. | |
| 22 | + | |
| 23 | +## Steps | |
| 24 | + | |
| 25 | +1. Translate the request into filters (ask for thresholds when vague: "cheap" → `pe<15`? `ev_ebitda<8`?). Convert percentages to decimals. | |
| 26 | +2. Screen + enrich: | |
| 27 | + `python3 scripts/screen.py --filters "pe<15,fcf_yield>0.06" --sort fcf_yield:desc --limit 25 --quarters 3 --out screen.csv` | |
| 28 | + - calls `/v1/fundamentals/screener` (cost: 2 requests), then for each hit `/v1/fundamentals/{t}/ratios` and `/v1/fundamentals/{t}/statements?statement=income&period=quarterly&limit=3` (`--no-enrich` to skip; keyless quota allows ~10 tickers/hour — set `HFMD_API_KEY`). | |
| 29 | + - prints the screen table, then per ticker the last N quarters (revenue, operating income, net income, EPS, filed_at) and a **point-in-time note**: latest `filed_at` used, days since, whether a newer period end exists without a filing yet. | |
| 30 | +3. Present: one table sorted as requested, then the quarterly mini-tables, then caveats. | |
| 31 | + | |
| 32 | +## Point-in-time discipline (always say it) | |
| 33 | + | |
| 34 | +- Every fundamentals row carries `period_end` **and** `filed_at`. A number is knowable only from `filed_at` (10-Q ≈ 40 days, 10-K ≈ 60-90 days after period end). Screens with `--as-of` use what was filed by that date — this is what makes a backtest of the screen honest. | |
| 35 | +- Restatements: `facts/{concept}` returns every reported value for a period; the statements endpoint returns the latest. Mention it when a number looks off. | |
| 36 | +- Price-based ratios (PE, FCF yield…) use the close of `as_of` (or the latest) and the last **reported** TTM figures — not analyst estimates. | |
| 37 | +- Coverage differs by company (`/v1/fundamentals/{t}/coverage`): small caps and recent IPOs have short histories; missing = `null`, never imputed. | |
| 38 | + | |
| 39 | +## Examples | |
| 40 | + | |
| 41 | +```bash | |
| 42 | +# Classic value + quality | |
| 43 | +python3 scripts/screen.py --filters "pe<15,roe>0.15,debt_to_equity<1" --sort pe:asc --limit 20 | |
| 44 | + | |
| 45 | +# Point-in-time: what the screen showed at the March 2020 low | |
| 46 | +python3 scripts/screen.py --filters "fcf_yield>0.08,net_margin>0.1" --as-of 2020-03-20 --no-enrich | |
| 47 | + | |
| 48 | +# Only enrich a list you already have | |
| 49 | +python3 scripts/screen.py --tickers AAPL MSFT GOOGL --quarters 4 | |
| 50 | +``` | |
| 51 | + | |
| 52 | +## Gotchas | |
| 53 | + | |
| 54 | +- A screener 404 means the fundamentals module is not deployed yet on the server; the script says so and exits 2. | |
| 55 | +- `sector` is a string filter (`sector=Technology`); everything else is numeric. | |
| 56 | +- Banks/insurers: EV/EBITDA and FCF yield are not meaningful — prefer PB, ROE. | |
| 57 | +- The screener costs 2 quota requests; with 30/h keyless, do not loop over many parameter sets without a key. | |
added
skills/hfmd-fundamentals-screen/scripts/hfmd.py
+156 −0
@@ -0,0 +1,156 @@ | ||
| 1 | +"""Tiny HF Market Data client shared by the hfmd-* skills (requests + pandas only). | |
| 2 | + | |
| 3 | +Copied verbatim into every skill's `scripts/` folder by `skills/scripts/build_skills.py` | |
| 4 | +so each skill stays self-contained. Do not edit the copies — edit `skills/_shared/hfmd.py`. | |
| 5 | + | |
| 6 | +Environment: HFMD_API_KEY (optional, free account = 120 req/min; keyless = 30 req/h), | |
| 7 | + HFMD_BASE_URL (default https://www.hfmarketdata.io). | |
| 8 | +""" | |
| 9 | +from __future__ import annotations | |
| 10 | + | |
| 11 | +import os | |
| 12 | +import sys | |
| 13 | +import time | |
| 14 | +from typing import Any | |
| 15 | + | |
| 16 | +import pandas as pd | |
| 17 | +import requests | |
| 18 | + | |
| 19 | +BASE_URL = os.environ.get("HFMD_BASE_URL", "https://www.hfmarketdata.io").rstrip("/") | |
| 20 | +API_KEY = os.environ.get("HFMD_API_KEY", "").strip() | |
| 21 | +UA = "hfmd-skills/1.0 (+https://www.hfmarketdata.io/integrations/skills)" | |
| 22 | + | |
| 23 | +TIMEFRAMES = {"1m": "1min", "5m": "5min", "30m": "30min", "1h": "1hour", "1d": "1day", | |
| 24 | + "1min": "1min", "5min": "5min", "30min": "30min", "1hour": "1hour", "1day": "1day"} | |
| 25 | +V2_INTERVAL = {"1min": "1m", "5min": "5m", "30min": "30m", "1hour": "1h", "1day": "1d"} | |
| 26 | + | |
| 27 | + | |
| 28 | +class HfmdError(RuntimeError): | |
| 29 | + def __init__(self, status: int, code: str, message: str, url: str): | |
| 30 | + super().__init__(f"HF Market Data error {status} {code}: {message} ({url})") | |
| 31 | + self.status, self.code, self.url = status, code, url | |
| 32 | + | |
| 33 | + | |
| 34 | +_session = requests.Session() | |
| 35 | +_session.headers.update({"Accept": "application/json", "User-Agent": UA}) | |
| 36 | +if API_KEY: | |
| 37 | + _session.headers["Authorization"] = f"Bearer {API_KEY}" | |
| 38 | + | |
| 39 | + | |
| 40 | +def log(msg: str) -> None: | |
| 41 | + print(msg, file=sys.stderr) | |
| 42 | + | |
| 43 | + | |
| 44 | +def get(path: str, params: dict[str, Any] | None = None, *, retries: int = 3) -> tuple[Any, dict[str, str]]: | |
| 45 | + """GET a JSON endpoint. Returns (body, headers). Retries on 429 honouring Retry-After.""" | |
| 46 | + url = f"{BASE_URL}{path}" | |
| 47 | + p = {k: (",".join(v) if isinstance(v, (list, tuple)) else v) for k, v in (params or {}).items() if v not in (None, "")} | |
| 48 | + p.setdefault("format", "json") | |
| 49 | + for attempt in range(retries + 1): | |
| 50 | + r = _session.get(url, params=p, timeout=120) | |
| 51 | + if r.status_code == 429 and attempt < retries: | |
| 52 | + wait = float(r.headers.get("Retry-After", "5")) | |
| 53 | + log(f"429 rate limited — sleeping {wait:.0f}s (set HFMD_API_KEY for 120 req/min)") | |
| 54 | + time.sleep(min(wait, 120)) | |
| 55 | + continue | |
| 56 | + if r.status_code >= 400: | |
| 57 | + try: | |
| 58 | + body = r.json() | |
| 59 | + except ValueError: | |
| 60 | + body = {} | |
| 61 | + err = body.get("error") or {} | |
| 62 | + raise HfmdError(r.status_code, err.get("code", "HTTP_ERROR"), err.get("message") or body.get("detail") or r.text[:200], r.url) | |
| 63 | + rem = r.headers.get("X-RateLimit-Remaining-Requests") | |
| 64 | + if rem is not None: | |
| 65 | + log(f"rate limit: {rem}/{r.headers.get('X-RateLimit-Limit-Requests')} requests left") | |
| 66 | + return r.json(), dict(r.headers) | |
| 67 | + raise HfmdError(429, "RATE_LIMIT_EXCEEDED", "gave up after retries", url) | |
| 68 | + | |
| 69 | + | |
| 70 | +def rows_of(body: Any) -> list[dict]: | |
| 71 | + """Rows from either envelope: v1 {count,data} or v2 {data,meta}.""" | |
| 72 | + if isinstance(body, list): | |
| 73 | + return body | |
| 74 | + if isinstance(body, dict): | |
| 75 | + data = body.get("data") | |
| 76 | + if isinstance(data, list): | |
| 77 | + return data | |
| 78 | + return [] | |
| 79 | + | |
| 80 | + | |
| 81 | +def meta_of(body: Any) -> dict: | |
| 82 | + if isinstance(body, dict): | |
| 83 | + m = dict(body.get("meta") or {}) | |
| 84 | + if "count" in body and "count" not in m: | |
| 85 | + m["count"] = body["count"] | |
| 86 | + return m | |
| 87 | + return {} | |
| 88 | + | |
| 89 | + | |
| 90 | +def to_df(body: Any, time_col: str | None = "datetime") -> pd.DataFrame: | |
| 91 | + df = pd.DataFrame(rows_of(body)) | |
| 92 | + if time_col and time_col in df.columns: | |
| 93 | + df[time_col] = pd.to_datetime(df[time_col]) | |
| 94 | + df = df.sort_values(time_col).reset_index(drop=True) | |
| 95 | + return df | |
| 96 | + | |
| 97 | + | |
| 98 | +def bars(asset: str, ticker: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, | |
| 99 | + adjustment: str | None = None, limit: int = 50_000) -> pd.DataFrame: | |
| 100 | + """v1 bars (stock/etf/crypto/index/fx, or futures = vendor continuous). Paginates by date when needed.""" | |
| 101 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 102 | + frames: list[pd.DataFrame] = [] | |
| 103 | + cursor_start = start | |
| 104 | + while True: | |
| 105 | + body, _ = get(f"/v1/bars/{asset}/{ticker}", {"timeframe": tf, "start": cursor_start, "end": end, "adjustment": adjustment, "limit": limit, "order": "asc"}) | |
| 106 | + df = to_df(body) | |
| 107 | + if df.empty: | |
| 108 | + break | |
| 109 | + frames.append(df) | |
| 110 | + if len(df) < limit: | |
| 111 | + break | |
| 112 | + last = df["datetime"].iloc[-1] | |
| 113 | + cursor_start = (last + pd.Timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M") if tf != "1day" else (last + pd.Timedelta(days=1)).strftime("%Y-%m-%d") | |
| 114 | + out = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(columns=["ticker", "datetime", "open", "high", "low", "close", "volume"]) | |
| 115 | + return out.drop_duplicates("datetime").reset_index(drop=True) | |
| 116 | + | |
| 117 | + | |
| 118 | +def contract_bars(symbol: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, limit: int = 50_000) -> pd.DataFrame: | |
| 119 | + """v2 individual futures contract bars (ESZ25…).""" | |
| 120 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 121 | + body, _ = get(f"/v1/futures/contract/{symbol}/bars", {"interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit}) | |
| 122 | + return to_df(body) | |
| 123 | + | |
| 124 | + | |
| 125 | +def continuous(root: str, roll: str = "volume", adjust: str = "back_adjusted", depth: int = 1, timeframe: str = "1day", | |
| 126 | + start: str | None = None, end: str | None = None, limit: int = 50_000) -> tuple[pd.DataFrame, dict]: | |
| 127 | + """v2 server-built continuous series. Returns (df, meta) — meta['roll_dates'] lists the rolls.""" | |
| 128 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 129 | + body, _ = get(f"/v1/futures/{root}/continuous", {"roll": roll, "adjust": adjust, "depth": depth, "interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit}) | |
| 130 | + return to_df(body), meta_of(body) | |
| 131 | + | |
| 132 | + | |
| 133 | +def contracts(root: str, status: str | None = None) -> pd.DataFrame: | |
| 134 | + body, _ = get(f"/v1/futures/{root}/contracts", {"status": status, "limit": 1000}) | |
| 135 | + return to_df(body, time_col=None) | |
| 136 | + | |
| 137 | + | |
| 138 | +def term_structure(root: str, as_of: str | None = None) -> tuple[pd.DataFrame, dict]: | |
| 139 | + body, _ = get(f"/v1/futures/{root}/term-structure", {"as_of": as_of}) | |
| 140 | + return to_df(body, time_col=None), meta_of(body) | |
| 141 | + | |
| 142 | + | |
| 143 | +def screener(filters: str, sort: str | None = None, as_of: str | None = None, limit: int = 100) -> tuple[pd.DataFrame, dict]: | |
| 144 | + body, _ = get("/v1/fundamentals/screener", {"filters": filters, "sort": sort, "as_of": as_of, "limit": limit}) | |
| 145 | + return to_df(body, time_col=None), meta_of(body) | |
| 146 | + | |
| 147 | + | |
| 148 | +def ratios(ticker: str) -> dict: | |
| 149 | + body, _ = get(f"/v1/fundamentals/{ticker}/ratios") | |
| 150 | + d = body.get("data", body) if isinstance(body, dict) else body | |
| 151 | + return d if isinstance(d, dict) else (d[0] if d else {}) | |
| 152 | + | |
| 153 | + | |
| 154 | +def statements(ticker: str, statement: str = "income", period: str = "quarterly", limit: int = 12) -> pd.DataFrame: | |
| 155 | + body, _ = get(f"/v1/fundamentals/{ticker}/statements", {"statement": statement, "period": period, "limit": limit}) | |
| 156 | + return to_df(body, time_col=None) | |
added
skills/hfmd-fundamentals-screen/scripts/screen.py
+114 −0
@@ -0,0 +1,114 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +"""Fundamental screener + enrichment with point-in-time notes. | |
| 3 | + | |
| 4 | + screen.py --filters "pe<15,fcf_yield>0.06" [--sort fcf_yield:desc] [--as-of 2020-03-20] [--limit 25] | |
| 5 | + [--quarters 3] [--no-enrich] [--out screen.csv] | |
| 6 | + screen.py --tickers AAPL MSFT --quarters 4 # enrich only | |
| 7 | +""" | |
| 8 | +from __future__ import annotations | |
| 9 | + | |
| 10 | +import argparse | |
| 11 | +import sys | |
| 12 | +from datetime import date | |
| 13 | +from pathlib import Path | |
| 14 | + | |
| 15 | +import pandas as pd | |
| 16 | + | |
| 17 | +sys.path.insert(0, str(Path(__file__).parent)) | |
| 18 | +import hfmd # noqa: E402 | |
| 19 | + | |
| 20 | +KEY_RATIOS = ["pe", "pb", "ev_ebitda", "fcf_yield", "dividend_yield", "roe", "roa", "gross_margin", "operating_margin", "net_margin", "debt_to_equity", "current_ratio", "revenue_growth", "eps_growth", "market_cap"] | |
| 21 | +INCOME_COLS = ["period_end", "fiscal_period", "revenue", "operating_income", "net_income", "eps_diluted", "filed_at", "form"] | |
| 22 | + | |
| 23 | + | |
| 24 | +def pit_note(q: pd.DataFrame) -> str: | |
| 25 | + if q.empty or "filed_at" not in q: | |
| 26 | + return "no quarterly statements available" | |
| 27 | + q = q.copy() | |
| 28 | + q["filed_at"] = pd.to_datetime(q["filed_at"], errors="coerce") | |
| 29 | + q["period_end"] = pd.to_datetime(q.get("period_end"), errors="coerce") | |
| 30 | + latest = q.sort_values("filed_at").iloc[-1] | |
| 31 | + lag = (latest["filed_at"] - latest["period_end"]).days if pd.notna(latest["filed_at"]) and pd.notna(latest["period_end"]) else None | |
| 32 | + age = (pd.Timestamp(date.today()) - latest["filed_at"]).days if pd.notna(latest["filed_at"]) else None | |
| 33 | + return (f"latest period {latest['period_end'].date() if pd.notna(latest['period_end']) else '?'} became public on " | |
| 34 | + f"{latest['filed_at'].date() if pd.notna(latest['filed_at']) else '?'} ({lag} days after period end; {age} days ago)" | |
| 35 | + + (" — a newer quarter has ended but is not filed yet" if age is not None and age > 100 else "")) | |
| 36 | + | |
| 37 | + | |
| 38 | +def main() -> int: | |
| 39 | + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) | |
| 40 | + ap.add_argument("--filters", help='e.g. "pe<15,roe>0.15,fcf_yield>0.06"') | |
| 41 | + ap.add_argument("--sort", help="metric:asc|desc") | |
| 42 | + ap.add_argument("--as-of", help="point-in-time date YYYY-MM-DD") | |
| 43 | + ap.add_argument("--limit", type=int, default=25) | |
| 44 | + ap.add_argument("--tickers", nargs="*", help="skip the screener, enrich these tickers") | |
| 45 | + ap.add_argument("--quarters", type=int, default=3) | |
| 46 | + ap.add_argument("--no-enrich", action="store_true") | |
| 47 | + ap.add_argument("--out") | |
| 48 | + a = ap.parse_args() | |
| 49 | + if not a.filters and not a.tickers: | |
| 50 | + ap.error("--filters or --tickers required") | |
| 51 | + | |
| 52 | + hits = pd.DataFrame() | |
| 53 | + if a.filters: | |
| 54 | + try: | |
| 55 | + hits, meta = hfmd.screener(a.filters, a.sort, a.as_of, a.limit) | |
| 56 | + except hfmd.HfmdError as e: | |
| 57 | + print(e, file=sys.stderr) | |
| 58 | + if e.status == 404: | |
| 59 | + print("→ the fundamentals module is not deployed on this server yet (v2 rollout). Nothing to screen.", file=sys.stderr) | |
| 60 | + return 2 | |
| 61 | + return 1 | |
| 62 | + print(f"screen `{a.filters}`{' as of ' + a.as_of if a.as_of else ''}: {meta.get('count', len(hits))} match(es)" | |
| 63 | + + (f" (universe {meta['universe']})" if 'universe' in meta else "")) | |
| 64 | + if hits.empty: | |
| 65 | + return 0 | |
| 66 | + with pd.option_context("display.width", 200, "display.max_columns", 30, "display.float_format", "{:.3f}".format): | |
| 67 | + print(hits.head(a.limit).to_string(index=False)) | |
| 68 | + tickers = list(hits["ticker"].head(a.limit)) if "ticker" in hits else [] | |
| 69 | + else: | |
| 70 | + tickers = [t.upper() for t in a.tickers] | |
| 71 | + | |
| 72 | + if a.no_enrich or not tickers: | |
| 73 | + if a.out and not hits.empty: | |
| 74 | + hits.to_csv(a.out, index=False) | |
| 75 | + print(f"wrote {a.out}") | |
| 76 | + return 0 | |
| 77 | + | |
| 78 | + if not hfmd.API_KEY and len(tickers) > 8: | |
| 79 | + hfmd.log(f"keyless mode: enriching {len(tickers)} tickers needs {2 * len(tickers)} requests (30/h quota) — set HFMD_API_KEY or use --limit 8") | |
| 80 | + | |
| 81 | + enriched = [] | |
| 82 | + for t in tickers: | |
| 83 | + print(f"\n=== {t} ===") | |
| 84 | + try: | |
| 85 | + r = hfmd.ratios(t) | |
| 86 | + line = " ".join(f"{k}={r[k]:.3g}" for k in KEY_RATIOS if isinstance(r.get(k), (int, float))) | |
| 87 | + print("ratios:", line or r) | |
| 88 | + if r.get("as_of") or r.get("period_end"): | |
| 89 | + print(f" based on period {r.get('period_end')} · filed {r.get('filed_at')} · price as of {r.get('as_of')}") | |
| 90 | + except hfmd.HfmdError as e: | |
| 91 | + r = {} | |
| 92 | + print(f"ratios: {e}") | |
| 93 | + try: | |
| 94 | + q = hfmd.statements(t, "income", "quarterly", a.quarters) | |
| 95 | + cols = [c for c in INCOME_COLS if c in q.columns] or list(q.columns)[:8] | |
| 96 | + with pd.option_context("display.width", 200, "display.float_format", "{:,.0f}".format): | |
| 97 | + print(q[cols].to_string(index=False) if not q.empty else " no statements") | |
| 98 | + print(" point-in-time:", pit_note(q)) | |
| 99 | + except hfmd.HfmdError as e: | |
| 100 | + q = pd.DataFrame() | |
| 101 | + print(f"statements: {e}") | |
| 102 | + enriched.append({"ticker": t, **{k: r.get(k) for k in KEY_RATIOS}, "latest_filed_at": (q["filed_at"].max() if "filed_at" in q else None)}) | |
| 103 | + | |
| 104 | + if a.out: | |
| 105 | + out = pd.DataFrame(enriched) | |
| 106 | + if not hits.empty and "ticker" in hits: | |
| 107 | + out = hits.merge(out, on="ticker", how="left", suffixes=("", "_latest")) | |
| 108 | + out.to_csv(a.out, index=False) | |
| 109 | + print(f"\nwrote {a.out}") | |
| 110 | + return 0 | |
| 111 | + | |
| 112 | + | |
| 113 | +if __name__ == "__main__": | |
| 114 | + sys.exit(main()) | |
added
skills/hfmd-quick-backtest/SKILL.md
+56 −0
@@ -0,0 +1,56 @@ | ||
| 1 | +--- | |
| 2 | +name: hfmd-quick-backtest | |
| 3 | +description: Backtest a moving-average crossover (or adapt the template to another signal) on HF Market Data bars with an honest walk-forward (parameters chosen on a training window, evaluated on the next out-of-sample window), transaction costs, and a full metrics report. Use when the user wants to "test a strategy", "backtest", or compare signal parameters on a symbol. | |
| 4 | +--- | |
| 5 | + | |
| 6 | +# hfmd-quick-backtest | |
| 7 | + | |
| 8 | +A small, honest backtest template on top of v1 bars. It exists to give a *defensible* first answer | |
| 9 | +(walk-forward, costs, benchmark, no look-ahead) — not to be a full trading framework. | |
| 10 | + | |
| 11 | +## When to use | |
| 12 | + | |
| 13 | +- "Backtest a 50/200 MA crossover on SPY", "does a 20/100 cross work on CL?", "which MA pair worked best on BTC — and did it hold out of sample?" | |
| 14 | +- Any quick strategy sanity check on daily or intraday bars | |
| 15 | + | |
| 16 | +## Method (what the script does) | |
| 17 | + | |
| 18 | +1. Fetch bars (`/v1/bars/{asset}/{ticker}`), keep `close`, compute daily/bar returns. | |
| 19 | +2. **Walk-forward**: split time into consecutive blocks (`--test-years`, default 1). For each test block, select the (fast, slow) pair with the best Sharpe over the preceding `--train-years` (default 3) of data from the `--grid`; apply *that* pair to the test block. Concatenate the out-of-sample test blocks → the reported equity curve. | |
| 20 | +3. Signal: long when `MA_fast > MA_slow`, flat otherwise (`--allow-short` for ±1). Position is applied to the **next bar** (no look-ahead). | |
| 21 | +4. Costs: `--cost-bps` per side per change of position (default 5 bps). | |
| 22 | +5. Report: CAGR, annualised volatility, Sharpe, max drawdown, exposure, number of trades, turnover, vs buy-and-hold on the same out-of-sample span; per-block table of the chosen parameters and the block's Sharpe (this is where you see whether the choice was stable). | |
| 23 | +6. Optional chart: equity vs benchmark + drawdown + the chosen pair over time. | |
| 24 | + | |
| 25 | +## Steps | |
| 26 | + | |
| 27 | +```bash | |
| 28 | +python3 scripts/ma_crossover.py --asset etf --ticker SPY --start 2010-01-01 \ | |
| 29 | + --grid 10,20,50 --grid-slow 100,150,200 --train-years 3 --test-years 1 --cost-bps 5 --plot spy_wf.png | |
| 30 | +``` | |
| 31 | + | |
| 32 | +Then explain: (a) the out-of-sample numbers *only*, (b) the stability of the selected parameters across blocks, (c) the in-sample vs out-of-sample gap (`--show-insample` prints the best in-sample pair on the full history for contrast), (d) what is not modelled (slippage beyond bps, borrow, dividends if UNADJUSTED, intraday fills). | |
| 33 | + | |
| 34 | +## Examples | |
| 35 | + | |
| 36 | +```bash | |
| 37 | +# Crude oil vendor continuous (ratio-adjusted so returns are meaningful across rolls) | |
| 38 | +python3 scripts/ma_crossover.py --asset futures --ticker CL --adjustment contin_adj_ratio --start 2012-01-01 --allow-short | |
| 39 | + | |
| 40 | +# Bitcoin, hourly bars, 6-month train / 2-month test | |
| 41 | +python3 scripts/ma_crossover.py --asset crypto --ticker BTCUSD --timeframe 1hour --start 2024-01-01 --train-years 0.5 --test-years 0.17 --grid 12,24,48 --grid-slow 96,168,336 | |
| 42 | + | |
| 43 | +# Single fixed pair, no optimisation (pure evaluation) | |
| 44 | +python3 scripts/ma_crossover.py --asset stock --ticker AAPL --fixed 50,200 --start 2015-01-01 | |
| 45 | +``` | |
| 46 | + | |
| 47 | +## Adapting the template | |
| 48 | + | |
| 49 | +`signal_ma_cross(close, fast, slow)` returns a position series in {0,1} (or {-1,0,1}). Replace it with any function of past data only; keep the `.shift(1)` when applying positions and keep the walk-forward loop untouched. | |
| 50 | + | |
| 51 | +## Gotchas | |
| 52 | + | |
| 53 | +- For futures use `contin_adj_ratio` (multiplicative back-adjustment): `contin_UNadj` has roll jumps that fake returns; `contin_adj_absolute` can go negative in long histories. | |
| 54 | +- Intraday bars are US/Eastern and include only the sessions the vendor covers; annualisation uses bars/year inferred from the median bar spacing — check the printed `periods_per_year`. | |
| 55 | +- Keyless quota (30 req/h) is enough for daily history; hourly since 2010 needs an API key (`HFMD_API_KEY`). | |
| 56 | +- A Sharpe above ~1.5 out-of-sample on a plain MA cross is a red flag for a bug or a tiny sample, not a discovery. | |
added
skills/hfmd-quick-backtest/scripts/hfmd.py
+156 −0
@@ -0,0 +1,156 @@ | ||
| 1 | +"""Tiny HF Market Data client shared by the hfmd-* skills (requests + pandas only). | |
| 2 | + | |
| 3 | +Copied verbatim into every skill's `scripts/` folder by `skills/scripts/build_skills.py` | |
| 4 | +so each skill stays self-contained. Do not edit the copies — edit `skills/_shared/hfmd.py`. | |
| 5 | + | |
| 6 | +Environment: HFMD_API_KEY (optional, free account = 120 req/min; keyless = 30 req/h), | |
| 7 | + HFMD_BASE_URL (default https://www.hfmarketdata.io). | |
| 8 | +""" | |
| 9 | +from __future__ import annotations | |
| 10 | + | |
| 11 | +import os | |
| 12 | +import sys | |
| 13 | +import time | |
| 14 | +from typing import Any | |
| 15 | + | |
| 16 | +import pandas as pd | |
| 17 | +import requests | |
| 18 | + | |
| 19 | +BASE_URL = os.environ.get("HFMD_BASE_URL", "https://www.hfmarketdata.io").rstrip("/") | |
| 20 | +API_KEY = os.environ.get("HFMD_API_KEY", "").strip() | |
| 21 | +UA = "hfmd-skills/1.0 (+https://www.hfmarketdata.io/integrations/skills)" | |
| 22 | + | |
| 23 | +TIMEFRAMES = {"1m": "1min", "5m": "5min", "30m": "30min", "1h": "1hour", "1d": "1day", | |
| 24 | + "1min": "1min", "5min": "5min", "30min": "30min", "1hour": "1hour", "1day": "1day"} | |
| 25 | +V2_INTERVAL = {"1min": "1m", "5min": "5m", "30min": "30m", "1hour": "1h", "1day": "1d"} | |
| 26 | + | |
| 27 | + | |
| 28 | +class HfmdError(RuntimeError): | |
| 29 | + def __init__(self, status: int, code: str, message: str, url: str): | |
| 30 | + super().__init__(f"HF Market Data error {status} {code}: {message} ({url})") | |
| 31 | + self.status, self.code, self.url = status, code, url | |
| 32 | + | |
| 33 | + | |
| 34 | +_session = requests.Session() | |
| 35 | +_session.headers.update({"Accept": "application/json", "User-Agent": UA}) | |
| 36 | +if API_KEY: | |
| 37 | + _session.headers["Authorization"] = f"Bearer {API_KEY}" | |
| 38 | + | |
| 39 | + | |
| 40 | +def log(msg: str) -> None: | |
| 41 | + print(msg, file=sys.stderr) | |
| 42 | + | |
| 43 | + | |
| 44 | +def get(path: str, params: dict[str, Any] | None = None, *, retries: int = 3) -> tuple[Any, dict[str, str]]: | |
| 45 | + """GET a JSON endpoint. Returns (body, headers). Retries on 429 honouring Retry-After.""" | |
| 46 | + url = f"{BASE_URL}{path}" | |
| 47 | + p = {k: (",".join(v) if isinstance(v, (list, tuple)) else v) for k, v in (params or {}).items() if v not in (None, "")} | |
| 48 | + p.setdefault("format", "json") | |
| 49 | + for attempt in range(retries + 1): | |
| 50 | + r = _session.get(url, params=p, timeout=120) | |
| 51 | + if r.status_code == 429 and attempt < retries: | |
| 52 | + wait = float(r.headers.get("Retry-After", "5")) | |
| 53 | + log(f"429 rate limited — sleeping {wait:.0f}s (set HFMD_API_KEY for 120 req/min)") | |
| 54 | + time.sleep(min(wait, 120)) | |
| 55 | + continue | |
| 56 | + if r.status_code >= 400: | |
| 57 | + try: | |
| 58 | + body = r.json() | |
| 59 | + except ValueError: | |
| 60 | + body = {} | |
| 61 | + err = body.get("error") or {} | |
| 62 | + raise HfmdError(r.status_code, err.get("code", "HTTP_ERROR"), err.get("message") or body.get("detail") or r.text[:200], r.url) | |
| 63 | + rem = r.headers.get("X-RateLimit-Remaining-Requests") | |
| 64 | + if rem is not None: | |
| 65 | + log(f"rate limit: {rem}/{r.headers.get('X-RateLimit-Limit-Requests')} requests left") | |
| 66 | + return r.json(), dict(r.headers) | |
| 67 | + raise HfmdError(429, "RATE_LIMIT_EXCEEDED", "gave up after retries", url) | |
| 68 | + | |
| 69 | + | |
| 70 | +def rows_of(body: Any) -> list[dict]: | |
| 71 | + """Rows from either envelope: v1 {count,data} or v2 {data,meta}.""" | |
| 72 | + if isinstance(body, list): | |
| 73 | + return body | |
| 74 | + if isinstance(body, dict): | |
| 75 | + data = body.get("data") | |
| 76 | + if isinstance(data, list): | |
| 77 | + return data | |
| 78 | + return [] | |
| 79 | + | |
| 80 | + | |
| 81 | +def meta_of(body: Any) -> dict: | |
| 82 | + if isinstance(body, dict): | |
| 83 | + m = dict(body.get("meta") or {}) | |
| 84 | + if "count" in body and "count" not in m: | |
| 85 | + m["count"] = body["count"] | |
| 86 | + return m | |
| 87 | + return {} | |
| 88 | + | |
| 89 | + | |
| 90 | +def to_df(body: Any, time_col: str | None = "datetime") -> pd.DataFrame: | |
| 91 | + df = pd.DataFrame(rows_of(body)) | |
| 92 | + if time_col and time_col in df.columns: | |
| 93 | + df[time_col] = pd.to_datetime(df[time_col]) | |
| 94 | + df = df.sort_values(time_col).reset_index(drop=True) | |
| 95 | + return df | |
| 96 | + | |
| 97 | + | |
| 98 | +def bars(asset: str, ticker: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, | |
| 99 | + adjustment: str | None = None, limit: int = 50_000) -> pd.DataFrame: | |
| 100 | + """v1 bars (stock/etf/crypto/index/fx, or futures = vendor continuous). Paginates by date when needed.""" | |
| 101 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 102 | + frames: list[pd.DataFrame] = [] | |
| 103 | + cursor_start = start | |
| 104 | + while True: | |
| 105 | + body, _ = get(f"/v1/bars/{asset}/{ticker}", {"timeframe": tf, "start": cursor_start, "end": end, "adjustment": adjustment, "limit": limit, "order": "asc"}) | |
| 106 | + df = to_df(body) | |
| 107 | + if df.empty: | |
| 108 | + break | |
| 109 | + frames.append(df) | |
| 110 | + if len(df) < limit: | |
| 111 | + break | |
| 112 | + last = df["datetime"].iloc[-1] | |
| 113 | + cursor_start = (last + pd.Timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M") if tf != "1day" else (last + pd.Timedelta(days=1)).strftime("%Y-%m-%d") | |
| 114 | + out = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(columns=["ticker", "datetime", "open", "high", "low", "close", "volume"]) | |
| 115 | + return out.drop_duplicates("datetime").reset_index(drop=True) | |
| 116 | + | |
| 117 | + | |
| 118 | +def contract_bars(symbol: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, limit: int = 50_000) -> pd.DataFrame: | |
| 119 | + """v2 individual futures contract bars (ESZ25…).""" | |
| 120 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 121 | + body, _ = get(f"/v1/futures/contract/{symbol}/bars", {"interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit}) | |
| 122 | + return to_df(body) | |
| 123 | + | |
| 124 | + | |
| 125 | +def continuous(root: str, roll: str = "volume", adjust: str = "back_adjusted", depth: int = 1, timeframe: str = "1day", | |
| 126 | + start: str | None = None, end: str | None = None, limit: int = 50_000) -> tuple[pd.DataFrame, dict]: | |
| 127 | + """v2 server-built continuous series. Returns (df, meta) — meta['roll_dates'] lists the rolls.""" | |
| 128 | + tf = TIMEFRAMES.get(timeframe, timeframe) | |
| 129 | + body, _ = get(f"/v1/futures/{root}/continuous", {"roll": roll, "adjust": adjust, "depth": depth, "interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit}) | |
| 130 | + return to_df(body), meta_of(body) | |
| 131 | + | |
| 132 | + | |
| 133 | +def contracts(root: str, status: str | None = None) -> pd.DataFrame: | |
| 134 | + body, _ = get(f"/v1/futures/{root}/contracts", {"status": status, "limit": 1000}) | |
| 135 | + return to_df(body, time_col=None) | |
| 136 | + | |
| 137 | + | |
| 138 | +def term_structure(root: str, as_of: str | None = None) -> tuple[pd.DataFrame, dict]: | |
| 139 | + body, _ = get(f"/v1/futures/{root}/term-structure", {"as_of": as_of}) | |
| 140 | + return to_df(body, time_col=None), meta_of(body) | |
| 141 | + | |
| 142 | + | |
| 143 | +def screener(filters: str, sort: str | None = None, as_of: str | None = None, limit: int = 100) -> tuple[pd.DataFrame, dict]: | |
| 144 | + body, _ = get("/v1/fundamentals/screener", {"filters": filters, "sort": sort, "as_of": as_of, "limit": limit}) | |
| 145 | + return to_df(body, time_col=None), meta_of(body) | |
| 146 | + | |
| 147 | + | |
| 148 | +def ratios(ticker: str) -> dict: | |
| 149 | + body, _ = get(f"/v1/fundamentals/{ticker}/ratios") | |
| 150 | + d = body.get("data", body) if isinstance(body, dict) else body | |
| 151 | + return d if isinstance(d, dict) else (d[0] if d else {}) | |
| 152 | + | |
| 153 | + | |
| 154 | +def statements(ticker: str, statement: str = "income", period: str = "quarterly", limit: int = 12) -> pd.DataFrame: | |
| 155 | + body, _ = get(f"/v1/fundamentals/{ticker}/statements", {"statement": statement, "period": period, "limit": limit}) | |
| 156 | + return to_df(body, time_col=None) | |
added
skills/hfmd-quick-backtest/scripts/ma_crossover.py
+182 −0
Large diff (182 lines) is not loaded by default to keep the page fast.Load diff
added
skills/hfmd-term-structure/SKILL.md
+45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added
skills/hfmd-term-structure/scripts/hfmd.py
+156 −0
Large diff (156 lines) is not loaded by default to keep the page fast.Load diff
added
skills/hfmd-term-structure/scripts/term_structure.py
+172 −0
Large diff (172 lines) is not loaded by default to keep the page fast.Load diff
added
skills/scripts/build_skills.py
+91 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff