SPB Git

spb/hfmarketdata

Public

Open high-frequency market data platform — FirstRate full-history downloader, DuckDB/Parquet lake, open REST API and React docs platform (www.hfmarketdata.io)

skills: pack de 5 skills Claude Code (analyse, backtest walk-forward, futures continus, screener fondamentaux, structure par terme) + zip téléchargeable

- chaque skill : SKILL.md (frontmatter, quand l'utiliser, étapes, exemples, pièges) + scripts Python requests/pandas
- helper commun skills/_shared/hfmd.py (enveloppes v1/v2, retry 429, pagination) copié dans chaque skill par le build
- skills/scripts/build_skills.py : synchro + zip déterministe → hfmarketdata/web/public/downloads/hfmarketdata-skills.zip
- scripts v1 validés en live (SPY 1 405 barres ; walk-forward 14 blocs) ; scripts v2 échouent proprement (404 = module pas encore déployé)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 4, 2026) parent 9f947ba Browse files

21 changed files +2,236 −0

added hfmarketdata/web/public/downloads/hfmarketdata-skills.zip +0 −0

Binary file not shown.

added skills/README.md +60 −0
@@ -0,0 +1,60 @@
1 +# HF Market Data — skills pack
2 +
3 +Five Claude Code / Claude skills that turn <https://www.hfmarketdata.io> into a data-analysis workbench.
4 +Each skill is a folder with a `SKILL.md` (when to use it, step-by-step, examples) and small Python scripts
5 +(`requests` + `pandas`, `matplotlib` optional). No other dependency.
6 +
7 +| Skill | Use it when you want to… |
8 +|---|---|
9 +| `hfmd-data-analysis` | pull bars for any symbol into pandas and get returns, volatility, drawdowns, seasonality, a chart |
10 +| `hfmd-quick-backtest` | test a moving-average crossover (or any signal) with an honest walk-forward, costs, and a report |
11 +| `hfmd-continuous-futures` | build or compare continuous futures series — roll rules, back-adjustment, vs the vendor series |
12 +| `hfmd-fundamentals-screen` | screen US stocks on ratios, enrich with statements, keep point-in-time discipline |
13 +| `hfmd-term-structure` | draw a futures curve, quantify contango/backwardation and roll yield, compare two dates |
14 +
15 +## Install
16 +
17 +```bash
18 +# download
19 +curl -LO https://www.hfmarketdata.io/downloads/hfmarketdata-skills.zip
20 +
21 +# for every project (personal skills)
22 +unzip -o hfmarketdata-skills.zip -d ~/.claude/skills/
23 +
24 +# or for one project only (shared with the team through git)
25 +unzip -o hfmarketdata-skills.zip -d .claude/skills/
26 +
27 +pip install requests pandas matplotlib # matplotlib only for the charts
28 +export HFMD_API_KEY=hfmd_live_… # optional: free account = 120 req/min (keyless = 30 req/h)
29 +```
30 +
31 +Claude Code lists them under `/skills`; you can also just ask — "backtest a 20/100 MA crossover on SPY since 2015" triggers `hfmd-quick-backtest`.
32 +
33 +## Layout
34 +
35 +```
36 +skills/
37 + _shared/hfmd.py single source of the API helper (copied into each skill by the build)
38 + hfmd-data-analysis/ SKILL.md · scripts/fetch_bars.py · scripts/analyze.py · scripts/hfmd.py
39 + hfmd-quick-backtest/ SKILL.md · scripts/ma_crossover.py · scripts/hfmd.py
40 + hfmd-continuous-futures/ SKILL.md · scripts/continuous_compare.py · scripts/stitch_local.py · scripts/hfmd.py
41 + hfmd-fundamentals-screen/ SKILL.md · scripts/screen.py · scripts/hfmd.py
42 + hfmd-term-structure/ SKILL.md · scripts/term_structure.py · scripts/hfmd.py
43 + scripts/build_skills.py syncs _shared/hfmd.py → each skill, zips → web/public/downloads/hfmarketdata-skills.zip
44 +```
45 +
46 +## Build the zip
47 +
48 +```bash
49 +python3 skills/scripts/build_skills.py # → hfmarketdata/web/public/downloads/hfmarketdata-skills.zip
50 +python3 skills/scripts/build_skills.py --check # fail if a copied hfmd.py drifted from _shared/
51 +```
52 +
53 +The zip is committed so the website can serve it without a build step.
54 +
55 +## Conventions the scripts follow
56 +
57 +- Never invent data: gaps stay `NaN`, and every script prints the row count and the date range it actually received.
58 +- Intraday timestamps are naive US/Eastern (as delivered by the API); daily bars are dates.
59 +- Respect the quota: keyless is 30 requests/hour — scripts paginate by date only when needed and honour `Retry-After` on 429.
60 +- Fundamentals are point-in-time: use `filed_at` (not `period_end`) to decide what was knowable on a given day.
added skills/_shared/hfmd.py +156 −0
@@ -0,0 +1,156 @@
1 +"""Tiny HF Market Data client shared by the hfmd-* skills (requests + pandas only).
2 +
3 +Copied verbatim into every skill's `scripts/` folder by `skills/scripts/build_skills.py`
4 +so each skill stays self-contained. Do not edit the copies — edit `skills/_shared/hfmd.py`.
5 +
6 +Environment: HFMD_API_KEY (optional, free account = 120 req/min; keyless = 30 req/h),
7 + HFMD_BASE_URL (default https://www.hfmarketdata.io).
8 +"""
9 +from __future__ import annotations
10 +
11 +import os
12 +import sys
13 +import time
14 +from typing import Any
15 +
16 +import pandas as pd
17 +import requests
18 +
19 +BASE_URL = os.environ.get("HFMD_BASE_URL", "https://www.hfmarketdata.io").rstrip("/")
20 +API_KEY = os.environ.get("HFMD_API_KEY", "").strip()
21 +UA = "hfmd-skills/1.0 (+https://www.hfmarketdata.io/integrations/skills)"
22 +
23 +TIMEFRAMES = {"1m": "1min", "5m": "5min", "30m": "30min", "1h": "1hour", "1d": "1day",
24 + "1min": "1min", "5min": "5min", "30min": "30min", "1hour": "1hour", "1day": "1day"}
25 +V2_INTERVAL = {"1min": "1m", "5min": "5m", "30min": "30m", "1hour": "1h", "1day": "1d"}
26 +
27 +
28 +class HfmdError(RuntimeError):
29 + def __init__(self, status: int, code: str, message: str, url: str):
30 + super().__init__(f"HF Market Data error {status} {code}: {message} ({url})")
31 + self.status, self.code, self.url = status, code, url
32 +
33 +
34 +_session = requests.Session()
35 +_session.headers.update({"Accept": "application/json", "User-Agent": UA})
36 +if API_KEY:
37 + _session.headers["Authorization"] = f"Bearer {API_KEY}"
38 +
39 +
40 +def log(msg: str) -> None:
41 + print(msg, file=sys.stderr)
42 +
43 +
44 +def get(path: str, params: dict[str, Any] | None = None, *, retries: int = 3) -> tuple[Any, dict[str, str]]:
45 + """GET a JSON endpoint. Returns (body, headers). Retries on 429 honouring Retry-After."""
46 + url = f"{BASE_URL}{path}"
47 + p = {k: (",".join(v) if isinstance(v, (list, tuple)) else v) for k, v in (params or {}).items() if v not in (None, "")}
48 + p.setdefault("format", "json")
49 + for attempt in range(retries + 1):
50 + r = _session.get(url, params=p, timeout=120)
51 + if r.status_code == 429 and attempt < retries:
52 + wait = float(r.headers.get("Retry-After", "5"))
53 + log(f"429 rate limited — sleeping {wait:.0f}s (set HFMD_API_KEY for 120 req/min)")
54 + time.sleep(min(wait, 120))
55 + continue
56 + if r.status_code >= 400:
57 + try:
58 + body = r.json()
59 + except ValueError:
60 + body = {}
61 + err = body.get("error") or {}
62 + raise HfmdError(r.status_code, err.get("code", "HTTP_ERROR"), err.get("message") or body.get("detail") or r.text[:200], r.url)
63 + rem = r.headers.get("X-RateLimit-Remaining-Requests")
64 + if rem is not None:
65 + log(f"rate limit: {rem}/{r.headers.get('X-RateLimit-Limit-Requests')} requests left")
66 + return r.json(), dict(r.headers)
67 + raise HfmdError(429, "RATE_LIMIT_EXCEEDED", "gave up after retries", url)
68 +
69 +
70 +def rows_of(body: Any) -> list[dict]:
71 + """Rows from either envelope: v1 {count,data} or v2 {data,meta}."""
72 + if isinstance(body, list):
73 + return body
74 + if isinstance(body, dict):
75 + data = body.get("data")
76 + if isinstance(data, list):
77 + return data
78 + return []
79 +
80 +
81 +def meta_of(body: Any) -> dict:
82 + if isinstance(body, dict):
83 + m = dict(body.get("meta") or {})
84 + if "count" in body and "count" not in m:
85 + m["count"] = body["count"]
86 + return m
87 + return {}
88 +
89 +
90 +def to_df(body: Any, time_col: str | None = "datetime") -> pd.DataFrame:
91 + df = pd.DataFrame(rows_of(body))
92 + if time_col and time_col in df.columns:
93 + df[time_col] = pd.to_datetime(df[time_col])
94 + df = df.sort_values(time_col).reset_index(drop=True)
95 + return df
96 +
97 +
98 +def bars(asset: str, ticker: str, timeframe: str = "1day", start: str | None = None, end: str | None = None,
99 + adjustment: str | None = None, limit: int = 50_000) -> pd.DataFrame:
100 + """v1 bars (stock/etf/crypto/index/fx, or futures = vendor continuous). Paginates by date when needed."""
101 + tf = TIMEFRAMES.get(timeframe, timeframe)
102 + frames: list[pd.DataFrame] = []
103 + cursor_start = start
104 + while True:
105 + body, _ = get(f"/v1/bars/{asset}/{ticker}", {"timeframe": tf, "start": cursor_start, "end": end, "adjustment": adjustment, "limit": limit, "order": "asc"})
106 + df = to_df(body)
107 + if df.empty:
108 + break
109 + frames.append(df)
110 + if len(df) < limit:
111 + break
112 + last = df["datetime"].iloc[-1]
113 + cursor_start = (last + pd.Timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M") if tf != "1day" else (last + pd.Timedelta(days=1)).strftime("%Y-%m-%d")
114 + out = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(columns=["ticker", "datetime", "open", "high", "low", "close", "volume"])
115 + return out.drop_duplicates("datetime").reset_index(drop=True)
116 +
117 +
118 +def contract_bars(symbol: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, limit: int = 50_000) -> pd.DataFrame:
119 + """v2 individual futures contract bars (ESZ25…)."""
120 + tf = TIMEFRAMES.get(timeframe, timeframe)
121 + body, _ = get(f"/v1/futures/contract/{symbol}/bars", {"interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit})
122 + return to_df(body)
123 +
124 +
125 +def continuous(root: str, roll: str = "volume", adjust: str = "back_adjusted", depth: int = 1, timeframe: str = "1day",
126 + start: str | None = None, end: str | None = None, limit: int = 50_000) -> tuple[pd.DataFrame, dict]:
127 + """v2 server-built continuous series. Returns (df, meta) — meta['roll_dates'] lists the rolls."""
128 + tf = TIMEFRAMES.get(timeframe, timeframe)
129 + body, _ = get(f"/v1/futures/{root}/continuous", {"roll": roll, "adjust": adjust, "depth": depth, "interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit})
130 + return to_df(body), meta_of(body)
131 +
132 +
133 +def contracts(root: str, status: str | None = None) -> pd.DataFrame:
134 + body, _ = get(f"/v1/futures/{root}/contracts", {"status": status, "limit": 1000})
135 + return to_df(body, time_col=None)
136 +
137 +
138 +def term_structure(root: str, as_of: str | None = None) -> tuple[pd.DataFrame, dict]:
139 + body, _ = get(f"/v1/futures/{root}/term-structure", {"as_of": as_of})
140 + return to_df(body, time_col=None), meta_of(body)
141 +
142 +
143 +def screener(filters: str, sort: str | None = None, as_of: str | None = None, limit: int = 100) -> tuple[pd.DataFrame, dict]:
144 + body, _ = get("/v1/fundamentals/screener", {"filters": filters, "sort": sort, "as_of": as_of, "limit": limit})
145 + return to_df(body, time_col=None), meta_of(body)
146 +
147 +
148 +def ratios(ticker: str) -> dict:
149 + body, _ = get(f"/v1/fundamentals/{ticker}/ratios")
150 + d = body.get("data", body) if isinstance(body, dict) else body
151 + return d if isinstance(d, dict) else (d[0] if d else {})
152 +
153 +
154 +def statements(ticker: str, statement: str = "income", period: str = "quarterly", limit: int = 12) -> pd.DataFrame:
155 + body, _ = get(f"/v1/fundamentals/{ticker}/statements", {"statement": statement, "period": period, "limit": limit})
156 + return to_df(body, time_col=None)
added skills/hfmd-continuous-futures/SKILL.md +53 −0
@@ -0,0 +1,53 @@
1 +---
2 +name: hfmd-continuous-futures
3 +description: Build and compare continuous futures series from HF Market Data — server-side v2 continuous endpoint (roll=volume/open_interest/calendar, adjust=none/back_adjusted/ratio, depth) versus the vendor continuous series, or stitched locally from individual contracts with explicit roll dates. Use when the user asks for a continuous futures price, questions how rolls/adjustments were done, or needs a series suitable for backtesting.
4 +---
5 +
6 +# hfmd-continuous-futures
7 +
8 +A futures "price history" is a construction: someone chose when to jump from one contract to the next
9 +(**roll**) and what to do with the price gap at the jump (**adjust**). This skill makes those choices
10 +explicit, reproducible and comparable.
11 +
12 +## Vocabulary (say this to the user)
13 +
14 +- **Roll rule** — `volume`: switch when the next contract's volume exceeds the front's (most common, tracks where liquidity is); `open_interest`: same with OI (smoother, lags a bit); `calendar`: N business days before expiry/first notice (deterministic, what many CTAs do).
15 +- **Adjustment** — `none`: raw prices, discontinuous at rolls (fine for *levels*, wrong for *returns*); `back_adjusted` (additive): shift all earlier history by the roll gap so the series is continuous in **points** (good for P&L in ticks, can go negative on long histories); `ratio` (multiplicative): scale earlier history by the gap ratio, continuous in **percent** (best for returns/backtests, levels are not real prices).
16 +- **Depth** — 1 = front month, 2 = second month … (depth 2 avoids expiry noise for spread work).
17 +- Vendor series (v1 `/v1/bars/futures/{ROOT}?adjustment=contin_UNadj|contin_adj_ratio|contin_adj_absolute`) are FirstRate Data's own construction — a fixed rule you cannot change. The v2 endpoint lets you pick.
18 +
19 +## When to use
20 +
21 +- "Get me continuous ES since 2015", "why does the crude series jump in April?", "back-adjust NG for a backtest", "compare volume-roll vs calendar-roll on CL"
22 +
23 +## Steps
24 +
25 +1. Confirm root (`/v1/futures/roots` or `search_symbols(asset=futures)`), timeframe, range, and the intended *use* (levels → `none`; P&L in points → `back_adjusted`; returns → `ratio`).
26 +2. Server-side build + comparison:
27 + `python3 scripts/continuous_compare.py --root CL --start 2018-01-01 --rolls volume,calendar --adjusts back_adjusted,ratio --vendor contin_adj_ratio --plot cl.png`
28 + - prints each series' roll dates (from `meta.roll_dates`), the number of rolls per year, the mean absolute gap at rolls, and the return correlation / tracking difference between the variants and the vendor series.
29 +3. If the v2 endpoint is not available on the server yet (404 `NOT_FOUND`), stitch locally to show the methodology:
30 + `python3 scripts/stitch_local.py --root CL --start 2023-01-01 --roll volume --adjust ratio --out cl_local.csv`
31 + (lists contracts via `/v1/futures/{root}/contracts`, pulls each contract's daily bars, rolls on volume crossover, adjusts; prints the roll table). This costs one request per contract — needs an API key for long histories.
32 +4. Report: chosen rule + why, roll dates table, adjusted vs raw last price (to remind that adjusted levels ≠ tradable prices), and any gap in coverage (`/v1/futures/contract/{symbol}/coverage`).
33 +
34 +## Examples
35 +
36 +```bash
37 +# ES front month, volume roll, ratio-adjusted, daily since 2015 → CSV for a backtest
38 +python3 scripts/continuous_compare.py --root ES --start 2015-01-01 --rolls volume --adjusts ratio --out es_cont.csv
39 +
40 +# Second-month natural gas vs front month (seasonal spread work)
41 +python3 scripts/continuous_compare.py --root NG --depth 2 --rolls open_interest --adjusts none --start 2020-01-01
42 +
43 +# How different are the vendor's absolute-adjusted and the v2 back_adjusted series?
44 +python3 scripts/continuous_compare.py --root CL --rolls volume --adjusts back_adjusted --vendor contin_adj_absolute --start 2019-01-01
45 +```
46 +
47 +## Gotchas
48 +
49 +- Never compute returns on `none`/`contin_UNadj`: the roll gap (often 0.5-3 %) is not a market move.
50 +- Back-adjusted (additive) series on long histories can go **negative** (CL 2000-2020 does) — use `ratio` for percent returns.
51 +- Roll dates depend on the timeframe used to measure volume (daily is standard); intraday continuous series inherit the daily roll calendar.
52 +- Coverage: contracts have data since 2010 (`archive` ≤ 2025 and `update` ≥ 2025 files are already merged by the API); a missing contract is reported as a gap, not filled.
53 +- FirstRate roots may differ from exchange codes (`E6` is euro FX `6E`; the API accepts both).
added skills/hfmd-continuous-futures/scripts/continuous_compare.py +118 −0
@@ -0,0 +1,118 @@
1 +#!/usr/bin/env python3
2 +"""Build server-side continuous futures series (v2) with several roll/adjust choices and compare them
3 +to each other and to the vendor continuous series (v1).
4 +
5 + continuous_compare.py --root CL --start 2018-01-01 --rolls volume,calendar --adjusts back_adjusted,ratio \
6 + [--depth 1] [--timeframe 1day] [--vendor contin_adj_ratio] [--out cl.csv] [--plot cl.png]
7 +"""
8 +from __future__ import annotations
9 +
10 +import argparse
11 +import sys
12 +from pathlib import Path
13 +
14 +import numpy as np
15 +import pandas as pd
16 +
17 +sys.path.insert(0, str(Path(__file__).parent))
18 +import hfmd # noqa: E402
19 +
20 +
21 +def describe_rolls(name: str, df: pd.DataFrame, meta: dict) -> None:
22 + rolls = meta.get("roll_dates") or []
23 + if df.empty:
24 + print(f"\n[{name}] no data")
25 + return
26 + years = max((df["datetime"].iloc[-1] - df["datetime"].iloc[0]).days / 365.25, 1e-9)
27 + roll_txt = f"{len(rolls)} rolls ({len(rolls) / years:.1f}/yr)" if rolls or not name.startswith("vendor") else "roll dates not exposed (vendor construction)"
28 + print(f"\n[{name}] {len(df):,} bars {df['datetime'].iloc[0].date()} → {df['datetime'].iloc[-1].date()} · {roll_txt}")
29 + if rolls:
30 + shown = rolls if len(rolls) <= 12 else rolls[:6] + ["…"] + rolls[-6:]
31 + print(" roll dates:", ", ".join(str(r) if isinstance(r, str) else str(r.get("date", r)) for r in shown))
32 + if "close" in df and len(df) > 1:
33 + print(f" last close {df['close'].iloc[-1]:.4f} · first close {df['close'].iloc[0]:.4f}"
34 + + (" (adjusted levels are NOT tradable prices)" if "adjust=none" not in name else ""))
35 +
36 +
37 +def main() -> int:
38 + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
39 + ap.add_argument("--root", required=True)
40 + ap.add_argument("--start", default="2015-01-01")
41 + ap.add_argument("--end")
42 + ap.add_argument("--timeframe", default="1day")
43 + ap.add_argument("--depth", type=int, default=1)
44 + ap.add_argument("--rolls", default="volume", help="comma list: volume, open_interest, calendar")
45 + ap.add_argument("--adjusts", default="back_adjusted", help="comma list: none, back_adjusted, ratio")
46 + ap.add_argument("--vendor", help="also fetch the v1 vendor continuous with this adjustment: contin_UNadj, contin_adj_ratio, contin_adj_absolute")
47 + ap.add_argument("--out", help="CSV with all series' closes aligned on datetime")
48 + ap.add_argument("--plot")
49 + a = ap.parse_args()
50 + root = a.root.upper()
51 +
52 + series: dict[str, pd.Series] = {}
53 + for roll in a.rolls.split(","):
54 + for adj in a.adjusts.split(","):
55 + name = f"v2 roll={roll} adjust={adj} depth={a.depth}"
56 + try:
57 + df, meta = hfmd.continuous(root, roll=roll, adjust=adj, depth=a.depth, timeframe=a.timeframe, start=a.start, end=a.end)
58 + except hfmd.HfmdError as e:
59 + print(f"\n[{name}] {e}", file=sys.stderr)
60 + if e.status == 404:
61 + print(" → the v2 continuous endpoint is not deployed on this server yet; use scripts/stitch_local.py", file=sys.stderr)
62 + continue
63 + describe_rolls(name, df, meta)
64 + if not df.empty:
65 + series[name] = df.set_index("datetime")["close"].astype(float)
66 +
67 + if a.vendor:
68 + name = f"vendor {a.vendor}"
69 + try:
70 + vdf = hfmd.bars("futures", root, a.timeframe, a.start, a.end, a.vendor)
71 + describe_rolls(name, vdf, {})
72 + if not vdf.empty:
73 + series[name] = vdf.set_index("datetime")["close"].astype(float)
74 + except hfmd.HfmdError as e:
75 + print(f"\n[{name}] {e}", file=sys.stderr)
76 +
77 + if not series:
78 + print("\nno series retrieved", file=sys.stderr)
79 + return 1
80 +
81 + wide = pd.concat(series, axis=1).sort_index()
82 + rets = wide.pct_change()
83 + print("\nreturn correlation between series (daily pct changes; 'none'/UNadj variants include roll gaps):")
84 + with pd.option_context("display.width", 200):
85 + print(rets.corr().round(4).to_string())
86 + if len(series) > 1:
87 + names = list(series)
88 + base = names[-1]
89 + print(f"\nannualised tracking difference of log-returns vs '{base}':")
90 + for n in names[:-1]:
91 + d = (np.log1p(rets[n]) - np.log1p(rets[base])).dropna()
92 + print(f" {n:<45} mean={d.mean() * 252:+.4%}/yr std={d.std() * np.sqrt(252):.4%} n={len(d)}")
93 +
94 + if a.out:
95 + wide.to_csv(a.out)
96 + print(f"\nwrote {a.out} ({len(wide):,} rows × {wide.shape[1]} series)")
97 + if a.plot:
98 + try:
99 + import matplotlib
100 + matplotlib.use("Agg")
101 + import matplotlib.pyplot as plt
102 + except ImportError:
103 + print("matplotlib not installed", file=sys.stderr)
104 + return 0
105 + fig, ax = plt.subplots(figsize=(11, 5))
106 + for n, s in series.items():
107 + ax.plot(s.index, s / s.dropna().iloc[0] * 100, lw=1.0, label=n)
108 + ax.set_title(f"{root} continuous series rebased to 100 (differences = roll/adjust methodology)")
109 + ax.grid(alpha=0.25)
110 + ax.legend(fontsize=8)
111 + fig.tight_layout()
112 + fig.savefig(a.plot, dpi=130)
113 + print(f"wrote {a.plot}")
114 + return 0
115 +
116 +
117 +if __name__ == "__main__":
118 + sys.exit(main())
added skills/hfmd-continuous-futures/scripts/hfmd.py +156 −0
@@ -0,0 +1,156 @@
1 +"""Tiny HF Market Data client shared by the hfmd-* skills (requests + pandas only).
2 +
3 +Copied verbatim into every skill's `scripts/` folder by `skills/scripts/build_skills.py`
4 +so each skill stays self-contained. Do not edit the copies — edit `skills/_shared/hfmd.py`.
5 +
6 +Environment: HFMD_API_KEY (optional, free account = 120 req/min; keyless = 30 req/h),
7 + HFMD_BASE_URL (default https://www.hfmarketdata.io).
8 +"""
9 +from __future__ import annotations
10 +
11 +import os
12 +import sys
13 +import time
14 +from typing import Any
15 +
16 +import pandas as pd
17 +import requests
18 +
19 +BASE_URL = os.environ.get("HFMD_BASE_URL", "https://www.hfmarketdata.io").rstrip("/")
20 +API_KEY = os.environ.get("HFMD_API_KEY", "").strip()
21 +UA = "hfmd-skills/1.0 (+https://www.hfmarketdata.io/integrations/skills)"
22 +
23 +TIMEFRAMES = {"1m": "1min", "5m": "5min", "30m": "30min", "1h": "1hour", "1d": "1day",
24 + "1min": "1min", "5min": "5min", "30min": "30min", "1hour": "1hour", "1day": "1day"}
25 +V2_INTERVAL = {"1min": "1m", "5min": "5m", "30min": "30m", "1hour": "1h", "1day": "1d"}
26 +
27 +
28 +class HfmdError(RuntimeError):
29 + def __init__(self, status: int, code: str, message: str, url: str):
30 + super().__init__(f"HF Market Data error {status} {code}: {message} ({url})")
31 + self.status, self.code, self.url = status, code, url
32 +
33 +
34 +_session = requests.Session()
35 +_session.headers.update({"Accept": "application/json", "User-Agent": UA})
36 +if API_KEY:
37 + _session.headers["Authorization"] = f"Bearer {API_KEY}"
38 +
39 +
40 +def log(msg: str) -> None:
41 + print(msg, file=sys.stderr)
42 +
43 +
44 +def get(path: str, params: dict[str, Any] | None = None, *, retries: int = 3) -> tuple[Any, dict[str, str]]:
45 + """GET a JSON endpoint. Returns (body, headers). Retries on 429 honouring Retry-After."""
46 + url = f"{BASE_URL}{path}"
47 + p = {k: (",".join(v) if isinstance(v, (list, tuple)) else v) for k, v in (params or {}).items() if v not in (None, "")}
48 + p.setdefault("format", "json")
49 + for attempt in range(retries + 1):
50 + r = _session.get(url, params=p, timeout=120)
51 + if r.status_code == 429 and attempt < retries:
52 + wait = float(r.headers.get("Retry-After", "5"))
53 + log(f"429 rate limited — sleeping {wait:.0f}s (set HFMD_API_KEY for 120 req/min)")
54 + time.sleep(min(wait, 120))
55 + continue
56 + if r.status_code >= 400:
57 + try:
58 + body = r.json()
59 + except ValueError:
60 + body = {}
61 + err = body.get("error") or {}
62 + raise HfmdError(r.status_code, err.get("code", "HTTP_ERROR"), err.get("message") or body.get("detail") or r.text[:200], r.url)
63 + rem = r.headers.get("X-RateLimit-Remaining-Requests")
64 + if rem is not None:
65 + log(f"rate limit: {rem}/{r.headers.get('X-RateLimit-Limit-Requests')} requests left")
66 + return r.json(), dict(r.headers)
67 + raise HfmdError(429, "RATE_LIMIT_EXCEEDED", "gave up after retries", url)
68 +
69 +
70 +def rows_of(body: Any) -> list[dict]:
71 + """Rows from either envelope: v1 {count,data} or v2 {data,meta}."""
72 + if isinstance(body, list):
73 + return body
74 + if isinstance(body, dict):
75 + data = body.get("data")
76 + if isinstance(data, list):
77 + return data
78 + return []
79 +
80 +
81 +def meta_of(body: Any) -> dict:
82 + if isinstance(body, dict):
83 + m = dict(body.get("meta") or {})
84 + if "count" in body and "count" not in m:
85 + m["count"] = body["count"]
86 + return m
87 + return {}
88 +
89 +
90 +def to_df(body: Any, time_col: str | None = "datetime") -> pd.DataFrame:
91 + df = pd.DataFrame(rows_of(body))
92 + if time_col and time_col in df.columns:
93 + df[time_col] = pd.to_datetime(df[time_col])
94 + df = df.sort_values(time_col).reset_index(drop=True)
95 + return df
96 +
97 +
98 +def bars(asset: str, ticker: str, timeframe: str = "1day", start: str | None = None, end: str | None = None,
99 + adjustment: str | None = None, limit: int = 50_000) -> pd.DataFrame:
100 + """v1 bars (stock/etf/crypto/index/fx, or futures = vendor continuous). Paginates by date when needed."""
101 + tf = TIMEFRAMES.get(timeframe, timeframe)
102 + frames: list[pd.DataFrame] = []
103 + cursor_start = start
104 + while True:
105 + body, _ = get(f"/v1/bars/{asset}/{ticker}", {"timeframe": tf, "start": cursor_start, "end": end, "adjustment": adjustment, "limit": limit, "order": "asc"})
106 + df = to_df(body)
107 + if df.empty:
108 + break
109 + frames.append(df)
110 + if len(df) < limit:
111 + break
112 + last = df["datetime"].iloc[-1]
113 + cursor_start = (last + pd.Timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M") if tf != "1day" else (last + pd.Timedelta(days=1)).strftime("%Y-%m-%d")
114 + out = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(columns=["ticker", "datetime", "open", "high", "low", "close", "volume"])
115 + return out.drop_duplicates("datetime").reset_index(drop=True)
116 +
117 +
118 +def contract_bars(symbol: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, limit: int = 50_000) -> pd.DataFrame:
119 + """v2 individual futures contract bars (ESZ25…)."""
120 + tf = TIMEFRAMES.get(timeframe, timeframe)
121 + body, _ = get(f"/v1/futures/contract/{symbol}/bars", {"interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit})
122 + return to_df(body)
123 +
124 +
125 +def continuous(root: str, roll: str = "volume", adjust: str = "back_adjusted", depth: int = 1, timeframe: str = "1day",
126 + start: str | None = None, end: str | None = None, limit: int = 50_000) -> tuple[pd.DataFrame, dict]:
127 + """v2 server-built continuous series. Returns (df, meta) — meta['roll_dates'] lists the rolls."""
128 + tf = TIMEFRAMES.get(timeframe, timeframe)
129 + body, _ = get(f"/v1/futures/{root}/continuous", {"roll": roll, "adjust": adjust, "depth": depth, "interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit})
130 + return to_df(body), meta_of(body)
131 +
132 +
133 +def contracts(root: str, status: str | None = None) -> pd.DataFrame:
134 + body, _ = get(f"/v1/futures/{root}/contracts", {"status": status, "limit": 1000})
135 + return to_df(body, time_col=None)
136 +
137 +
138 +def term_structure(root: str, as_of: str | None = None) -> tuple[pd.DataFrame, dict]:
139 + body, _ = get(f"/v1/futures/{root}/term-structure", {"as_of": as_of})
140 + return to_df(body, time_col=None), meta_of(body)
141 +
142 +
143 +def screener(filters: str, sort: str | None = None, as_of: str | None = None, limit: int = 100) -> tuple[pd.DataFrame, dict]:
144 + body, _ = get("/v1/fundamentals/screener", {"filters": filters, "sort": sort, "as_of": as_of, "limit": limit})
145 + return to_df(body, time_col=None), meta_of(body)
146 +
147 +
148 +def ratios(ticker: str) -> dict:
149 + body, _ = get(f"/v1/fundamentals/{ticker}/ratios")
150 + d = body.get("data", body) if isinstance(body, dict) else body
151 + return d if isinstance(d, dict) else (d[0] if d else {})
152 +
153 +
154 +def statements(ticker: str, statement: str = "income", period: str = "quarterly", limit: int = 12) -> pd.DataFrame:
155 + body, _ = get(f"/v1/fundamentals/{ticker}/statements", {"statement": statement, "period": period, "limit": limit})
156 + return to_df(body, time_col=None)
added skills/hfmd-continuous-futures/scripts/stitch_local.py +124 −0
@@ -0,0 +1,124 @@
1 +#!/usr/bin/env python3
2 +"""Stitch a continuous futures series locally from individual contracts (transparent methodology,
3 +and a fallback when the v2 /continuous endpoint is unavailable).
4 +
5 + stitch_local.py --root CL --start 2023-01-01 [--roll volume|open_interest|calendar] [--roll-days 5]
6 + [--adjust none|back_adjusted|ratio] [--out cl_local.csv]
7 +
8 +Costs one request per contract in the range (≈ 12/yr for CL, 4/yr for ES) — set HFMD_API_KEY.
9 +"""
10 +from __future__ import annotations
11 +
12 +import argparse
13 +import sys
14 +from pathlib import Path
15 +
16 +import numpy as np
17 +import pandas as pd
18 +
19 +sys.path.insert(0, str(Path(__file__).parent))
20 +import hfmd # noqa: E402
21 +
22 +
23 +def main() -> int:
24 + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
25 + ap.add_argument("--root", required=True)
26 + ap.add_argument("--start", default="2023-01-01")
27 + ap.add_argument("--end")
28 + ap.add_argument("--roll", default="volume", choices=["volume", "open_interest", "calendar"])
29 + ap.add_argument("--roll-days", type=int, default=5, help="calendar rule: business days before expiration")
30 + ap.add_argument("--adjust", default="ratio", choices=["none", "back_adjusted", "ratio"])
31 + ap.add_argument("--out")
32 + a = ap.parse_args()
33 + root = a.root.upper()
34 +
35 + cons = hfmd.contracts(root)
36 + if cons.empty:
37 + print("no contracts returned", file=sys.stderr)
38 + return 1
39 + cons["expiration_date"] = pd.to_datetime(cons["expiration_date"])
40 + cons["last_data_date"] = pd.to_datetime(cons.get("last_data_date"))
41 + lo = pd.Timestamp(a.start) - pd.Timedelta(days=120)
42 + hi = pd.Timestamp(a.end) if a.end else pd.Timestamp.today()
43 + sel = cons[(cons["expiration_date"] >= lo) & (cons["first_data_date"].pipe(pd.to_datetime) <= hi)].sort_values("expiration_date") if "first_data_date" in cons else cons[cons["expiration_date"] >= lo].sort_values("expiration_date")
44 + print(f"{root}: {len(sel)} contracts in scope ({sel['symbol'].iloc[0]} … {sel['symbol'].iloc[-1]})")
45 +
46 + panels: dict[str, pd.DataFrame] = {}
47 + for sym in sel["symbol"]:
48 + df = hfmd.contract_bars(sym, "1day", a.start and str((pd.Timestamp(a.start) - pd.Timedelta(days=120)).date()), a.end)
49 + if df.empty:
50 + hfmd.log(f" {sym}: no bars (gap reported, not filled)")
51 + continue
52 + panels[sym] = df.set_index("datetime")
53 + if not panels:
54 + return 1
55 + order = [s for s in sel["symbol"] if s in panels]
56 + expiry = dict(zip(sel["symbol"], sel["expiration_date"]))
57 +
58 + # --- choose the active contract on each date ---------------------------------------------------
59 + dates = sorted(set().union(*[p.index for p in panels.values()]))
60 + dates = [d for d in dates if d >= pd.Timestamp(a.start)]
61 + active_idx, active, rolls = 0, order[0], []
62 + chosen: list[tuple[pd.Timestamp, str]] = []
63 + for d in dates:
64 + # skip contracts already expired
65 + while active_idx + 1 < len(order) and expiry[active] < d:
66 + active_idx += 1
67 + active = order[active_idx]
68 + rolls.append((d, active, "expired"))
69 + if active_idx + 1 < len(order):
70 + nxt = order[active_idx + 1]
71 + switch = False
72 + if a.roll == "calendar":
73 + switch = d >= expiry[active] - pd.tseries.offsets.BDay(a.roll_days)
74 + elif d in panels[active].index and d in panels[nxt].index:
75 + col = "volume" if a.roll == "volume" else "open_interest"
76 + if col in panels[nxt].columns and col in panels[active].columns:
77 + switch = float(panels[nxt].loc[d, col] or 0) > float(panels[active].loc[d, col] or 0)
78 + if switch:
79 + active_idx += 1
80 + active = nxt
81 + rolls.append((d, active, a.roll))
82 + chosen.append((d, active))
83 +
84 + # --- assemble + adjust --------------------------------------------------------------------------
85 + rows = []
86 + for d, sym in chosen:
87 + if d in panels[sym].index:
88 + r = panels[sym].loc[d]
89 + rows.append({"datetime": d, "contract": sym, "open": r["open"], "high": r["high"], "low": r["low"], "close": r["close"], "volume": r.get("volume"), "open_interest": r.get("open_interest")})
90 + else:
91 + rows.append({"datetime": d, "contract": sym, "open": np.nan, "high": np.nan, "low": np.nan, "close": np.nan, "volume": np.nan, "open_interest": np.nan})
92 + out = pd.DataFrame(rows).set_index("datetime")
93 +
94 + if a.adjust != "none":
95 + adj = out.copy()
96 + # walk backwards: at each roll, compute the gap between the new and old contract on the roll date
97 + roll_dates = [d for d, _, _ in rolls if d in out.index]
98 + for d in reversed(roll_dates):
99 + i = out.index.get_loc(d)
100 + new_sym, old_sym = out["contract"].iloc[i], out["contract"].iloc[i - 1] if i > 0 else None
101 + if not old_sym or d not in panels[old_sym].index or d not in panels[new_sym].index:
102 + continue
103 + new_c, old_c = float(panels[new_sym].loc[d, "close"]), float(panels[old_sym].loc[d, "close"])
104 + before = adj.index < d
105 + for col in ("open", "high", "low", "close"):
106 + if a.adjust == "back_adjusted":
107 + adj.loc[before, col] = adj.loc[before, col] + (new_c - old_c)
108 + else:
109 + adj.loc[before, col] = adj.loc[before, col] * (new_c / old_c if old_c else 1.0)
110 + out = adj
111 +
112 + print(f"\nrolls ({a.roll}{' ' + str(a.roll_days) + 'bd' if a.roll == 'calendar' else ''}, adjust={a.adjust}): {len(rolls)}")
113 + for d, sym, why in rolls:
114 + print(f" {d.date()} → {sym} ({why})")
115 + print(f"\n{len(out):,} rows {out.index[0].date()} → {out.index[-1].date()} · missing closes: {int(out['close'].isna().sum())}")
116 + print(f"last close (adjusted={a.adjust}): {out['close'].iloc[-1]:.4f} · contract {out['contract'].iloc[-1]}")
117 + if a.out:
118 + out.to_csv(a.out)
119 + print(f"wrote {a.out}")
120 + return 0
121 +
122 +
123 +if __name__ == "__main__":
124 + sys.exit(main())
added skills/hfmd-data-analysis/SKILL.md +57 −0
@@ -0,0 +1,57 @@
1 +---
2 +name: hfmd-data-analysis
3 +description: Fetch historical OHLCV bars (stocks, ETFs, futures, crypto, indices, FX; 1-minute to daily) from HF Market Data into pandas and produce summary statistics, return/volatility/drawdown analysis and charts. Use when the user asks to analyse, describe, chart or export price history for a symbol or a basket.
4 +---
5 +
6 +# hfmd-data-analysis
7 +
8 +Pull price history from `https://www.hfmarketdata.io` (v1 `/v1/bars/{asset}/{ticker}`) into a DataFrame and
9 +answer "what did this instrument do?" questions with numbers, not impressions.
10 +
11 +## When to use
12 +
13 +- "Show me AAPL daily since 2020 and summarise it", "how volatile was BTC in 2024?", "export SPY 5-minute bars for last week to CSV"
14 +- Any request that needs bars in pandas before something else (feature engineering, correlations, seasonality)
15 +- Not for backtests (use `hfmd-quick-backtest`), continuous futures methodology (`hfmd-continuous-futures`) or curves (`hfmd-term-structure`)
16 +
17 +## Inputs to confirm with the user
18 +
19 +| Parameter | Values | Default |
20 +|---|---|---|
21 +| asset | `stock` `etf` `crypto` `index` `fx` `futures` (vendor continuous) | infer from the symbol |
22 +| timeframe | `1min` `5min` `30min` `1hour` `1day` | `1day` |
23 +| start / end | `YYYY-MM-DD` (intraday: keep ranges short — one day of 1-min bars ≈ 390 rows RTH, 1 440 for crypto) | last 5 years for daily |
24 +| adjustment | stock/etf: `UNADJUSTED` `adj_split` `adj_splitdiv` · futures: `contin_UNadj` `contin_adj_ratio` `contin_adj_absolute` | API default (`adj_splitdiv` for stocks) |
25 +
26 +## Steps
27 +
28 +1. Resolve the symbol if unsure: `GET /v1/{asset}/tickers?search=AAP` (or ask).
29 +2. Fetch: `python3 scripts/fetch_bars.py --asset stock --ticker AAPL --timeframe 1day --start 2020-01-01 --out aapl.csv`
30 + - paginates automatically (keyless max 5 000 rows/request) and prints the exact range received — never assume the range you asked for is the range you got.
31 +3. Analyse: `python3 scripts/analyze.py aapl.csv --plot aapl.png`
32 + - prints: rows, first/last bar, CAGR, annualised volatility, Sharpe (rf = 0), max drawdown (with dates), best/worst bar, skew/kurtosis, gap count, monthly return table; writes a price + drawdown chart if `--plot`.
33 +4. Report the numbers with their window and adjustment; flag missing sessions instead of filling them.
34 +
35 +## Examples
36 +
37 +```bash
38 +# 1. Five years of daily SPY, summary + chart
39 +python3 scripts/fetch_bars.py --asset etf --ticker SPY --start 2021-01-01 --out spy.csv && python3 scripts/analyze.py spy.csv --plot spy.png
40 +
41 +# 2. One week of 5-minute bars for TSLA (intraday: naive US/Eastern timestamps)
42 +python3 scripts/fetch_bars.py --asset stock --ticker TSLA --timeframe 5min --start 2025-08-25 --end 2025-08-29 --out tsla_5m.csv
43 +
44 +# 3. Vendor continuous crude oil, ratio-adjusted, to parquet
45 +python3 scripts/fetch_bars.py --asset futures --ticker CL --adjustment contin_adj_ratio --start 2015-01-01 --out cl.parquet
46 +
47 +# 4. Several tickers → one long DataFrame
48 +python3 scripts/fetch_bars.py --asset stock --ticker AAPL MSFT NVDA --start 2024-01-01 --out mega.csv
49 +```
50 +
51 +## Gotchas
52 +
53 +- Intraday `datetime` is naive **US/Eastern** (exchange time); daily bars are plain dates. Localise before joining with UTC data.
54 +- Stocks default to split+dividend adjusted prices; say so when quoting historical levels.
55 +- Keyless quota: 30 requests/hour, 5 000 rows/request. Set `HFMD_API_KEY` (free account, 120 req/min, 50 000 rows/request) before pulling intraday history.
56 +- `volume` for FX/indices may be 0/absent — do not compute volume statistics on them.
57 +- Error `TICKER_NOT_FOUND` usually means the symbol exists in another asset class (e.g. `SPY` is `etf`, not `stock`).
added skills/hfmd-data-analysis/scripts/analyze.py +119 −0
@@ -0,0 +1,119 @@
1 +#!/usr/bin/env python3
2 +"""Summary statistics for a bars file produced by fetch_bars.py (one or several tickers).
3 +
4 + analyze.py aapl.csv [--plot aapl.png] [--periods-per-year 252]
5 +
6 +Prints, per ticker: rows, range, CAGR, annualised volatility, Sharpe (rf=0), max drawdown (+ dates),
7 +best/worst bar, skew, kurtosis, number of gaps > 3 calendar days, monthly returns table.
8 +"""
9 +from __future__ import annotations
10 +
11 +import argparse
12 +import sys
13 +
14 +import numpy as np
15 +import pandas as pd
16 +
17 +
18 +def infer_periods_per_year(dt: pd.Series) -> int:
19 + if len(dt) < 3:
20 + return 252
21 + step = dt.diff().dropna().median()
22 + if step >= pd.Timedelta(days=1):
23 + return 252 if dt.dt.dayofweek.max() <= 4 else 365
24 + per_day = int(pd.Timedelta(hours=6.5) / step) if step < pd.Timedelta(hours=1) else int(pd.Timedelta(hours=24) / step)
25 + return max(per_day, 1) * 252
26 +
27 +
28 +def stats(df: pd.DataFrame, ppy: int | None) -> dict:
29 + df = df.sort_values("datetime").reset_index(drop=True)
30 + ppy = ppy or infer_periods_per_year(df["datetime"])
31 + close = df["close"].astype(float)
32 + ret = close.pct_change().dropna()
33 + years = max((df["datetime"].iloc[-1] - df["datetime"].iloc[0]).days / 365.25, 1e-9)
34 + equity = (1 + ret).cumprod()
35 + dd = equity / equity.cummax() - 1
36 + trough = dd.idxmin() if len(dd) else None
37 + peak = equity.loc[:trough].idxmax() if trough is not None else None
38 + gaps = (df["datetime"].diff() > pd.Timedelta(days=3)).sum()
39 + out = {
40 + "rows": len(df),
41 + "first": str(df["datetime"].iloc[0]),
42 + "last": str(df["datetime"].iloc[-1]),
43 + "periods_per_year": ppy,
44 + "total_return": close.iloc[-1] / close.iloc[0] - 1,
45 + "cagr": (close.iloc[-1] / close.iloc[0]) ** (1 / years) - 1,
46 + "ann_vol": ret.std() * np.sqrt(ppy),
47 + "sharpe_rf0": (ret.mean() / ret.std() * np.sqrt(ppy)) if ret.std() > 0 else np.nan,
48 + "max_drawdown": dd.min() if len(dd) else np.nan,
49 + "dd_peak": str(df["datetime"].iloc[peak]) if peak is not None else None,
50 + "dd_trough": str(df["datetime"].iloc[trough]) if trough is not None else None,
51 + "best_bar": ret.max(),
52 + "worst_bar": ret.min(),
53 + "skew": ret.skew(),
54 + "kurtosis": ret.kurt(),
55 + "gaps_gt_3d": int(gaps),
56 + "avg_volume": float(df["volume"].mean()) if "volume" in df else None,
57 + }
58 + return out
59 +
60 +
61 +def monthly_table(df: pd.DataFrame) -> pd.DataFrame:
62 + s = df.set_index("datetime")["close"].astype(float).resample("ME").last().pct_change().dropna()
63 + t = s.to_frame("r")
64 + t["year"], t["month"] = t.index.year, t.index.month
65 + return (t.pivot(index="year", columns="month", values="r") * 100).round(1)
66 +
67 +
68 +def main() -> int:
69 + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
70 + ap.add_argument("file")
71 + ap.add_argument("--plot", help="write a price + drawdown PNG (needs matplotlib)")
72 + ap.add_argument("--periods-per-year", type=int)
73 + a = ap.parse_args()
74 +
75 + df = pd.read_parquet(a.file) if a.file.endswith(".parquet") else pd.read_csv(a.file)
76 + df["datetime"] = pd.to_datetime(df["datetime"])
77 + tickers = df["ticker"].unique() if "ticker" in df else ["?"]
78 + for t in tickers:
79 + sub = df[df["ticker"] == t] if "ticker" in df else df
80 + s = stats(sub, a.periods_per_year)
81 + print(f"\n=== {t} ===")
82 + for k, v in s.items():
83 + if isinstance(v, float):
84 + print(f"{k:>18}: {v:,.4f}" if abs(v) < 1000 else f"{k:>18}: {v:,.0f}")
85 + else:
86 + print(f"{k:>18}: {v}")
87 + if s["periods_per_year"] <= 365 and len(sub) > 40:
88 + print("\nmonthly returns (%):")
89 + print(monthly_table(sub).to_string())
90 +
91 + if a.plot:
92 + try:
93 + import matplotlib
94 + matplotlib.use("Agg")
95 + import matplotlib.pyplot as plt
96 + except ImportError:
97 + print("matplotlib not installed: pip install matplotlib", file=sys.stderr)
98 + return 0
99 + fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(11, 6.5), sharex=True, gridspec_kw={"height_ratios": [3, 1]})
100 + for t in tickers:
101 + sub = (df[df["ticker"] == t] if "ticker" in df else df).sort_values("datetime")
102 + close = sub["close"].astype(float)
103 + norm = close / close.iloc[0] * 100 if len(tickers) > 1 else close
104 + ax1.plot(sub["datetime"], norm, lw=1.1, label=t)
105 + eq = (1 + close.pct_change().fillna(0)).cumprod()
106 + ax2.fill_between(sub["datetime"], (eq / eq.cummax() - 1) * 100, 0, alpha=0.35)
107 + ax1.set_title(f"{', '.join(tickers)} — {'rebased to 100' if len(tickers) > 1 else 'close'}")
108 + ax1.grid(alpha=0.25)
109 + ax1.legend(loc="upper left")
110 + ax2.set_ylabel("drawdown %")
111 + ax2.grid(alpha=0.25)
112 + fig.tight_layout()
113 + fig.savefig(a.plot, dpi=130)
114 + print(f"\nwrote {a.plot}")
115 + return 0
116 +
117 +
118 +if __name__ == "__main__":
119 + sys.exit(main())
added skills/hfmd-data-analysis/scripts/fetch_bars.py +52 −0
@@ -0,0 +1,52 @@
1 +#!/usr/bin/env python3
2 +"""Fetch OHLCV bars from HF Market Data into CSV or Parquet.
3 +
4 +Examples:
5 + fetch_bars.py --asset stock --ticker AAPL --start 2020-01-01 --out aapl.csv
6 + fetch_bars.py --asset stock --ticker AAPL MSFT --timeframe 1hour --start 2025-08-01 --out both.csv
7 + fetch_bars.py --asset futures --ticker CL --adjustment contin_adj_ratio --out cl.parquet
8 +"""
9 +from __future__ import annotations
10 +
11 +import argparse
12 +import sys
13 +from pathlib import Path
14 +
15 +import pandas as pd
16 +
17 +sys.path.insert(0, str(Path(__file__).parent))
18 +import hfmd # noqa: E402
19 +
20 +
21 +def main() -> int:
22 + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
23 + ap.add_argument("--asset", required=True, choices=["stock", "etf", "crypto", "index", "fx", "futures"])
24 + ap.add_argument("--ticker", required=True, nargs="+", help="one or more symbols")
25 + ap.add_argument("--timeframe", default="1day", help="1min 5min 30min 1hour 1day (aliases 1m 5m 30m 1h 1d)")
26 + ap.add_argument("--start")
27 + ap.add_argument("--end")
28 + ap.add_argument("--adjustment", help="stock/etf: UNADJUSTED adj_split adj_splitdiv · futures: contin_UNadj contin_adj_ratio contin_adj_absolute")
29 + ap.add_argument("--out", required=True, help=".csv or .parquet")
30 + a = ap.parse_args()
31 +
32 + frames = []
33 + for t in a.ticker:
34 + df = hfmd.bars(a.asset, t.upper(), a.timeframe, a.start, a.end, a.adjustment)
35 + if df.empty:
36 + hfmd.log(f"{t}: no rows returned")
37 + continue
38 + hfmd.log(f"{t}: {len(df):,} bars from {df['datetime'].iloc[0]} to {df['datetime'].iloc[-1]}")
39 + frames.append(df)
40 + if not frames:
41 + return 1
42 + out = pd.concat(frames, ignore_index=True)
43 + if a.out.endswith(".parquet"):
44 + out.to_parquet(a.out, index=False)
45 + else:
46 + out.to_csv(a.out, index=False)
47 + print(f"wrote {a.out}: {len(out):,} rows, {out['ticker'].nunique()} ticker(s), columns={list(out.columns)}")
48 + return 0
49 +
50 +
51 +if __name__ == "__main__":
52 + sys.exit(main())
added skills/hfmd-data-analysis/scripts/hfmd.py +156 −0
@@ -0,0 +1,156 @@
1 +"""Tiny HF Market Data client shared by the hfmd-* skills (requests + pandas only).
2 +
3 +Copied verbatim into every skill's `scripts/` folder by `skills/scripts/build_skills.py`
4 +so each skill stays self-contained. Do not edit the copies — edit `skills/_shared/hfmd.py`.
5 +
6 +Environment: HFMD_API_KEY (optional, free account = 120 req/min; keyless = 30 req/h),
7 + HFMD_BASE_URL (default https://www.hfmarketdata.io).
8 +"""
9 +from __future__ import annotations
10 +
11 +import os
12 +import sys
13 +import time
14 +from typing import Any
15 +
16 +import pandas as pd
17 +import requests
18 +
19 +BASE_URL = os.environ.get("HFMD_BASE_URL", "https://www.hfmarketdata.io").rstrip("/")
20 +API_KEY = os.environ.get("HFMD_API_KEY", "").strip()
21 +UA = "hfmd-skills/1.0 (+https://www.hfmarketdata.io/integrations/skills)"
22 +
23 +TIMEFRAMES = {"1m": "1min", "5m": "5min", "30m": "30min", "1h": "1hour", "1d": "1day",
24 + "1min": "1min", "5min": "5min", "30min": "30min", "1hour": "1hour", "1day": "1day"}
25 +V2_INTERVAL = {"1min": "1m", "5min": "5m", "30min": "30m", "1hour": "1h", "1day": "1d"}
26 +
27 +
28 +class HfmdError(RuntimeError):
29 + def __init__(self, status: int, code: str, message: str, url: str):
30 + super().__init__(f"HF Market Data error {status} {code}: {message} ({url})")
31 + self.status, self.code, self.url = status, code, url
32 +
33 +
34 +_session = requests.Session()
35 +_session.headers.update({"Accept": "application/json", "User-Agent": UA})
36 +if API_KEY:
37 + _session.headers["Authorization"] = f"Bearer {API_KEY}"
38 +
39 +
40 +def log(msg: str) -> None:
41 + print(msg, file=sys.stderr)
42 +
43 +
44 +def get(path: str, params: dict[str, Any] | None = None, *, retries: int = 3) -> tuple[Any, dict[str, str]]:
45 + """GET a JSON endpoint. Returns (body, headers). Retries on 429 honouring Retry-After."""
46 + url = f"{BASE_URL}{path}"
47 + p = {k: (",".join(v) if isinstance(v, (list, tuple)) else v) for k, v in (params or {}).items() if v not in (None, "")}
48 + p.setdefault("format", "json")
49 + for attempt in range(retries + 1):
50 + r = _session.get(url, params=p, timeout=120)
51 + if r.status_code == 429 and attempt < retries:
52 + wait = float(r.headers.get("Retry-After", "5"))
53 + log(f"429 rate limited — sleeping {wait:.0f}s (set HFMD_API_KEY for 120 req/min)")
54 + time.sleep(min(wait, 120))
55 + continue
56 + if r.status_code >= 400:
57 + try:
58 + body = r.json()
59 + except ValueError:
60 + body = {}
61 + err = body.get("error") or {}
62 + raise HfmdError(r.status_code, err.get("code", "HTTP_ERROR"), err.get("message") or body.get("detail") or r.text[:200], r.url)
63 + rem = r.headers.get("X-RateLimit-Remaining-Requests")
64 + if rem is not None:
65 + log(f"rate limit: {rem}/{r.headers.get('X-RateLimit-Limit-Requests')} requests left")
66 + return r.json(), dict(r.headers)
67 + raise HfmdError(429, "RATE_LIMIT_EXCEEDED", "gave up after retries", url)
68 +
69 +
70 +def rows_of(body: Any) -> list[dict]:
71 + """Rows from either envelope: v1 {count,data} or v2 {data,meta}."""
72 + if isinstance(body, list):
73 + return body
74 + if isinstance(body, dict):
75 + data = body.get("data")
76 + if isinstance(data, list):
77 + return data
78 + return []
79 +
80 +
81 +def meta_of(body: Any) -> dict:
82 + if isinstance(body, dict):
83 + m = dict(body.get("meta") or {})
84 + if "count" in body and "count" not in m:
85 + m["count"] = body["count"]
86 + return m
87 + return {}
88 +
89 +
90 +def to_df(body: Any, time_col: str | None = "datetime") -> pd.DataFrame:
91 + df = pd.DataFrame(rows_of(body))
92 + if time_col and time_col in df.columns:
93 + df[time_col] = pd.to_datetime(df[time_col])
94 + df = df.sort_values(time_col).reset_index(drop=True)
95 + return df
96 +
97 +
98 +def bars(asset: str, ticker: str, timeframe: str = "1day", start: str | None = None, end: str | None = None,
99 + adjustment: str | None = None, limit: int = 50_000) -> pd.DataFrame:
100 + """v1 bars (stock/etf/crypto/index/fx, or futures = vendor continuous). Paginates by date when needed."""
101 + tf = TIMEFRAMES.get(timeframe, timeframe)
102 + frames: list[pd.DataFrame] = []
103 + cursor_start = start
104 + while True:
105 + body, _ = get(f"/v1/bars/{asset}/{ticker}", {"timeframe": tf, "start": cursor_start, "end": end, "adjustment": adjustment, "limit": limit, "order": "asc"})
106 + df = to_df(body)
107 + if df.empty:
108 + break
109 + frames.append(df)
110 + if len(df) < limit:
111 + break
112 + last = df["datetime"].iloc[-1]
113 + cursor_start = (last + pd.Timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M") if tf != "1day" else (last + pd.Timedelta(days=1)).strftime("%Y-%m-%d")
114 + out = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(columns=["ticker", "datetime", "open", "high", "low", "close", "volume"])
115 + return out.drop_duplicates("datetime").reset_index(drop=True)
116 +
117 +
118 +def contract_bars(symbol: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, limit: int = 50_000) -> pd.DataFrame:
119 + """v2 individual futures contract bars (ESZ25…)."""
120 + tf = TIMEFRAMES.get(timeframe, timeframe)
121 + body, _ = get(f"/v1/futures/contract/{symbol}/bars", {"interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit})
122 + return to_df(body)
123 +
124 +
125 +def continuous(root: str, roll: str = "volume", adjust: str = "back_adjusted", depth: int = 1, timeframe: str = "1day",
126 + start: str | None = None, end: str | None = None, limit: int = 50_000) -> tuple[pd.DataFrame, dict]:
127 + """v2 server-built continuous series. Returns (df, meta) — meta['roll_dates'] lists the rolls."""
128 + tf = TIMEFRAMES.get(timeframe, timeframe)
129 + body, _ = get(f"/v1/futures/{root}/continuous", {"roll": roll, "adjust": adjust, "depth": depth, "interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit})
130 + return to_df(body), meta_of(body)
131 +
132 +
133 +def contracts(root: str, status: str | None = None) -> pd.DataFrame:
134 + body, _ = get(f"/v1/futures/{root}/contracts", {"status": status, "limit": 1000})
135 + return to_df(body, time_col=None)
136 +
137 +
138 +def term_structure(root: str, as_of: str | None = None) -> tuple[pd.DataFrame, dict]:
139 + body, _ = get(f"/v1/futures/{root}/term-structure", {"as_of": as_of})
140 + return to_df(body, time_col=None), meta_of(body)
141 +
142 +
143 +def screener(filters: str, sort: str | None = None, as_of: str | None = None, limit: int = 100) -> tuple[pd.DataFrame, dict]:
144 + body, _ = get("/v1/fundamentals/screener", {"filters": filters, "sort": sort, "as_of": as_of, "limit": limit})
145 + return to_df(body, time_col=None), meta_of(body)
146 +
147 +
148 +def ratios(ticker: str) -> dict:
149 + body, _ = get(f"/v1/fundamentals/{ticker}/ratios")
150 + d = body.get("data", body) if isinstance(body, dict) else body
151 + return d if isinstance(d, dict) else (d[0] if d else {})
152 +
153 +
154 +def statements(ticker: str, statement: str = "income", period: str = "quarterly", limit: int = 12) -> pd.DataFrame:
155 + body, _ = get(f"/v1/fundamentals/{ticker}/statements", {"statement": statement, "period": period, "limit": limit})
156 + return to_df(body, time_col=None)
added skills/hfmd-fundamentals-screen/SKILL.md +57 −0
@@ -0,0 +1,57 @@
1 +---
2 +name: hfmd-fundamentals-screen
3 +description: Screen US stocks on fundamental ratios with HF Market Data (PE, PB, EV/EBITDA, FCF yield, ROE, margins, leverage, growth), enrich the hits with their latest ratios and recent quarterly statements, and document the point-in-time caveats (filing dates, restatements). Use when the user asks to "find stocks with…", "screen for…", or compare companies on fundamentals.
4 +---
5 +
6 +# hfmd-fundamentals-screen
7 +
8 +Run a screener → pull ratios and statements for the survivors → produce a table the user can defend,
9 +with the date each number became public.
10 +
11 +## When to use
12 +
13 +- "Screen US stocks with PE<15 and FCF yield>6% and show their last 3 quarters"
14 +- "Cheapest software companies by EV/EBITDA with ROE > 20 %", "which of these tickers has the best balance sheet?"
15 +- Historical: "what would this screen have returned on 2020-03-20?" (`--as-of`, point-in-time)
16 +
17 +## Filter syntax
18 +
19 +`filters="pe<15,roe>0.15,fcf_yield>0.06,market_cap>1e9"` — comma-separated `metric<op>value`, ops `< <= > >= =`.
20 +Ratios are **decimals** (0.15 = 15 %). Known metrics: `pe pb ps ev_ebitda fcf_yield dividend_yield roe roa gross_margin operating_margin net_margin debt_to_equity current_ratio revenue_growth eps_growth market_cap sector`.
21 +Sort: `--sort fcf_yield:desc`.
22 +
23 +## Steps
24 +
25 +1. Translate the request into filters (ask for thresholds when vague: "cheap" → `pe<15`? `ev_ebitda<8`?). Convert percentages to decimals.
26 +2. Screen + enrich:
27 + `python3 scripts/screen.py --filters "pe<15,fcf_yield>0.06" --sort fcf_yield:desc --limit 25 --quarters 3 --out screen.csv`
28 + - calls `/v1/fundamentals/screener` (cost: 2 requests), then for each hit `/v1/fundamentals/{t}/ratios` and `/v1/fundamentals/{t}/statements?statement=income&period=quarterly&limit=3` (`--no-enrich` to skip; keyless quota allows ~10 tickers/hour — set `HFMD_API_KEY`).
29 + - prints the screen table, then per ticker the last N quarters (revenue, operating income, net income, EPS, filed_at) and a **point-in-time note**: latest `filed_at` used, days since, whether a newer period end exists without a filing yet.
30 +3. Present: one table sorted as requested, then the quarterly mini-tables, then caveats.
31 +
32 +## Point-in-time discipline (always say it)
33 +
34 +- Every fundamentals row carries `period_end` **and** `filed_at`. A number is knowable only from `filed_at` (10-Q ≈ 40 days, 10-K ≈ 60-90 days after period end). Screens with `--as-of` use what was filed by that date — this is what makes a backtest of the screen honest.
35 +- Restatements: `facts/{concept}` returns every reported value for a period; the statements endpoint returns the latest. Mention it when a number looks off.
36 +- Price-based ratios (PE, FCF yield…) use the close of `as_of` (or the latest) and the last **reported** TTM figures — not analyst estimates.
37 +- Coverage differs by company (`/v1/fundamentals/{t}/coverage`): small caps and recent IPOs have short histories; missing = `null`, never imputed.
38 +
39 +## Examples
40 +
41 +```bash
42 +# Classic value + quality
43 +python3 scripts/screen.py --filters "pe<15,roe>0.15,debt_to_equity<1" --sort pe:asc --limit 20
44 +
45 +# Point-in-time: what the screen showed at the March 2020 low
46 +python3 scripts/screen.py --filters "fcf_yield>0.08,net_margin>0.1" --as-of 2020-03-20 --no-enrich
47 +
48 +# Only enrich a list you already have
49 +python3 scripts/screen.py --tickers AAPL MSFT GOOGL --quarters 4
50 +```
51 +
52 +## Gotchas
53 +
54 +- A screener 404 means the fundamentals module is not deployed yet on the server; the script says so and exits 2.
55 +- `sector` is a string filter (`sector=Technology`); everything else is numeric.
56 +- Banks/insurers: EV/EBITDA and FCF yield are not meaningful — prefer PB, ROE.
57 +- The screener costs 2 quota requests; with 30/h keyless, do not loop over many parameter sets without a key.
added skills/hfmd-fundamentals-screen/scripts/hfmd.py +156 −0
@@ -0,0 +1,156 @@
1 +"""Tiny HF Market Data client shared by the hfmd-* skills (requests + pandas only).
2 +
3 +Copied verbatim into every skill's `scripts/` folder by `skills/scripts/build_skills.py`
4 +so each skill stays self-contained. Do not edit the copies — edit `skills/_shared/hfmd.py`.
5 +
6 +Environment: HFMD_API_KEY (optional, free account = 120 req/min; keyless = 30 req/h),
7 + HFMD_BASE_URL (default https://www.hfmarketdata.io).
8 +"""
9 +from __future__ import annotations
10 +
11 +import os
12 +import sys
13 +import time
14 +from typing import Any
15 +
16 +import pandas as pd
17 +import requests
18 +
19 +BASE_URL = os.environ.get("HFMD_BASE_URL", "https://www.hfmarketdata.io").rstrip("/")
20 +API_KEY = os.environ.get("HFMD_API_KEY", "").strip()
21 +UA = "hfmd-skills/1.0 (+https://www.hfmarketdata.io/integrations/skills)"
22 +
23 +TIMEFRAMES = {"1m": "1min", "5m": "5min", "30m": "30min", "1h": "1hour", "1d": "1day",
24 + "1min": "1min", "5min": "5min", "30min": "30min", "1hour": "1hour", "1day": "1day"}
25 +V2_INTERVAL = {"1min": "1m", "5min": "5m", "30min": "30m", "1hour": "1h", "1day": "1d"}
26 +
27 +
28 +class HfmdError(RuntimeError):
29 + def __init__(self, status: int, code: str, message: str, url: str):
30 + super().__init__(f"HF Market Data error {status} {code}: {message} ({url})")
31 + self.status, self.code, self.url = status, code, url
32 +
33 +
34 +_session = requests.Session()
35 +_session.headers.update({"Accept": "application/json", "User-Agent": UA})
36 +if API_KEY:
37 + _session.headers["Authorization"] = f"Bearer {API_KEY}"
38 +
39 +
40 +def log(msg: str) -> None:
41 + print(msg, file=sys.stderr)
42 +
43 +
44 +def get(path: str, params: dict[str, Any] | None = None, *, retries: int = 3) -> tuple[Any, dict[str, str]]:
45 + """GET a JSON endpoint. Returns (body, headers). Retries on 429 honouring Retry-After."""
46 + url = f"{BASE_URL}{path}"
47 + p = {k: (",".join(v) if isinstance(v, (list, tuple)) else v) for k, v in (params or {}).items() if v not in (None, "")}
48 + p.setdefault("format", "json")
49 + for attempt in range(retries + 1):
50 + r = _session.get(url, params=p, timeout=120)
51 + if r.status_code == 429 and attempt < retries:
52 + wait = float(r.headers.get("Retry-After", "5"))
53 + log(f"429 rate limited — sleeping {wait:.0f}s (set HFMD_API_KEY for 120 req/min)")
54 + time.sleep(min(wait, 120))
55 + continue
56 + if r.status_code >= 400:
57 + try:
58 + body = r.json()
59 + except ValueError:
60 + body = {}
61 + err = body.get("error") or {}
62 + raise HfmdError(r.status_code, err.get("code", "HTTP_ERROR"), err.get("message") or body.get("detail") or r.text[:200], r.url)
63 + rem = r.headers.get("X-RateLimit-Remaining-Requests")
64 + if rem is not None:
65 + log(f"rate limit: {rem}/{r.headers.get('X-RateLimit-Limit-Requests')} requests left")
66 + return r.json(), dict(r.headers)
67 + raise HfmdError(429, "RATE_LIMIT_EXCEEDED", "gave up after retries", url)
68 +
69 +
70 +def rows_of(body: Any) -> list[dict]:
71 + """Rows from either envelope: v1 {count,data} or v2 {data,meta}."""
72 + if isinstance(body, list):
73 + return body
74 + if isinstance(body, dict):
75 + data = body.get("data")
76 + if isinstance(data, list):
77 + return data
78 + return []
79 +
80 +
81 +def meta_of(body: Any) -> dict:
82 + if isinstance(body, dict):
83 + m = dict(body.get("meta") or {})
84 + if "count" in body and "count" not in m:
85 + m["count"] = body["count"]
86 + return m
87 + return {}
88 +
89 +
90 +def to_df(body: Any, time_col: str | None = "datetime") -> pd.DataFrame:
91 + df = pd.DataFrame(rows_of(body))
92 + if time_col and time_col in df.columns:
93 + df[time_col] = pd.to_datetime(df[time_col])
94 + df = df.sort_values(time_col).reset_index(drop=True)
95 + return df
96 +
97 +
98 +def bars(asset: str, ticker: str, timeframe: str = "1day", start: str | None = None, end: str | None = None,
99 + adjustment: str | None = None, limit: int = 50_000) -> pd.DataFrame:
100 + """v1 bars (stock/etf/crypto/index/fx, or futures = vendor continuous). Paginates by date when needed."""
101 + tf = TIMEFRAMES.get(timeframe, timeframe)
102 + frames: list[pd.DataFrame] = []
103 + cursor_start = start
104 + while True:
105 + body, _ = get(f"/v1/bars/{asset}/{ticker}", {"timeframe": tf, "start": cursor_start, "end": end, "adjustment": adjustment, "limit": limit, "order": "asc"})
106 + df = to_df(body)
107 + if df.empty:
108 + break
109 + frames.append(df)
110 + if len(df) < limit:
111 + break
112 + last = df["datetime"].iloc[-1]
113 + cursor_start = (last + pd.Timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M") if tf != "1day" else (last + pd.Timedelta(days=1)).strftime("%Y-%m-%d")
114 + out = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(columns=["ticker", "datetime", "open", "high", "low", "close", "volume"])
115 + return out.drop_duplicates("datetime").reset_index(drop=True)
116 +
117 +
118 +def contract_bars(symbol: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, limit: int = 50_000) -> pd.DataFrame:
119 + """v2 individual futures contract bars (ESZ25…)."""
120 + tf = TIMEFRAMES.get(timeframe, timeframe)
121 + body, _ = get(f"/v1/futures/contract/{symbol}/bars", {"interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit})
122 + return to_df(body)
123 +
124 +
125 +def continuous(root: str, roll: str = "volume", adjust: str = "back_adjusted", depth: int = 1, timeframe: str = "1day",
126 + start: str | None = None, end: str | None = None, limit: int = 50_000) -> tuple[pd.DataFrame, dict]:
127 + """v2 server-built continuous series. Returns (df, meta) — meta['roll_dates'] lists the rolls."""
128 + tf = TIMEFRAMES.get(timeframe, timeframe)
129 + body, _ = get(f"/v1/futures/{root}/continuous", {"roll": roll, "adjust": adjust, "depth": depth, "interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit})
130 + return to_df(body), meta_of(body)
131 +
132 +
133 +def contracts(root: str, status: str | None = None) -> pd.DataFrame:
134 + body, _ = get(f"/v1/futures/{root}/contracts", {"status": status, "limit": 1000})
135 + return to_df(body, time_col=None)
136 +
137 +
138 +def term_structure(root: str, as_of: str | None = None) -> tuple[pd.DataFrame, dict]:
139 + body, _ = get(f"/v1/futures/{root}/term-structure", {"as_of": as_of})
140 + return to_df(body, time_col=None), meta_of(body)
141 +
142 +
143 +def screener(filters: str, sort: str | None = None, as_of: str | None = None, limit: int = 100) -> tuple[pd.DataFrame, dict]:
144 + body, _ = get("/v1/fundamentals/screener", {"filters": filters, "sort": sort, "as_of": as_of, "limit": limit})
145 + return to_df(body, time_col=None), meta_of(body)
146 +
147 +
148 +def ratios(ticker: str) -> dict:
149 + body, _ = get(f"/v1/fundamentals/{ticker}/ratios")
150 + d = body.get("data", body) if isinstance(body, dict) else body
151 + return d if isinstance(d, dict) else (d[0] if d else {})
152 +
153 +
154 +def statements(ticker: str, statement: str = "income", period: str = "quarterly", limit: int = 12) -> pd.DataFrame:
155 + body, _ = get(f"/v1/fundamentals/{ticker}/statements", {"statement": statement, "period": period, "limit": limit})
156 + return to_df(body, time_col=None)
added skills/hfmd-fundamentals-screen/scripts/screen.py +114 −0
@@ -0,0 +1,114 @@
1 +#!/usr/bin/env python3
2 +"""Fundamental screener + enrichment with point-in-time notes.
3 +
4 + screen.py --filters "pe<15,fcf_yield>0.06" [--sort fcf_yield:desc] [--as-of 2020-03-20] [--limit 25]
5 + [--quarters 3] [--no-enrich] [--out screen.csv]
6 + screen.py --tickers AAPL MSFT --quarters 4 # enrich only
7 +"""
8 +from __future__ import annotations
9 +
10 +import argparse
11 +import sys
12 +from datetime import date
13 +from pathlib import Path
14 +
15 +import pandas as pd
16 +
17 +sys.path.insert(0, str(Path(__file__).parent))
18 +import hfmd # noqa: E402
19 +
20 +KEY_RATIOS = ["pe", "pb", "ev_ebitda", "fcf_yield", "dividend_yield", "roe", "roa", "gross_margin", "operating_margin", "net_margin", "debt_to_equity", "current_ratio", "revenue_growth", "eps_growth", "market_cap"]
21 +INCOME_COLS = ["period_end", "fiscal_period", "revenue", "operating_income", "net_income", "eps_diluted", "filed_at", "form"]
22 +
23 +
24 +def pit_note(q: pd.DataFrame) -> str:
25 + if q.empty or "filed_at" not in q:
26 + return "no quarterly statements available"
27 + q = q.copy()
28 + q["filed_at"] = pd.to_datetime(q["filed_at"], errors="coerce")
29 + q["period_end"] = pd.to_datetime(q.get("period_end"), errors="coerce")
30 + latest = q.sort_values("filed_at").iloc[-1]
31 + lag = (latest["filed_at"] - latest["period_end"]).days if pd.notna(latest["filed_at"]) and pd.notna(latest["period_end"]) else None
32 + age = (pd.Timestamp(date.today()) - latest["filed_at"]).days if pd.notna(latest["filed_at"]) else None
33 + return (f"latest period {latest['period_end'].date() if pd.notna(latest['period_end']) else '?'} became public on "
34 + f"{latest['filed_at'].date() if pd.notna(latest['filed_at']) else '?'} ({lag} days after period end; {age} days ago)"
35 + + (" — a newer quarter has ended but is not filed yet" if age is not None and age > 100 else ""))
36 +
37 +
38 +def main() -> int:
39 + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
40 + ap.add_argument("--filters", help='e.g. "pe<15,roe>0.15,fcf_yield>0.06"')
41 + ap.add_argument("--sort", help="metric:asc|desc")
42 + ap.add_argument("--as-of", help="point-in-time date YYYY-MM-DD")
43 + ap.add_argument("--limit", type=int, default=25)
44 + ap.add_argument("--tickers", nargs="*", help="skip the screener, enrich these tickers")
45 + ap.add_argument("--quarters", type=int, default=3)
46 + ap.add_argument("--no-enrich", action="store_true")
47 + ap.add_argument("--out")
48 + a = ap.parse_args()
49 + if not a.filters and not a.tickers:
50 + ap.error("--filters or --tickers required")
51 +
52 + hits = pd.DataFrame()
53 + if a.filters:
54 + try:
55 + hits, meta = hfmd.screener(a.filters, a.sort, a.as_of, a.limit)
56 + except hfmd.HfmdError as e:
57 + print(e, file=sys.stderr)
58 + if e.status == 404:
59 + print("→ the fundamentals module is not deployed on this server yet (v2 rollout). Nothing to screen.", file=sys.stderr)
60 + return 2
61 + return 1
62 + print(f"screen `{a.filters}`{' as of ' + a.as_of if a.as_of else ''}: {meta.get('count', len(hits))} match(es)"
63 + + (f" (universe {meta['universe']})" if 'universe' in meta else ""))
64 + if hits.empty:
65 + return 0
66 + with pd.option_context("display.width", 200, "display.max_columns", 30, "display.float_format", "{:.3f}".format):
67 + print(hits.head(a.limit).to_string(index=False))
68 + tickers = list(hits["ticker"].head(a.limit)) if "ticker" in hits else []
69 + else:
70 + tickers = [t.upper() for t in a.tickers]
71 +
72 + if a.no_enrich or not tickers:
73 + if a.out and not hits.empty:
74 + hits.to_csv(a.out, index=False)
75 + print(f"wrote {a.out}")
76 + return 0
77 +
78 + if not hfmd.API_KEY and len(tickers) > 8:
79 + hfmd.log(f"keyless mode: enriching {len(tickers)} tickers needs {2 * len(tickers)} requests (30/h quota) — set HFMD_API_KEY or use --limit 8")
80 +
81 + enriched = []
82 + for t in tickers:
83 + print(f"\n=== {t} ===")
84 + try:
85 + r = hfmd.ratios(t)
86 + line = " ".join(f"{k}={r[k]:.3g}" for k in KEY_RATIOS if isinstance(r.get(k), (int, float)))
87 + print("ratios:", line or r)
88 + if r.get("as_of") or r.get("period_end"):
89 + print(f" based on period {r.get('period_end')} · filed {r.get('filed_at')} · price as of {r.get('as_of')}")
90 + except hfmd.HfmdError as e:
91 + r = {}
92 + print(f"ratios: {e}")
93 + try:
94 + q = hfmd.statements(t, "income", "quarterly", a.quarters)
95 + cols = [c for c in INCOME_COLS if c in q.columns] or list(q.columns)[:8]
96 + with pd.option_context("display.width", 200, "display.float_format", "{:,.0f}".format):
97 + print(q[cols].to_string(index=False) if not q.empty else " no statements")
98 + print(" point-in-time:", pit_note(q))
99 + except hfmd.HfmdError as e:
100 + q = pd.DataFrame()
101 + print(f"statements: {e}")
102 + enriched.append({"ticker": t, **{k: r.get(k) for k in KEY_RATIOS}, "latest_filed_at": (q["filed_at"].max() if "filed_at" in q else None)})
103 +
104 + if a.out:
105 + out = pd.DataFrame(enriched)
106 + if not hits.empty and "ticker" in hits:
107 + out = hits.merge(out, on="ticker", how="left", suffixes=("", "_latest"))
108 + out.to_csv(a.out, index=False)
109 + print(f"\nwrote {a.out}")
110 + return 0
111 +
112 +
113 +if __name__ == "__main__":
114 + sys.exit(main())
added skills/hfmd-quick-backtest/SKILL.md +56 −0
@@ -0,0 +1,56 @@
1 +---
2 +name: hfmd-quick-backtest
3 +description: Backtest a moving-average crossover (or adapt the template to another signal) on HF Market Data bars with an honest walk-forward (parameters chosen on a training window, evaluated on the next out-of-sample window), transaction costs, and a full metrics report. Use when the user wants to "test a strategy", "backtest", or compare signal parameters on a symbol.
4 +---
5 +
6 +# hfmd-quick-backtest
7 +
8 +A small, honest backtest template on top of v1 bars. It exists to give a *defensible* first answer
9 +(walk-forward, costs, benchmark, no look-ahead) — not to be a full trading framework.
10 +
11 +## When to use
12 +
13 +- "Backtest a 50/200 MA crossover on SPY", "does a 20/100 cross work on CL?", "which MA pair worked best on BTC — and did it hold out of sample?"
14 +- Any quick strategy sanity check on daily or intraday bars
15 +
16 +## Method (what the script does)
17 +
18 +1. Fetch bars (`/v1/bars/{asset}/{ticker}`), keep `close`, compute daily/bar returns.
19 +2. **Walk-forward**: split time into consecutive blocks (`--test-years`, default 1). For each test block, select the (fast, slow) pair with the best Sharpe over the preceding `--train-years` (default 3) of data from the `--grid`; apply *that* pair to the test block. Concatenate the out-of-sample test blocks → the reported equity curve.
20 +3. Signal: long when `MA_fast > MA_slow`, flat otherwise (`--allow-short` for ±1). Position is applied to the **next bar** (no look-ahead).
21 +4. Costs: `--cost-bps` per side per change of position (default 5 bps).
22 +5. Report: CAGR, annualised volatility, Sharpe, max drawdown, exposure, number of trades, turnover, vs buy-and-hold on the same out-of-sample span; per-block table of the chosen parameters and the block's Sharpe (this is where you see whether the choice was stable).
23 +6. Optional chart: equity vs benchmark + drawdown + the chosen pair over time.
24 +
25 +## Steps
26 +
27 +```bash
28 +python3 scripts/ma_crossover.py --asset etf --ticker SPY --start 2010-01-01 \
29 + --grid 10,20,50 --grid-slow 100,150,200 --train-years 3 --test-years 1 --cost-bps 5 --plot spy_wf.png
30 +```
31 +
32 +Then explain: (a) the out-of-sample numbers *only*, (b) the stability of the selected parameters across blocks, (c) the in-sample vs out-of-sample gap (`--show-insample` prints the best in-sample pair on the full history for contrast), (d) what is not modelled (slippage beyond bps, borrow, dividends if UNADJUSTED, intraday fills).
33 +
34 +## Examples
35 +
36 +```bash
37 +# Crude oil vendor continuous (ratio-adjusted so returns are meaningful across rolls)
38 +python3 scripts/ma_crossover.py --asset futures --ticker CL --adjustment contin_adj_ratio --start 2012-01-01 --allow-short
39 +
40 +# Bitcoin, hourly bars, 6-month train / 2-month test
41 +python3 scripts/ma_crossover.py --asset crypto --ticker BTCUSD --timeframe 1hour --start 2024-01-01 --train-years 0.5 --test-years 0.17 --grid 12,24,48 --grid-slow 96,168,336
42 +
43 +# Single fixed pair, no optimisation (pure evaluation)
44 +python3 scripts/ma_crossover.py --asset stock --ticker AAPL --fixed 50,200 --start 2015-01-01
45 +```
46 +
47 +## Adapting the template
48 +
49 +`signal_ma_cross(close, fast, slow)` returns a position series in {0,1} (or {-1,0,1}). Replace it with any function of past data only; keep the `.shift(1)` when applying positions and keep the walk-forward loop untouched.
50 +
51 +## Gotchas
52 +
53 +- For futures use `contin_adj_ratio` (multiplicative back-adjustment): `contin_UNadj` has roll jumps that fake returns; `contin_adj_absolute` can go negative in long histories.
54 +- Intraday bars are US/Eastern and include only the sessions the vendor covers; annualisation uses bars/year inferred from the median bar spacing — check the printed `periods_per_year`.
55 +- Keyless quota (30 req/h) is enough for daily history; hourly since 2010 needs an API key (`HFMD_API_KEY`).
56 +- A Sharpe above ~1.5 out-of-sample on a plain MA cross is a red flag for a bug or a tiny sample, not a discovery.
added skills/hfmd-quick-backtest/scripts/hfmd.py +156 −0
@@ -0,0 +1,156 @@
1 +"""Tiny HF Market Data client shared by the hfmd-* skills (requests + pandas only).
2 +
3 +Copied verbatim into every skill's `scripts/` folder by `skills/scripts/build_skills.py`
4 +so each skill stays self-contained. Do not edit the copies — edit `skills/_shared/hfmd.py`.
5 +
6 +Environment: HFMD_API_KEY (optional, free account = 120 req/min; keyless = 30 req/h),
7 + HFMD_BASE_URL (default https://www.hfmarketdata.io).
8 +"""
9 +from __future__ import annotations
10 +
11 +import os
12 +import sys
13 +import time
14 +from typing import Any
15 +
16 +import pandas as pd
17 +import requests
18 +
19 +BASE_URL = os.environ.get("HFMD_BASE_URL", "https://www.hfmarketdata.io").rstrip("/")
20 +API_KEY = os.environ.get("HFMD_API_KEY", "").strip()
21 +UA = "hfmd-skills/1.0 (+https://www.hfmarketdata.io/integrations/skills)"
22 +
23 +TIMEFRAMES = {"1m": "1min", "5m": "5min", "30m": "30min", "1h": "1hour", "1d": "1day",
24 + "1min": "1min", "5min": "5min", "30min": "30min", "1hour": "1hour", "1day": "1day"}
25 +V2_INTERVAL = {"1min": "1m", "5min": "5m", "30min": "30m", "1hour": "1h", "1day": "1d"}
26 +
27 +
28 +class HfmdError(RuntimeError):
29 + def __init__(self, status: int, code: str, message: str, url: str):
30 + super().__init__(f"HF Market Data error {status} {code}: {message} ({url})")
31 + self.status, self.code, self.url = status, code, url
32 +
33 +
34 +_session = requests.Session()
35 +_session.headers.update({"Accept": "application/json", "User-Agent": UA})
36 +if API_KEY:
37 + _session.headers["Authorization"] = f"Bearer {API_KEY}"
38 +
39 +
40 +def log(msg: str) -> None:
41 + print(msg, file=sys.stderr)
42 +
43 +
44 +def get(path: str, params: dict[str, Any] | None = None, *, retries: int = 3) -> tuple[Any, dict[str, str]]:
45 + """GET a JSON endpoint. Returns (body, headers). Retries on 429 honouring Retry-After."""
46 + url = f"{BASE_URL}{path}"
47 + p = {k: (",".join(v) if isinstance(v, (list, tuple)) else v) for k, v in (params or {}).items() if v not in (None, "")}
48 + p.setdefault("format", "json")
49 + for attempt in range(retries + 1):
50 + r = _session.get(url, params=p, timeout=120)
51 + if r.status_code == 429 and attempt < retries:
52 + wait = float(r.headers.get("Retry-After", "5"))
53 + log(f"429 rate limited — sleeping {wait:.0f}s (set HFMD_API_KEY for 120 req/min)")
54 + time.sleep(min(wait, 120))
55 + continue
56 + if r.status_code >= 400:
57 + try:
58 + body = r.json()
59 + except ValueError:
60 + body = {}
61 + err = body.get("error") or {}
62 + raise HfmdError(r.status_code, err.get("code", "HTTP_ERROR"), err.get("message") or body.get("detail") or r.text[:200], r.url)
63 + rem = r.headers.get("X-RateLimit-Remaining-Requests")
64 + if rem is not None:
65 + log(f"rate limit: {rem}/{r.headers.get('X-RateLimit-Limit-Requests')} requests left")
66 + return r.json(), dict(r.headers)
67 + raise HfmdError(429, "RATE_LIMIT_EXCEEDED", "gave up after retries", url)
68 +
69 +
70 +def rows_of(body: Any) -> list[dict]:
71 + """Rows from either envelope: v1 {count,data} or v2 {data,meta}."""
72 + if isinstance(body, list):
73 + return body
74 + if isinstance(body, dict):
75 + data = body.get("data")
76 + if isinstance(data, list):
77 + return data
78 + return []
79 +
80 +
81 +def meta_of(body: Any) -> dict:
82 + if isinstance(body, dict):
83 + m = dict(body.get("meta") or {})
84 + if "count" in body and "count" not in m:
85 + m["count"] = body["count"]
86 + return m
87 + return {}
88 +
89 +
90 +def to_df(body: Any, time_col: str | None = "datetime") -> pd.DataFrame:
91 + df = pd.DataFrame(rows_of(body))
92 + if time_col and time_col in df.columns:
93 + df[time_col] = pd.to_datetime(df[time_col])
94 + df = df.sort_values(time_col).reset_index(drop=True)
95 + return df
96 +
97 +
98 +def bars(asset: str, ticker: str, timeframe: str = "1day", start: str | None = None, end: str | None = None,
99 + adjustment: str | None = None, limit: int = 50_000) -> pd.DataFrame:
100 + """v1 bars (stock/etf/crypto/index/fx, or futures = vendor continuous). Paginates by date when needed."""
101 + tf = TIMEFRAMES.get(timeframe, timeframe)
102 + frames: list[pd.DataFrame] = []
103 + cursor_start = start
104 + while True:
105 + body, _ = get(f"/v1/bars/{asset}/{ticker}", {"timeframe": tf, "start": cursor_start, "end": end, "adjustment": adjustment, "limit": limit, "order": "asc"})
106 + df = to_df(body)
107 + if df.empty:
108 + break
109 + frames.append(df)
110 + if len(df) < limit:
111 + break
112 + last = df["datetime"].iloc[-1]
113 + cursor_start = (last + pd.Timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M") if tf != "1day" else (last + pd.Timedelta(days=1)).strftime("%Y-%m-%d")
114 + out = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame(columns=["ticker", "datetime", "open", "high", "low", "close", "volume"])
115 + return out.drop_duplicates("datetime").reset_index(drop=True)
116 +
117 +
118 +def contract_bars(symbol: str, timeframe: str = "1day", start: str | None = None, end: str | None = None, limit: int = 50_000) -> pd.DataFrame:
119 + """v2 individual futures contract bars (ESZ25…)."""
120 + tf = TIMEFRAMES.get(timeframe, timeframe)
121 + body, _ = get(f"/v1/futures/contract/{symbol}/bars", {"interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit})
122 + return to_df(body)
123 +
124 +
125 +def continuous(root: str, roll: str = "volume", adjust: str = "back_adjusted", depth: int = 1, timeframe: str = "1day",
126 + start: str | None = None, end: str | None = None, limit: int = 50_000) -> tuple[pd.DataFrame, dict]:
127 + """v2 server-built continuous series. Returns (df, meta) — meta['roll_dates'] lists the rolls."""
128 + tf = TIMEFRAMES.get(timeframe, timeframe)
129 + body, _ = get(f"/v1/futures/{root}/continuous", {"roll": roll, "adjust": adjust, "depth": depth, "interval": V2_INTERVAL.get(tf, "1d"), "from": start, "to": end, "limit": limit})
130 + return to_df(body), meta_of(body)
131 +
132 +
133 +def contracts(root: str, status: str | None = None) -> pd.DataFrame:
134 + body, _ = get(f"/v1/futures/{root}/contracts", {"status": status, "limit": 1000})
135 + return to_df(body, time_col=None)
136 +
137 +
138 +def term_structure(root: str, as_of: str | None = None) -> tuple[pd.DataFrame, dict]:
139 + body, _ = get(f"/v1/futures/{root}/term-structure", {"as_of": as_of})
140 + return to_df(body, time_col=None), meta_of(body)
141 +
142 +
143 +def screener(filters: str, sort: str | None = None, as_of: str | None = None, limit: int = 100) -> tuple[pd.DataFrame, dict]:
144 + body, _ = get("/v1/fundamentals/screener", {"filters": filters, "sort": sort, "as_of": as_of, "limit": limit})
145 + return to_df(body, time_col=None), meta_of(body)
146 +
147 +
148 +def ratios(ticker: str) -> dict:
149 + body, _ = get(f"/v1/fundamentals/{ticker}/ratios")
150 + d = body.get("data", body) if isinstance(body, dict) else body
151 + return d if isinstance(d, dict) else (d[0] if d else {})
152 +
153 +
154 +def statements(ticker: str, statement: str = "income", period: str = "quarterly", limit: int = 12) -> pd.DataFrame:
155 + body, _ = get(f"/v1/fundamentals/{ticker}/statements", {"statement": statement, "period": period, "limit": limit})
156 + return to_df(body, time_col=None)
added skills/hfmd-quick-backtest/scripts/ma_crossover.py +182 −0
Large diff (182 lines) is not loaded by default to keep the page fast.Load diff
added skills/hfmd-term-structure/SKILL.md +45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added skills/hfmd-term-structure/scripts/hfmd.py +156 −0
Large diff (156 lines) is not loaded by default to keep the page fast.Load diff
added skills/hfmd-term-structure/scripts/term_structure.py +172 −0
Large diff (172 lines) is not loaded by default to keep the page fast.Load diff
added skills/scripts/build_skills.py +91 −0
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme