# Accounts, API keys & rate limiting (chantier 2) — guide for the web team

Backend modules: hfmarketdata/api/accounts/ (users, keys, auth, dashboard API, admin, CLI) and hfmarketdata/api/ratelimit/ (tiers, Redis sliding windows, ASGI middleware, usage accounting). Everything below is additive under /v1/; the legacy endpoints are unchanged. The OpenAPI document (/openapi.json) carries the full description of every endpoint (summary, examples, x-errors).

# 1. Tiers and costs

Costs: every request costs 1 (screener/frames: 2 — request.state.request_cost = 2); every returned data row costs 1, Parquet costs ½ row (ceil(rows/2)), bulk endpoints (request.state.quota_exempt) and HTTP 304 cost 0 rows. Errors (4xx/5xx) cost the request but no rows. /v1/limits, /v1/me/* and /v1/admin/* are never charged. Keyless principals get 401 AUTH_REQUIRED on endpoints tagged stream/screener (and on any handler that sets request.state.requires_key = True).

GET /v1/limits (public, free) returns the table above plus the caller's live counters — use it for the pricing page and the playground banner. The constants live in ratelimit/tiers.py and are also exported as info.x-tiers in the OpenAPI document.

# 2. Headers on every /v1/* response

X-RateLimit-Limit-Requests: 120        requests allowed in the window
X-RateLimit-Remaining-Requests: 118    after this response
X-RateLimit-Limit-Rows: 1000000
X-RateLimit-Remaining-Rows: 997512
X-RateLimit-Reset: 1788012345          unix seconds when the oldest bucket of the sliding window expires
X-Row-Count: 2488                      rows in THIS response (set by frame_response / legacy endpoints)

Sliding window with 1-second buckets: Reset is the moment the oldest counted second leaves the window (i.e. when capacity starts coming back), not a fixed calendar boundary. All headers are exposed through CORS. When Redis is unreachable the API fails open: no X-RateLimit-* headers, request served, warning logged.

# 429 response

HTTP/1.1 429 Too Many Requests
Retry-After: 41
X-RateLimit-Remaining-Requests: 0
{"error": {"code": "RATE_LIMIT_EXCEEDED",
           "type": "requests_per_hour",          # requests_per_hour | rows_per_hour | requests_per_minute | rows_per_minute
           "message": "Rate limit exceeded: 30 requests per hour for keyless (per IP). Retry in 41 s. Create a free account at https://www.hfmarketdata.io/signup for 120 requests/min and 1,000,000 rows/min with an API key.",
           "docs": "https://www.hfmarketdata.io/docs/errors#rate_limit_exceeded",
           "details": {"limit": 30, "window_seconds": 3600, "reset": 1788012345, "tier": "keyless"}},
 "detail": "Rate limit exceeded: …"}

/v1/auth/* has its own throttle: 10 requests / hour / IP (all auth endpoints except logout) → same envelope, type: requests_per_hour, message "Too many authentication attempts…".

Other quota-related errors: 400 ROW_LIMIT_EXCEEDED when limit exceeds the tier's max rows per request (details.max_rows), 401 INVALID_API_KEY (unknown/revoked key — the request is NOT downgraded to keyless), 403 ACCOUNT_DISABLED, 401 AUTH_REQUIRED.

# The site's own requests (/charts) have no quota — principal site

The charts page must work without an account and without any limit. The middleware recognises the site's own data requests and takes them out of the quota system entirely (ratelimit/middleware.py, is_site_request). A request is the site's own when all three hold, on a GET of a charts data path (/v1/bars/*, /v1/futures/*, /v1/{asset}/tickers, /v1/options/*):

1. X-HFMD-Client: charts (or web) — set by the page's data layer (web/src/charts/data/*.js);
2. Sec-Fetch-Site: same-origin (or same-site) — set by the browser, never by a fetch() from another origin;
3. Origin (or Referer) whose host is the host of HFMD_PUBLIC_URL (with/without www.); localhost, 127.0.0.1, ::1 are also accepted when HFMD_ENV != production.

A Bearer key / ?api_key= always wins (the caller asked for its own principal). Anything short of the three criteria — a curl, a third-party site, a missing header — is a regular keyless / key / session request.

What changes for a site request:

• not counted, not row-charged, never 429 for quota reasons; no X-RateLimit-* header at all;
• request.state.principal = "site", principal_kind = "site", max_rows = 200 000 (tiers.SITE_MAX_ROWS; the page asks for 5 000–20 000 bars at once; the JSON ceiling of the legacy bars endpoint, 50 000, still applies through bound_limit);
• usage is recorded under the site principal for internal statistics (requests, rows, bytes) — never under the visitor's IP, whose keyless counters stay untouched;
• the only guard is a DoS protection, not a product limit: tiers.SITE_BURST_PER_MINUTE_PER_IP = 1 200 requests per minute per IP on this path (Redis sliding window, fail-open). Above it the API answers 429 RATE_LIMIT_EXCEEDED with a dedicated message ("an abuse guard, not a quota: the charts have no data limit"), Retry-After, details.guard = "site_burst" and no X-RateLimit-* header. A human on the page cannot reach it (a full 1-minute history load is ~10 requests).

site is not a tier: it is absent from /v1/limits (which, not being a data path, still answers as keyless / key / session). Tests: tests/test_ratelimit_site.py.

# 3. Authentication model

• API key: Authorization: Bearer hfmd_live_… or ?api_key=… — for data endpoints and programmatic access to /v1/me*. Format hfmd_live_ + 32 base62 chars; only sha256(salt+key) and the display prefix (hfmd_live_ab12cd34) are stored; shown once at creation. Lookups are cached 60 s per worker; revoke/rotate/tier changes invalidate the cache immediately on the worker that performed them.
• Session cookie hfmd_session (itsdangerous-signed, HttpOnly, SameSite=Lax, Secure in production, 30 days) — for the web dashboard. Set by verify, login, reset, accept-invite; cleared by logout.
• CSRF: every POST/PATCH/DELETE on /v1/auth, /v1/me, /v1/admin must send Content-Type: application/json (use {} as body when there is nothing to send) → otherwise 415 UNSUPPORTED_MEDIA_TYPE. Cookies are never sent cross-origin (CORS without credentials).

# 4. Flows (what the SPA must implement)

# Self-service signup

1. POST /v1/auth/signup {email, name, password} → 202 {"status":"verification_sent"} (or "invitation_sent" if the address had been invited by an admin). Password ≥ 10 chars (400 WEAK_PASSWORD), existing verified account → 409 EMAIL_TAKEN.
2. The e-mail links to /verify?token=… (SPA route). The page calls GET /v1/auth/verify?token=… → 200 profile + session cookie → redirect to /dashboard/keys and prompt to create the first key. 400 INVALID_TOKEN = expired (48 h) or already used → offer "sign up again" (re-sends).
3. POST /v1/auth/login {email, password} → 200 profile + cookie. Errors: 401 INVALID_CREDENTIALS, 403 EMAIL_NOT_VERIFIED, 403 ACCOUNT_DISABLED.
4. POST /v1/auth/logout {}.

# Password reset

POST /v1/auth/forgot {email} → always 202. Link → /reset-password?token=…; page posts POST /v1/auth/reset {token, password} → 200 + session. Reset tokens last 1 hour.

# Invitation (admin / CLI created users)

Link → /accept-invite?token=…; page posts POST /v1/auth/accept-invite {token, password} → 200 + session; the account becomes active, e-mail counts as verified, and an API key already exists (created at invitation, visible by prefix under /v1/me/keys; the user should rotate or create a new one to obtain a full key). Invitation tokens last 7 days.

# Dashboard

• GET /v1/me → {user:{…, keys_active}, limits:{tier, window_seconds, requests, rows, max_rows_per_request, upgrade}}
• GET /v1/me/keys · POST /v1/me/keys {name} (201, data.key = full key, once) · DELETE /v1/me/keys/{id} · POST /v1/me/keys/{id}/rotate {} (201, new data.key, rotated_from). Max 10 active keys (409 KEY_LIMIT_REACHED), 404 KEY_NOT_FOUND.
• GET /v1/me/usage?range=24h|7d|30d → {range, step_seconds, from, to, points:[{t, requests, rows}], totals, principals} (per minute / per hour / per day; UTC; includes the live minute).
• Playground with key injected: read X-RateLimit-* from every response for the quota widget.

# Admin (role: admin)

• GET /v1/admin/users?search=&tier=&status=&role=&limit=&cursor= · POST /v1/admin/users (invite when no password; response has key.prefix and invitation.{delivered, link?}) · GET/PATCH /v1/admin/users/{id} (tier, role, status, name) · POST …/{id}/invite {} · POST …/{id}/reset-password {} · POST …/{id}/keys {name, tier_override?} (full key once) · DELETE …/{id}/keys/{kid}.
• GET /v1/admin/usage?days=30&top=20 → per_day[] + top[] (principals mapped to users).
• GET /v1/admin/audit?limit=&cursor=&action= (newest first).

In development (HFMD_ENV != production and no Resend key) auth responses include debug_link so the flows can be tested without a mailbox. Never in production.

# 5. E-mail (Resend)

accounts/mailer.py posts to https://api.resend.com/emails with httpx. Templates (English, text + minimal HTML): verification, invitation, password reset, "new key created" notice. API keys are never e-mailed.

Configuration: HFMD_RESEND_API_KEY=re_… and HFMD_MAIL_FROM="HF Market Data <noreply@hfmarketdata.io>" (the domain must be verified in Resend — DKIM/SPF records — before the from address works). Without a key, mails are logged and the action link is stored in email_tokens.meta; admins see it in GET /v1/admin/users/{id} (pending_invite_link) / creation responses, and the CLI prints it.

# 6. Usage accounting

Per request the middleware increments usage:{principal}:{minute} in Redis (requests, rows, rows_parquet, bytes, status_2xx, status_429; TTL 8 days). A background task (started with the first request, one Redis lock across the 2 uvicorn workers) folds finished minutes every 60 s into SQLite usage_minute (7 rolling days) and usage_daily (kept forever). Series endpoints merge folded data with the live minute.

# 7. CLI hfmd (scripts/hfmd)

scripts/hfmd seed [--show-key] [--no-mail]                 # idempotent: 3 users + admin, prints prefixes + invitation links
scripts/hfmd users add "Name" email [--tier free|high_usage] [--admin] [--show-key] [--no-mail]
scripts/hfmd users list | set-tier <email> <tier> | disable <email> | enable <email> | invite-resend <email>
scripts/hfmd keys list <email>

The wrapper picks .venv/bin/python or hfmarketdata/venv/bin/python, puts hfmarketdata/api on PYTHONPATH and honours the same HFMD_* variables as the API (in particular HFMD_STATE_DB). Full keys are only printed at creation with --show-key; re-running seed/users add never re-creates users, keys or tokens (an existing pending invitation link is printed again).

# 8. Production configuration

Behind ngrok the client IP is the first hop of X-Forwarded-For; make sure nothing upstream lets clients spoof it (ngrok overwrites it).

# Inventaire des routes comptes (généré 2026-09-06 depuis /openapi.json)