docs: comptes, clés API et quotas — guide pour l'équipe web (flux, en-têtes, Resend, CLI, variables prod)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 changed file +162 −0
added
docs/accounts-ratelimit.md
+162 −0
@@ -0,0 +1,162 @@ | ||
| 1 | +# Accounts, API keys & rate limiting (chantier 2) — guide for the web team | |
| 2 | + | |
| 3 | +Backend modules: `hfmarketdata/api/accounts/` (users, keys, auth, dashboard API, admin, CLI) and | |
| 4 | +`hfmarketdata/api/ratelimit/` (tiers, Redis sliding windows, ASGI middleware, usage accounting). | |
| 5 | +Everything below is additive under `/v1/`; the legacy endpoints are unchanged. The OpenAPI document | |
| 6 | +(`/openapi.json`) carries the full description of every endpoint (summary, examples, `x-errors`). | |
| 7 | + | |
| 8 | +## 1. Tiers and costs | |
| 9 | + | |
| 10 | +| tier | who | window | requests | rows | max rows / request | | |
| 11 | +|---|---|---|---|---|---| | |
| 12 | +| `keyless` | per IP (first hop of `X-Forwarded-For`, hashed) | 1 h | 30 | 100 000 | 5 000 | | |
| 13 | +| `free` | per API key | 1 min | 120 | 1 000 000 | 50 000 | | |
| 14 | +| `high_usage` | per API key (on request to contact@spboucher.ai) | 1 min | 600 | 10 000 000 | 200 000 | | |
| 15 | + | |
| 16 | +Costs: every request costs 1 (screener/frames: 2 — `request.state.request_cost = 2`); every returned data | |
| 17 | +row costs 1, **Parquet costs ½ row** (`ceil(rows/2)`), **bulk endpoints (`request.state.quota_exempt`) and | |
| 18 | +HTTP 304 cost 0 rows**. Errors (4xx/5xx) cost the request but no rows. `/v1/limits`, `/v1/me/*` and | |
| 19 | +`/v1/admin/*` are never charged. Keyless principals get `401 AUTH_REQUIRED` on endpoints tagged | |
| 20 | +`stream`/`screener` (and on any handler that sets `request.state.requires_key = True`). | |
| 21 | + | |
| 22 | +`GET /v1/limits` (public, free) returns the table above plus the caller's live counters — use it for the | |
| 23 | +pricing page and the playground banner. The constants live in `ratelimit/tiers.py` and are also exported | |
| 24 | +as `info.x-tiers` in the OpenAPI document. | |
| 25 | + | |
| 26 | +## 2. Headers on every `/v1/*` response | |
| 27 | + | |
| 28 | +``` | |
| 29 | +X-RateLimit-Limit-Requests: 120 requests allowed in the window | |
| 30 | +X-RateLimit-Remaining-Requests: 118 after this response | |
| 31 | +X-RateLimit-Limit-Rows: 1000000 | |
| 32 | +X-RateLimit-Remaining-Rows: 997512 | |
| 33 | +X-RateLimit-Reset: 1788012345 unix seconds when the oldest bucket of the sliding window expires | |
| 34 | +X-Row-Count: 2488 rows in THIS response (set by frame_response / legacy endpoints) | |
| 35 | +``` | |
| 36 | + | |
| 37 | +Sliding window with 1-second buckets: `Reset` is the moment the oldest counted second leaves the window | |
| 38 | +(i.e. when capacity starts coming back), not a fixed calendar boundary. All headers are exposed through CORS. | |
| 39 | +When Redis is unreachable the API **fails open**: no `X-RateLimit-*` headers, request served, warning logged. | |
| 40 | + | |
| 41 | +### 429 response | |
| 42 | + | |
| 43 | +```http | |
| 44 | +HTTP/1.1 429 Too Many Requests | |
| 45 | +Retry-After: 41 | |
| 46 | +X-RateLimit-Remaining-Requests: 0 | |
| 47 | +{"error": {"code": "RATE_LIMIT_EXCEEDED", | |
| 48 | + "type": "requests_per_hour", # requests_per_hour | rows_per_hour | requests_per_minute | rows_per_minute | |
| 49 | + "message": "Rate limit exceeded: 30 requests per hour for keyless (per IP). Retry in 41 s. Create a free account at https://www.hfmarketdata.io/signup for 120 requests/min and 1,000,000 rows/min with an API key.", | |
| 50 | + "docs": "https://www.hfmarketdata.io/docs/errors#rate_limit_exceeded", | |
| 51 | + "details": {"limit": 30, "window_seconds": 3600, "reset": 1788012345, "tier": "keyless"}}, | |
| 52 | + "detail": "Rate limit exceeded: …"} | |
| 53 | +``` | |
| 54 | + | |
| 55 | +`/v1/auth/*` has its own throttle: **10 requests / hour / IP** (all auth endpoints except logout) → same | |
| 56 | +envelope, `type: requests_per_hour`, message "Too many authentication attempts…". | |
| 57 | + | |
| 58 | +Other quota-related errors: `400 ROW_LIMIT_EXCEEDED` when `limit` exceeds the tier's max rows per request | |
| 59 | +(`details.max_rows`), `401 INVALID_API_KEY` (unknown/revoked key — the request is NOT downgraded to keyless), | |
| 60 | +`403 ACCOUNT_DISABLED`, `401 AUTH_REQUIRED`. | |
| 61 | + | |
| 62 | +## 3. Authentication model | |
| 63 | + | |
| 64 | +* **API key**: `Authorization: Bearer hfmd_live_…` or `?api_key=…` — for data endpoints and programmatic | |
| 65 | + access to `/v1/me*`. Format `hfmd_live_` + 32 base62 chars; only `sha256(salt+key)` and the display prefix | |
| 66 | + (`hfmd_live_ab12cd34`) are stored; shown **once** at creation. Lookups are cached 60 s per worker; | |
| 67 | + revoke/rotate/tier changes invalidate the cache immediately on the worker that performed them. | |
| 68 | +* **Session cookie** `hfmd_session` (itsdangerous-signed, HttpOnly, SameSite=Lax, Secure in production, | |
| 69 | + 30 days) — for the web dashboard. Set by `verify`, `login`, `reset`, `accept-invite`; cleared by `logout`. | |
| 70 | +* **CSRF**: every POST/PATCH/DELETE on `/v1/auth`, `/v1/me`, `/v1/admin` must send | |
| 71 | + `Content-Type: application/json` (use `{}` as body when there is nothing to send) → otherwise | |
| 72 | + `415 UNSUPPORTED_MEDIA_TYPE`. Cookies are never sent cross-origin (CORS without credentials). | |
| 73 | + | |
| 74 | +## 4. Flows (what the SPA must implement) | |
| 75 | + | |
| 76 | +### Self-service signup | |
| 77 | +1. `POST /v1/auth/signup {email, name, password}` → `202 {"status":"verification_sent"}` (or | |
| 78 | + `"invitation_sent"` if the address had been invited by an admin). Password ≥ 10 chars (`400 WEAK_PASSWORD`), | |
| 79 | + existing verified account → `409 EMAIL_TAKEN`. | |
| 80 | +2. The e-mail links to **`/verify?token=…`** (SPA route). The page calls `GET /v1/auth/verify?token=…` → | |
| 81 | + `200` profile + session cookie → redirect to `/dashboard/keys` and prompt to create the first key. | |
| 82 | + `400 INVALID_TOKEN` = expired (48 h) or already used → offer "sign up again" (re-sends). | |
| 83 | +3. `POST /v1/auth/login {email, password}` → `200` profile + cookie. Errors: `401 INVALID_CREDENTIALS`, | |
| 84 | + `403 EMAIL_NOT_VERIFIED`, `403 ACCOUNT_DISABLED`. | |
| 85 | +4. `POST /v1/auth/logout {}`. | |
| 86 | + | |
| 87 | +### Password reset | |
| 88 | +`POST /v1/auth/forgot {email}` → always `202`. Link → **`/reset-password?token=…`**; page posts | |
| 89 | +`POST /v1/auth/reset {token, password}` → `200` + session. Reset tokens last 1 hour. | |
| 90 | + | |
| 91 | +### Invitation (admin / CLI created users) | |
| 92 | +Link → **`/accept-invite?token=…`**; page posts `POST /v1/auth/accept-invite {token, password}` → | |
| 93 | +`200` + session; the account becomes `active`, e-mail counts as verified, and an API key already exists | |
| 94 | +(created at invitation, visible by prefix under `/v1/me/keys`; the user should rotate or create a new one to | |
| 95 | +obtain a full key). Invitation tokens last 7 days. | |
| 96 | + | |
| 97 | +### Dashboard | |
| 98 | +* `GET /v1/me` → `{user:{…, keys_active}, limits:{tier, window_seconds, requests, rows, max_rows_per_request, upgrade}}` | |
| 99 | +* `GET /v1/me/keys` · `POST /v1/me/keys {name}` (201, `data.key` = full key, once) · | |
| 100 | + `DELETE /v1/me/keys/{id}` · `POST /v1/me/keys/{id}/rotate {}` (201, new `data.key`, `rotated_from`). | |
| 101 | + Max 10 active keys (`409 KEY_LIMIT_REACHED`), `404 KEY_NOT_FOUND`. | |
| 102 | +* `GET /v1/me/usage?range=24h|7d|30d` → `{range, step_seconds, from, to, points:[{t, requests, rows}], totals, principals}` | |
| 103 | + (per minute / per hour / per day; UTC; includes the live minute). | |
| 104 | +* Playground with key injected: read `X-RateLimit-*` from every response for the quota widget. | |
| 105 | + | |
| 106 | +### Admin (`role: admin`) | |
| 107 | +* `GET /v1/admin/users?search=&tier=&status=&role=&limit=&cursor=` · `POST /v1/admin/users` (invite when | |
| 108 | + no `password`; response has `key.prefix` and `invitation.{delivered, link?}`) · `GET/PATCH /v1/admin/users/{id}` | |
| 109 | + (`tier`, `role`, `status`, `name`) · `POST …/{id}/invite {}` · `POST …/{id}/reset-password {}` · | |
| 110 | + `POST …/{id}/keys {name, tier_override?}` (full key once) · `DELETE …/{id}/keys/{kid}`. | |
| 111 | +* `GET /v1/admin/usage?days=30&top=20` → `per_day[]` + `top[]` (principals mapped to users). | |
| 112 | +* `GET /v1/admin/audit?limit=&cursor=&action=` (newest first). | |
| 113 | + | |
| 114 | +In development (`HFMD_ENV != production` **and** no Resend key) auth responses include `debug_link` so the | |
| 115 | +flows can be tested without a mailbox. Never in production. | |
| 116 | + | |
| 117 | +## 5. E-mail (Resend) | |
| 118 | + | |
| 119 | +`accounts/mailer.py` posts to `https://api.resend.com/emails` with httpx. Templates (English, text + minimal | |
| 120 | +HTML): verification, invitation, password reset, "new key created" notice. **API keys are never e-mailed.** | |
| 121 | + | |
| 122 | +Configuration: `HFMD_RESEND_API_KEY=re_…` and `HFMD_MAIL_FROM="HF Market Data <noreply@hfmarketdata.io>"` | |
| 123 | +(the domain must be verified in Resend — DKIM/SPF records — before the `from` address works). | |
| 124 | +Without a key, mails are logged and the action link is stored in `email_tokens.meta`; admins see it in | |
| 125 | +`GET /v1/admin/users/{id}` (`pending_invite_link`) / creation responses, and the CLI prints it. | |
| 126 | + | |
| 127 | +## 6. Usage accounting | |
| 128 | + | |
| 129 | +Per request the middleware increments `usage:{principal}:{minute}` in Redis (requests, rows, rows_parquet, | |
| 130 | +bytes, status_2xx, status_429; TTL 8 days). A background task (started with the first request, one Redis lock | |
| 131 | +across the 2 uvicorn workers) folds finished minutes every 60 s into SQLite `usage_minute` (7 rolling days) | |
| 132 | +and `usage_daily` (kept forever). Series endpoints merge folded data with the live minute. | |
| 133 | + | |
| 134 | +## 7. CLI `hfmd` (scripts/hfmd) | |
| 135 | + | |
| 136 | +```bash | |
| 137 | +scripts/hfmd seed [--show-key] [--no-mail] # idempotent: 3 users + admin, prints prefixes + invitation links | |
| 138 | +scripts/hfmd users add "Name" email [--tier free|high_usage] [--admin] [--show-key] [--no-mail] | |
| 139 | +scripts/hfmd users list | set-tier <email> <tier> | disable <email> | enable <email> | invite-resend <email> | |
| 140 | +scripts/hfmd keys list <email> | |
| 141 | +``` | |
| 142 | + | |
| 143 | +The wrapper picks `.venv/bin/python` or `hfmarketdata/venv/bin/python`, puts `hfmarketdata/api` on | |
| 144 | +`PYTHONPATH` and honours the same `HFMD_*` variables as the API (in particular `HFMD_STATE_DB`). Full keys | |
| 145 | +are only printed at creation with `--show-key`; re-running `seed`/`users add` never re-creates users, keys | |
| 146 | +or tokens (an existing pending invitation link is printed again). | |
| 147 | + | |
| 148 | +## 8. Production configuration | |
| 149 | + | |
| 150 | +| variable | purpose | | |
| 151 | +|---|---| | |
| 152 | +| `HFMD_SECRET_KEY` | signs session cookies — long random string, rotate = everyone signed out | | |
| 153 | +| `HFMD_KEY_SALT` | salt of `sha256(salt+key)` for API keys/tokens and of the IP hash — **never change after keys exist** | | |
| 154 | +| `HFMD_REDIS_URL` | `redis://127.0.0.1:6379/0` (quotas + usage); `fakeredis://` in tests | | |
| 155 | +| `HFMD_RESEND_API_KEY` | Resend API key (absent → links printed/stored, nothing sent) | | |
| 156 | +| `HFMD_MAIL_FROM` | verified sender, e.g. `HF Market Data <noreply@hfmarketdata.io>` | | |
| 157 | +| `HFMD_STATE_DB` | SQLite path (default `<data_root>/state/hfmd.db`) | | |
| 158 | +| `HFMD_RATELIMIT` | `0` disables the middleware entirely (emergency only) | | |
| 159 | +| `HFMD_PUBLIC_URL` | base of the action links in e-mails (`/verify`, `/accept-invite`, `/reset-password`) | | |
| 160 | + | |
| 161 | +Behind ngrok the client IP is the first hop of `X-Forwarded-For`; make sure nothing upstream lets clients | |
| 162 | +spoof it (ngrok overwrites it). | |
| 163 | ||