# Accounts, API keys & rate limiting (chantier 2) — guide for the web team

Backend modules: `hfmarketdata/api/accounts/` (users, keys, auth, dashboard API, admin, CLI) and
`hfmarketdata/api/ratelimit/` (tiers, Redis sliding windows, ASGI middleware, usage accounting).
Everything below is additive under `/v1/`; the legacy endpoints are unchanged. The OpenAPI document
(`/openapi.json`) carries the full description of every endpoint (summary, examples, `x-errors`).

## 1. Tiers and costs

| tier | who | window | requests | rows | max rows / request |
|---|---|---|---|---|---|
| `keyless` | per IP (first hop of `X-Forwarded-For`, hashed) | 1 h | 30 | 100 000 | 5 000 |
| `free` | per API key | 1 min | 120 | 1 000 000 | 50 000 |
| `high_usage` | per API key (on request to contact@spboucher.ai) | 1 min | 600 | 10 000 000 | 200 000 |

Costs: every request costs 1 (screener/frames: 2 — `request.state.request_cost = 2`); every returned data
row costs 1, **Parquet costs ½ row** (`ceil(rows/2)`), **bulk endpoints (`request.state.quota_exempt`) and
HTTP 304 cost 0 rows**. Errors (4xx/5xx) cost the request but no rows. `/v1/limits`, `/v1/me/*` and
`/v1/admin/*` are never charged. Keyless principals get `401 AUTH_REQUIRED` on endpoints tagged
`stream`/`screener` (and on any handler that sets `request.state.requires_key = True`).

`GET /v1/limits` (public, free) returns the table above plus the caller's live counters — use it for the
pricing page and the playground banner. The constants live in `ratelimit/tiers.py` and are also exported
as `info.x-tiers` in the OpenAPI document.

## 2. Headers on every `/v1/*` response

```
X-RateLimit-Limit-Requests: 120        requests allowed in the window
X-RateLimit-Remaining-Requests: 118    after this response
X-RateLimit-Limit-Rows: 1000000
X-RateLimit-Remaining-Rows: 997512
X-RateLimit-Reset: 1788012345          unix seconds when the oldest bucket of the sliding window expires
X-Row-Count: 2488                      rows in THIS response (set by frame_response / legacy endpoints)
```

Sliding window with 1-second buckets: `Reset` is the moment the oldest counted second leaves the window
(i.e. when capacity starts coming back), not a fixed calendar boundary. All headers are exposed through CORS.
When Redis is unreachable the API **fails open**: no `X-RateLimit-*` headers, request served, warning logged.

### 429 response

```http
HTTP/1.1 429 Too Many Requests
Retry-After: 41
X-RateLimit-Remaining-Requests: 0
{"error": {"code": "RATE_LIMIT_EXCEEDED",
           "type": "requests_per_hour",          # requests_per_hour | rows_per_hour | requests_per_minute | rows_per_minute
           "message": "Rate limit exceeded: 30 requests per hour for keyless (per IP). Retry in 41 s. Create a free account at https://www.hfmarketdata.io/signup for 120 requests/min and 1,000,000 rows/min with an API key.",
           "docs": "https://www.hfmarketdata.io/docs/errors#rate_limit_exceeded",
           "details": {"limit": 30, "window_seconds": 3600, "reset": 1788012345, "tier": "keyless"}},
 "detail": "Rate limit exceeded: …"}
```

`/v1/auth/*` has its own throttle: **10 requests / hour / IP** (all auth endpoints except logout) → same
envelope, `type: requests_per_hour`, message "Too many authentication attempts…".

Other quota-related errors: `400 ROW_LIMIT_EXCEEDED` when `limit` exceeds the tier's max rows per request
(`details.max_rows`), `401 INVALID_API_KEY` (unknown/revoked key — the request is NOT downgraded to keyless),
`403 ACCOUNT_DISABLED`, `401 AUTH_REQUIRED`.

### The site's own requests (`/charts`) have no quota — principal `site`

The charts page must work without an account and **without any limit**. The middleware recognises the
site's own data requests and takes them out of the quota system entirely (`ratelimit/middleware.py`,
`is_site_request`). A request is the site's own when **all three** hold, on a **GET** of a charts data path
(`/v1/bars/*`, `/v1/futures/*`, `/v1/{asset}/tickers`, `/v1/options/*`):

1. `X-HFMD-Client: charts` (or `web`) — set by the page's data layer (`web/src/charts/data/*.js`);
2. `Sec-Fetch-Site: same-origin` (or `same-site`) — set by the browser, never by a `fetch()` from another origin;
3. `Origin` (or `Referer`) whose host is the host of `HFMD_PUBLIC_URL` (with/without `www.`); `localhost`,
   `127.0.0.1`, `::1` are also accepted when `HFMD_ENV != production`.

A Bearer key / `?api_key=` always wins (the caller asked for its own principal). Anything short of the three
criteria — a `curl`, a third-party site, a missing header — is a regular keyless / key / session request.

What changes for a `site` request:

* **not counted, not row-charged, never 429 for quota reasons**; no `X-RateLimit-*` header at all;
* `request.state.principal = "site"`, `principal_kind = "site"`, `max_rows = 200 000`
  (`tiers.SITE_MAX_ROWS`; the page asks for 5 000–20 000 bars at once; the JSON ceiling of the legacy bars
  endpoint, 50 000, still applies through `bound_limit`);
* usage is recorded under the `site` principal for internal statistics (requests, rows, bytes) — **never under
  the visitor's IP**, whose keyless counters stay untouched;
* the only guard is a **DoS protection**, not a product limit: `tiers.SITE_BURST_PER_MINUTE_PER_IP` = 1 200
  requests per minute per IP on this path (Redis sliding window, fail-open). Above it the API answers
  `429 RATE_LIMIT_EXCEEDED` with a dedicated message ("an abuse guard, not a quota: the charts have no data
  limit"), `Retry-After`, `details.guard = "site_burst"` and no `X-RateLimit-*` header. A human on the page
  cannot reach it (a full 1-minute history load is ~10 requests).

`site` is not a tier: it is absent from `/v1/limits` (which, not being a data path, still answers as keyless /
key / session). Tests: `tests/test_ratelimit_site.py`.

## 3. Authentication model

* **API key**: `Authorization: Bearer hfmd_live_…` or `?api_key=…` — for data endpoints and programmatic
  access to `/v1/me*`. Format `hfmd_live_` + 32 base62 chars; only `sha256(salt+key)` and the display prefix
  (`hfmd_live_ab12cd34`) are stored; shown **once** at creation. Lookups are cached 60 s per worker;
  revoke/rotate/tier changes invalidate the cache immediately on the worker that performed them.
* **Session cookie** `hfmd_session` (itsdangerous-signed, HttpOnly, SameSite=Lax, Secure in production,
  30 days) — for the web dashboard. Set by `verify`, `login`, `reset`, `accept-invite`; cleared by `logout`.
* **CSRF**: every POST/PATCH/DELETE on `/v1/auth`, `/v1/me`, `/v1/admin` must send
  `Content-Type: application/json` (use `{}` as body when there is nothing to send) → otherwise
  `415 UNSUPPORTED_MEDIA_TYPE`. Cookies are never sent cross-origin (CORS without credentials).

## 4. Flows (what the SPA must implement)

### Self-service signup
1. `POST /v1/auth/signup {email, name, password}` → `202 {"status":"verification_sent"}` (or
   `"invitation_sent"` if the address had been invited by an admin). Password ≥ 10 chars (`400 WEAK_PASSWORD`),
   existing verified account → `409 EMAIL_TAKEN`.
2. The e-mail links to **`/verify?token=…`** (SPA route). The page calls `GET /v1/auth/verify?token=…` →
   `200` profile + session cookie → redirect to `/dashboard/keys` and prompt to create the first key.
   `400 INVALID_TOKEN` = expired (48 h) or already used → offer "sign up again" (re-sends).
3. `POST /v1/auth/login {email, password}` → `200` profile + cookie. Errors: `401 INVALID_CREDENTIALS`,
   `403 EMAIL_NOT_VERIFIED`, `403 ACCOUNT_DISABLED`.
4. `POST /v1/auth/logout {}`.

### Password reset
`POST /v1/auth/forgot {email}` → always `202`. Link → **`/reset-password?token=…`**; page posts
`POST /v1/auth/reset {token, password}` → `200` + session. Reset tokens last 1 hour.

### Invitation (admin / CLI created users)
Link → **`/accept-invite?token=…`**; page posts `POST /v1/auth/accept-invite {token, password}` →
`200` + session; the account becomes `active`, e-mail counts as verified, and an API key already exists
(created at invitation, visible by prefix under `/v1/me/keys`; the user should rotate or create a new one to
obtain a full key). Invitation tokens last 7 days.

### Dashboard
* `GET /v1/me` → `{user:{…, keys_active}, limits:{tier, window_seconds, requests, rows, max_rows_per_request, upgrade}}`
* `GET /v1/me/keys` · `POST /v1/me/keys {name}` (201, `data.key` = full key, once) ·
  `DELETE /v1/me/keys/{id}` · `POST /v1/me/keys/{id}/rotate {}` (201, new `data.key`, `rotated_from`).
  Max 10 active keys (`409 KEY_LIMIT_REACHED`), `404 KEY_NOT_FOUND`.
* `GET /v1/me/usage?range=24h|7d|30d` → `{range, step_seconds, from, to, points:[{t, requests, rows}], totals, principals}`
  (per minute / per hour / per day; UTC; includes the live minute).
* Playground with key injected: read `X-RateLimit-*` from every response for the quota widget.

### Admin (`role: admin`)
* `GET /v1/admin/users?search=&tier=&status=&role=&limit=&cursor=` · `POST /v1/admin/users` (invite when
  no `password`; response has `key.prefix` and `invitation.{delivered, link?}`) · `GET/PATCH /v1/admin/users/{id}`
  (`tier`, `role`, `status`, `name`) · `POST …/{id}/invite {}` · `POST …/{id}/reset-password {}` ·
  `POST …/{id}/keys {name, tier_override?}` (full key once) · `DELETE …/{id}/keys/{kid}`.
* `GET /v1/admin/usage?days=30&top=20` → `per_day[]` + `top[]` (principals mapped to users).
* `GET /v1/admin/audit?limit=&cursor=&action=` (newest first).

In development (`HFMD_ENV != production` **and** no Resend key) auth responses include `debug_link` so the
flows can be tested without a mailbox. Never in production.

## 5. E-mail (Resend)

`accounts/mailer.py` posts to `https://api.resend.com/emails` with httpx. Templates (English, text + minimal
HTML): verification, invitation, password reset, "new key created" notice. **API keys are never e-mailed.**

Configuration: `HFMD_RESEND_API_KEY=re_…` and `HFMD_MAIL_FROM="HF Market Data <noreply@hfmarketdata.io>"`
(the domain must be verified in Resend — DKIM/SPF records — before the `from` address works).
Without a key, mails are logged and the action link is stored in `email_tokens.meta`; admins see it in
`GET /v1/admin/users/{id}` (`pending_invite_link`) / creation responses, and the CLI prints it.

## 6. Usage accounting

Per request the middleware increments `usage:{principal}:{minute}` in Redis (requests, rows, rows_parquet,
bytes, status_2xx, status_429; TTL 8 days). A background task (started with the first request, one Redis lock
across the 2 uvicorn workers) folds finished minutes every 60 s into SQLite `usage_minute` (7 rolling days)
and `usage_daily` (kept forever). Series endpoints merge folded data with the live minute.

## 7. CLI `hfmd` (scripts/hfmd)

```bash
scripts/hfmd seed [--show-key] [--no-mail]                 # idempotent: 3 users + admin, prints prefixes + invitation links
scripts/hfmd users add "Name" email [--tier free|high_usage] [--admin] [--show-key] [--no-mail]
scripts/hfmd users list | set-tier <email> <tier> | disable <email> | enable <email> | invite-resend <email>
scripts/hfmd keys list <email>
```

The wrapper picks `.venv/bin/python` or `hfmarketdata/venv/bin/python`, puts `hfmarketdata/api` on
`PYTHONPATH` and honours the same `HFMD_*` variables as the API (in particular `HFMD_STATE_DB`). Full keys
are only printed at creation with `--show-key`; re-running `seed`/`users add` never re-creates users, keys
or tokens (an existing pending invitation link is printed again).

## 8. Production configuration

| variable | purpose |
|---|---|
| `HFMD_SECRET_KEY` | signs session cookies — long random string, rotate = everyone signed out |
| `HFMD_KEY_SALT` | salt of `sha256(salt+key)` for API keys/tokens and of the IP hash — **never change after keys exist** |
| `HFMD_REDIS_URL` | `redis://127.0.0.1:6379/0` (quotas + usage); `fakeredis://` in tests |
| `HFMD_RESEND_API_KEY` | Resend API key (absent → links printed/stored, nothing sent) |
| `HFMD_MAIL_FROM` | verified sender, e.g. `HF Market Data <noreply@hfmarketdata.io>` |
| `HFMD_STATE_DB` | SQLite path (default `<data_root>/state/hfmd.db`) |
| `HFMD_RATELIMIT` | `0` disables the middleware entirely (emergency only) |
| `HFMD_PUBLIC_URL` | base of the action links in e-mails (`/verify`, `/accept-invite`, `/reset-password`) |

Behind ngrok the client IP is the first hop of `X-Forwarded-For`; make sure nothing upstream lets clients
spoof it (ngrok overwrites it).

## Inventaire des routes comptes (généré 2026-09-06 depuis /openapi.json)

| Route | Méthode | Résumé |
|---|---|---|
| `/v1/admin/audit` | GET | Audit log |
| `/v1/admin/usage` | GET | Global usage: totals per day + top principals |
| `/v1/admin/users` | GET | List users |
| `/v1/admin/users` | POST | Create or invite a user |
| `/v1/admin/users/{user_id}` | DELETE | Delete (anonymise) a user |
| `/v1/admin/users/{user_id}` | GET | User detail: keys, pending links, 7-day usage |
| `/v1/admin/users/{user_id}` | PATCH | Update tier / role / status / name |
| `/v1/admin/users/{user_id}/invite` | POST | (Re-)send the invitation |
| `/v1/admin/users/{user_id}/keys` | POST | Create an API key for a user (shown once) |
| `/v1/admin/users/{user_id}/keys/{key_id}` | DELETE | Revoke a user's API key |
| `/v1/admin/users/{user_id}/reset-password` | POST | Send a password-reset link to a user |
| `/v1/auth/accept-invite` | POST | Accept an invitation (set your password) |
| `/v1/auth/forgot` | POST | Request a password-reset link |
| `/v1/auth/login` | POST | Sign in with e-mail + password |
| `/v1/auth/logout` | POST | Sign out (this browser) |
| `/v1/auth/reset` | POST | Set a new password from a reset link |
| `/v1/auth/signup` | POST | Create an account (sends a verification e-mail) |
| `/v1/auth/verify` | GET | Verification link landing (redirects to the web page, does not consume the token) |
| `/v1/auth/verify` | POST | Verify an e-mail address (or confirm an e-mail change) |
| `/v1/limits` | GET | Tier table + your current counters (works without a key) |
| `/v1/me` | DELETE | Delete your account |
| `/v1/me` | GET | Your profile, tier and limits |
| `/v1/me` | PATCH | Update your profile (name, quota alerts) |
| `/v1/me/email` | POST | Change your e-mail address (confirmation link sent to the new address) |
| `/v1/me/keys` | GET | List your API keys |
| `/v1/me/keys` | POST | Create an API key (shown once) |
| `/v1/me/keys/{key_id}` | DELETE | Revoke an API key |
| `/v1/me/keys/{key_id}` | PATCH | Rename a key / edit its note |
| `/v1/me/keys/{key_id}/rotate` | POST | Rotate an API key (revoke + create) |
| `/v1/me/limits` | GET | Live quota of your keys (remaining requests / rows in the current window) |
| `/v1/me/password` | POST | Change your password |
| `/v1/me/sessions/revoke-all` | POST | Sign out everywhere |
| `/v1/me/usage` | GET | Your usage series (requests, rows, 429s) |
| `/v1/me/usage.csv` | GET | Your usage series as CSV |