# Fetcha v0.2 — contract for the access model, browser mode and crawling

This document is the single source of truth for the v0.2 changes while the backend (packages/core,
packages/providers, packages/browser, packages/routing, apps/api) and the web app are built in
parallel. Field names below are final.

## 1. Access model (private platform)

- There is **one plan**: `unlimited` (`PLANS = ["unlimited"]`, `PLAN_LIMITS.unlimited`). Legacy values in
  `organizations.plan` are migrated to `unlimited`; always go through `normalizePlan(org.plan)` from
  `@fetcha/core` instead of casting.
- Limits: unlimited monthly requests, 200 concurrent requests, 120 s max timeout, 5 retries, all
  network classes, 90-day log retention, browser rendering (8 concurrent renders), crawl jobs
  (2,000 pages/job, 5 concurrent jobs). No prices, no invoices, no checkout: billing UI must say
  "private platform, no billing" (do not show plan grids, upgrade dialogs or "Start free").
- **Signup is invitation-only**: table `signup_allowlist(email pk lower-cased, note, invited_by_user_id,
  invited_at, used_at, user_id, created_at)`. Better Auth `databaseHooks.user.create.before` must
  refuse (`APIError("FORBIDDEN", { message })`) any email that is neither in the allowlist nor in
  `ADMIN_EMAILS`. The `after` hook sets `used_at`/`user_id`.
- Emails listed in `ADMIN_EMAILS` (env, comma-separated; prod = `spbou4@icloud.com`) get
  `users.role = 'admin'` at creation, and `pnpm db:seed` (packages/db/src/seed.ts) promotes existing
  users with those emails + inserts them in the allowlist (idempotent).
- Admin UI `/admin/access`: list allowlist (email, note, invited by, invited at, account created?),
  add one or many emails (textarea, comma/newline separated) with optional note and "send
  invitation email" checkbox, resend invitation, remove entry (only if no account yet). Server
  actions in `apps/web/src/actions/access.ts`, audited via `admin.action` metadata types
  `access.allow`, `access.invite`, `access.revoke`.
- Invitation email: `EmailService.sendInvite(to, { inviterName, signupUrl })` in packages/email
  (template `InviteEmail`): "You have been invited to Fetcha", button → `${siteUrl}/signup?email=…`.
- Signup form: pre-fill email from `?email=`, copy "Fetcha is invitation-only. Use the address your
  administrator approved." On a 403 from sign-up show: "This email is not on the access list. Ask
  your Fetcha administrator to invite you." Marketing CTAs "Start free" → "Log in" (primary) and
  "Request access" → `mailto:hello@fetcha.co?subject=Fetcha%20access`.
- `/pricing` becomes an "Access" page (same route, nav label "Access"): explains private access,
  what is included (unlimited, browser, crawl, sessions, geo), how to get invited.

## 2. `POST /v1/fetch` additions (all optional, strict schema)

| Field | Type | Default | Meaning |
|---|---|---|---|
| `format` | `"html" \| "text" \| "markdown" \| "json" \| "raw"` | `"html"` | `markdown` returns the page converted to Markdown in `markdown` (main content first, boilerplate removed). |
| `browser` | boolean | false | Render in the managed headless Chromium routed through the same proxy network/geo/session. Live. |
| `browser_fallback` | boolean | true | If an HTTP attempt is blocked by a JS challenge/anti-bot, automatically retry in the browser. |
| `wait_for` | string (CSS) | — | Browser: selector to wait for before capturing. |
| `wait_ms` | 0–30000 | — | Browser: extra settle time. |
| `wait_until` | `"load" \| "domcontentloaded" \| "networkidle"` | `"domcontentloaded"` | Browser navigation wait condition. |
| `javascript` | boolean | true | Browser: disable scripting when false. |
| `block_resources` | boolean | true | Browser: skip images/fonts/media. |
| `screenshot` | boolean | false | Browser: PNG base64 in `screenshot`. |
| `links` | boolean | false | Return `links[]` (all hyperlinks, absolute). |
| `referer` | `"auto" \| "none" \| url` | `"auto"` | Referer strategy (auto = none first, search-engine referer on retries). |

Response additions:

```jsonc
{
  "markdown": "# Title…",          // only for format=markdown
  "page": { "title": "…", "description": "…", "canonical": "…", "lang": "en", "og": { "og:title": "…" }, "links_count": 42 },
  "links": [{ "url": "https://…", "text": "About", "internal": true, "nofollow": false }],   // only with links:true
  "screenshot": "iVBORw0…",        // only with browser + screenshot
  "metadata": { "mode": "http" | "browser", …, "debug": { "attempts": [{ "provider", "network", "mode", "country", "outcome", "block_reason", "status", "duration_ms" }] } }
}
```

New error semantics: `BROWSER_UNAVAILABLE` is now only returned when the browser pool is disabled or
down (503-ish situations), `BROWSER_TIMEOUT` when the page does not settle in time. Scope
`browser:use` is **not** required (browser is part of fetch); keep the scope name for compatibility.

## 3. Crawl API

`POST /v1/crawl` (scope `fetch:execute`) — body = `crawlCreateSchema` (`@fetcha/core`):
`url`, `max_pages` (1–5000, default 25), `max_depth` (0–10, default 2), `same_domain` (true),
`allow_subdomains` (false), `include_patterns[]`, `exclude_patterns[]` (glob with `*` or `/regex/`),
`respect_robots` (true), `use_sitemap` (false), `concurrency` (1–10, default 3), `delay_ms`, `timeout`,
`format` (`markdown|text|html`, default markdown), `main_content` (true), `country`, `network`,
`browser`, `browser_fallback`, `headers`, `webhook_url`, `label`.

Returns `202`:

```json
{ "id": "crawl_…", "status": "queued", "seed_url": "…", "created_at": "…", "options": { … } }
```

`GET /v1/crawl/:id` → job:

```json
{ "id", "status": "queued|running|completed|failed|cancelled", "label", "seed_url", "domain", "options",
  "stats": { "discovered", "fetched", "ok", "blocked", "failed", "bytes" },
  "error": { "code", "message" } | null, "created_at", "started_at", "completed_at" }
```

`GET /v1/crawl/:id/pages?cursor=&limit=100&status=success` → `{ "data": [page…], "next_cursor": "…" | null }` where page =
`{ "id", "url", "final_url", "depth", "status", "http_status", "error_code", "title", "description", "content_type", "content", "links_count", "bytes", "duration_ms", "mode", "fetched_at" }`.

`DELETE /v1/crawl/:id` → `{ "id", "status": "cancelled" }`. `GET /v1/crawl?limit=50` → `{ "data": [job…] }`.

Every crawled page is also a normal fetch request (visible in the Requests log with `source:
"crawl"`), so quotas, retries, escalation and routing intelligence apply.

`POST /v1/map` (sync, ≤ 60 s) — body = `mapCreateSchema`: `url`, `limit` (default 1000), `use_sitemap`
(true), `use_links` (true), `same_domain`, `allow_subdomains`, `search`, `country`, `network`, `timeout`.
Returns `{ "url", "count", "urls": ["…"], "sources": { "sitemap": n, "links": n }, "truncated": bool }`.

## 4. Internal routes (dashboard → API, service token)

- `POST /internal/crawls` `{ project_id, user_id, options }` → job (202 body as above)
- `GET /internal/crawls?project_id&user_id&limit=50` → `{ data: [job…] }`
- `GET /internal/crawls/:id?project_id&user_id` → job
- `GET /internal/crawls/:id/pages?project_id&user_id&cursor&limit&status` → pages page
- `DELETE /internal/crawls/:id?project_id&user_id` → cancel
- `POST /internal/map` `{ project_id, user_id, options }` → map result
- `GET /internal/browser` → `{ enabled, running, capacity, queue }` (admin System page)

Dashboard: new page `/dashboard/crawls` (list jobs of the current project, status badges, stats,
create dialog) and `/dashboard/crawls/[id]` (job detail, pages table with content preview, cancel).
Sidebar item "Crawls" after "Requests". Add crawl/map client methods to `apps/web/src/lib/api.ts`.

## 5. SDKs

JS (`packages/sdk`) and Python (`sdk-python`): add `format: "markdown"`, `browser*`, `links`,
`referer` to fetch options and `markdown`/`page`/`links`/`screenshot` to results; add
`crawl.create(options)`, `crawl.get(id)`, `crawl.pages(id, { cursor, limit })`, `crawl.cancel(id)`,
`crawl.wait(id, { pollMs, timeoutMs })` (polls until terminal) and `map(options)`. Bump SDK user-agent
to `fetcha-sdk-js/0.2.0` / `fetcha-sdk-python/0.2.0`.