spb/fetcha

1# Fetcha v0.2 — contract for the access model, browser mode and crawling23This document is the single source of truth for the v0.2 changes while the backend (packages/core,4packages/providers, packages/browser, packages/routing, apps/api) and the web app are built in5parallel. Field names below are final.67## 1. Access model (private platform)89- There is **one plan**: `unlimited` (`PLANS = ["unlimited"]`, `PLAN_LIMITS.unlimited`). Legacy values in10  `organizations.plan` are migrated to `unlimited`; always go through `normalizePlan(org.plan)` from11  `@fetcha/core` instead of casting.12- Limits: unlimited monthly requests, 200 concurrent requests, 120 s max timeout, 5 retries, all13  network classes, 90-day log retention, browser rendering (8 concurrent renders), crawl jobs14  (2,000 pages/job, 5 concurrent jobs). No prices, no invoices, no checkout: billing UI must say15  "private platform, no billing" (do not show plan grids, upgrade dialogs or "Start free").16- **Signup is invitation-only**: table `signup_allowlist(email pk lower-cased, note, invited_by_user_id,17  invited_at, used_at, user_id, created_at)`. Better Auth `databaseHooks.user.create.before` must18  refuse (`APIError("FORBIDDEN", { message })`) any email that is neither in the allowlist nor in19  `ADMIN_EMAILS`. The `after` hook sets `used_at`/`user_id`.20- Emails listed in `ADMIN_EMAILS` (env, comma-separated; prod = `spbou4@icloud.com`) get21  `users.role = 'admin'` at creation, and `pnpm db:seed` (packages/db/src/seed.ts) promotes existing22  users with those emails + inserts them in the allowlist (idempotent).23- Admin UI `/admin/access`: list allowlist (email, note, invited by, invited at, account created?),24  add one or many emails (textarea, comma/newline separated) with optional note and "send25  invitation email" checkbox, resend invitation, remove entry (only if no account yet). Server26  actions in `apps/web/src/actions/access.ts`, audited via `admin.action` metadata types27  `access.allow`, `access.invite`, `access.revoke`.28- Invitation email: `EmailService.sendInvite(to, { inviterName, signupUrl })` in packages/email29  (template `InviteEmail`): "You have been invited to Fetcha", button → `${siteUrl}/signup?email=…`.30- Signup form: pre-fill email from `?email=`, copy "Fetcha is invitation-only. Use the address your31  administrator approved." On a 403 from sign-up show: "This email is not on the access list. Ask32  your Fetcha administrator to invite you." Marketing CTAs "Start free" → "Log in" (primary) and33  "Request access" → `mailto:hello@fetcha.co?subject=Fetcha%20access`.34- `/pricing` becomes an "Access" page (same route, nav label "Access"): explains private access,35  what is included (unlimited, browser, crawl, sessions, geo), how to get invited.3637## 2. `POST /v1/fetch` additions (all optional, strict schema)3839| Field | Type | Default | Meaning |40|---|---|---|---|41| `format` | `"html" \| "text" \| "markdown" \| "json" \| "raw"` | `"html"` | `markdown` returns the page converted to Markdown in `markdown` (main content first, boilerplate removed). |42| `browser` | boolean | false | Render in the managed headless Chromium routed through the same proxy network/geo/session. Live. |43| `browser_fallback` | boolean | true | If an HTTP attempt is blocked by a JS challenge/anti-bot, automatically retry in the browser. |44| `wait_for` | string (CSS) | — | Browser: selector to wait for before capturing. |45| `wait_ms` | 0–30000 | — | Browser: extra settle time. |46| `wait_until` | `"load" \| "domcontentloaded" \| "networkidle"` | `"domcontentloaded"` | Browser navigation wait condition. |47| `javascript` | boolean | true | Browser: disable scripting when false. |48| `block_resources` | boolean | true | Browser: skip images/fonts/media. |49| `screenshot` | boolean | false | Browser: PNG base64 in `screenshot`. |50| `links` | boolean | false | Return `links[]` (all hyperlinks, absolute). |51| `referer` | `"auto" \| "none" \| url` | `"auto"` | Referer strategy (auto = none first, search-engine referer on retries). |5253Response additions:5455```jsonc56{57  "markdown": "# Title…",          // only for format=markdown58  "page": { "title": "…", "description": "…", "canonical": "…", "lang": "en", "og": { "og:title": "…" }, "links_count": 42 },59  "links": [{ "url": "https://…", "text": "About", "internal": true, "nofollow": false }],   // only with links:true60  "screenshot": "iVBORw0…",        // only with browser + screenshot61  "metadata": { "mode": "http" | "browser", …, "debug": { "attempts": [{ "provider", "network", "mode", "country", "outcome", "block_reason", "status", "duration_ms" }] } }62}63```6465New error semantics: `BROWSER_UNAVAILABLE` is now only returned when the browser pool is disabled or66down (503-ish situations), `BROWSER_TIMEOUT` when the page does not settle in time. Scope67`browser:use` is **not** required (browser is part of fetch); keep the scope name for compatibility.6869## 3. Crawl API7071`POST /v1/crawl` (scope `fetch:execute`) — body = `crawlCreateSchema` (`@fetcha/core`):72`url`, `max_pages` (1–5000, default 25), `max_depth` (0–10, default 2), `same_domain` (true),73`allow_subdomains` (false), `include_patterns[]`, `exclude_patterns[]` (glob with `*` or `/regex/`),74`respect_robots` (true), `use_sitemap` (false), `concurrency` (1–10, default 3), `delay_ms`, `timeout`,75`format` (`markdown|text|html`, default markdown), `main_content` (true), `country`, `network`,76`browser`, `browser_fallback`, `headers`, `webhook_url`, `label`.7778Returns `202`:7980```json81{ "id": "crawl_…", "status": "queued", "seed_url": "…", "created_at": "…", "options": { … } }82```8384`GET /v1/crawl/:id` → job:8586```json87{ "id", "status": "queued|running|completed|failed|cancelled", "label", "seed_url", "domain", "options",88  "stats": { "discovered", "fetched", "ok", "blocked", "failed", "bytes" },89  "error": { "code", "message" } | null, "created_at", "started_at", "completed_at" }90```9192`GET /v1/crawl/:id/pages?cursor=&limit=100&status=success` → `{ "data": [page…], "next_cursor": "…" | null }` where page =93`{ "id", "url", "final_url", "depth", "status", "http_status", "error_code", "title", "description", "content_type", "content", "links_count", "bytes", "duration_ms", "mode", "fetched_at" }`.9495`DELETE /v1/crawl/:id` → `{ "id", "status": "cancelled" }`. `GET /v1/crawl?limit=50` → `{ "data": [job…] }`.9697Every crawled page is also a normal fetch request (visible in the Requests log with `source:98"crawl"`), so quotas, retries, escalation and routing intelligence apply.99100`POST /v1/map` (sync, ≤ 60 s) — body = `mapCreateSchema`: `url`, `limit` (default 1000), `use_sitemap`101(true), `use_links` (true), `same_domain`, `allow_subdomains`, `search`, `country`, `network`, `timeout`.102Returns `{ "url", "count", "urls": ["…"], "sources": { "sitemap": n, "links": n }, "truncated": bool }`.103104## 4. Internal routes (dashboard → API, service token)105106- `POST /internal/crawls` `{ project_id, user_id, options }` → job (202 body as above)107- `GET /internal/crawls?project_id&user_id&limit=50` → `{ data: [job…] }`108- `GET /internal/crawls/:id?project_id&user_id` → job109- `GET /internal/crawls/:id/pages?project_id&user_id&cursor&limit&status` → pages page110- `DELETE /internal/crawls/:id?project_id&user_id` → cancel111- `POST /internal/map` `{ project_id, user_id, options }` → map result112- `GET /internal/browser` → `{ enabled, running, capacity, queue }` (admin System page)113114Dashboard: new page `/dashboard/crawls` (list jobs of the current project, status badges, stats,115create dialog) and `/dashboard/crawls/[id]` (job detail, pages table with content preview, cancel).116Sidebar item "Crawls" after "Requests". Add crawl/map client methods to `apps/web/src/lib/api.ts`.117118## 5. SDKs119120JS (`packages/sdk`) and Python (`sdk-python`): add `format: "markdown"`, `browser*`, `links`,121`referer` to fetch options and `markdown`/`page`/`links`/`screenshot` to results; add122`crawl.create(options)`, `crawl.get(id)`, `crawl.pages(id, { cursor, limit })`, `crawl.cancel(id)`,123`crawl.wait(id, { pollMs, timeoutMs })` (polls until terminal) and `map(options)`. Bump SDK user-agent124to `fetcha-sdk-js/0.2.0` / `fetcha-sdk-python/0.2.0`.125