# Fetcha v0.2 — contract for the access model, browser mode and crawling

This document is the single source of truth for the v0.2 changes while the backend (packages/core, packages/providers, packages/browser, packages/routing, apps/api) and the web app are built in parallel. Field names below are final.

# 1. Access model (private platform)

• There is one plan: unlimited (PLANS = ["unlimited"], PLAN_LIMITS.unlimited). Legacy values in organizations.plan are migrated to unlimited; always go through normalizePlan(org.plan) from @fetcha/core instead of casting.
• Limits: unlimited monthly requests, 200 concurrent requests, 120 s max timeout, 5 retries, all network classes, 90-day log retention, browser rendering (8 concurrent renders), crawl jobs (2,000 pages/job, 5 concurrent jobs). No prices, no invoices, no checkout: billing UI must say "private platform, no billing" (do not show plan grids, upgrade dialogs or "Start free").
• Signup is invitation-only: table signup_allowlist(email pk lower-cased, note, invited_by_user_id, invited_at, used_at, user_id, created_at). Better Auth databaseHooks.user.create.before must refuse (APIError("FORBIDDEN", { message })) any email that is neither in the allowlist nor in ADMIN_EMAILS. The after hook sets used_at/user_id.
• Emails listed in ADMIN_EMAILS (env, comma-separated; prod = spbou4@icloud.com) get users.role = 'admin' at creation, and pnpm db:seed (packages/db/src/seed.ts) promotes existing users with those emails + inserts them in the allowlist (idempotent).
• Admin UI /admin/access: list allowlist (email, note, invited by, invited at, account created?), add one or many emails (textarea, comma/newline separated) with optional note and "send invitation email" checkbox, resend invitation, remove entry (only if no account yet). Server actions in apps/web/src/actions/access.ts, audited via admin.action metadata types access.allow, access.invite, access.revoke.
• Invitation email: EmailService.sendInvite(to, { inviterName, signupUrl }) in packages/email (template InviteEmail): "You have been invited to Fetcha", button → ${siteUrl}/signup?email=….
• Signup form: pre-fill email from ?email=, copy "Fetcha is invitation-only. Use the address your administrator approved." On a 403 from sign-up show: "This email is not on the access list. Ask your Fetcha administrator to invite you." Marketing CTAs "Start free" → "Log in" (primary) and "Request access" → mailto:hello@fetcha.co?subject=Fetcha%20access.
• /pricing becomes an "Access" page (same route, nav label "Access"): explains private access, what is included (unlimited, browser, crawl, sessions, geo), how to get invited.

# 2. POST /v1/fetch additions (all optional, strict schema)

Response additions:

{
  "markdown": "# Title…",          // only for format=markdown
  "page": { "title": "…", "description": "…", "canonical": "…", "lang": "en", "og": { "og:title": "…" }, "links_count": 42 },
  "links": [{ "url": "https://…", "text": "About", "internal": true, "nofollow": false }],   // only with links:true
  "screenshot": "iVBORw0…",        // only with browser + screenshot
  "metadata": { "mode": "http" | "browser", …, "debug": { "attempts": [{ "provider", "network", "mode", "country", "outcome", "block_reason", "status", "duration_ms" }] } }
}

New error semantics: BROWSER_UNAVAILABLE is now only returned when the browser pool is disabled or down (503-ish situations), BROWSER_TIMEOUT when the page does not settle in time. Scope browser:use is not required (browser is part of fetch); keep the scope name for compatibility.

# 3. Crawl API

POST /v1/crawl (scope fetch:execute) — body = crawlCreateSchema (@fetcha/core): url, max_pages (1–5000, default 25), max_depth (0–10, default 2), same_domain (true), allow_subdomains (false), include_patterns[], exclude_patterns[] (glob with * or /regex/), respect_robots (true), use_sitemap (false), concurrency (1–10, default 3), delay_ms, timeout, format (markdown|text|html, default markdown), main_content (true), country, network, browser, browser_fallback, headers, webhook_url, label.

Returns 202:

{ "id": "crawl_…", "status": "queued", "seed_url": "…", "created_at": "…", "options": { … } }

GET /v1/crawl/:id → job:

{ "id", "status": "queued|running|completed|failed|cancelled", "label", "seed_url", "domain", "options",
  "stats": { "discovered", "fetched", "ok", "blocked", "failed", "bytes" },
  "error": { "code", "message" } | null, "created_at", "started_at", "completed_at" }

GET /v1/crawl/:id/pages?cursor=&limit=100&status=success → { "data": [page…], "next_cursor": "…" | null } where page = { "id", "url", "final_url", "depth", "status", "http_status", "error_code", "title", "description", "content_type", "content", "links_count", "bytes", "duration_ms", "mode", "fetched_at" }.

DELETE /v1/crawl/:id → { "id", "status": "cancelled" }. GET /v1/crawl?limit=50 → { "data": [job…] }.

Every crawled page is also a normal fetch request (visible in the Requests log with source: "crawl"), so quotas, retries, escalation and routing intelligence apply.

POST /v1/map (sync, ≤ 60 s) — body = mapCreateSchema: url, limit (default 1000), use_sitemap (true), use_links (true), same_domain, allow_subdomains, search, country, network, timeout. Returns { "url", "count", "urls": ["…"], "sources": { "sitemap": n, "links": n }, "truncated": bool }.

# 4. Internal routes (dashboard → API, service token)

• POST /internal/crawls { project_id, user_id, options } → job (202 body as above)
• GET /internal/crawls?project_id&user_id&limit=50 → { data: [job…] }
• GET /internal/crawls/:id?project_id&user_id → job
• GET /internal/crawls/:id/pages?project_id&user_id&cursor&limit&status → pages page
• DELETE /internal/crawls/:id?project_id&user_id → cancel
• POST /internal/map { project_id, user_id, options } → map result
• GET /internal/browser → { enabled, running, capacity, queue } (admin System page)

Dashboard: new page /dashboard/crawls (list jobs of the current project, status badges, stats, create dialog) and /dashboard/crawls/[id] (job detail, pages table with content preview, cancel). Sidebar item "Crawls" after "Requests". Add crawl/map client methods to apps/web/src/lib/api.ts.

# 5. SDKs

JS (packages/sdk) and Python (sdk-python): add format: "markdown", browser*, links, referer to fetch options and markdown/page/links/screenshot to results; add crawl.create(options), crawl.get(id), crawl.pages(id, { cursor, limit }), crawl.cancel(id), crawl.wait(id, { pollMs, timeoutMs }) (polls until terminal) and map(options). Bump SDK user-agent to fetcha-sdk-js/0.2.0 / fetcha-sdk-python/0.2.0.